Does generative AI deepen learning, or does it merely raise performance during support?

If the educational effect of generative AI is expressed by a single score, performance during support and the ability remaining with the individual become mixed. A 2026 meta-analysis synthesizing programming research reported a positive effect on productivity, but the pooled effect on learning outcomes was not significant. Furthermore, in studies where AI could be used during exams, the effect was large, but in exams where AI was removed, the effect was nearly zero.[4] This suggests that scores including AI may not measure independent skills.

Assessments should be divided into at least three layers. The first is speed and quality in the AI-assisted state; the second is an immediate test solving the same scope without AI; the third is retention and transfer to other tasks after several weeks. The first layer has value as a measure of job performance capability, but one cannot call it an improvement in human ability without measuring the second and third layers.

Before training begins, fix which of the following will be the passing condition: performance during AI use, reproduction immediately after removing AI, retention after a period of time, or transfer to other tasks. If only speed during support increases and independent evaluation decreases, report productivity improvement and skill formation separately. When transferring the pooled results of programming research to other professions, also use the condition of whether AI was available during the exam as a comparison condition.[4]

• Performance during support: Measure the output of the person and the entire system including AI
• Independent reproduction: Remove AI and measure whether principles and procedures remain with the individual
• Retention and transfer: Measure after a period of time and whether it can be applied to different problems

Wu’s formal meta-analysis synthesized 35 experimental and quasi-experimental studies published between 2022 and 2024, involving 4,193 participants, and reported an average effect of ChatGPT use as Hedges’s g = 0.670.[1] On the other hand, the heterogeneity among studies was I² = 91.44%. Since results vary significantly depending on subject, duration, instructional format, and type of knowledge, this average value cannot be applied as the expected effect for our company’s training.

Maier’s meta-analysis limited to programming synthesized 10 studies on learning, 1,069 participants, and 11 effect sizes, reporting g = 0.14 with a 95% confidence interval of −0.18 to 0.47.[4] This is an unpublished preprint released in May 2026, and since the number of studies on learning is still small, it does not constitute a definitive conclusion. Nevertheless, the point that whether the outcome metric includes exams during AI use affects results is directly relevant to implementation evaluation.

In a preregistered RCT involving approximately 1,000 high school students in Turkey, three groups were compared: GPT Base (similar to standard ChatGPT), GPT Tutor (incorporating teacher-provided correct answers, typical errors, and hint constraints), and no AI. Performance during practice increased by 48% for Base and 127% for Tutor, but in the no-AI exam, the Base group was 17% lower than the control group, and while this decrease was nearly eliminated in the Tutor group, it did not exceed the control group.[2]

The same experiment shows cases where success during support becomes an illusion of learning and cases where harm is mitigated by design.

In a crossover RCT involving 194 physics students at Harvard, an AI Tutor designed by experts with materials, videos, problem-specific prompts, and staged questioning showed higher immediate post-test scores than face-to-face active learning. The effect size from regression estimation was 0.63 standard deviations, and the median learning time for the AI group was 49 minutes.[3] However, the subjects were limited to two physics themes, and long-term retention was not measured. This is the effect of a precisely designed instructional system, not of a general-purpose chat.

We do not treat the Tutor group’s results as the effect of the base model alone, because the teacher-provided correct answers, typical errors, staged hints, and instructional structure are simultaneously intervening.[2] [3] In our company’s implementation, we will separate conditions for general chat, support with hint constraints, and materials alone, to verify which design elements contributed to error discovery and independent performance. The expert preparation effort will also remain as an operational condition.

Comparing only the final test with AI allowed and the final test without AI does not reveal at which support stage the individual became independent. The recommended withdrawal curve involves gradually reducing support from free AI, to hint-limited, to questioning only, to no reference, measuring explanation, error correction, and completion at each stage. This is not a common procedure directly verified by existing research, but a design proposal to observe the transition to independent ability.

Promotion conditions will be based not on the number of times support was used, but on whether the same errors could be self-corrected in the next stage. If performance drops as support is reduced, record insufficient knowledge, problem understanding, procedure memory, and low confidence as competing hypotheses. Do not immediately return to answer presentation; return only one stage to the minimum necessary clue and confirm changes in a re-test.

This is a hypothetical example. If correct answers can be given with hints but reasons cannot be explained with questioning only, distinguish between passing job performance and unmet independent understanding. Classify processes that are sufficient with AI-inclusive performance by role and those requiring independent reproduction in case of obstacles before starting. Do not advance everyone to the same stage; determine withdrawal points according to the responsibility left to the individual.

Even if other tasks are given, if terminology, numbers, question order, and solution patterns are close to the practice, procedural memory may be mistaken for transfer. The recommended assessment distinguishes between reproduction tasks that change only the surface and transfer tasks that require selecting the same principle in a different situation, and does not inform graders of the practice conditions. This is not a claim of new learning effects, but an exam design to prevent contaminating passing criteria.

Do not leave transfer tasks to be automatically generated by AI; the instructional material manager must verify that phrases from practice answers and correct answer structures have not leaked. To prevent graders who viewed the conversation history with AI from changing scores due to expectation, evaluate answers independently. If results are high for reproduction tasks and low for transfer tasks, do not conclude insufficient understanding; also check for differences in task difficulty and prerequisite knowledge.

The recommended judgment connects the performance during support, during withdrawal, immediate independent, delayed retention, and transfer into one learning history for each person. Do not offset with average scores; set the lower limit of the stage required for business first. If only transfer is unmet, do not stop AI use entirely, but return to practice that requires selecting principles in different contexts, and re-confirm with the same unused task set.

• Reproduction tasks: Change the surface and measure whether the same knowledge and procedures can be used independently
• Transfer tasks: Measure whether one can independently select the necessary principles in a different situation
• Blinding: Separate practice conditions and AI history from graders and check for leaks
• Re-test: Return learning only to unmet stages and confirm with unused tasks

When transferring to business training, do not ask whether to ban AI or open it fully, but separate learning stages and performance stages. Explain basic concepts independently, find errors in AI proposals, solve with hints only, and finally produce business quality with AI. With this sequence, one can clarify the abilities remaining with people and those delegated to the system.

Before implementation, set practical tests with AI allowed, independent tests without AI, and delayed tests for the same subjects. Record not only the AI proposal adoption rate, but also the percentage of explanations given, the error discovery rate, the drop when support is removed, and retention after several weeks. What can be safely derived from research is not the conclusion that generative AI always enhances learning, but that the constraints and assessment designs that protect learning determine part of the effect.[2][3]

Learning managers will separately monitor drops in non-AI evaluations, failures to detect incorrect proposals, and work stopping when support is removed. If any falls below the pre-established ability floor, keep direct answers disabled, step back only to the minimum necessary questioning or hint stage, and retest. Because average research effects vary substantially by condition, exclude occupation-task groups that remain worse even when the company-wide average is positive.[1] [2]

• Fix first: Knowledge to remain with the individual, processes to delegate to AI, confirmation responsibility
• Compare: Three conditions of AI allowed, hint-limited, and AI-free
• Measure later: Not just immediately, but retention and transfer to other tasks
• Update: Change support design when error logs and excessive dependence are found
• Training designer: Fix the evaluation purpose for each AI availability condition first
• Instructional material manager: Record the effort for designing hints, correct answers, and typical errors
• Learning manager: Stop deployment at the lower limits of independent performance, retention, and transfer