When does AI coding accelerate development, and when does it slow it down?

Maier et al.'s preprint meta-analysis systematically explored studies from 2019 to 2025, integrating 23 studies and 27 effect sizes. For productivity, analyzing 16 effect sizes, they estimated the combined effect of generative AI assistance at Hedges' g = 0.33, with a 95% confidence interval of 0.09 to 0.58. Meanwhile, the heterogeneity across studies was I² = 99%.[1] The fact that the mean value is positive is a different question from whether it will be positive for specific tasks within your own organization.

In the same meta-analysis, the effect for controlled laboratory tasks was significant at g = 0.73, whereas for enterprise environments it was g = 0.19 and for OSS environments g = 0.01, with the latter two not significantly different from zero.[1] Research environment alone explains about 36% of the heterogeneity. Extrapolating the improvement rates from short, structured tasks to the entire development process, including maintenance, adjustment, and review of existing systems, risks overestimating expectations.

• Can the boundaries of the task and pass/fail criteria be fixed in advance?
• Can the AI reference the necessary specifications, dependencies, and organizational rules?
• Can the generated results be verified at low cost through testing or type checking?
• Does the measurement window cover post-implementation review, rework, and maintenance?

Cui et al. randomly assigned the timing of GitHub Copilot access to 4,867 developers across Microsoft, Accenture, and an anonymous Fortune 100 company. Using random access assignment as an instrumental variable, the estimates showed a 26.08% increase in weekly pull requests, a 13.55% increase in commits, and a 38.38% increase in builds.[2]

However, enterprise-specific estimates were imprecise, and among the aggregated primary metrics, only the pull request rate reached conventional levels of statistical significance. This does not estimate a 26% output increase for everyone, but rather an effect closer to those who started using it due to the experiment.

Peng et al.'s initial preprint working paper randomized 95 professional developers recruited from Upwork to build a JavaScript HTTP server. Comparing only the 35 completers in each group, the average time was 71.17 minutes versus 160.89 minutes, with the AI group completing tasks 55.8% faster. However, the 7-percentage-point difference in success rate was not statistically significant.[4]

While significant time savings are possible for small, self-contained new tasks where correctness can be verified via automated tests, it is important to note that this excludes incompleters and long-term quality.

• Tasks with clear output forms, such as boilerplate code, scaffolding, and test creation.
• Tasks involving exploring unknown APIs or unfamiliar code to obtain candidates quickly.
• Tasks where errors can be immediately mechanically verified through testing, type checking, or static analysis.
• Tasks where the assignee can easily decide not to adopt the proposal and revert to manual implementation.

Becker et al.'s preprint study assigned 246 real-world issues to 16 experienced developers continuously maintaining mature OSS, randomizing AI availability per issue. Participants had an average of 5 years of experience in the target repositories and primarily used Cursor Pro and Claude 3.5/3.7 Sonnet. For issues where AI was permitted, the completion time including implementation and post-review corrections increased by an average of 19%, with a 95% confidence interval of 2% to 39% increase.[3]

Interestingly, there is a discrepancy with perception. Participants predicted a 24% reduction before starting and estimated a 20% reduction after the study.[3] However, screen recordings and interviews suggested that the burden of implicit context in large codebases, low generation confidence, and the verification/correction of outputs may have contributed to this. Although the mechanism analysis is exploratory, it aligns with the interpretation that for tasks experts can solve accurately from memory, explanation, waiting, and verification add new workloads when using AI.

• Many design reasons or exceptions known only to the assignee.
• Locally correct changes may have side effects in other areas.
• Review criteria are strict, demanding explainability and maintainability.
• The time to identify incorrect candidates is longer than manual implementation.

If developers choose which cases to use AI for, the difficulty and unfamiliarity of used versus non-used cases may differ from the start. Simple user comparisons risk misattributing these differences to AI effects. The recommended internal validation separates the period of pre-assigned availability from the period of free choice, storing assignment, actual use, and mid-course dropout as separate variables. This is not a claim for a new effect size.

If assignment cannot be implemented, group cases by the same developer, similar task type, comparable code scope, and identical review criteria. Since unobserved difficulty remains, report the difference as conditional rather than causal. Verify reasons for not using AI despite permission and possibilities of using it via other channels despite prohibition, ensuring access and use are not treated identically.

This is a hypothetical example. In a team using AI only for unfamiliar areas and manual implementation for known maintenance, longer completion times for AI-assisted cases are not necessarily detrimental. Conversely, sending only simple boilerplate cases to AI may overstate effects. Fix difficulty estimates and assignee familiarity before assignment; if classifications are changed after viewing results, re-evaluate as a new comparison version.

While the interpretation that explanation, waiting, and verification for AI became additional work aligns with the observed increase in completion times in the mature OSS experiment, the causal contribution of individual factors has not been determined.[3] In your own organization, observe not only the creator's input time but also review waiting time and others' confirmation time to verify if this explanation holds.

If a creator submits a PR quickly but the change is large, extending the review queue and delaying subsequent cases, individual time savings do not translate to team throughput. The recommended measurement places the creation, initial review waiting, comment response, and re-approval of AI-assisted and non-assisted PRs on the same clock. It also records whether reviewers' confirmation behavior changes upon knowing if the output is AI-generated.

This is a hypothetical example. If implementation time decreases but review time for two people increases, do not simply offset them; investigate which types of comments increased. If due to format or test deficiencies, adjust generation conditions; if due to insufficient explanation of design intent, improve handover; if due to broad side effects, narrow the scope of application. Reduce the change volume and retest; do not automatically expand categories where waiting time does not return.

• Creation process: Separate time spent on understanding, generation, verification, and manual correction.
• Queue: Record time to first review and concurrent backlog volume.
• Other burden: Connect the assignee's time for review, questions, rework, and re-approval.
• Decision: Determine correction targets (generation conditions, handover, scope of application) for each type of comment.

Instead of setting a company-wide improvement rate, stratify work by verifiability, codebase familiarity, contextual depth, and failure impact. Set comparison periods with and without AI for each stratum, measuring from issue initiation through merge, review, and rework on the same clock. If PRs increase but only due to finer granularity, customer value has not increased.

Also determine continuation conditions before adoption. For example, exclude strata where rework or defects increased despite shorter completion times, narrowing the scope to exploration or boilerplate implementation only. Conversely, incorporate strata where unfamiliar areas ramped up faster and quality was maintained into education and standard procedures. Calculate ROI based on quality-adjusted throughput, not account count.

The engineering owner fixes the work stratum, task set, and tool version in the adoption record, with a minimum quality-adjusted throughput and maxima for rework, defects, critical vulnerabilities, and verification time for each stratum. Only the stratum that breaches a limit returns to the previous standard workflow. Restart follows changed training or process, two observation windows meeting every criterion on the same tasks, and approval by both the DevEx and AppSec owners.

• Speed: Lead time from initiation to merge and human actual work time.
• Quality: Review comments, rework, defects, incidents, and subsequent correction volume.
• Usage: Adoption rate, rejection rate, and point of reverting to manual implementation.
• Capability: Whether the state of being able to explain, debug, and change without AI was maintained.