The ROI of Generative AI Is Determined by Pre-Implementation Design

Noy and Zhang randomly divided 453 college-educated professionals into groups and assigned them job-relevant text tasks. The group with access to generative AI saw an average time reduction of 40% and a 18% improvement in quality as assessed by independent evaluators. [1] However, the tasks were short and did not require much enterprise-specific context or rigorous fact-checking. What was measured here was productivity for specific tasks.

Dillon et al. randomized access to generative AI integrated into daily applications across 66 companies and 7,137 people. During months 4–6, the estimated effect of assigned access reduced measured email-session time by about 1.4 hours per week. The instrumental-variable estimate of tool use, using randomized access as the instrument, was about 2 hours. This estimate requires additional assumptions; it is not a simple comparison restricted to actual users. [4]

Quality and revenue were not measured directly, so time savings alone cannot establish ROI without measuring where the saved time went.

To convert time savings into value, one must track what happened in the subsequent steps after the shortened process. If processing volume does not change, alternative explanations include insufficient demand, constraints in downstream processes, or the absorption of saved time into other tasks. Record the same period for both the AI-using group and the conventional procedure group; if downstream wait times or confirmed outputs do not improve, do not count the time savings directly as benefits. [1] [4]

In the ROI ledger, explicitly map the conversion path from usage to business outcomes. Link AI usage, work time, review, confirmed quality, rework, processing volume, and final outcomes under the same business ID. If intermediate changes cannot be observed, leave them as gaps in causality. Even if usage costs decrease, if the verification burden is merely shifted to another department, treat it as a cost transfer between departments rather than an improvement.

• Usage: Who used it and in which business process among the target population
• Efficiency: How work time, wait time, and lead time changed
• Quality: How accuracy, rejections, rework, and customer evaluations changed
• Business Outcomes: How they connected to sales, gross profit, processing capacity, and opportunity loss
• Total Cost: Whether it included model usage fees, verification, training, data preparation, and controls

In a study by Brynjolfsson et al. analyzing 5,172 customer support agents and approximately 3 million conversations, AI assistance increased the average number of resolutions per hour by 15%. The effect was not uniform: it increased by 36% for the lowest skill tier prior to implementation, while no significant productivity improvement was confirmed for the highest tier. [2] Deciding investment based solely on average effects causes one to lose sight of which businesses and people derive value.

The ROI ledger should be maintained at the business process level, not as a single company-wide sheet. Align input data, difficulty, user proficiency, and the rate of adopting AI suggestions to observe the distribution of effects. This is because the same tool yields value differently depending on whether it is transferring knowledge to inexperienced staff or supporting exploration for skilled workers.

Before attributing differences in effect solely to proficiency, verify task difficulty, self-selection in choosing AI, reviewers, and work shifts. While studies showing significant improvements in low-proficiency tiers are important, extrapolating to other professions requires reproducing stratified comparisons under the same business conditions. [2] Conversely, even if there is no average difference for skilled workers, investigate the possibility that value remains in parts of exploration or exception handling using separate metrics.

In a randomized experiment involving 758 consultants, 385 were assigned to 18 tasks designed to be within the AI capability frontier and 373 to one task designed to be outside it. Within the frontier, AI groups improved task completion, quality and time. On the outside-frontier task, AI groups were on average 19 percentage points less likely than the control group to answer correctly. [3] KPIs based only on plausibility or speed can miss losses from incorrect answers.

Before implementation, fix the baseline of conventional processes and the criteria for continuation, expansion, or termination. Instead of converting time savings into monetary value alone, place accuracy, rework, review time, and the impact of failures on the same denominator. ROI is not an explanatory document created after implementation, but an observation device for updating investment decisions.

Do not conveniently change the criteria for expansion and withdrawal after implementation. Even if time, completion, and quality improve within the capability range, accuracy may drop outside it; therefore, place quality guardrails for each target business. [3] Segments that violate guardrails revert to conventional procedures regardless of usage rate or time savings, and only segments where improvements are reproduced are expanded to the next investment unit. The decision to resume is recorded by the quality owner, not solely by the cost owner.

• Compare conventional procedures and AI-assisted procedures under the same difficulty, duration, and quality standards
• Record not just adoption rates, but also the volume of corrections and review time
• Observe distributions by business process, proficiency, and difficulty in addition to averages
• Estimate both the loss from incorrect answers and the opportunity loss from not using AI
• Expand only reproducible conditions to the next investment unit

Experiments showing both speed/quality improvements and accuracy drops within the same implementation due to capability differences demonstrate the peril of consolidating results into a single average. [3] However, this study did not observe double counting in corporate accounting. Below, we treat the overlap of benefits as an accounting inference when transferring research results to investment decisions adjusted for quality.

This is a hypothetical example. If the reduction in work time is fully recorded as labor cost savings, and the increased processing volume achieved in the same time is added to sales benefits, there is a risk of counting the same surplus twice. The same applies to reduced rework due to quality improvements and the shortened time for that rework. Link the source, benefiting department, and realization conditions for each benefit candidate, and treat candidates sharing the same change as mutually exclusive scenarios.

The recommended judgment is to divide benefits into four categories: reduction in cash outflows, increase in processing capacity, avoidance of quality losses, and future options, summing only those categories where realization has been confirmed. Even if surplus is generated, if there is no demand or reallocation, it remains as unused capacity rather than cash benefit. After excluding duplicates, only cases that meet quality floors and total costs are considered investment candidates.

It is not necessary to reduce the uncertainty of ROI to the same precision for all variables. Separate variables that can reverse investment decisions from those that do not change the conclusion even when varied widely. For the former, compare whether the cost and duration required for additional measurement are less than the value of avoiding incorrect expansion or deferral. This is a decision-making procedure for allocating resources to the next experiment, not an estimation of effect size.

This is a hypothetical example. If investment recovery collapses with a slight drop in actual usage rate, but the conclusion remains the same even with significant changes in model pricing, prioritize measuring the reproducibility of usage over price negotiation. Conversely, if the probability of violating the quality floor dictates the decision, add samples of borderline cases before increasing the number of users. Record which measurements were chosen and which uncertainties were left in the investment ticket.

The recommended operation is to place on a single sheet the current range, the boundary where the decision reverses, the minimum unit of additional observation, and the possible actions after observation for each variable. Measurements that cannot change expansion, contraction, or termination even after receiving results are deprioritized. Do not separate validation costs from implementation costs; include them in total costs as investments to improve decision-making.

• Target Variable: Select one of usage rate, quality, loss, or cost that can reverse the decision
• Uncertainty Range: Record the current lower bound, benchmark, upper bound, and their basis
• Additional Observation: Link required sample size, duration, cost, and actions selectable after observation
• Stop Rule: Terminate measurement once the range where the conclusion does not change is confirmed

ROI should be presented as a range under varied assumptions, not a single point prediction. Treat target users, actual usage rate, the proportion of time savings redistributable to outcomes, quality improvements, incorrect answer losses, review time, model fees, and training/data preparation/control costs as separate variables. Recalculate results under baseline, conservative, and adverse conditions; cases where the investment decision reverses due to the failure of a single optimistic assumption should remain in small-scale validation.

The cost owner owns total costs and recovery conditions, the business owner owns processing volume and final outcomes, and the quality owner owns accuracy, rework, and critical failures. In determining continuation, verify not only if benefits increased, but also if the quality floor was maintained, the verification burden was not shifted to other departments, and segments worse than conventional procedures were reverted. Do not fill unmeasurable variables with zeros or averages; leave them as uncertain assumptions for decision-makers.

• Benefits: Count only the portion of time savings that shifted to processing volume, quality, and customer outcomes
• Costs: Include usage fees plus verification, rework, training, data preparation, and controls
• Sensitivity: Search for boundaries where the decision reverses by varying actual usage rate, redistribution rate, and incorrect answer losses
• Withdrawal: Revert segments that violate any of the quality floor, total cost, or recovery conditions to conventional procedures