A Pilot Comparison Design for Moving Generative AI into Production

In Cui et al.’s developer experiments across three firms and 4,867 people, the estimate concerning AI tool use indicated more completed tasks, while individual experiments were noisy and results varied across firms. [1] Even with the same product, changing the work, training, or review procedure changes the intervention being compared. Define the local workflow as a unit rather than replacing its measured effect with a research average.

For customer responses, decide whether follow-up includes review, revision, approval, and repeat contact after sending, rather than drafting alone. Align eligible cases, intake periods, quality criteria, and follow-up horizons across the existing and AI-assisted arms. Record training and added review as part of the assisted procedure so they do not disappear from its evaluation.

Choose the assignment unit to reflect how influence is shared. Shared AI drafts or a common reviewer may make teams or shifts worth considering alongside individuals. Record both the randomized assignment unit and the work-case unit; many cases from one team are not many independent randomized participants. Report an ordinary phased rollout separately from a randomized staggered rollout.

Dillon et al. randomized access to generative AI embedded in everyday work applications across 66 firms and 7,137 people, following them for six months. In the second half of the experiment, estimated weekly email-session time fell by about 1.4 hours under the assignment-based analysis and about 2 hours when assignment was used as an instrument for use. The latter was not a simple comparison restricted to users. [2]

If the decision concerns giving all eligible people access in production, make the originally assigned comparison the main report. Retain nonusers and record reasons such as no eligible work, unclear rules, or unhelpful assistance. Use rate diagnoses whether the intervention reached people; it does not replace workflow outcomes.

Selecting only actual users after assignment can mix motivation, case mix, and managerial support into a comparison. Dillon et al. used random assignment as an instrument for tool use. [2] Its causal interpretation requires additional assumptions; a simple user–nonuser comparison does not produce the same estimate. If the additional assumptions cannot be justified locally, present actual-use summaries as supplementary observations.

Dillon et al. reported less email-session time but did not detect a statistically clear change in the quantity or composition of measured digital tasks. [2] This does not establish that saved time has no value. The operating proposal here links the improved step to confirmed outcomes with the same case ID, and reports how far improvement was actually observed.

Add the active time all people spend drafting, reviewing, revising, and handling exceptions. Elapsed time runs from intake to confirmed closure and includes waiting. Parallel work can make total person-time exceed elapsed time. Before starting, define when waiting for AI counts as human effort, including whether other work is possible.

Calculating time only for successful cases can erase effort spent on difficult, failed, abandoned, and unfinished work. Retain predeclared eligible intake as the denominator, and distinguish quality-approved closure, unfinished work, known failure, and missing records at the same case-relative follow-up horizon. Do not call every unfinished case a failure or replace missing time with zero. Hold a confirmed time-saving judgment when incomplete follow-up or recording could reverse it.

The following is a Catalysphere measurement proposal. Record outside AI use in the control arm, shared generated artifacts, training exposure, and load on common reviewers. These effects change the contrast and its interpretation, but need not always reduce it. Removing users after observing results changes the compared population; preserve original-assignment reporting and supplement it with sharing and other-use records.

When models, reference data, prompts, review, or eligible work change, record the date and affected cases. Splitting periods does not remove calendar, learning, or case-mix differences. Reassess what the original comparison supports; if a decision about the new procedure is needed, retest under a new version and comparison plan. Report outcomes or exclusions changed after results became visible separately as exploratory analyses.

In Dell’Acqua et al.’s BCG experiment, AI assistance improved performance on one set of tasks, while correctness fell on one managerial task placed outside the AI capability frontier. [3] This does not guarantee the same boundary for current models. Predefine a small set of important segments using difficulty, experience, reference-information condition, verifiability, and consequences, and report both aggregate and segment results.

Record quality-pass definitions, severe errors, returns, rework, and repeat contact separately. Where feasible, blind reviewers to the procedure and apply the same rubric. If review uses a sample, preserve its selection method and reviewed count. Design sample size and follow-up around material deterioration and decision-relevant improvement. Neither nonsignificance in a small segment nor zero observed severe errors proves production safety.

NIST AI RMF 1.0 is a voluntary framework that organizes context of use, expected benefits and costs, test sets and metrics, deployment-like evaluation, and generalization limits. [4] The record below is a Catalysphere proposal informed by those principles, not an empirically validated form or a NIST scoring template. Fill in the conditions before launch; then put the existing and AI-assisted results side by side at the end.

Attach measurement source, observation date, and denominator to every value. Mark unmeasured values as unobserved, and give the basis for estimates or future assumptions. Retain the primary outcome, quality floors, comparison deviations, and immature outcomes in the same record rather than selecting only favorable metrics. Keep case state and missing records in separate fields: a case that is both unfinished and missing effort records still counts once in eligible intake.

Fill in the table by metric and add the data source, observation date, difference, and uncertainty. If uncertainty cannot be calculated, state why and which observations are missing. Do not substitute zero for an unobserved value.

• Purpose and scope: Workflow proposed for production, eligible people and cases, exclusions, primary outcome, minimum meaningful improvement, and quality conditions
• Comparison plan: Existing and AI-assisted procedures, assignment method and unit, analysis unit, intervention version, intake period, and case-relative follow-up horizon
• Denominators and state: Assigned units, eligible intake, confirmed closures, known failures, unfinished cases, missing data, and predeclared exclusions with reasons
• Outcomes and load: Primary outcome, total person-time and role breakdown, elapsed time, reviewed count, severe errors, rework, and repeat contact
• Evidence and reproducibility: Evaluator, data origin, observation date, aggregation-procedure version, missing-data treatment, and location of records for rechecking
• Interpretation and costs: Segment results and uncertainty, sharing and contamination, mid-pilot changes, unobserved outcomes, measured costs, and assumptions handed to finance
• Decision and responsibility: Deploy, hold, retest, or stop; rationale and approved scope; business, quality, and cost owners; stop and recovery owners; next review date

NIST describes deployment-like evaluation, production monitoring, determinations about intended purpose, and responsibilities and mechanisms for deactivation. [4] To connect those management principles with a workflow comparison, this article proposes four decisions. Check comparison validity, outcome and quality conditions, and the operating team’s ability to monitor, stop, and restore the existing process separately.

When proceeding, specify eligible people and cases, procedure version, usage volume, and the next review date. Give the quality owner authority to stop use on a breach, and have the business owner confirm the handling route and capacity during a stop. Define who restores which scope if evidence remains insufficient by the deadline or quality falls below its floor.

The record assembles operating evidence for considering production deployment. Final budget approval also needs a financial evaluation using measured person-time and costs. Monetization is covered in the separate article “Generative AI ROI Is Determined Before Deployment.” Begin with the business and measurement owners drafting the comparison plan, and the quality owner checking floors and stop conditions. Tie the next investment to the scope actually tested and to the next test that resolves the remaining unknowns.