Human–AI collaboration can beat humans alone but does not necessarily beat the best solo performer

Vaccaro et al. systematically collected peer-reviewed research from 2020 to June 2023, synthesizing 106 experiments and 370 effect sizes from 74 papers. When comparing human+AI to humans alone, the pooled effect size was Hedges' g of 0.64. In contrast, when benchmarked against the better-performing of humans alone or AI alone, the pooled effect size was g = −0.23, with human+AI significantly underperforming.[1]

The former measures human extension, while the latter measures synergy. While introducing improvements to human work has value, it does not necessarily mean that 'pairing humans and AI is optimal.' There are tasks where it is better for AI to process alone and humans to handle exceptions, and tasks where it is better for humans to judge without using AI. Fixing the comparison benchmark to humans alone causes us to lose sight of these options.

Differences were also observed by task type. In decision-making tasks with a finite set of choices to select an answer, synergy was g = −0.27, while in open-ended creative tasks, it was g = 0.19, but the confidence interval for the latter crossed zero.[1] It is safer to interpret this not as 'synergy always occurs in creativity,' but as the difficulty of integration being particularly observed in decision-making tasks.

In Vaccaro et al.'s moderator analysis, in tasks where humans alone outperformed AI alone, human+AI exceeded both, yielding a synergy of g = 0.46. Conversely, in tasks where AI alone outperformed humans alone, human+AI fell below AI alone, with g = −0.54.[1] It is not merely that the human is the final approver, but that the human possesses relative ability sufficient to discern cases where AI errs.

In an example from the same study, in a fake hotel review detection task where AI accuracy was 73%, humans alone achieved 55% and human+AI achieved 69%. Adding human judgment to tasks where AI is stronger resulted in lower performance than AI alone. On the other hand, in bird image classification, AI achieved 73%, humans achieved 81%, and human+AI achieved 90%.[1] This is an observed example from an individual study and not a general rule, but it shows that the combined result varies depending on the relationship between the baseline capabilities of humans and AI, even with the same screen configuration.

Explanatory displays and AI confidence displays did not significantly change synergy across the entire meta-analysis. Only three experiments pre-assigned sub-tasks to humans and AI, and the average was positive but uncertain.[1] Therefore, it is necessary to verify not only the addition of explanations but also which sub-tasks are assigned to whom under three conditions.

• Measure first: baseline performance and error types for humans alone and AI alone
• Measure separately: adoption of correct AI proposals and rejection of incorrect AI proposals
• Change delegation methods: advice, second opinion, exception handover, sub-task assignment
• Do not rely solely on displays: verify whether explanations or confidence levels improved business performance

Dell’Acqua et al. randomly assigned 758 BCG consultants to three conditions: no AI, with GPT-4, and with GPT-4 plus an overview of prompt design. In 18 tasks designed within the scope of AI capabilities, AI users completed 12.2% more tasks, were on average 25.1% faster, and produced significantly higher answer quality.[2] This was a scenario where AI's strengths in exploration, idea expansion, and documentation could be connected to human work.

However, in complex management tasks placed outside the scope of AI capabilities, AI users had a 19 percentage point lower probability of providing correct answers compared to the control group.[2] Even within the same profession, using the same model, and with similar difficulty, tasks that AI helps and tasks that AI harms coexist. The 'jagged frontier' shown by the authors indicates the need to decompose work into sub-tasks rather than dichotomizing the entire job as AI-favorable or unfavorable.

Since this experiment lacked an AI-alone group, it cannot determine synergy in exceeding the best individual entity. What it shows is that the extension effect relative to humans alone reverses depending on task boundaries. Additionally, only one task was verified as being outside the boundary, and the model was GPT-4 as of 2023.[2] Instead of creating a fixed prohibition list, re-measure the capability boundary with each model update.

• AI advantage: Generation of large numbers of candidates, organization of known information, format conversion
• Human advantage: Goal setting, adjustment of ambiguous constraints, exception judgment involving responsibility
• Collaboration candidates: Tasks with distinct strengths, verifiable outputs, and low handover costs
• Hold candidates: Tasks where AI errors are difficult to detect and failures are irreversible

Existing meta-analyses did not confirm that displaying explanations or AI confidence significantly changes synergy.[1] However, this is not proof that displays have no value, but rather the result that no clear average difference could be detected within the scope of the target studies and implementation conditions. We re-verify displays not as a universal solution nor as an invalid function, but as part of the handover conditions.

The added value of combined systems is evaluated not only by the difference from the best individual performance but also by including resources used for consultation, verification, integration, waiting time, and responsibility transfer. Even if quality improves slightly, if handovers are long and exception handling increases, single processing with exception handover may be better than full collaboration. This is a hypothesis about adjustment costs to be verified in one's own processes, not a research average effect.

The recommended 'Synergy Ledger' records the quality and time for humans alone, AI alone, and human+AI, in addition to separately accounting for steps such as reading proposals, falsifying them, correcting them, and passing them to the responsible person. If a combined system exceeds the best individual but relies on unpaid additional work from specific users, it is not considered reproducible synergy. We expand only the scope where quality differences and adjustment loads can be observed for the same task and period.

If we create rules like 'this task for humans, this task for AI' while looking at the results of the three conditions, we may fit them conveniently to the observed sample. We must separate the sample used to create the assignment rules from the sample used to evaluate them with fixed rules, confirming in the latter whether it exceeds the best individual. This is not a new effect claim but a recommended procedure to avoid overestimating task boundaries.

This is a hypothetical example. Suppose that in the training sample, only cases with high information volume appeared to favor humans. We then fix information volume as the assignment condition and assign a different group of cases to humans, AI, and the combined system. If the results are not replicated, it remains possible that the assignee's expertise or the verifiability of errors explained the outcome. If rules are changed afterward, they must be re-verified as a new version.

The recommended judgment involves recording not only the overall average but also the direction in which misassignment occurred. We separate cases where AI was entrusted and human advantage was lost, cases where tasks were passed to humans and AI advantage was lowered, and cases where combination fell below both. For rule-uncertain categories, we do not escape to full collaboration but pre-determine the safe side for humans, AI, or hold based on the impact of failure.

• Rule candidate: Place task characteristics and assignment conditions for humans, AI, and the combined system in one sentence
• Training sample: Save the cases used to create the rules and the observed three-condition performance
• Verification sample: Fix the rules and measure the difference from the best individual using unused cases
• Misassignment: Record human advantage loss, AI advantage loss, and combined degradation separately

The minimum unit for evaluating collaboration is the three conditions: humans alone, AI alone, and human+AI. We measure correct answers, quality, time, rework, and error detection rates for the same task. If human+AI exceeds humans alone but falls below AI alone, there is room to reconsider the design of returning everything to humans. Conversely, even if AI alone is stronger on average, if humans can spot specific exceptions, those exception conditions and handover methods become the value of collaboration.

Next, we decide who hands over work and when. The results vary even with the same model depending on whether tasks are passed to humans only when the model is uncertain, whether humans make provisional judgments first and then view AI as a second opinion, or whether AI is entrusted only with candidate generation. We record not only the authority table but also inputs, judgment order, basis display, rejection, stoppage, and recovery as a single experimental condition.

Synergy is not present in a state of high AI utilization rate. It is a state where synergy is designed to maintain individual strengths and prevent weaknesses from overlapping, and where the combined performance truly exceeds the best individual entity. Until we can measure up to that point, it is better to report separately 'extended humans' and 'synergy between humans and AI' to avoid misleading investment decisions.

The workflow owner fixes the versions of the human-only, AI-only, and combined conditions, plus the common tasks and budget, in a three-condition comparison record. The record preregisters a minimum quality gain over the best standalone actor and a maximum severe-error rate. If the combined system misses the gain or exceeds the error limit, operations revert to the best standalone method. Restart follows a redesigned handoff, two observation windows replicating quality-adjusted additional value, and joint approval by the quality and finance owners.