What Is Lost in Organizations That Rely Too Heavily on AI

Lyell et al. had 120 medical students in the later stages of their curriculum perform simulated electronic prescriptions, varying the correctness of the decision support. In low-complexity scenarios, correct support reduced oversights by 38.3%, whereas flawed support increased them by 33.3%. Flawed support produced 65.8% more commission errors than the no-support control condition. [1] The 65.8% is a relative difference in error counts between conditions, not the proportion of participants who accepted false warnings. Correct automation increases safety, but the same trust becomes a risk when support is flawed.

This study targeted medical students and simulated prescriptions, and is not a study of generative AI itself. Nevertheless, it indicates the unit of observation organizations should focus on. We must measure the accuracy of the human-AI coupled system, including whether humans can spot errors, not just the accuracy of the AI alone.

In evaluating the coupled system, solve the same pending cases by humans alone, AI alone, and the human-AI combination, recording oversights and incorrect adoptions separately. Error types where the human-AI combination performs worse than humans alone should be subject to usage suspension or independent verification, not just alerts. Do not directly transfer the amplification rates observed in medical simulation tasks; instead, redraw boundaries based on your organization's definition of correctness and loss. [1]

Among the 373 participants assigned to one outside-frontier task in the 758-person consultant experiment, the control group's accuracy was 84.5%, while the AI-using group scored 60.0% and 70.6%, an average drop of 19 points. Nevertheless, the AI-using group's responses received higher subjective evaluations for structure and persuasiveness. [2] This article treats fluent errors passing review as an operating hypothesis; the experiment did not measure review acceptance.

Instead of providing reviewers with only the final text, separately display input data, citations, assumptions, alternatives, and areas where the AI is uncertain. For high-impact judgments, design the process to prevent anchoring by having humans leave a provisional judgment before viewing AI output, having independent parties verify, or assigning roles to seek disconfirming evidence.

• Evaluate correctness and text quality separately
• Enable tracking of inputs, evidence, assumptions, and revision history
• Record provisional judgments before viewing AI output for high-stakes cases
• Strengthen independent review for judgments with high impact and low reversibility

Lee et al. collected 936 practical examples from 319 knowledge workers using generative AI. Higher trust in AI correlated with lower self-reported critical thinking and lower effort. Meanwhile, the center of critical thinking shifted from information creation to verification, integration, and oversight of the entire work. [3]

This is a cross-sectional survey based on self-reports and does not demonstrate a causal relationship that AI reduces critical thinking. However, it provides grounds for shifting training focus. Evaluate not only the skill to generate well but also the skills to verify sources, find contradictions, explain adoption reasons, and reject when necessary.

Reasons for the decrease in self-reported critical thinking include not only overconfidence in AI but also routing simpler cases to AI, and reviewers not counting verification actions as thinking. [3] To verify implementation effects, standardize case difficulty and track action logs spent on source verification, disconfirming evidence search, revision, and rejection. If self-evaluations and logs diverge, suspend conclusions about training effects.

Interventions requiring prior judgment reduced overconfidence in flawed AI, but stronger interventions were also reported to have lower usability scores. [4] However, this result does not directly indicate increased avoidance of use. The possibility that poor usability leads to reflexive rejection or procedural avoidance should be treated as an operational hypothesis for the organization to verify separately.

Safe judgment is not determined by the frequency of questioning AI. Underuse, where correct advice is discarded without basis, and overconfidence, where flawed advice is accepted, both lower the performance of the coupled system. Therefore, distinguish between correct adoption, correct rejection, incorrect adoption, and incorrect rejection, comparing them with standardized case difficulty and verifiability. Do not use usage or rejection rates alone as safety indicators.

The recommended adjudication is to display evidence and require independent verification when incorrect adoptions increase, and to review judgment reasons and necessary information when incorrect rejections increase. If both increase, consider capability gaps, time constraints, screen sequence, and ambiguous responsibility as competing hypotheses. Also verify that controls are not causing workarounds, and retest safety and executability in the same change unit.

Cases where humans and AI reach different conclusions are not to be resolved by majority vote but are samples for observing dependency risk. By separating the initial judgment, changes after viewing AI, referenced evidence, and final responsibility, one can distinguish between anchored changes and evidence-based corrections. This is a recommendation to ensure doubts do not disappear within the organization, rather than an evaluation to optimize synergy.

Consider a hypothetical example. If a staff member opposes an AI proposal but a superior adopts it without explanation, saving only the final correctness hides the weakness of the dissent channel. Link the reason for opposition, additional verification, the adjudicator, and the later result to the same case, and verify operations where speaking up does not lead to disadvantage, regardless of whether the opposition was justified. Leave unresolved disagreements in a pending state, not approved.

In recommended periodic reviews, do not aim to reduce the number of disagreements. Instead, observe the percentage of recorded dissent, the percentage of added independent evidence, time to adjudication, and recurrence of the same disagreement. If dissent from specific roles is always rejected, consider not just capability gaps but also power structures. Preserve adjudication insights as counterexamples for future cases, not as a universal answer key.

• Origin: Save the human's provisional judgment and AI proposal as separate, timestamped versions before they see each other
• Dissent: Record reasons for opposition, required additional evidence, impact, and whether it can be held
• Adjudication: Record the independent judge, reasons for adoption/rejection, conditional approval, and deadlines
• Learning: Link later results and recurrence to review the function of the dissent channel

Buçinca et al. compared designs that merely added explanations with cognitive interventions such as requiring independent judgment before viewing AI proposals in an online experiment with 199 participants. Cognitive interventions reduced overconfidence in flawed AI, but the designs that reduced overconfidence the most also had lower subjective usability scores. [4] Human verification is a matter of screen and procedure design, not just alerts.

Applying strong controls to all cases increases load and leads to hollowing out. Categorize cases by impact, discoverability of errors, and reversibility, combining sampling, evidence verification, dual approval, and usage suspension. Also, regularly test fallback procedures without AI in preparation for failures or quality degradation. Maturity is measured not by how much is delegated, but by the ability to question, stop, and revert when needed.

Business owners should periodically reenact representative cases that can be processed without AI, recording solo correctness, time to suspect anomalies, and completion of fallback procedures. Since cognitive interventions may impair usability while suppressing overconfidence, also place the increase in avoidance usage on the same monitoring table. [4] If solo capability or fallback procedures fall below the pre-set lower limit, the quality owner makes the decision to revert the scope of automation.

• Low impact / Reversible: Post-event sampling and log audit
• Medium impact: Evidence display, exception detection, explicit approval
• High impact / Irreversible: Independent dual confirmation, stop authority, fallback procedures
• All areas: Record errors and dissent as learning assets
• Quality Owner: Submit the error gap between humans alone and the coupled system to the approval meeting
• Business Owner: Correct incomplete AI-free exercises and fallback procedures
• Auditor: Record not only adoptions but also revisions, rejections, and dissent