More explanation does not automatically calibrate reliance on AI

Appropriate reliance cannot be measured by AI favorability or utilization rates. It is a state where proposals are adopted in cases where AI is correct and rejected in cases where AI is wrong. Increasing only the former increases overconfidence, while increasing only the latter increases underutilization, discarding even useful advice. Therefore, we preserve the four categories of correct adoption, correct rejection, incorrect adoption, and incorrect rejection, rather than using average agreement rates.

We also distinguish subjective trust from actual reliance behavior. Users may report trusting AI but not adopt its proposals, or report low trust while following AI on screen. In 2026 experiments, support methods were changed based on trust levels reported during iterative tasks, treating underutilization at low trust and overconfidence at high trust separately.[7] Organizations should not judge calibration by questionnaires alone but link them to logs of adoption, modification, and rejection.

This question differs from the synergy question of whether AI-human combinations exceed the best single entity. It also differs from controls preventing long-term skill loss due to AI dependence. The unit of this article is whether a user receiving a single piece of advice could correctly change adoption or rejection based on explanations or screen controls. Long-term capability and organizational recovery are left to adjacent articles; here, we make the moment of reliance experimentally observable.

• Correct adoption: In cases where AI is correct, the final judgment matches the AI proposal.
• Correct rejection: In cases where AI is wrong, the user rejects the proposal and reaches the correct answer.
• Incorrect adoption: In cases where AI is wrong, the user follows the proposal.
• Incorrect rejection: In cases where AI is correct, the user rejects the proposal without basis.

Cecil et al. conducted five pre-registered experiments with 1,403 participants, including students and HR practitioners, selecting candidates from resumes. They mixed correct and incorrect advice, varying the source (human or AI) and whether AI explanations were heatmaps or charts. Incorrect advice consistently lowered performance, and adding explainability had limited effects on suppressing overconfidence toward incorrect advice.[1] Increased visible information is distinct from the ability to distinguish correctness.

Explanations can become a second persuasive material rather than verification material. If the model merely reinforces its conclusion plausibly using the same features or text that generated it, incorrect answers and explanations appear coherent together. Although the HR selection experiment did not target free-text generation by generative AI, it shows that the mere fact of implementing an explanation should not be taken as evidence of safety.[1] Even in explanation conditions, incorrect adoption and incorrect rejection must be re-aggregated by correctness.

On the other hand, explanations serve purposes beyond performance, such as accountability, appeal, audit, and learning. This judgment does not negate those purposes. We avoid the holistic evaluation that 'transparency implies trust' and define who could verify what by use case. If displaying model feature contributions does not allow verification of business grounds, that display is transparency documentation, not an operation for reliance calibration.

In contrast, Senoner et al. compared black-box predictions with heatmap-assisted predictions in two pre-registered experiments using 9,600 product inspections by factory workers and 5,650 chest X-ray judgments by radiologists. The balanced accuracy of the heatmap group increased by 7.7 points in manufacturing and 4.7 points in healthcare.[2] It was reported that the rate of adopting correct predictions and overturning incorrect predictions moved in a favorable direction.

In this study, explanations showed where the image was viewed. Users possess professional knowledge and can return to the original image to verify if the indicated areas are valid as defects or lesions. Even with the same 'explanation present' condition as feature highlighting in HR selection, the target object, user knowledge, and localization of the ground truth differ. The research subjects were also limited to visual inspection with 14% positive images and heatmaps, preventing the transfer of effect sizes to text generation or business judgments.[2]

In five studies by Vasconcelos et al. using maze tasks totaling 731 people, overconfidence increased as tasks became difficult, explanation verification became difficult, and benefits of verification decreased. Conversely, in conditions where explanations sufficiently lowered the cost for users to independently verify AI proposals, overconfidence was reduced.[6] This is a behavioral science interpretation that the value of an explanation depends not on information volume, but on the cost and benefit for the user to reach falsification.

• Verification target: Ability to return from explanation to specific parts of the original data
• Independent knowledge: Users can judge correctness using grounds other than AI
• Verification cost: Time, screen transitions, and specialized work fall within acceptable limits
• Failure benefit: The value of finding an error is sufficiently greater than the verification cost

Steyvers et al. investigated whether users could distinguish answer correctness from explanations in multiple-choice and short-answer questions using GPT-3.5, PaLM2, and GPT-4o. The AUC for correct/incorrect discrimination by those reading default explanations was 0.589, 0.602, and 0.592 for the respective models and tasks, barely exceeding chance levels. In contrast, the AUC based on internal confidence of each model ranged from 0.746 to 0.781.[3] Fluent explanations did not accurately convey the model's uncertainty to humans.

Lengthening explanations increased human confidence but did not improve the ability to distinguish correct from incorrect answers. Conversely, aligning confidence expressions in explanations with model confidence improved human calibration and discrimination. However, final accuracy in multiple-choice tasks did not improve.[3] If users do not possess correct knowledge independent of AI, a gap remains between 'knowing it is suspicious' and 'correctly fixing it.'

Confidence displays do not solve the problem if they present unverified numbers. In two experiments by Zhang et al., model confidence for individual cases helped calibrate trust, but local explanations did not improve joint human-AI performance, and calibration alone was insufficient to improve final performance.[4] If displayed, the relationship between confidence and actual accuracy must be verified for each model and business task, and recalibration must occur after distribution changes. Do not use assertive tone in text as a substitute for probability.

Buçinca et al. compared cognitive interventions in an online experiment with 199 participants: requiring users to request AI proposals, presenting their own judgment first, and waiting before display, versus immediately showing AI proposals and explanations. The intervention groups reduced overconfidence toward incorrect AI more than simple explanation groups, but designs that strongly suppressed overconfidence had lower subjective evaluations, and benefits varied due to tendencies to prefer cognitive effort.[5] While friction is effective, applying the same intensity to all cases may lead to avoidance.

Prior judgment has the advantage of observing anchoring after seeing AI. By separately storing the initial answer, AI proposal, and final answer, one can determine whether AI corrected human errors or changed human correct answers to incorrect ones. However, presenting answers first does not necessarily reduce overconfidence. For users unable to answer tasks independently, fixation on initial judgments may increase underutilization, so correct adoption and incorrect rejection must be viewed simultaneously.

In 2026, Srinivasan and Thomason tested interventions providing supportive explanations when trust was low and opposing explanations or forced stops when trust was high during iterative tasks. In science questions and synthetic diagnosis tasks, conditional interventions reduced inappropriate reliance by up to 38% and improved final judgment accuracy by up to 20%. Constant explanations showed no such improvement and, in some diagnosis task conditions, worsened performance.[7] However, simple trust thresholds and small condition groups cannot be directly transferred to production rules.

• Low impact, easy verification: Display explanations optionally and conduct sample audits of incorrect adoption afterward
• Medium impact, human capable of independent judgment: Retain provisional judgment and reasons for change before viewing AI
• High impact, high trust state: Present opposing grounds, forced stop, and independent verification
• Low trust, AI superiority confirmed: Present supportive grounds and measure if incorrect rejection decreases

In pilot tests, randomly compare no AI, proposal only, constant explanation, conditional explanation, prior judgment, or forced stop. Include cases where AI is correct and wrong in advance, and measure the baseline accuracy of humans alone. If AI errors are too few, overconfidence cannot be estimated; if tasks are too difficult for humans to correct, the effect of explanations cannot be evaluated. Separate the main sample, close to actual usage frequency, from the safety sample designed to detect behavior during errors.

The primary metrics are correct adoption rate and correct rejection rate. Secondary metrics include final accuracy, judgment time, ground-viewing, changes from initial judgment, holding, re-verification, and subjective trust. If accuracy increases but incorrect adoption concentrates on critical cases, it is not a pass. If explanation viewing rates are high but discrimination does not change, it is interpreted as formal viewing rather than understanding. Stratify effects by user expertise, task difficulty, AI confidence, and explanation verification time.

Explanation design is not evaluated solely on text readability. Record the original data pointed to by explanations, falsifiable conditions, alternative candidates, uncertainty, and next available actions. Tests combining incorrect AI proposals with incorrect explanations are also necessary. Under the premise that explanations are always correct, one cannot measure whether users can doubt the explanation itself. Audit transparency documentation and calibration interfaces for use must be version-controlled with separate purposes.

In practice, first create a small set of representative cases from core business, linking initial human judgment, AI proposal, explanation viewing, final judgment, and ground truth or later results for each case with the same identifier. Next, fix 'how many points of incorrect adoption are tolerable' and 'how much incorrect rejection should not increase' before adoption. Do not offset one harm with the other in a total score; set two upper limits for harm as separate gates.

If explanations increase only correct adoption without changing incorrect adoption, they are candidates for adoption within that task scope. If both incorrect adoption and incorrect rejection decrease, they can be expanded as conditional interventions. If either worsens, causes include explanation content, display order, verification time, user independent knowledge, and AI confidence calibration. Until causes are separated, narrow the scope of use or revert AI proposals to second opinions.

If any of the model, prompt, search, explanation generator, confidence calculation, or screen order changes, the same calibration may not continue. Upon update, re-run the fixed safety sample and compare both correct adoption and correct rejection with the previous version. Appropriate reliance is a state where AI capability, task, explanation, time, and responsibility combine, not user personality. Therefore, manage not only training but also interfaces and measurements in the same unit of change.

• Adoption: Correct adoption increases, and incorrect adoption and incorrect rejection remain within pre-set upper limits for harm
• Conditional adoption: Use only within the range where effects are replicated by expertise, task, confidence, and verifiability
• Reduction: Categories where incorrect adoption increased are reverted to second opinions, independent verification, or holding
• Withdrawal: Stop support in uses where major incorrect adoption exceeds pre-set upper limits for harm even after redesigning explanations
• Re-test: Re-measure the four categories after changes to model, explanation generation, screen order, or thresholds