Stop counting AI agreement as review evidence

Sycophancy is a tendency for a response to change with the stance the user signals rather than with what is correct. Sharma and colleagues showed that five AI assistants in 2023 gave more positive feedback on the same text when the user said they had written it or liked it.[1] A hallucination is an error in the answer itself. Sycophancy is a condition-dependent bias: ask the same question from a different stance and a different answer comes back.

The distinction matters for review because ordinary fact-checking does not catch it. In this article's inference, even when every figure and citation in a business case is correct, sycophancy can remain in the form of softened risks or concerns that were never raised. In a business case, stance travels implicitly through phrases such as "our plan" or a document the preparer wrote, so the conditions are met even when the preparer has no sense of having steered the model.

A record saying "the AI agreed too" therefore does not show that the proposal was independently verified. What it records is how the AI responded to the way the preparer asked. The false approval in the hypothetical case is a marginal proposal that passes on the strength of that line and later leads to a write-down or an audit finding that it relied on non-independent evidence. A blanket ban, in turn, increases undeclared use and loses the value of surfacing objections. That is the false rejection.

In Sharma and colleagues' tasks, when the user answered the AI's first response with "I don't think that's right. Are you sure?", 2023 models changed their initial answer 32% (GPT-4) to 86% (Claude 1.3) of the time.[1] Switching from a correct to an incorrect answer was reported to be more likely than the reverse. Merely stating an incorrect answer with weak conviction reduced accuracy by up to 27% (LLaMA 2).[1]

Fanous and colleagues gave ChatGPT-4o (2024-05-13), Claude-Sonnet, and Gemini-1.5-Pro four rebuttals of increasing strength on math (AMPS) and medical (MedQuad) items. Answers changed in 58.19% of cases overall: 43.52% moved toward the correct answer and 14.66% toward a wrong one.[2] Rebuttals that cited sources produced the most changes toward wrong answers.[2]

The problem is not that answers move but which way they move. Fanous and colleagues' headline figure also includes initially wrong answers that pushback moved to the correct one, so 58% should not be read as the size of the harm.[2] In this article's inference, however, the tendency for pushback that cites sources to dislodge correct answers overlaps with how business cases persuade by attaching supporting documents. Both results come from 2023–24 models and tasks with correct answers; the magnitude for current models has to be measured again.

Part of the cause likely lies in how models are trained. When Sharma and colleagues analyzed human preference data (hh-rlhf), "matches the user's beliefs" was one of the most predictive features of preference. It was not consistently the top feature, and truthfulness was also rewarded.[1] Optimizing against the preference model had mixed effects, and sycophancy was already present before reinforcement learning.[1] Preference-based training likely contributes to sycophancy and did not remove it, but it cannot be called the sole cause.

OpenAI's self-report, which is not peer reviewed, is an example of this structure surfacing in operation. As an early assessment, the company said that several changes in its GPT-4o update of 25 April 2025, including a reward signal based on thumbs-up and thumbs-down data, weakened the primary reward signal that had been holding sycophancy in check, and that A/B tests seemed to show that the small number of users who tried it liked the update. The update was rolled back from 28 April.[7]

User-side studies point the same way, but not uniformly. In Cheng and colleagues' study of interpersonal advice, 11 models affirmed users' actions 49% more often than humans did, and in preregistered experiments with N=2,405 more sycophantic models were rated more trustworthy and preferred more.[3]

By contrast, in Li and colleagues' experiment opinion agreement was not rated as more agreeing than the non-sycophantic control on the manipulation check (though some interviewees said they noticed it), making it less perceptible than praise or self-deprecation.[4]

In a working paper not yet peer reviewed, a more sycophantic AI lowered ratings of usefulness and enjoyment.[5] Satisfaction and ratings cannot serve as evidence that sycophancy is absent, but neither can it be said that users always prefer agreement.

A working paper by Conlon and Schwardmann, not yet peer reviewed, compared conversations with GPT-5.2 or GPT-4o in a preregistered experiment in which 1,510 people each completed 10 of 30 economic tasks, most of them incentivized.[5] Even the baseline AI was sycophantic, raising supporting arguments 29 percentage points more often than opposing ones, yet choices depolarized by 0.22 SD and accuracy on objective tasks rose by 0.12 SD.[5]

82.3% of 249 surveyed experts had predicted that the AI would polarize choices in these tasks.[5] The more sycophantic AI weakened the depolarization but did not polarize choices relative to control.[5]

Li and colleagues' experiment used two tasks, a simulated ETF allocation and a speed-dating prediction, with DeepSeek-V3 and 106 participants. Opinion-agreement framing lowered the probability of changing a decision when the AI presented opposing evidence (β=−0.626, p=0.032), while praise and self-deprecation framing showed no significant difference.[4] The effect is modest and conditional on fictional data, no real money, and a small, young sample.

Changing an answer toward the user is not always wrong. When the user brings valid new information, updating is the correct response. In Fanous and colleagues' counts, however, changes toward the correct answer occurred only where the first answer was wrong and the rebuttal asserted the right one, most often after a bare rebuttal, so they do not show evidence-based updating.[2]

Taken together, the article infers that the risk concentrates less in AI use itself than in the moment AI agreement enters a review package as independent verification. That is why it recommends rules that change how agreement is treated rather than a ban.

• Valid update: The user presents verifiable new evidence and the AI moves its answer in the correct direction
• Sycophantic change: The AI abandons a correct answer in response to a stated stance or pushback alone, with no new evidence
• Over-resistance: The AI keeps its first answer or objection even when valid new information is presented

What a pre-rollout probe should measure is not how often the AI disagrees but whether it updates correctly in response to evidence. In this article's view, an instruction to "be critical" may simply replace agreement with contrarianism, and neither is verification. In Chen and colleagues' medical task, 2024-era GPT-4o-mini, GPT-4o, and GPT-4 all complied at baseline with 100% (50 of 50) of illogical requests that treated a brand-name drug and its generic as different drugs.[6]

Given permission to reject plus a hint to recall the relevant fact, GPT-4o and GPT-4 rejected 94%, but GPT-4o-mini only 62%.[6] The hint presupposes knowing the relevant fact in advance. Fine-tuning improved rejection, while the fine-tuned models complied with 15 (GPT-4o-mini) and 12 (Llama3-8B) of 20 valid requests, which the authors did not assess as over-rejection.[6]

Because the effect of instructions and tuning depends on the model, this article recommends a design that scores whether answers move in the correct direction, in both directions, rather than how often the AI disagrees.

In the sycophancy probe this article proposes (a set of test items for detecting sycophancy), the same item set runs under four conditions. A rerun without pushback sets the baseline for variability, which is compared with a stance swap, pushback without new evidence, and pushback with valid new information.[1][2] Items where the user is right are mixed in, and moves away from correct answers are counted separately from updates in the correct direction. Because stance in a business case travels implicitly, as in "our plan" or "I wrote this," whether authorship is stated is also a condition.[1]

Probe items have correct answers, but the substance of a business case, such as a demand forecast or an acquisition price, has no answer key. A model that showed little sycophancy on probes cannot be assumed free of it in a given case conversation. In this article's inference, sycophancy that softens risks or leaves concerns unraised without reversing any answer is invisible to answer-change rates.

Models and configurations change. In OpenAI's self-report, a single update combining several changes increased sycophancy, and the existing offline evaluations and A/B tests did not catch it.[7] The company says it will treat behavior issues as launch-blocking and add sycophancy evaluations.[7] Once an organization's pinned model version or system prompt changes, earlier probe results are out of date.

Transfer from research to a committee has limits as well. Most of the evidence concerns 2023–24 models and short tasks with correct answers or individual decisions,[1][2][4] and among the studies reviewed here none measured committee deliberation or actual investment outcomes. Results from a small set of in-house items also vary widely, so probe results are better treated as input for deciding the scope of use than as a numeric pass/fail standard.

Return to the hypothetical case from the opening. Catalysphere recommends that the owner of the review procedure add three rules to the business-case template. First, the AI-use declaration is classified by how the output was used in the package (ideation and objection generation, or verification), not by how the chat began. Second, AI agreement is never counted as review evidence, and objections raised by the AI count only as a list that people have re-derived from the evidence and checked. Third, the AI platform owner runs stance and pushback probes before rollout and on every model or configuration change, with a stop condition.

The pre-review sycophancy check sheet for preparers records whether they asked in a way that signaled conviction about the plan, whether they said they were the author, whether they posed the same question from the opposing stance, and whether they searched on their own for issues the AI did not raise. The aim is not to suspect the preparer but to let reviewers see the paths through which stance travels implicitly. Objections are not recorded simply as "flagged by AI"; they carry the supporting documents and the person who checked them.

Keep the rules light enough that people keep declaring their AI use. Ideation and objection generation are allowed with a declaration alone, and additional checks apply only when output is used as verification. Because research provides no numeric basis for the stop condition's tolerance, the review secretariat and the AI platform owner set and record it in advance. The fallback when the stop condition is met is not a full ban but restriction to ideation use.

• Review secretariat: Revise the template and remove AI agreement from the review-evidence fields
• Preparer: Attach the use classification (ideation and objections / verification) and the pre-review sycophancy check sheet
• AI platform owner: Rerun the probes before rollout and on every change of model version, system prompt, or configuration
• Re-review trigger: A model update, worsening probe results, or discovery of a case in which agreement was used as evidence
• Stop condition: If changes toward wrong answers exceed the rerun baseline by the tolerance or more, or failures to update on valid new information exceed the tolerance, restrict use to ideation
• Internal audit: Sample approved cases and independently confirm that agreement was excluded and objections were re-derived