Gathering any number of AIs does not result in independent wisdom.

Du et al. demonstrated that discussions in which multiple language models exchange answers and critiques improved performance on selected reasoning and factual tasks. [1] This is important evidence indicating that mutual review can be used for error correction.

On the other hand, the evaluation is based on specific models, tasks, and discussion protocols. [1] This does not conclude that discussions are better than single answers for all tasks, or that longer discussions are always better.

Upon implementation, track not only the final correct answer but also at which turn the error was found and on what basis it was corrected. Do not count mere paraphrasing of answers as the number of critiques.

Divergent initial hypotheses or new independently verified evidence may help debate improve an answer. Check whether agents identify others' evidence and update toward warranted reasoning. If everyone shares an error and no new verification evidence enters, debate may merely strengthen confidence.

Candidates for application are tasks where basis comparison and counterexample examination are possible. In tasks where external correct answers can be verified immediately, verification tools may be cheaper and more reliable than discussions.

• Initial hypothesis: Are different candidates actually present?
• Critique content: Which aspects of the basis, calculation, or premise were examined?
• Reason for update: Was it changed based on evidence rather than the authority of the other party?
• Final improvement: Did it become more accurate than simple regeneration?

Answers created from the same base model, same prompt, and same search results may have correlated errors even if their formats differ. I do not assume that the independence required for majority vote is satisfied merely by differences in agent names.

Measure the semantic diversity of initial answers, the overlap of bases, and the alignment of error types. Even if expressions are diverse, if the conclusions and premises are the same, the number of independent hypotheses has not increased.

Diversity can be increased by changing the model series, temperature, and role instructions, but this also increases random noise. Evaluate not diversity itself, but the probability that correct candidates are included and their identifiability.

When separating external data, record the difference in the quality of the bases. Do not confuse a situation where only one person has older materials with pure reasoning diversity. In the final judgment, compare the timeliness and authority of the bases.

Even if everyone agrees on the same incorrect answer, do not make the agreement rate equal to reliability. Measure how often unanimous incorrect answers occur in a known set of correct answers, separating agreement from accuracy.

Khan et al. investigated a setting where discussions between strong models with information are judged by weak models or people without information, reporting results where discussions helped identify truth. [2] This addresses the important question of how to evaluate models that are more advanced than supervisors.

However, in the research setting, correct answers are managed, and discussants advocate for different positions. [2] This cannot be directly transferred to ambiguous judgments in practice or situations where both parties have the same missing information.

To separate persuasiveness from correctness, have judges present the basis locations, counterarguments, and unresolved points. Test whether winners are chosen solely based on fluency or confidence using known incorrect answer discussions.

If the judge is an LLM, audit its position, redundancy, and self-preference. Do not close the judgment within the same series as the discussion participants; set conditions for disagreement by sending to humans or a different series.

Even when humans judge, there is a load in reading long discussions. Since summaries may miss important counterarguments, provide displays that allow tracing basis links and changes per turn.

• Basis presentation: Link claims to verifiable locations
• Counterargument presentation: Retain the opponent's strongest counterexample
• Unresolved points: Do not hide premises that could not be agreed upon
• Judgment limits: Send conditions that the judge cannot understand to humans

In discussions, an agent that was initially correct may change to the majority's incorrect answer. Models may treat the other party's expressions of confidence or the number of people as evidence, potentially following social cues over content.

Save each agent's initial answer privately and track changes per turn. Relying solely on the final correct answer rate offsets the number of errors corrected by the discussion against the number of correct answers destroyed.

Set up a control where the opponent's name, model name, and past win rate are hidden, and re-evaluation is done based on content alone. If judgment changes with anonymization, weight is being placed on identity.

When making majority vote the final rule, do not discard the bases of minority opinions. If only the minority cites primary sources, prioritize basis verification over vote count.

In areas with strong conformity, try reducing continuous dialogue and performing basis comparison only once after independent generation. Do not consider increasing the amount of conversation as improvement.

Becker et al. analyzed problem drift, where multiple agent discussions drift from the original problem as turns increase, harming performance. [3] Long discussions accumulate not only additional information but also misunderstandings and digressions.

At each turn, keep the original question, constraints, and unresolved points fixed, and record which requirements each statement advanced. If new claims are made, request bases, and do not bring unverifiable branches into the final answer.

Repetition of the same points, low-quality critiques, and ambiguous questions are signs of drift. [3] Set not only a turn limit but also early stopping if there is no progress.

Re-injecting a long discussion into a single prompt may bury important initial conditions in the middle. Structure requirements and bases, and do not rely solely on the full conversation history.

In subjective creative tasks, the answer space is wide, and determining drift is difficult. Do not mechanically transfer stopping rules obtained from tasks with clear answers to value judgments.

Multiple models and long discussions increase inference costs and response times. The comparison target should not be just one cheap answer, but also self-consistency, search, verification tools, and human verification under the same budget.

Even if discussions outperform majority vote, if external calculators can easily detect the same errors, the reason for choosing discussions becomes weaker. Prioritize the most direct verification method for each task.

Costs include not only tokens but also log storage, judgment, latency, and failure investigation. Even if discussions enhance explainability, generating a large amount of incorrect reasoning increases the verification load.

Added value is measured by the difference in correct answers compared to simple methods, reduction in critical errors, improvement in abstentions, and verification time. Even if the absolute score is high, do not adopt it if it does not exceed the control under the same budget.

Also evaluate routers that send only some difficult cases to discussion. Include the cost of mistakenly sending many easy cases and the loss of missing truly difficult cases.

Production logs should save initial answers, each critique, reasons for change, citations, final aggregation, and judges. The final text alone cannot reproduce the process by which correct minority opinions were lost.

Minimize necessary information by role to avoid replicating personal or confidential data across multiple agents. Consider that as the number of discussion participants increases, so does the data exposure surface.

When updating models or prompts, re-evaluate initial diversity, correct answer retention, drift, and costs. Even changing just one participant alters interactions, so do not consider partial changes minor.

Stopping conditions include increased destruction of correct answers, drift, conformity, critical errors, and cost overruns. After stopping, maintain a path to revert to a single model or human process.

The value of multiple AIs is the difference by which independent hypotheses and verifiable critiques improved the final judgment. Use discussions only within the range where this difference exceeds simple methods under the same budget.

The same-budget comparison record fixes the task set, agent roles, aggregation method, and budget, with a minimum quality gain over the simple method and maxima for cost, latency, and severe errors. Debate stops for a use case that misses the gain or exceeds a maximum, reverting to the simple method under the same budget. Restart requires joint approval by the evaluation and FinOps owners of an independent retest after changing roles or aggregation.