When using an LLM as a judge, who audits the judge?

Zheng et al. demonstrated that evaluation using strong LLMs highly agrees with human preference under certain settings and can be used for large-scale model comparison. At the same time, they reported biases such as position, redundancy, and self-enhancement. [1] Utility and bias coexist within the same study.

The agreement rate with humans varies depending on the difficulty of the evaluation examples, human consensus, and scoring criteria. Even when agreeing on answers clearly preferred by the majority, it does not necessarily mean that missing legal grounds or subtle logical leaps can be identified. We do not use the average value as the sole indicator for acceptability.

First, we decompose the judgments entrusted to the evaluator. Fluency, instruction following, factuality, grounding fidelity, and safety are separate criteria. Combining them into a single overall score may allow long, well-structured answers to obscure significant factual errors.

The criteria table includes not only passing examples but also borderline cases that are superficially good but fail. We fix scoring rules with counterexamples such as having citations that do not support the content, being polite but not answering the question, appearing safe but enabling prohibited acts.

• Criteria separation: Score fluency, accuracy, grounding, and safety separately
• Consensus confirmation: Separate examples where human evaluators disagree into a distinct category
• Difficulty stratification: View accuracy separately for clear differences and subtle errors
• Overall score limitation: Prevent major failures from being offset by average scores

In pairwise comparisons, position bias becomes an issue where the judgment changes if the presentation order of the same two answers is swapped. Wang et al. analyzed the lack of fairness regarding answer order and demonstrated the need for evaluation with swapped orders. [2] A single judgment cannot separate preference for content from preference for position.

In production audits, we hide candidate and model names and perform two judgments with reversed answer orders. Examples where the winner changes due to reversal are not automatically determined but are saved as a set of disagreements. Before proceeding to simple majority voting, we verify whether the judgment reasons refer to the same criteria.

Simply offsetting position bias with averages leaves errors in individual high-impact judgments. We use the disagreement rate as a quality indicator for evaluators and look for where biases concentrate by domain, answer length, and criteria. In high-impact domains, we send reversal disagreements to humans.

Reversal tests also verify the reproducibility of the same evaluation call. To avoid confusing output fluctuation with position effects, we separate repetitions with fixed settings from those with only the order changed. If probabilistic fluctuation is large, we do not treat a single vote as a definitive judgment.

Redundant answers may appear more detailed and be preferred even if the content is equivalent. Additionally, if the evaluator and generator are from the same model lineage, self-preference for similar writing styles or answers may occur. [1] This systematically distorts comparison results, even if not intentional misconduct.

We compare a summarized version with controlled length to the original text to see if the conclusion changes. We include human-created reference answers to ensure that necessary grounds are not lost due to shortening. This is not a task where length itself is quality, but an audit among answers meeting the same requirements.

We record the relationship between the generation model and the evaluation model, and do not close evaluation solely with the same lineage. We maintain a hold set using different lineages or human evaluation. If multiple evaluators share the same learning tendency, increasing the number of evaluators does not necessarily increase independent evidence.

To investigate self-preference, we keep the generator hidden while reformatting the same content into different styles to see if the outcome changes. Regeneration involving content changes mixes causes, so we separate manipulation of surface form from manipulation of the issue. We also save the changed judgment reasons.

• Anonymization: Remove model names, product names, and candidate symbols from evaluation inputs
• Length control: View judgment stability with shortened versions that maintain requirements
• Lineage separation: Do not finalize based solely on evaluation by the same lineage as the generator
• Reason preservation: Record which criteria and points were used to determine superiority or inferiority

JudgeBench collected difficult answer pairs with objective ground truth labels and showed that even strong evaluation models see significant performance drops on difficult cases. [3] This does not mean LLM evaluation is worthless, but that simple preference sets overestimate capabilities.

Difficult cases are created from errors that occurred in production. They include subtle differences such as correct citations but leap in conclusion, missing one premise, different numerical denominators, and euphemistic violation of prohibitions. Experts fix the reasons for the correct answer and cross-check them with the evaluator's explanations.

Even if the model selects the correct winner, reproducibility remains weak if the reason is incorrect. We separate the win/loss, criterion-specific scores, grounding points, and correctness of the reason. We do not treat the reason as the final truth but as an auditable intermediate record.

Setting the ratio of difficult cases too high may lead to an underestimation of overall production processing performance. We report the representative set and stress set separately, with the former measuring operational frequency and the latter measuring safety boundaries. We do not mix them into a single average.

Returning everything to humans loses the value of automated evaluation, while leaving everything to models makes bias invisible. We pre-define conditions for sending to humans as reversal disagreement, inter-evaluator disagreement, low scores on major criteria, known difficult case types, and insufficient grounding.

Human judgments are not perfect either. We provide human evaluators with the same criteria table and examples, and record disagreements among multiple people. In areas with low human consensus, we review the evaluation task and labels themselves before calculating the LLM evaluator's accuracy.

We return decisions after escalation to the next audit set. However, we do not pass production data directly to learning or external services. We verify anonymization, usage rights, and retention periods, ensuring evaluation improvement and confidentiality protection follow the same procedures.

We place capacity planning rather than a cap on the proportion returned to humans. If the number of disagreements increases but the submission quota is fixed, important examples will remain in the queue. We link re-review deadlines and priorities to impact to prevent unprocessed items from becoming automatic approvals.

If the evaluation model, prompt, temperature, answer order, or criteria table changes, the judgment of the same answer pair may also change. Rather than viewing the evaluator as a transparent fixed object, we version control it like the generation model and re-run the reversal and difficult case sets before and after changes.

Operational metrics are not the number of processed cases, but the stability of order reversal, difficult case identification, missed major failures, submission rate to humans, and overturn rate in re-review. If the submission rate decreases but the overturn rate increases, we do not judge that automation has improved.

LLM evaluators are powerful tools for large-scale comparisons with low impact and clear criteria. Precisely for this reason, we determine where trust is possible and where to return to humans not through the model's self-reporting, but through reversal tests and known ground truths.

In approving changes to the evaluator, we first check if missed major failures have increased even if average agreement has improved. We extract disagreements with the old evaluator, and humans review the differential reasons. We record both the basis for improvement and remaining biases in the change log.

From evaluation logs, we track not only the generated answer but also the input to the judge, criteria table, output, reversal results, and final human judgment. When bias is discovered later, we identify which acceptance or rejection decisions were affected and re-score if necessary.

If the vendor updates the evaluation model without notice, we cannot expect reproducibility equivalent to the fixed version. We place update detection, holds, and regression audits in both contracts and technology. After an unauditable update, we do not simply inherit the existing scope of automatic approvals.

The regression-audit record separates evaluator version and input order from acceptance criteria: maxima for position effect and scoring delta, and a minimum agreement rate with blind human review. A version that breaches a maximum or misses agreement is removed from automatic passing, which returns to the fixed evaluator or human review. Restart requires the evaluation and model-risk owners to approve clean regression on both the fixed item set and blind human review after the cause is identified.