Top public benchmark rankings do not determine production quality

HELM applied seven metrics, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, to sixteen core scenarios to the extent possible. Furthermore, it standardized and compared thirty language models across forty-two scenarios, explicitly identifying areas that were not evaluated. [1]

In production, equal accuracy does not necessarily mean equal value; for customer service, harmful expressions and deferral or abstention rates matter; for extraction, evidence consistency is key; and for development support, test pass rates and maintainability are important. If we do not view quality, failure impact, latency, cost, and reproducibility on the same table using the same business samples for each candidate model, we will overlook conditions that have worsened behind the scenes of improved metrics.

Before creating the evaluation table, we separate the metrics prioritized by business judgment from the guardrails that must not be exceeded. This is to prevent candidates with improved accuracy but worsened critical errors or harmful outputs from being rescued by a total score. If the differences between candidates reverse depending on the user or failure impact category, the condition for translating HELM's multi-faceted evaluation into practice is to shift from a single ranking to adoption decisions based on use case. [1]

Zheng et al. reported that GPT-4 scoring aligned with human preference in over 80% of cases using MT-Bench and Chatbot Arena, matching the level of agreement between humans. The publicly available evaluation resources include approximately 3,000 expert votes and approximately 30,000 human-preference-labeled conversations. [2] As a method for comparing a large number of candidate responses against fixed criteria, LLM scoring is a strong candidate.

However, the same study analyzed tendencies such as preference for answer order, verbosity, preference for answers similar to itself, and limitations in reasoning ability. [2] Therefore, do not treat the scorer model's scores as ground truth labels; instead, verify the agreement rate and types of disagreement using human samples for each evaluation criterion. Cases with high-impact errors, small differences between models, or scoring reasons that do not match the evidence should remain as layers returned to human review.

Before adoption, the LLM scorer creates a disagreement table with human evaluation. If scores change when reversing answer order, equalizing length, or hiding the model name, scoring bias serves as an alternative explanation to quality difference. The criteria for remaining disagreements that miss critical errors are not passed to automated scoring; the owner determines the frequency of human sample re-verification and the scorer's stop conditions. [2]

Sclar et al. evaluated multiple open models in few-shot settings by changing the delimiters and formatting of examples while preserving meaning. For LLaMA-2-13B, formatting differences alone caused a gap of up to 76 accuracy points, and sensitivity remained even when the model size was increased or the number of examples was expanded. [3]

The score obtained from a single template is a local observation of that template. In practical evaluation, prepare a group of valid prompts with different word orders, delimiters, examples, output formats, and lengths, and examine not only the average but also the lower-tier performance and variance. When changing prompts, raise the evaluation version as with model updates, and operation must ensure that previous passing results are not automatically inherited.

Prompt robustness is viewed not only by the average of valid expression groups but also by the worst results and rank reversals. If only specific tasks drop when formatting changes, template dependency is suspected rather than overall model degradation. When shifting to uses where users input free text, do not inherit passing results from fixed templates; re-evaluate based on the actual range of expressions. [3]

Multi-faceted evaluation, LLM scoring bias, and prompt format sensitivity indicate that model rankings are not attributes independent of evaluation conditions. [1] [2] [3] Therefore, instead of asking "which model is best," we freeze the full set of evidence for each conclusion unit of "which candidate to adopt for which use case under which conditions," and also set an application expiration date.

The conclusion ticket links the compared candidates, users and business, input samples, prompt groups, scoring criteria, scorers, impact of failure, and unevaluated areas. Do not extend rankings to uses not included in the conclusion; for categories with small candidate differences or high scoring disagreement, leave them as equivalent only if the estimate meets a predefined equivalence margin with adequate precision; otherwise inconclusive rather than superior/inferior. Display user groups not in the samples as out of scope.

In the sensitivity column, record whether the adoption decision reverses when only one element among scorer, answer order, answer length, or prompt format is changed. If it reverses, do not conclude a quality difference between candidates; instead, rule on which conditions are considered representative of the business. Save all conditions to avoid keeping only favorable ones in the final conclusion, and verify the combination of multiple conditions and the rationale for selecting the representative condition.

Cases where human and LLM, or human and human, scorers disagree are not averaged out but sent to a disagreement ticket. First, anonymize the answers, and have each scorer independently indicate the evidence location and the applied criteria. Classify differences in ground truth knowledge, criterion interpretation, preferences, oversights, and ambiguity in the problem itself into separate cause codes. If the disagreement cannot be resolved, leave it as undetermined.

The adjudicator judges by returning to the pre-determined criteria and evidence, not by the scorer's title or model name. If the problem is ambiguous, do not finalize it as a deduction for the candidate; instead, modify the evaluation sample and retain the comparison with the old version. Differences in preference are not treated as quality differences; add to the evaluation contract what is required for the business. If adjudication is impossible, narrow the scope of the adoption decision.

Even after the disagreement ticket is closed, save the original scoring, adjudication results, and cause codes. If the same cause recurs, review the criterion text, counterexamples, and sample ambiguity before adjusting the scorer's weights. When adding adjudicated cases to the correction samples, verify that they do not bias toward examples favorable to specific candidates, and leave a one-line record of the adoption reason.

• CLAIM: Fix the adoption conclusion including use case, conditions, and candidates in one sentence.
• FREEZE: Link samples, prompts, scoring criteria, and scorers to the conclusion.
• DISAGREE: Attach evidence and cause codes, and adjudicate disagreements independently.
• REVERSE: Record evaluation conditions that reverse the adoption decision and unevaluated areas.

First, create representative samples from the work the user actually performs. Include not only typical cases but also ambiguity, insufficient information, contradictions, requests to be rejected, and critical errors. For generative tasks with no single correct answer, place specific evaluation criteria and counterexamples for good/bad, allowing scorers to record what they saw.

Next, unify pre-publication fixed evaluation, shadow operation using real traffic, limited release, and post-operation sampling into one flow. Perform differential evaluation every time the input distribution, model version, system prompt, search index, or scorer changes, and determine conditions for rollback if critical failures increase. Benchmarks retain value when used as observation devices to detect changes rather than as selection tables.

The evaluation contract specifies the person responsible for approving changes, the person responsible for maintaining failure samples, and the person responsible for executing rollbacks. If any of the model, prompt, index, or scorer changes and breaks the critical failure guardrail, revert to the previous combination regardless of average quality improvement. Include failures that occurred in production in the fixed evaluation for recovery judgment.

• Target: Include ambiguity, insufficient information, contradiction, rejection, and critical errors in addition to representative tasks.
• Scoring: Record disagreements between humans and LLMs, sensitivity to position and verbosity, and consistency with evidence.
• Robustness: Compare averages, lower-tier performance, and rank fluctuations using prompt variants that preserve meaning.
• Change: Maintain separate versions for models, prompts, indices, and scorers to allow independent rollback.