Does that benchmark score measure unseen capabilities?

Test contamination, where evaluation examples enter training, obscures whether a model has generalized to future unknown examples or reproduced known examples. Deng et al. investigated contamination by combining search duplication with public corpora and Testset Slot Guessing, which fills in incorrect answer choices. [1]

Merely knowing the benchmark name is not evidence of contamination. The problem text may be widely cited or completable from common sense. Conversely, even without exact matches, the model may have learned the substantive answer from explanations, translations, or derived problems.

In decision-making, we assign evidence categories of 'clean,' 'suspect,' and 'unknown' to scores. Instead of immediately excluding models that cannot prove they are unknown, we lower the weight given to the score and verify it with other evaluations.

We also avoid assuming that high scores are solely due to contamination, as actual capability, contamination, and adaptation to problem formats may contribute simultaneously. We compare explanations by examining whether scores excluding contaminated items and scores measuring the same capability in different formats are maintained.

• Publication history: Record when and where the problem and answer were published.
• Duplication inspection: Distinguish between exact matches, approximate matches, and explanations/translations.
• Behavior inspection: Investigate abnormal sensitivity to completion and order.
• Evidence categorization: Report clean, suspect, and unknown alongside scores.

In open training corpora, one can search for string duplication between evaluation problems, correct answers, and explanations. Deng et al.'s search-based inspection provides more direct clues than looking only inside the model, but is limited to available corpora. [1] It does not constitute an examination of non-public data portions.

We consider not only exact matches but also removal of punctuation, changes in option order, translations, and the publication of problems and answers on separate pages. However, we do not conclude that learning was utilized solely based on high similarity. We link the publication time, crawl time, and training end time to stage the possibility of inclusion.

Upon discovering duplicated examples, we do not invalidate the entire benchmark en masse, but instead examine the contamination rate and score contribution per item. We recalculate rankings excluding contaminated items, domain-specific rankings, and confidence intervals to verify whether management decisions change.

Searchable corpora have issues of omission and versioning. We do not treat non-finding as zero duplication, but record the search targets, index date, normalization rules, and similarity thresholds. We ensure the state is such that re-inspection can be performed using the same procedure when new corpora become available later.

Golchin and Surdeanu proposed guided instruction, which provides the dataset name, split name, and beginning portion, and asks the model to complete the latter half. They reported high detection accuracy on a limited validation set. [2] However, completion matches are a sign of memory, not complete provenance proof of training data.

We compare the same problem under general instructions versus instructions specifying the dataset, observing whether the latter unnaturally reproduces the continuation only. If the model can use web search or external tools, we isolate those factors. We do not rely solely on LLMs to judge output similarity, but combine string metrics with human verification.

The inability to complete is also not proof of non-contamination. The model may have learned but forgotten, refused, or changed its expression. The inspection is not a proof of innocence, but is treated as asymmetric evidence to suspect scores.

We also verify that the inspection prompt itself does not guide the answer. We create controls by removing each element—dataset name, split name, and beginning sentence—one by one, and separate which clues increased the matches. We do not collapse the results into a single contamination probability.

• Comparison with general instructions: Observe if reproduction occurs only when the dataset name is specified.
• Blocking external functions: Do not confuse answers from search/tools with training memory.
• Multiple metrics: Distinguish between exact matches, semantic matches, and human verification.
• Asymmetric interpretation: Positives indicate suspicion; negatives do not prove non-contamination.

Oren et al. presented a method to statistically test contamination in black boxes using the model's likelihood for permutations of interchangeable test examples. [3] This approach uses a different sign than verbatim reproduction of problem text, based on the idea that the model gives high likelihood to specific orders seen during training.

This method also has premises: that necessary probability information can be obtained from the model, that examples are originally interchangeable, and that there are no other rules for the publication order. [3] It cannot be directly applied to APIs that do not return probabilities or to evaluations where order has meaning.

When different inspections suspect the same item, the evidence strengthens. When results conflict, we do not conveniently adopt one, but leave the lack of training data disclosure, inspection sensitivity, and existence of derived examples as unresolved.

When repeating statistical tests on multiple sets, false positives increase. We fix the inspection targets and criteria in advance, separating exploratory discovery from confirmatory judgment. We show not only significance but also which scores are affected in practice.

Rather than just continuing contamination inspections, we create evaluations that are hard to contaminate. We combine problems created after the model's training end, problems with restricted access, and problems updated periodically. After publication, future cleanliness decreases, so we specify the version and usage period.

Simple paraphrasing of existing problems may still favor models that have learned the same solution or explanation. We change not only the surface but also the target cases, necessary reasoning, and structure of incorrect answers. If difficulty changes, we do not directly compare with old scores.

Non-public sets also have leakage risks. We manage accessors, distribution destinations, submissions to evaluation services, and log storage. We do not treat secrecy alone as quality, but independently verify validity, representativeness, and correct answer justification by experts.

If evaluations are sent to external APIs, there is a concern that they may enter future learning or retention depending on the provider's terms. We verify contracts and settings, and execute in isolated environments if necessary. After use, we record which version was passed to which model and update the clean period.

In the procurement table, we place contamination evidence, uninspectable portions, publication time, and re-evaluation results next to the benchmark score. Mixing suspect scores into the average hides uncertain evidence with precise numbers. We aggregate separately by cleanliness category.

If the ranking changes upon excluding contaminated items, we suspend model selection and prioritize our own non-public business sets. Even if the ranking does not change, we do not finalize adoption until the quality lower bound for unknown problems is met. Benchmarks are for narrowing candidates, not for production approval.

Contamination is an uncertainty that cannot be completely eliminated. Therefore, we layer multiple inspections, time separation, non-public sets, and production regression. It is important to be able to track which evidence a score depends on, rather than asserting 'no inclusion.'

The final report includes both the uninspected training scope and the performance basis remaining even after excluding contamination. Rather than just reprinting scores from sales materials, the person in charge updates which evaluation sets the adoption decision depends on and how long those sets remain valid.

In inter-model comparisons, we also indicate whether contamination inspection was possible or not. To avoid the reversal where only transparent models are found to have more suspicions, we do not count uninspectable areas as safety but treat them as uncertainty. We do not penalize for the amount of disclosure.

The contamination-confidence record stores the model, benchmark version, inspectable scope, contamination hypothesis, and retest deadline, while quarantining suspect scores from the selection total. Quarantine remains until provenance review or independent-item retesting satisfies the preregistered release evidence. A model without that evidence by the deadline leaves the comparison and returns only after the evaluation and data-provenance owners jointly approve a new test version and release decision.