Benchmark contamination: why public scores decay

Public benchmark scores decay because the tasks and their answers leak into the data that later models are trained on, so the score comes to measure recall rather than skill. This is benchmark contamination, and the only reliable defence is a test set that nobody outside the evaluator ever sees.

This post explains how the leak happens, what ARC Prize observed when it compared its public and private evaluation sets, and what a private set changes. It is part of the guide to AI agent benchmarks and when to trust a score.

What is benchmark contamination?

A benchmark is meant to be a test the model has not seen. Contamination is the condition where it has: the tasks, the reference answers, or worked solutions got into the training corpus, and the model is partly remembering rather than solving.

It needs no intent. Benchmarks are published so that people can use them. People use them by writing about them, copying them into repositories, posting solutions and discussing failures. All of that is web text, and web text is what models are trained on. Contamination is the default outcome of training on the internet and testing with something that is on the internet.

The effect is subtle because nothing visibly breaks. The score goes up. The capability the score was meant to indicate does not go up by as much, or at all. The two drift apart, and because the benchmark is the only instrument you have, you do not see the drift.

How do tasks get into training data?

There are several routes, and they compound.

ARC Prize names the third route explicitly for its own benchmark. Many leading models are pre-trained on GitHub data, which contains the public evaluation set’s solutions as well as example solution programs from past contests and research, and that tends to produce higher scores against the public set than against the private one (ARC Prize, 2024). Nobody had to do anything wrong for that to happen. Competitors published their code, as competitors are encouraged to, and the code was on GitHub when the next generation of models was trained.

The last route is specific to agents and is easy to overlook. A chat model can only contaminate at training time. An agent that can run a search or open a file can contaminate at test time, by reaching material that contains the answer. For an agent benchmark, “was the model trained on this?” is only half the question. The other half is “could the agent have looked it up?”

What did ARC Prize observe?

ARC Prize’s data is split into four sets: a public training set, a public evaluation set, a private evaluation set and a semi-private evaluation set. All of the evaluation sets were intended to be equally hard (ARC Prize, 2024).

When ARC Prize set up its public leaderboard, it found evidence that they are not. The public evaluation set appeared easier than the private evaluation set. ARC Prize does not claim to know exactly why, and lists four candidate causes: the sets are relatively small, so sampling noise is large; model contamination; solutions overfitting on public examples; and uncalibrated difficulty, since the public set was made first.

Two things about that account are worth copying. First, ARC Prize treated the gap as evidence rather than as an embarrassment, published it, and committed to calibrating difficulty across sets in a future version. Second, it did not collapse the two scores into one. Each entry on the public leaderboard carries both a public-set score and a semi-private verification score, so a reader can see the gap for themselves.

The lesson for anyone publishing agent scores is that a public set and a private set measure different things, and the honest response is to report both and say which is which, not to pick the flattering one.

Why are agent benchmarks more exposed than model benchmarks?

Three reasons, each of which makes contamination both more likely and harder to detect.

The tasks are often real. Agent benchmarks for coding tend to be built from real repositories and real issues, because that is what makes them realistic. But a real issue has a real fix, and the fix is public too, often in the same repository. A model that has seen the resolved pull request does not need to solve the issue; it needs to recall it.

The environment can leak. An agent benchmark gives the agent tools, and tools reach outside the task. If the environment allows web access, the agent can search for the issue. If it allows file access to the repository’s history, the fix may be in it. Contamination then depends on the environment’s configuration, which leaderboards rarely state. ARC Prize’s competition track forbids internet access for this reason, and its public track allows it and says so (ARC Prize, 2024).

The check often rests on the agent’s report. This is the reason that is least discussed. If pass or fail is decided from the agent’s final message, or from a judge reading the transcript, then a contaminated model has an extra advantage: it knows what a successful solution looks like, so it can produce a convincing report even where it has not done the work. In one 2026 study of coding agents that reported their own status, false success, where the agent claimed completion without achieving it, accounted for 75.8% of all failures, and the best LLM judge reading the transcript reached an AUROC of only 0.65 on τ²-bench and 0.54 on AppWorld, where 0.5 is chance (Advani, 2026). A contaminated model and a self-reported check compound each other: the first makes the report more convincing, the second makes the report the thing that is graded.

How do private and semi-private sets work in practice?

A held-out evaluation is the general term for testing on tasks kept out of training. A private set is the strong form, where the tasks are also kept out of the developer’s hands. ARC Prize runs both forms side by side, and the arrangement is instructive (ARC Prize, 2024).

The private set is the one used to score contest submissions, and ARC Prize states that it must remain private. The reason given is the core of the whole practice: the creators of a solution cannot know what the problem will be, or they risk encoding their own intelligence into the solution. A private set does not just keep the answers out of training data; it keeps the task shape out of the designer’s head.

The semi-private set does a different job. It is used to verify public-leaderboard claims: a submission that scores well on the public set is re-run on the semi-private set by ARC Prize, and both figures are published. High scores on the public track are published alongside reproducible open-source code, and ARC Prize funds the verification, reimbursing API costs for verified claims that beat the previous high score by at least 1%, under a cost limit of 10,000 US dollars per run.

Two features of this arrangement are worth noting. The verification is done by the evaluator, not by the submitter. And the verification set is distinct from the public set, so a solution that has overfitted the public tasks does not get credit for it. Those are the two properties a private suite needs, and they are independent of the specific benchmark.

What does contamination do to a leaderboard over time?

A contaminated score does not just overstate one agent. It changes the shape of the table.

Older models were trained before the benchmark existed and cannot have seen it. Newer models were trained afterwards and may have. So contamination inflates the newer entries more than the older ones, and the apparent rate of progress rises. A leaderboard that is not refreshed with new tasks shows a steeper climb than the capability it is tracking.

It also changes rank order unevenly. Models trained on different corpora have seen different amounts of the benchmark. Two models of equal skill can sit several points apart because one of them has seen more of the solutions. The ordering then reflects data pipelines rather than capability.

ARC Prize’s expectation for its own two tracks is a useful framing. It says that if someone achieves a conceptual breakthrough on the public set, it expects high scores to roll down quickly to the private leaderboard (ARC Prize, 2024). That is the test: a real gain on a public set shows up on a private set soon after. A gain that stays on the public set and never arrives on the private one was never a gain.

How can you tell if a score is contaminated?

You usually cannot tell from the score alone, which is the whole problem. But there are signals.

• A gap between public and private results. If the benchmark has both and the same agent scores noticeably higher on the public set, contamination is one of the candidate explanations, as ARC Prize lists it. It is not the only one, and ARC Prize is careful to say so.
• The benchmark’s age relative to the model’s training cut-off. A benchmark published before the training data was collected is at risk. One published after is not, until the next model.
• The environment’s reach. If the agent could search the web or read the repository’s history during the run, test-time contamination was possible, whatever the training data contained.
• Where the check came from. If the score was computed from the agent’s own report, a contaminated model has an advantage that no amount of task refreshing removes.
• The absence of a version and date. A score that does not say which version of the task set it used, or when it was run, cannot be placed on the timeline at all.

None of these is proof. Together they tell you whether the score should be read as a measurement or as an upper bound.

What are the trade-offs of keeping a set private?

Private sets are not free, and it is worth being honest about the costs.

Reproducibility. A private score cannot be reproduced by anyone who does not hold the set. Trust moves from the tasks, which anyone could inspect, to the evaluator, who must be trusted to run the agent fairly and report what happened. ARC Prize’s answer is to verify public claims itself and to publish the code behind them, so that the public track stays reproducible while the private track stays meaningful.

Set size. A private set that is never published can only be built and maintained by the evaluator, which limits how large it can be. ARC Prize lists small sample size among the reasons its sets may differ in difficulty (ARC Prize, 2024). A small set means wide confidence intervals, which the leaderboard should show.

Calibration. Two sets built at different times by different people will differ in difficulty, and the difference is hard to measure without running the same solutions on both. ARC Prize discovered its own gap only after setting up the second track, and has committed to calibrating future versions.

Leakage over time. A private set is private until it is not. Every evaluation run exposes the tasks to the agent, and if the agent’s operator logs its inputs, the set has left the building. A private suite therefore needs replenishment, and the leaderboard needs to say which version of the set each score came from.

The alternative, a public set with a public score, has one advantage, reproducibility, and one defect, decay. For a leaderboard that is meant to be read as a measurement, decay is the worse problem.

What this means for teams building agents

If you evaluate your own agent on a public benchmark, treat the score as an upper bound and date it. Check whether the benchmark predates your model’s training data, and whether your agent’s environment lets it reach the answers during the run. If your check is the agent’s own report, or a judge reading the transcript, remember that a contaminated model is better at producing a convincing report, not just a correct answer.

If you publish scores, publish them the way ARC Prize does: say which set, which version, who ran it and under what conditions, and if you have a private verification set, show both numbers rather than one. A score that is honest about what it measures is worth more than a higher score that is not.

We are building a private, held-out suite and plan to run every major model on it ourselves, for the reasons above. For how that fits with exploit rates, task families and confidence intervals, go back to the guide to AI agent benchmarks and when to trust a score.