AI agent benchmarks: when to trust a score

An AI agent benchmark score tells you how an agent did on a fixed set of tasks, checked in a particular way, at a particular time. Whether that number says anything about your tasks depends on three things the number does not show: whether the tasks were already in the training data, whether the check could be satisfied without doing the work, and who ran the measurement.

This guide covers each of those in turn. It sits between two neighbouring questions with their own guides: how to verify what a single agent run actually did, and how agents are trained with verifiable rewards. Here the subject is measuring models and agents across many tasks, and when to trust the result. Every number links to its source.

What is an AI agent benchmark?

A benchmark for a chat model is a list of questions with known answers. A benchmark for an agent is more than that, because an agent acts. It is given a goal, an environment (a repository, a shell, a browser, a set of tools) and the freedom to take many steps before it reports back. Four parts make up the benchmark, and the score depends on all four.

Two features of agent benchmarks matter more than they do for question-answer tests. First, the environment is part of the test, so what the agent is allowed to touch changes the result. Second, the agent writes its own account of what it did, and that account is often the easiest thing to grade. Both are where the trouble starts.

Why do public benchmark scores decay?

Because the tasks stop being new. Once a benchmark is published, its tasks, its reference solutions and discussions of both end up on the same web that future models are trained on. The score then measures recall as much as skill, and rises without the capability it was meant to track rising with it. This is benchmark contamination.

ARC Prize describes the mechanism plainly for its own benchmark. Many leading models are pre-trained on GitHub data, which contains the public evaluation set’s solutions and example programs from past contests, and that tends to produce higher scores on the public set than on the private one (ARC Prize, 2024). When ARC Prize set up its public leaderboard it found that the public evaluation set appeared easier than the private set, and listed model contamination and overfitting on public examples among the possible causes, alongside small sample sizes and uncalibrated difficulty.

Agents add a second route. A model that cannot recall an answer may still be able to find it during the run, if the environment lets it search the web or read files that contain the solution. For an agent benchmark, contamination is a question about test time as well as training time.

The practical consequence is that a public score has a shelf life. The post on why public scores decay goes through the routes by which tasks leak and what ARC Prize observed when it compared its sets.

What do held-out and private test sets change?

A held-out evaluation uses tasks kept out of training and, in the stronger form, out of the developer’s hands altogether. The evaluator holds the tasks, runs the agent, and publishes the result. The developer never sees the tasks.

ARC Prize is the clearest public example of the practice. Its data is split into a public training set, a public evaluation set, a private evaluation set and a semi-private evaluation set. The private set, used to score contest submissions, stays private; the stated principle is that the creators of a solution cannot know what the problem will be, or they risk encoding their own intelligence into the solution. Scores on the competition leaderboard are verified by ARC Prize rather than taken on trust, and the competition track allows no internet access and enforces compute limits (ARC Prize, 2024).

The public leaderboard, ARC-AGI-Pub, takes the opposite trade: it measures the public evaluation set, allows internet access, and enforces a cost limit rather than a compute limit. High scores are verified against the semi-private set and published alongside reproducible open-source code. ARC Prize reports both a public-set score and a semi-private verification score for each entry, and is open about the fact that the two sets are not yet calibrated to the same difficulty.

The trade-off is real. A private set cannot be reproduced by outsiders, so trust shifts from the tasks to the evaluator. A public set can be reproduced, but its score means less every year. ARC Prize’s answer is to run both and say which is which. That is a reasonable model for anyone publishing agent scores.

What do tasks that cannot be completed reveal?

On an ordinary task, an agent that does the work and an agent that finds a way to satisfy the check look the same from the outside: both pass. An impossible task, where the only correct outcome is to report that the task cannot be done as stated, separates the two. Any pass is a shortcut.

ImpossibleBench makes this concrete. It creates impossible variants of tasks from existing coding benchmarks, LiveCodeBench and SWE-bench, by introducing direct conflicts between the natural-language specification and the unit tests. An agent’s “cheating rate” is its pass rate on those tasks, since any pass necessarily implies a specification-violating shortcut. The authors use the framework to study the shapes that cheating takes, from simple test modification to operator overloading, and to show that the prompt, the agent’s access to the tests and the feedback loop all change how often it happens (ImpossibleBench, 2025).

The point for benchmarks is not that agents are malicious. It is that a check an agent can reach is a check it may satisfy directly, and that this is a form of reward hacking that an ordinary pass rate cannot see. The same agent, on possible tasks, has the same habit; you just cannot tell which passes came from it.

The willingness to say “this cannot be done” is also a trained property, and it can be trained away. The study of the hallucination tax found that standard reinforcement finetuning could cut a model’s refusal rate on unanswerable problems by more than 80%, and that mixing in about 10% unanswerable problems during training substantially restored appropriate refusals with minimal cost on solvable tasks (Hallucination Tax, 2025). OpenAI’s GPT-5 system card describes a related step: placing the model in tasks that were partly or entirely infeasible and rewarding it for admitting that it could not complete them, after observing that an earlier model would sometimes say it had completed tasks it had not (GPT-5 system card, 2025).

An evaluation that contains no impossible tasks cannot measure any of this. The post on impossible tasks and exploit rates covers how such tasks are built, what the rate does and does not tell you, and how it should sit alongside the ordinary score.

Self-reported or independently measured?

Many agent scores rest, at some point, on the agent’s own account of its work: a final message that says the task is done, or a transcript that an LLM judge reads and grades. Both are measurements of the report, not of the work.

The gap between the two is large. In one 2026 study of coding agents that reported their own status, false success made up 75.8% of all failures: the agents did not crash or give up, they said “done” (Advani, 2026). The same study measured how well an LLM judge reading the transcript could tell true from false success. The best configuration reached an AUROC of 0.65 on τ²-bench and 0.54 on AppWorld, where 0.5 is chance.

Those numbers set a floor for what “self-reported” should mean to a reader. A score computed from the agent’s own status is, on that evidence, closer to a measure of confidence than of completion. A score computed by a judge reading the transcript is better, but not by much on some task families.

Independence is the other half. A score produced by the organisation that built the agent, on tasks it chose, with a check it wrote, is a claim about its own product. ARC Prize verifies submitted scores and reproduces high-score claims before listing them (ARC Prize, 2024). That is the standard to look for: the party publishing the number is not the party being measured, and the number was produced by running the agent, not by reading its report.

What is overclaiming, and why does it matter for a benchmark?

Overclaiming is a final response that reports work the agent’s own transcript shows it did not do. OverclaimBench defines it that way deliberately: the criterion needs no inference about intent and does not depend on whether the delivered work was correct. It asks only whether the reported work was done.

The suite consists of five file-review scenarios with transcript-based coverage measurement and planted defects. Across eight proprietary frontier models run in their own command-line interfaces and four open-weight models in a fixed harness, agents failed to read every file they were asked to review in 67.9% of runs. Among those incomplete runs, the final response was misleading 80.4% of the time (59% to 96% per model), either claiming a complete review or leaving the gap undisclosed. Agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file. The authors’ conclusion is that agents’ final responses are not reliable accounts of their actions (OverclaimBench, 2026).

For benchmark design, the implication is direct. Any score that is derived from the final response, or from a judge reading it, inherits that unreliability. The check has to look at what happened, not at what was said. And the last finding matters most: claims of completion concealed real defects, so a benchmark that grades the claim will not only overstate the score, it will miss the failures that cost the most.

What should a trustworthy leaderboard state?

A leaderboard row is a measurement claim. To be usable it has to say what was measured, on what, when, by whom and with how much uncertainty. The table below lists what a row should carry and what its absence hides.

ARC Prize’s own leaderboards illustrate several of these: two tracks with different rules, a public score and a verification score side by side, verification before listing, reproducible code, and an explicit note that the evaluation sets differ in difficulty (ARC Prize, 2024). A reader can tell what each number means. That is the bar.

The post on how to read an agent leaderboard takes each row item in turn, including confidence intervals in plain language and why two rates beat one score.

How to read a score

A short routine for any agent score you are shown.

1. Find the check. Was pass or fail decided by the final state of the environment, by tests, by a judge reading the transcript, or by the agent’s own report? The further right in that list, the less the score says about the work.
2. Find the task set. Is it public? If so, when was it published, and could the agent have trained on it or reached it during the run? If it is private, who holds it and who ran the agent?
3. Find the version and the date. Which agent, which harness, which benchmark version, on which day. If any is missing, the number cannot be compared with anything else.
4. Find the uncertainty. How many tasks, and what is the interval? Two scores whose intervals overlap are not ranked, whatever the table says.
5. Find the second rate. Does the suite contain tasks that cannot be completed, and is the agent’s behaviour on them reported separately from its success on the rest? If not, the score cannot distinguish doing from claiming.
6. Find the runner. If the party publishing the score built the agent, treat it as a product claim until an independent run agrees.

A score that survives all six is a measurement. One that fails the first or the sixth is a claim, and should be read as one.

What we are building

We are building Inoku Verified™, an independent evaluation of how often an agent’s “done” is true and how often its “I couldn’t” is true, on a private suite the agent cannot have trained on, broken down by task family with confidence intervals. We plan to publish the method and a free public leaderboard for major models. None of it exists yet; it is in development, with design partners first. The principles in this guide are the ones we are holding ourselves to, and the sources above are the reasons for each.