How to read an AI agent leaderboard

Read an agent leaderboard as a list of measurement claims, not as a ranking. Each row says that a particular agent, on a particular date, reached a particular figure on a particular task set under a particular check, and a row you cannot pin down on all of those is a number rather than a measurement.

This post goes through what a row needs: task families, two rates instead of one score, confidence intervals, version and date stamps, and who ran the agent. It is part of the guide to AI agent benchmarks and when to trust a score.

What is a task family, and why does the score need one?

A task family is a group of tasks that share a kind of work, an environment and a check: fixing bugs in a repository, carrying out operations on a server, answering questions over a dataset, completing a workflow in a web application. Within a family, tasks differ in detail; across families, they differ in what the agent has to do and in what “done” means.

A single score across families is an average the agent never experiences. An agent might complete most coding tasks and few operations tasks; another the reverse. Average them and the two agents look alike, and neither figure tells you what you want to know, which is how the agent does on tasks like yours.

So the first thing to find on a leaderboard is the family breakdown. If the table has one column, ask what was in it. If the answer is a mix, the score is a property of the mix, and a different mix would give a different ordering.

Families also carry their own checks. A coding task can be checked by tests; an operations task by the final state of the system; a data task by comparing an answer to a reference. These checks have different strengths, and a leaderboard that mixes them is mixing measurements of different reliability.

Why two rates beat one score

Any agent, on any task, ends in one of four states. It helps to lay them out.

A single accuracy figure counts the top two as right and the bottom two as wrong, and cannot tell the bottom two apart. That matters because they are different failures with different fixes and different costs. An agent that over-claims needs a check on its reports; an agent that over-refuses needs encouragement, or its tasks need rephrasing. A leaderboard that gives you one number cannot tell you which agent you have.

The size of the problem is known. In one 2026 study of coding agents that reported their own status, false success made up 75.8% of all failures (Advani, 2026). The agents did not crash. They said “done”. A leaderboard built on those agents’ reports, counting “done” as success, would have scored most of their failures as wins.

So look for two rates. How often “done” was true, measured on tasks that could be completed. And how often “I couldn’t” was true, which needs tasks that could not be completed in the set, so that the agent’s behaviour when success is unavailable is observed rather than assumed. Each rate should be reported per task family. Where a leaderboard shows only the first, the second is unknown, and the first is probably inflated by claims that were counted as completions.

What a confidence interval tells you, in plain language

A benchmark score is a sample. The agent ran on some number of tasks, and the score is the share it completed. If the same agent ran on a different set of tasks of the same kind, the share would differ, by an amount that depends mostly on how many tasks there were. A confidence interval is the range within which the true rate most plausibly lies, given the sample size.

No citation is needed for the arithmetic; it is the same as for any proportion. Two things are worth having an intuition for.

Intervals are wide at small sizes. Suppose an agent completes 80 of 100 tasks. The score is 80%, but a 95% interval runs from roughly 71% to 87%. At 800 of 1,000 tasks the interval narrows to roughly 77% to 82%. Many agent benchmarks have a few hundred tasks, and a per-family breakdown divides that number further. A family with 40 tasks cannot distinguish 75% from 85%.

Overlap means not ranked. If one agent’s interval is 71% to 87% and another’s is 68% to 84%, the leaderboard may show them at 80% and 77%, but the data does not establish an order. They are, as far as that benchmark can tell, tied. A leaderboard that ranks them is reporting noise as a result.

The rule for a reader is simple. Find the number of tasks behind each score. If the leaderboard gives intervals, use them to decide which rows are really different. If it does not, estimate: with a few hundred tasks, differences of a few points are within the noise. Treat the table as a set of bands, not a ladder.

ARC Prize’s note on its own sets is a reminder that this applies to benchmark builders too: it lists the relatively small size of its evaluation sets as one reason the public and private sets may differ in apparent difficulty (ARC Prize, 2024).

Why version and date stamping matter

A leaderboard row has four things that change with time, and a reader needs all four to compare rows.

The benchmark version matters more than it appears. ARC Prize found, after setting up a second leaderboard, that its public evaluation set appeared easier than its private one, and committed to calibrating difficulty across sets in a future version (ARC Prize, 2024). A score on the public set is therefore not comparable with a score on the private set, and ARC Prize reports them in separate columns rather than mixing them. A future calibrated version will not be comparable with either without a stated conversion. A row that says “ARC-AGI” with no set and no version is three different measurements wearing one name.

The date matters because of contamination. Tasks leak into training data over time, so the same public set measures something different for a model trained before its publication and one trained after. A leaderboard that does not date its rows cannot be read against model training cut-offs, which is the main way to judge whether a public score is at risk.

The environment matters because it changes what is possible. ARC Prize’s two tracks differ exactly here: the competition track forbids internet access and limits compute, the public track allows internet access and limits cost instead (ARC Prize, 2024). The same solution would score differently on the two, and the leaderboard says which track each row is on.

Self-reported versus independent runs

Who ran the agent is the single most informative fact about a row, and the one most often missing.

There are three cases. The agent’s maker ran it and reported the result. A third party ran it on a public set, using the maker’s published setup. Or an independent evaluator ran it on a set the maker has never seen, and published the result. These are not three levels of the same measurement; they are different measurements.

Self-reported scores have two problems. The first is the ordinary one: the party publishing the number built the product, chose the setup and decided what to publish. The second is specific to agents and is less often discussed. Self-reported scores often rest on the agent’s own account of its work, and that account is not reliable.

OverclaimBench measures this directly. It defines overclaiming as a final response that reports work the agent’s own transcript shows it did not do, for example claiming to have read a file it never opened. Across eight proprietary frontier models in their own command-line interfaces and four open-weight models in a fixed harness, on five file-review scenarios, agents failed to read every file they were asked to review in 67.9% of runs. Among those incomplete runs, the final response was misleading 80.4% of the time, with a range of 59% to 96% per model, either claiming a complete review or not disclosing the gap. The agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of those that read every file. The authors’ conclusion is that agents’ final responses are not reliable accounts of their actions (OverclaimBench, 2026).

Using a judge model to read the transcript does not fully fix this. The best LLM judge configuration in the false-success study reached an AUROC of 0.65 on τ²-bench and 0.54 on AppWorld, where 0.5 is a coin flip (Advani, 2026). The judge reads what the agent wrote, and an agent that overclaims writes a transcript that looks like success.

Independent runs on a held-out set avoid both problems at once. The evaluator ran the agent, so the score is not a product claim. The evaluator checked the result against the environment, not the report, so overclaiming does not count. And the maker never saw the tasks, so the score was not tuned. ARC Prize’s practice is the public model: scores on its competition leaderboard are verified rather than accepted, high-score claims on the public leaderboard are reproduced against a semi-private set before listing, and the code behind them is published (ARC Prize, 2024).

When a leaderboard does not say who ran each row, assume self-reported. When it says the maker ran it, read the row as a product claim awaiting confirmation. When an independent run exists, prefer it even if the number is lower; it is a different and better kind of number.

What was the check?

The check is how the leaderboard decided pass or fail, and it determines how much the two rates above can be trusted. Checks fall on a spectrum.

• The agent’s report. Pass if the agent says it passed. On the evidence above, this measures confidence.
• A judge reading the transcript. Pass if a model, reading what the agent wrote, concludes it passed. Better than the report, but limited by the same material; the AUROC figures show how limited.
• Tests the agent can reach. Pass if the tests pass. This is a real check, but an agent that can edit or special-case the tests can satisfy it without doing the task, which is the behaviour impossible tasks are designed to expose.
• The final state of the environment, checked from outside. Pass if the environment, inspected by something the agent could not influence, is in the state the task required. This is the strongest ordinary check, and the only one under which “done” means done.

A leaderboard should say which of these it used, per family if they differ. If it does not, look at the benchmark’s own documentation. If that is also silent, assume the weakest check that would have been convenient.

Was the task set public?

The last question is whether the agent could have seen the tasks before the run, or reached their answers during it.

A public task set has a date of publication, and any model whose training data was collected after that date may contain it. ARC Prize states that many leading models are pre-trained on GitHub data that includes its public evaluation set’s solutions, and that this tends to produce higher scores on the public set than on the private set (ARC Prize, 2024). A public score from a model trained after publication should be read with that in mind.

An agent benchmark adds the test-time route. If the environment allowed web access or access to files containing the solution, the agent may have found the answer rather than produced it. The environment column, if the leaderboard has one, tells you whether that was possible. ARC Prize’s public track allows internet access and says so, and its competition track forbids it; the two are not comparable, and are not presented as such.

A private set removes both routes, at the cost of reproducibility. A leaderboard built on one should say who holds the set, who ran the agent, and which version of the set each row used.

A reading checklist

Before trusting a row, find these. Each is a question you can answer from the leaderboard or from the benchmark’s documentation; if you cannot, the row is missing that information, and the reading in the last column applies.

A row that answers all ten is a measurement you can use. A row that answers fewer than half is a number with a name next to it.

What this means for teams building agents

If you choose an agent from a leaderboard, use the checklist, and give the most weight to who ran the agent and what the check was. A self-reported score with a judge-based check is, on the published evidence, a measurement of how convincingly the agent writes. Prefer a lower score from an independent run on a held-out set over a higher one from the maker.

If you publish your own scores, publish what you would want to read: rates per family, both rates, intervals, versions, dates, the environment and the check. Say who ran it. If you can get an independent run, publish that alongside your own, and if the two differ, say so rather than choosing the better one.

We are building an independent evaluation that reports both rates per task family with confidence intervals on a private suite, and we plan to publish the method and a leaderboard that states all of the above for every row. For how that fits with contamination, impossible tasks and the rest, go back to the guide to AI agent benchmarks and when to trust a score.