Private evaluations for model builders

The problem for model builders

A lab that publishes its own agent scores runs into two problems at once.

Contamination. Public benchmarks leak. Their tasks end up in training data, and a score on them says less with every model generation. This is benchmark contamination. It is why ARC Prize keeps evaluation sets that are never published and runs the tests itself rather than accepting self-reported results (ARC Prize, 2024).

Self-grading. A score produced by the organisation that built the model, using a judge that reads the model’s own transcript, is hard for anyone outside to trust. The measured accuracy of such judges at catching false success is also poor: the best AUROC was 0.65 on τ²-bench and 0.54 on AppWorld (Advani, 2026). And the behaviour being judged is now a measured property of frontier models. Researchers quantify how prone agents are to overclaiming, reporting more than they did (OverclaimBench, 2026).

A private evaluation by an independent party, on tasks the model has never seen, avoids both problems.

What we test

We test agentic tasks on real systems: an agent is given a goal, a working environment and tools, and asked to get something done. We then compare what the agent reported with what actually happened.

Two rates are reported separately.

• False “done”. The agent said the task was complete, and it wasn’t.
• False “can’t”. The agent said the task could not be done, and it could.

The split matters because the fixes differ. Measuring the second rate needs tasks that genuinely cannot be completed, so the suite includes them. Agents given tests they cannot legitimately pass have been shown to find ways to pass them anyway (ImpossibleBench, 2025), and reinforcement finetuning has been found to make models less willing to say that a problem cannot be solved (Hallucination Tax, 2025). An agent that says “I couldn’t” when that is true is behaving well, and we count it that way. See honest failure and impossible task.

Task families are agreed with you in advance: for example coding, operations or data work. Within each family the specific tasks are held out and never published.

How an engagement works

1. Scoping call. What your agent does, which task families matter, what you want to learn. About an hour.
2. Agreed task families. We propose the families and the size of the run. You confirm.
3. A private run. We run your agent as you ship it, through its API or as a deployed agent. No model weights, no code changes.
4. A report. Both rates by task family, with confidence intervals, plus the task-level record behind each figure, so your team can see where the claims and the outcomes diverged.

Confidentiality and data handling

• Results are yours. We publish nothing about a private evaluation without your written permission. NDAs are available.
• Your tasks, prompts and outputs are used only for your evaluation. We delete them when the engagement ends unless you ask us to keep them.
• A private engagement does not change where a model stands on the public leaderboard we plan to publish. Payment never changes a score.

Status

Private evaluations go to design partners first. The programme is free for participants and gives you a say in what the benchmark measures. The same measurement underlies the Inoku Verified™ mark, and our custom task environments are built on the same independent ground truth. Larger organisations pay once the programme ends; see pricing.

To request an evaluation, contact us and choose the topic “Evaluation”.