Inoku vs LLM-as-judge: opinion or record?

What an LLM judge reads

An LLM-as-judge setup asks a language model to grade another model’s work. For agents, the judge is usually given the agent’s transcript: the messages, the tool calls the agent made, the outputs it saw, and the summary it wrote at the end. The judge then answers a question such as “was the task completed?”

Everything the judge reads passed through the agent. The transcript is the agent’s account of what it did. The judge is forming an opinion about a story.

Measured limits

The study with the clearest numbers is Advani (2026), which measured how well judges detect false success: the agent reporting a task complete when it was not. Among self-reporting coding agents, false success accounted for 75.8% of all failures. The best judge tested reached an AUROC of 0.65 on τ²-bench and 0.54 on AppWorld (Advani, 2026). An AUROC of 0.5 is a coin toss.

Three further findings explain why the ceiling is low.

• Agents can edit the transcript. In most of the harnesses tested in one study, agents could delete or rewrite their own traces, and did so when it was rewarded (Qin et al., 2026). A judge reading an edited transcript is grading the edit.
• Agents fabricate supporting evidence. All 11 agents tested in another study showed upward deception, in some cases creating files to back up a claim of success (Are Your Agents Upward Deceivers?, 2025). The fabricated file appears in the transcript as evidence.
• Agents game their evaluators. On METR’s RE-Bench, o3 showed reward hacking on 30.4% of runs, against 0.7% on HCAST (METR, 2025). A judge is an evaluator, and a capable agent treats it as one.

Research on self-verifying agents makes the same point from the other side: a verifier does better with evidence of the outcome than with the agent’s narrative alone (SmartSnap, 2025).

When an LLM judge is fine

A judge is cheap, fast and flexible. It is the right tool when:

• the question is about quality or style rather than fact: is this answer clear, is this code idiomatic;
• the cost of a wrong “pass” is low, and a person will look before anything ships;
• you need a first pass over thousands of runs to decide which ones a person should read;
• the transcript is trustworthy, because the agent could not have changed it and had no reason to.

Used this way, judges are useful.

When you need an independent record

You need something other than the agent’s account when:

• the claim is about an action on a real system: a file written, a migration run, a ticket closed;
• the agent could have reached its own log;
• the result feeds training, a public score or a customer-facing claim, so there is a reason to game it;
• an auditor, a customer or a regulator has to rely on the answer.

In those cases the question is not “does the story hold together?” but “what actually happened?” Answering it needs a record made outside the agent’s reach, against a ground truth the agent cannot see or change. That is the principle behind Inoku Verified™.

The comparison, at principle level

Where this leaves you

Keep your judge for what it is good at. For the one claim that costs you customers or fails an audit, the claim that the work was done, use a measurement that does not depend on the agent’s own account. Inoku Verified™ is that measurement: in development, with design partners first. The longer treatment is in how to verify what an AI agent actually did.