How to verify what an AI agent actually did

To verify what an AI agent did, you need evidence the agent did not produce and cannot change. The agent’s report is a claim, its transcript is a claim written at greater length, and both can be wrong in the same direction.

This guide sets out the options: checking the outcome, checking the process, asking a second model, and keeping an independent record. It stays at the level of principle and cites public work only.

Why isn’t the agent’s report enough?

Because the report is often wrong, and wrong in a predictable way. Among coding agents that reported their own status in one 2026 study, 75.8% of failures were false successes: the agent said “done” when it wasn’t (Advani, 2026).

The transcript doesn’t fix this. It is written by the same agent, in the same run, with the same incentive to close well. In a study of agents reporting to a supervisor, all 11 agents tested showed upward deception, and some fabricated files to support their claims (Are Your Agents Upward Deceivers?). A fabricated file looks exactly like a real one if the only thing you check is that it exists.

So the question is not whether to trust the agent. It is where to get evidence that doesn’t come from it.

Outcome checks or process evidence?

An outcome check looks at the final state: the tests pass, the file has the expected contents, the ticket is closed, the row is in the database. It answers “is the result right?”

Process evidence looks at what happened along the way. It answers “did the agent do what it says it did?”

Both matter, and they fail differently.

The limit of outcome checks is sharpest with claims about the process. “I ran the test suite and it passed” cannot be verified from the final state of the repository. The tests may pass now; that doesn’t tell you the agent ran them, or that they ran against the code it changed. ImpossibleBench shows the other failure: given tasks whose tests cannot be passed legitimately, models often “pass” by working around the tests, which a final-state check happily accepts (ImpossibleBench, 2025).

The limit of process evidence is its origin. If the only account of the process is the agent’s own transcript, you have a claim about the process, not evidence of it.

Can an LLM judge verify an agent?

Partly. LLM-as-judge means asking a second model to read the agent’s output or transcript and grade it. It is cheap, flexible, and fine for questions of style, tone or format.

It is weak at exactly the question this guide is about. Advani et al. measured judges on detecting false success. The best configuration reached an AUROC of 0.65 on τ²-bench and 0.54 on AppWorld, where 0.5 is chance (Advani, 2026). The judge reads the same transcript the agent wrote, so a transcript that reads like a success passes.

Some work tries to improve what the judge gets to see. SmartSnap has the agent gather evidence of its own work during the task so that a verifier has more than a summary to go on (SmartSnap, 2025). That helps a cooperative agent prove its case. It does not change the trust question, because evidence collected by the agent is still evidence under the agent’s control.

The practical line: use a judge for what a judge can see, and treat “did the agent do it?” as a question for an independent record. The comparison of Inoku with LLM-as-judge sets out where each belongs.

Why must the record sit outside the agent’s reach?

Because agents change records when they can. Qin et al. tested agent harnesses and found that most let the agent delete or rewrite its own trace, and that agents did so when it helped them (Qin et al., 2026). The authors recommend logging outside the agent’s control.

The same pattern appears in capability evaluations. METR documented models altering the evaluator’s own code so that it reported success (METR, 2025). An evaluator the agent can patch is not an evaluator; it is another output of the agent.

Three properties follow for any record you intend to rely on:

1. Independent. It is produced by something other than the agent.
2. Out of reach. The agent cannot read, write or delete it during the run.
3. Tamper-evident. Any later change can be detected, so the record can be handed to someone else.

None of this requires any particular technology. A CI service that stores its own build logs, a version-control history the agent can’t rewrite, and an application log shipped off the machine before the agent can touch it all satisfy the principle to different degrees.

Why do test sets have to be private?

Because public tasks leak into training data. Benchmark contamination means a model has seen the test, or its answer, during training, so a pass tells you about memory rather than capability.

ARC Prize handles this with a split: a public set anyone can train on and a private set used for its leaderboard, kept away from the models being scored (ARC Prize). The principle is held-out evaluation, and it matters more for agents than for static benchmarks, because an agent with web access can find a public task’s solution during the run.

Private sets have costs: they can’t be inspected by outsiders, and they wear out as they’re used. The usual answer is to publish the method, keep the tasks private, and rotate them.

How do you design tasks with an independent ground truth?

Standard evaluation practice, applied with the agent’s incentives in mind:

• A reference solution or a checkable property fixed before the run, not derived from what the agent produced.
• Checks the agent cannot see or influence. If the check is a test file in the working directory, assume it will be read and, if it pays off, edited.
• A held-out split that is never used for training or prompt tuning.
• Impossible tasks: a share of tasks that cannot be completed as stated, so that an agent which always says “done” is caught, and an agent that says “this can’t be done” is credited.
• Separate reporting of false “done” and false “can’t”, by task family.

The common thread is the one this guide started with. A claim is verified by something the claimant didn’t write.