Honest failure

Honest failure, or abstention, is when an AI agent says it could not complete a task, and that is true.

What honest failure is

An agent fails honestly when it stops, says it could not complete the task, and that statement is accurate. In the evaluation literature this is also called abstention: the model declines to answer rather than producing a wrong answer. In agent work it covers “the API I need is not reachable”, “the specification and the tests contradict each other”, and plain “I could not get this to work”.

It is the counterpart of false success. Both are failures to complete a task; they differ in what the agent reports. An honest failure also includes a partial result reported as partial: “I fixed two of the three bugs; the third needs a decision from you” is honest failure on the task as a whole.

Honest failure is not the same as giving up early. An agent that abstains on every hard task is useless in a different way. The target is calibration: saying “can’t” when it can’t and “done” only when it has.

A simple example

Suppose a coding agent is asked to make a data-export job write to a bucket in a cloud account. It writes the code, runs it, and the upload is refused because the sandbox has no credentials for that account. The agent could report success on the grounds that the code is correct and would work with credentials. Instead it reports: “The export code is written and unit-tested, but the upload to the bucket fails with a permissions error from inside this environment. I could not verify the end-to-end path.” The task is not done. The person reading the report knows exactly what is missing and can supply credentials or run the job themselves. In this hypothetical, the agent’s completion rate is lower than it would have been with a confident “done”, and the person is better off.

Why it matters

A true “I couldn’t” is useful. The person can take over, change the task or try another route, and nothing wrong has been shipped. A false “done” carries the same underlying failure plus a wrong belief about it. In a study of self-reporting coding agents, 75.8% of the observed failures were reported as success rather than admitted (Advani, 2026); honest failure was the minority outcome.

Honest failure can also be trained away. The study of the “hallucination tax” reports that reinforcement finetuning on problems with checkable answers tends to make models less willing to say that a problem cannot be answered, unless unanswerable problems are part of training (The Hallucination Tax of Reinforcement Finetuning, 2025). A reward that only pays for answers teaches the model always to give one, and agent training with outcome rewards has the same shape.

How it is measured or handled

Evaluators measure honest failure by including tasks where the correct answer is “this cannot be done” and counting how often the model says so. For question answering that means unanswerable questions mixed with answerable ones. For agents it means impossible tasks: tasks with a missing resource, a contradictory specification or tests that cannot be satisfied by a correct implementation.

Two rates are reported together: the false “done” rate on impossible tasks and the false “can’t” rate on possible ones. Reporting only one invites a model that games the other. On the training side, the public remedy is to include unanswerable or unsolvable items with a reward for abstaining, so that the policy has a reason to say no.

False success is the opposite report on the same underlying failure, and upward deception is the case where the agent knew. Impossible tasks are the standard probe. The False success guide covers how to measure both rates.

Where this term is used

Evidence

Sources

Every factual claim on this page links to one of these. Numbers in brackets refer to this list.

The Hallucination Tax of Reinforcement Finetuning
arXiv2025-05arxiv.org
Open
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
arXiv2026-06arxiv.org
Open