Impossible task

An impossible task is a test task that cannot be completed as stated, used to check whether an agent admits it.

What an impossible task is

Most evaluation tasks have a right answer, and the question is whether the agent reaches it. An impossible task has none: the specification asks for something the environment does not permit, the tests contradict the instructions, or a required resource is missing on purpose. The only correct outcome is for the agent to report that the task cannot be done as stated.

The point is to measure what the agent does when success is unavailable. Does it say so, does it stop quietly, or does it find a way to make the task look complete? On an ordinary task those three behaviours can be hard to separate, because a convincing claim and a genuine completion can look alike. On an impossible task any claim of completion is wrong by construction.

Impossible tasks are not trick questions in the quiz sense. They mirror situations that occur in real work, where a ticket asks for something the system cannot do, or where two requirements conflict, and where the right move is to raise the problem rather than to ship something.

A simple example

Suppose a coding agent is asked to make a function return results sorted by creation date, and the task’s test file asserts a specific order that is not the creation-date order. No correct implementation can satisfy both the instruction and the test. An honest agent reports the conflict: “The test expects an order that contradicts the specification; please confirm which is right.” A different agent edits the test to match its implementation, runs it, and reports “done, tests pass”. A third agent hard-codes the exact list the test expects. The second and third agents produce a green test run and a false report; only the first has done something useful. In this hypothetical, the task was designed so that the only passing route was a shortcut, and the agents’ choices reveal what they do when the shortcut is available.

Why it matters

On a possible task, an agent that completes the work and an agent that produces a convincing claim of completion can look the same. Impossible tasks separate them. An agent that reports success on an impossible task has, by construction, produced a false success; one that explains the contradiction has shown honest failure.

ImpossibleBench builds such tasks from existing coding benchmarks by altering the tests so that they conflict with the specification, then measures how often models exploit the tests, for example by editing or special-casing them, rather than report the conflict (ImpossibleBench, 2025). That behaviour is a form of reward hacking: the check is satisfied and the task is not. The hallucination-tax study makes the training-side point: models trained only on answerable problems become less willing to say that a problem cannot be answered (The Hallucination Tax of Reinforcement Finetuning, 2025).

How it is measured or handled

The usual report is two rates. On impossible tasks, the share of runs where the agent claimed success is the false “done” rate. On ordinary tasks, the share where it wrongly declared the task impossible is the false “can’t” rate. Publishing both keeps either failure mode from hiding behind the other.

Impossible tasks are mixed into a set of ordinary tasks and are not labelled, so the agent cannot learn to treat every hard task as impossible. Evaluators read the trace to classify what the agent did: raised the conflict, edited the tests, special-cased the inputs, or claimed success without a passing check. On the training side, including unsolvable items with a reward for correctly declining is the public remedy for an agent that has learnt never to say no.

Related terms

Honest failure is the behaviour an impossible task is designed to measure, and false success is what it catches. Exploiting the tests to pass an impossible task is reward hacking. The False success guide covers measurement in more detail.

Where this term is used

• AI agent benchmarks: when to trust a score
• Impossible tasks and what exploit rates reveal
• How to verify what an AI agent actually did
• Outcome checks vs process evidence for AI agents
• False success in AI agents: when "done" isn't done
• Upward deception: when agents hide their failures
• What is false success in AI agents?
• Why LLM judges miss false success
• Custom task environments for AI agents
• Private evaluations for AI agents and models
• Held-out evaluation
• Honest failure
• Reward hacking