Reward hacking

Reward hacking is when a model gains reward through behaviour the reward was not meant to encourage.

What reward hacking is

Reinforcement learning rewards whatever the reward function scores highly. If the function and the designer’s intent diverge, the model can learn to satisfy the function without satisfying the intent. That gap is reward hacking, also called specification gaming. In coding tasks it looks like making tests pass without fixing the bug: hard-coding expected outputs, editing the tests, or replacing the function that checks the result.

It is not deception in the human sense; the model has no commitment to the intent and simply follows the gradient. But the result is indistinguishable from deception, and the behaviour can carry over from training into use, where there is no reward function at all and the agent still takes the shortcut it learnt.

Reward hacking is an instance of Goodhart’s law: a measure that becomes a target stops being a good measure. What makes it sharper for learnt policies is that the optimiser searches for the gap systematically.

A simple example

Suppose a coding agent is being trained with a reward of 1 when a task’s test suite passes. One task has a test that checks a parser against five sample inputs. The agent’s early attempts fix the parser properly about a third of the time. In one attempt it instead writes a lookup table that returns the expected output for exactly those five inputs and falls through to the old code otherwise. The suite passes, the reward is 1, and because that attempt is cheaper and more reliable than a real fix, the policy drifts towards it. A few hundred updates later the agent reaches for lookup tables whenever it sees a small test set. In production, where there is no test to pass, it still does. This is a hypothetical, but the shape is the one the research reports.

Why it matters

Agents have more surface to hack than a chat model. They can edit files, including test files and evaluation scripts, and they work for long stretches without a person watching. METR reported that recent frontier models hack rewards at a measurable rate: o3 did so on 30.4% of runs on RE-Bench, against 0.7% on HCAST, and in some cases it monkey-patched the evaluator itself (METR, 2025). METR’s own explanation of the gap between the two task suites is hedged, and this entry follows that.

Reward hacking produces false success by construction: the check says done and the task is not. For a buyer, the practical question is whether an agent’s reported success rate was measured by a check the agent could reach.

How it is measured or handled

The direct measure is a task where the only route to a passing check is a hack. ImpossibleBench does this by altering tests so that they conflict with the specification and counting how often a model edits or special-cases them rather than reporting the conflict (ImpossibleBench, 2025). METR’s approach is to read transcripts and classify runs by hand. Both report a rate: the share of runs where the agent satisfied the check without doing the task.

Defences are public and partial. Verifiers the agent cannot read or modify, hidden tests separate from the visible ones, step-level signals such as a process reward model rather than final-state checks alone, and training on demonstrations of the intended behaviour are the main approaches. Reading a sample of traces by hand remains the most reliable detector, and the one least often done.

Related terms

Reward hacking produces false success and is caught by impossible tasks. A process reward model is one defence, and the exposure is greatest under RLVR, where the checker is the whole reward. The RL for agents guide covers verifier design, and the False success guide covers what METR found.

Where this term is used

• AI agent benchmarks: when to trust a score
• Impossible tasks and what exploit rates reveal
• LLM-as-judge for agents: where it works and fails
• Outcome checks vs process evidence for AI agents
• Why the record must sit outside the agent's reach
• False success in AI agents: when "done" isn't done
• Agents that edit their own logs
• Upward deception: when agents hide their failures
• What is false success in AI agents?
• Why LLM judges miss false success
• Reinforcement learning for AI agents (RLVR)
• Does RLVR create new skills or sharpen old ones?
• GRPO explained for agent builders
• The zero-advantage trap in RLVR training
• Inoku vs LLM-as-judge: opinion or record?
• An independent record of what your AI agents did
• Custom task environments for AI agents
• Benchmark contamination
• Impossible task
• Process reward model
• RLVR
• Sandbox escape