Process reward model

A process reward model scores the intermediate steps of a solution rather than only its final answer.

What a process reward model is

An outcome reward looks at the end: did the answer match, did the tests pass. A process reward model (PRM) looks at the route. It assigns a score to each intermediate step of a solution, so that a correct answer reached through a wrong argument scores lower than one reached cleanly, and a wrong answer with sound early steps gets some credit for them.

PRMs are usually trained from step-level labels. Those labels come from people marking each step of a solution as correct or not, or from automatic estimates of how often a given step leads to a correct final answer when the rest of the solution is sampled many times. The trained model then scores new solutions step by step, and the scores are used either to pick the best of several candidates at inference time or as a reward during reinforcement learning.

The contrast is with an outcome reward model, which sees only the final result, and with a verifiable outcome reward, which replaces the learnt model with a checker.

A simple example

Suppose a coding agent is being trained to resolve issues in a repository, and a typical run takes thirty tool calls. With an outcome reward, a run that fixes the bug scores 1 and a run that does not scores 0, and every call in the run gets the same credit or blame. Now suppose a PRM has been trained to score each call. In a failed run it gives high scores to the first ten calls, which located the bug correctly, and a low score to the eleventh, where the agent edited the wrong file and never recovered. The policy learns that the search strategy was sound and the edit was the mistake, rather than discarding the whole run. In this hypothetical, the PRM turned one bit of feedback into thirty, at the cost of training and trusting a second model.

Why it matters

Agent tasks are long. A coding agent may take dozens of tool calls before anything can be checked, and a single pass-or-fail reward at the end says little about which of those calls helped. That is the credit-assignment problem, and it is why training agents with outcome rewards alone is slow and noisy. A process reward gives a signal per step and shortens the feedback loop.

It also offers a defence against reward hacking: an agent that makes the tests pass by editing them has a bad process despite a good outcome, and a step-level signal can see that where an outcome check cannot. For teams buying trained agents, whether the training used any process signal is a reasonable question to ask, because it affects how the agent behaves when the outcome check is weak.

How it is measured or handled

A PRM is evaluated like any classifier: on held-out solutions with step labels, by how well its scores separate correct from incorrect steps, and downstream by whether selecting or training with it improves final-answer accuracy over an outcome reward. Step labels are expensive, which is why automatic estimation from rollouts is the usual route at scale.

The known failure is that a learnt PRM is itself a reward model that can be gamed. If the PRM rewards steps that look right, the policy learns to produce steps that look right. Work on tree search for agent reinforcement learning avoids hand-labelled steps by sampling trajectories as a tree and deriving step-level signals from how the branches turn out, using only the outcome reward (Tree Search for LLM Agent Reinforcement Learning, 2025). Keeping the outcome check as the anchor, and treating the process signal as a shaping term rather than the whole reward, is the common hedge.

Related terms

A PRM supplements the outcome reward in RLVR and improves the credit assignment that GRPO lacks on long trajectories. It is one defence against reward hacking, and one new surface for it. The RL for agents guide covers the promise and the pitfalls.

Where this term is used

• Reinforcement learning for AI agents (RLVR)
• GRPO explained for agent builders
• GRPO
• Reward hacking
• RLVR
• Zero-advantage problem