GRPO explained for agent builders

Group Relative Policy Optimization (GRPO) trains a language model by sampling several answers to the same prompt, scoring each one, and pushing the model towards the answers that beat their own group’s average. It replaces the separate value model that PPO needs with a baseline computed from the group itself, which is what makes it cheap enough to run at scale.

That one design choice explains most of what agent builders need to know: why the method is affordable, why a task the model always solves or never solves teaches it nothing, and why the method is best understood as amplifying successes the model can already produce. This post walks through the algorithm in plain language. The wider context, including process rewards and reward-hacking defences, is in the guide to reinforcement learning for AI agents.

Where does GRPO come from?

GRPO was introduced in DeepSeekMath, a 7B mathematics model built by continuing the pre-training of DeepSeek-Coder-Base-v1.5 7B on 120B maths-related tokens from Common Crawl, together with natural language and code (DeepSeekMath, 2024). The paper credits two things for the model’s results: a data selection pipeline for public web data, and GRPO, which it describes as a variant of Proximal Policy Optimization (PPO) that improves mathematical reasoning while reducing PPO’s memory use (DeepSeekMath, 2024).

The headline numbers give a sense of the setting. DeepSeekMath 7B scored 51.7% on the competition-level MATH benchmark without external tools or voting, and 60.9% with self-consistency over 64 samples (DeepSeekMath, 2024). The second number matters for what follows. It shows that the base model could already solve many problems sometimes, if you let it try often enough. GRPO is a way of turning “sometimes” into “usually”.

Since then GRPO has become the default algorithm for reinforcement learning with verifiable rewards (RLVR), where the reward comes from a checker rather than a learned preference model. Most agent training recipes you will meet are GRPO or a close relative.

What problem does GRPO solve compared with PPO?

Policy-gradient methods share a problem: a raw reward is a poor training signal. If a maths answer earns reward 1 for correct and 0 for wrong, then every correct answer looks equally good and every wrong answer equally bad, regardless of whether the prompt was trivial or nearly impossible. What you want is to reward an answer relative to what you expected for that prompt. The gap between the reward and the expectation is the advantage.

PPO estimates the expectation with a value model: a second network, trained alongside the policy, that predicts the expected reward from a given state. For large language models that means a second model of comparable size, with its own memory, its own training, and its own errors.

GRPO removes the value model. Instead of predicting what reward to expect for a prompt, it measures it directly by sampling. Generate a group of answers to the prompt, score them all, and use the group’s mean as the expectation. The advantage of each answer is how far its reward sits above or below its siblings. That is the whole trick, and it is the reason the method is called group relative.

The last row of the table is the one that shapes agent training. Removing the value model saves memory but makes the signal depend entirely on the group having a spread of rewards.

How does a GRPO update work, step by step?

The loop is short enough to describe in full.

1. Pick a batch of prompts. For an agent, a prompt is a task: a repository with a failing test, a ticket to resolve, a server to configure.
2. Sample a group for each prompt. Run the current policy several times on the same task, independently. Each run produces a trajectory: for a chat model a single answer, for an agent a sequence of tool calls and messages.
3. Score every sample. Under RLVR the score comes from a checker. Did the tests pass? Does the final answer match? Did the end state meet the specification? The reward is usually binary.
4. Normalise within the group. Compute the group’s mean reward and its standard deviation. Each sample’s advantage is (reward minus group mean) divided by group standard deviation. Samples above the mean get a positive advantage, those below a negative one.
5. Update the policy. Increase the probability of the tokens in positively weighted samples and decrease the probability of those in negatively weighted ones, scaled by the advantage. The update is clipped so the new policy does not move too far from the one that produced the samples, and a KL penalty against a reference model keeps it from drifting too far from where it started.
6. Repeat with fresh prompts, or the same prompts again, using the updated policy to sample the next groups.

Two features deserve a second look. First, the normalisation step divides by the group’s standard deviation, so a group with a small spread of rewards gets its differences magnified and a group with a large spread gets them damped. Second, nothing in the loop asks what an answer looks like, only what it scored. The method is indifferent to style, structure or length except as those happen to correlate with reward.

What does “group-relative advantage” mean in practice?

A worked example makes the behaviour concrete. Suppose the group size is eight and the reward is 1 for pass, 0 for fail.

The first two rows are the zero-advantage problem. When every sample scores the same, the mean equals every reward, every advantage is zero, and the update from that prompt is nothing. The model has paid for eight rollouts and eight checks and learnt nothing. For agent tasks, where a single rollout may mean minutes of tool use in a sandbox, that is an expensive nothing.

The remaining rows show why GRPO is a contrastive method. The prompt does not teach “this is a good answer”. It teaches “this answer is better than those answers, for this prompt”. The rarer the pass, the harder it is pushed, which is exactly what you want for a task the model can only just manage.

What does the maths say GRPO is really doing?

Mroueh’s analysis of GRPO under binary verifiable rewards makes the contrastive reading precise. It shows that normalising the rewards by their mean and variance induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy (Mroueh, 2025). In plain terms: the “negatives” GRPO learns from are the model’s own earlier failures on the same prompt. It is being trained against itself.

The analysis also separates out the design choices that implementations vary. GRPO was originally paired with clipping to keep updates near the old policy; Mroueh analyses variants that differ in reward normalisation (mean only versus mean plus variance) and in how they regularise with KL divergence: penalising divergence from the previous model, from a fixed reference model, or both (Mroueh, 2025). For each variant the optimal policy at a given step has an explicit form in terms of the binary reward, the first and second-order statistics of the reward under the previous policy, and the previous and reference policies (Mroueh, 2025).

The result that matters most for agent builders concerns what happens over many steps. Iterating the update gives a sequence of policies whose probability of success obeys a simple recurrence, which converges to a fixed point determined by the reference policy’s probability of success and the regularisation strength, and that fixed point exceeds the reference (Mroueh, 2025). The paper’s name for this is success amplification: GRPO raises the policy’s probability of success on a prompt, and how far it can raise it depends on where the reference started and how strongly the update is held back.

Two consequences follow directly. If the reference probability of success on a prompt is zero, there is nothing to amplify; the recurrence starts at zero and stays there. And the regularisation strength is not a mere stability knob. It sets the ceiling the model converges towards. Loosen it and the fixed point moves up, at the cost of letting the policy drift further from the reference. Whether that drift creates new capability or only sharpens existing capability is the subject of a separate note on this topic.

How does GRPO change when the task is an agent task?

Everything above was described for a maths problem with a single answer. Agent tasks differ in four ways, and each one makes the method harder to run well.

Trajectories are long. An agent’s sample is not one answer but a sequence of tool calls, observations and messages. The update treats every token in a trajectory as sharing the trajectory’s advantage. A 40-step run that passed gets all 40 steps pushed up, including the three that were mistakes it recovered from; a run that failed gets all 40 pushed down, including the 37 that were fine. This is the credit-assignment problem, and it is why process reward models and tree-structured sampling exist.

Rollouts are expensive. Sampling a group of eight for a maths prompt costs eight short generations. Sampling a group of eight for a coding task costs eight sandboxes, eight test runs and eight sets of tool latency. The price of a zero-advantage group is correspondingly higher, and so is the value of choosing tasks at the right difficulty.

The reward comes from an environment the agent can act on. A maths checker compares a string with a reference. An agent’s checker inspects an end state that the agent has been modifying for the whole episode. If the agent can reach the checker’s inputs, the contrastive signal will happily reward whatever reaches them. That is reward hacking, and GRPO has no defence against it beyond the one you build into the environment.

The environment is stochastic. Two runs of the same agent on the same task can diverge because of timing, tool output and inference noise. That is partly a feature, since it supplies the spread the group needs, and partly a hazard, since it adds variance the advantage cannot distinguish from skill.

The practical upshot is that the environment, not the optimiser, is where most of the difficulty lives. GRPO will amplify whatever the checker rewards, on whichever tasks give it a spread to work with.

What are the knobs, and what do they do?

Implementations expose a handful of settings. The table gives the plain-language effect of each, drawn from how the update is constructed rather than from any single paper’s tuning.

The last two rows are not optimiser settings at all. They are environment design, and they dominate. A perfectly tuned GRPO run on a task suite that is 90% too easy or too hard is a perfectly tuned way of spending 90% of the budget on nothing.

What goes wrong, and how would you notice?

Three failure modes account for most bad GRPO runs on agents.

The signal disappears. The share of groups with zero spread climbs as the model learns the easy tasks, and training slows to a crawl while compute use stays flat. You notice it by logging, per batch, the fraction of groups whose rewards were all identical. If that fraction is high, the data is the problem, not the learning rate.

The signal is real but points the wrong way. The checker rewards something that is not the task: a test file the agent edited, an output it hard-coded, a “done” it reported without doing. Reward goes up, the model gets worse at the actual job. You notice it only if something other than the checker measures the agent, which is why we treat a checker the agent cannot reach as the starting point of any task environment we build.

Length drifts. Because every token in a trajectory shares its advantage, there is a mild pressure on long failed trajectories to get shorter and on long passing ones to stay long. Depending on the implementation’s token-level weighting, this can push the model towards verbosity or terseness unrelated to quality. You notice it by tracking trajectory length alongside reward.

None of these is exotic. They follow from the three ingredients of the method: a group baseline, a contrastive update, and a checker that supplies the reward.

What this means for teams training agents

GRPO is a good default because it is cheap and because its behaviour is now well understood: a weighted contrastive loss, with the model’s own previous attempts as the negatives, that amplifies the probability of success towards a ceiling set by the reference and the regularisation (Mroueh, 2025). That understanding tells you where to spend effort.

• Spend it on task difficulty. The method only learns from prompts where the group splits. Measure the split rate and keep it high.
• Spend it on the checker. The method amplifies whatever the checker rewards. Make sure the checker rewards the task and nothing cheaper.
• Spend it on credit assignment only once the first two are solved. Long trajectories are a real problem, but a dense signal on a gameable checker is worse than a sparse signal on a sound one.
• Expect sharpening. The clearest theoretical result says GRPO raises the probability of outcomes the base model could already produce; a prompt the base model never solves gives it nothing to work with (Mroueh, 2025). Plan your task suite around what the model can sometimes do.

GRPO is one piece of the picture. The rest, including why most training data gives zero advantage, whether RLVR creates new skills, and how to defend the checker, is in the guide to reinforcement learning for AI agents.