GRPO

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that scores each sample against the mean of a group of samples for the same prompt.

What GRPO is

Policy-gradient methods need a baseline: an estimate of how good a response was expected to be, so that the update rewards responses that beat expectation and penalises those that fall short. Proximal policy optimisation (PPO), the previous default for language models, learns that baseline with a separate value model, usually as large as the policy itself.

GRPO drops the value model. For each prompt it samples a group of responses, scores each with the reward function, and uses the group’s mean and standard deviation as the baseline. A response’s advantage is how far its reward sits above or below its siblings, in units of the group’s spread. The policy is then pushed towards the above-average responses and away from the below-average ones, with the same clipping and reference-model penalty that PPO uses to keep updates small.

GRPO was introduced in DeepSeekMath, where it trained a mathematics model without the memory and compute a value model would have needed (DeepSeekMath, 2024). It has since become the usual algorithm for RLVR, and most open training libraries ship an implementation.

A simple example

Suppose a team is training a coding agent and the current prompt asks it to make a failing integration test pass. GRPO samples eight attempts. Five end with the test passing and three do not, so the rewards are 1, 1, 1, 1, 1, 0, 0, 0. The group mean is 0.625. The five successes get a positive advantage, the three failures a negative one, and the update raises the probability of whatever the successful attempts did differently. Now suppose the next prompt is one the model always solves: eight rewards of 1, a mean of 1, and an advantage of zero for every sample. The team has spent eight agent runs and learnt nothing from that prompt. In this hypothetical, the first prompt was informative and the second was wasted compute, and nothing in the loss curve would have told the team which was which.

Why it matters

The group baseline is what makes GRPO cheap, and also what gives it its main weakness. If every response in a group earns the same reward, every advantage is zero and the prompt teaches nothing. For agent tasks with binary success rewards this happens often. MagicBeanStalk reports that about 88% of a 78K-problem mathematics corpus produced zero advantage for the model under training (MagicBeanStalk, 2026). That is the zero-advantage problem, and managing it is a large part of running GRPO well.

Agent trajectories are also long. A single outcome reward spread across dozens of tool calls is a weak signal for which calls helped, so credit assignment is poor and training is slow. And because the reward is whatever the checker returns, GRPO will amplify any way of satisfying the checker, including ways the task designer did not intend.

How it is measured or handled

Practitioners watch a few quantities during a GRPO run: the share of groups with non-zero advantage, the mean reward per prompt, response length, and the divergence from the reference model. A falling share of informative groups is the usual sign that the curriculum needs refreshing.

Common adjustments are public and well documented. Prompts that are always or never solved are filtered out, or sampling continues until a batch holds enough informative groups, a practice DAPO calls dynamic sampling (DAPO, 2025). Group size is tuned so that mixed outcomes are likely. Clipping ranges are widened on the upside to keep exploration alive. For long tasks, step-level signals from a process reward model or from tree-structured sampling supplement the outcome reward.

Related terms

GRPO is the usual optimiser for RLVR. Its group baseline causes the zero-advantage problem, and a process reward model is one response to its weak credit assignment on long tasks. The RL for agents guide explains GRPO for agent builders in more detail.

Where this term is used

• Reinforcement learning for AI agents (RLVR)
• Does RLVR create new skills or sharpen old ones?
• GRPO explained for agent builders
• The zero-advantage trap in RLVR training
• Custom task environments for AI agents
• Process reward model
• RLVR
• Zero-advantage problem
• For AI labs and model builders