Zero-advantage problem

The zero-advantage problem is when every sample in a group gets the same reward, so the training update carries no signal.

What the zero-advantage problem is

GRPO and related methods compute each sample’s advantage relative to the other samples for the same prompt: the sample’s reward minus the group mean, divided by the group’s spread. With a binary reward, a prompt where all samples succeed gives every sample an advantage of zero; so does a prompt where all samples fail. In both cases the gradient from that prompt is zero. The model has spent the compute to generate and score the group and learnt nothing from it.

The problem is structural, not a bug. It follows from using the group as its own baseline, which is the thing that made the method cheap. A value model would have given a non-zero signal on an all-fail prompt; the group cannot, because there is nothing within the group to compare against.

It is also a moving target. A prompt that teaches today becomes all-pass once the model has learnt it, and a corpus that was well matched to the model at the start of training drifts out of range as training proceeds.

A simple example

Suppose a team is training an agent on 1,000 environment tasks with a group of eight samples per task and a reward of 1 for success. On the first pass they log the per-task outcomes. Four hundred tasks score eight out of eight: the base model already handles them. Four hundred score zero out of eight: they are beyond it. Two hundred land somewhere in between. Only those two hundred move the policy; the other eight hundred cost 6,400 agent runs and contributed nothing. After a few epochs, fifty of the informative tasks have become all-pass, and the useful set has shrunk to one hundred and fifty. In this hypothetical, four fifths of the compute was wasted from the start and the fraction grows unless the team refreshes the task set.

Why it matters

The share of wasted prompts can be large. MagicBeanStalk reports that about 88% of a 78K-problem mathematics corpus produced zero advantage for the model under training (MagicBeanStalk, 2026): the problems were solved every time or never. Only the remaining fraction, where the model sometimes succeeded, moved the policy.

For agents the issue is sharper. Agent tasks are expensive to run, success is usually binary, and difficulty is hard to set in advance, so a large share of an environment suite may be dead weight for any given model. For a team buying or building training environments, the question is not how many tasks a suite has but how many sit at the edge of what the model can do, and that number is specific to the model and changes as it trains.

How it is measured or handled

The diagnostic is simple: the share of groups per batch with non-zero advantage, tracked over training. A falling share means the curriculum is drifting out of range. Per-task pass rates from an initial sweep show which tasks are informative for the current model.

Handling it has three public forms. Filtering out all-pass and all-fail prompts before training saves compute but adds no signal. Dynamic sampling, as in DAPO, keeps sampling until a batch contains enough informative groups (DAPO, 2025). Generating new tasks at the right difficulty adds signal: MagicBeanStalk’s calibrated synthesis of problems gave a gain of +8.6 points, against +5.8 from the best filtered subset of real data (MagicBeanStalk, 2026). Ordering tasks by difficulty and mixing in demonstrations on all-fail prompts are related approaches.

Related terms

The problem is a property of GRPO under RLVR. A process reward model gives step-level signal on prompts where the outcome reward is uninformative. The RL for agents guide covers the trap and the ways round it.

Where this term is used

• Reinforcement learning for AI agents (RLVR)
• Does RLVR create new skills or sharpen old ones?
• GRPO explained for agent builders
• The zero-advantage trap in RLVR training
• Custom task environments for AI agents
• GRPO
• RLVR
• For AI labs and model builders