The zero-advantage trap in RLVR training

Most of the data in a typical reinforcement-learning corpus teaches the model nothing, because the model either solves every problem in it every time or never solves it at all. Under group-based methods such as GRPO, a task where every sample gets the same reward produces an advantage of zero for every sample, and a zero advantage is a zero gradient.

That is the zero-advantage problem, and the scale of it is larger than most teams expect: one study found that roughly 88% of a pooled 78K-problem maths corpus fell into it for the policy being trained (MagicBeanStalk, 2026). This post explains why it happens, why it is worse for agents, and what the two main responses, filtering and calibrated synthesis, can and cannot do. The algorithm itself is covered in the guide to reinforcement learning for AI agents.

What is the zero-advantage trap?

Reinforcement learning with verifiable rewards (RLVR) scores each sample with a checker, usually giving 1 for pass and 0 for fail. GRPO and its relatives then compare each sample with the other samples for the same prompt. The baseline for a prompt is the mean reward of its group, and a sample’s advantage is its reward minus that mean, divided by the group’s standard deviation.

Now consider a prompt the model always solves. Every reward is 1, the mean is 1, and every advantage is 1 minus 1, which is zero. Consider a prompt the model never solves. Every reward is 0, the mean is 0, and every advantage is 0 minus 0, which is also zero. In both cases the prompt contributes nothing to the update. The compute to generate and score the group has been spent, and the policy is exactly where it was.

The trap is that this happens silently. A training loop reports that it processed the batch. Reward curves look fine, because the all-pass prompts are counted as passes. Nothing flags that the batch carried no signal. MagicBeanStalk’s phrase for a corpus in this state is that it has fallen into a zero-advantage trap (MagicBeanStalk, 2026).

Why does a task every sample passes or fails give no gradient?

The intuition is simple, but it is worth seeing why it is not a quirk that a cleverer implementation could avoid.

A policy-gradient update increases the probability of each sampled token in proportion to the advantage of the sample it belongs to. Positive advantage, push up; negative advantage, push down; zero advantage, no push. The advantage exists to tell the model which of its own attempts was better than its other attempts. It is a relative quantity by construction.

Group-based methods make the relativity explicit by using the group as the baseline. A group that all passed has nothing to contrast: there is no worse attempt to push away from. A group that all failed has nothing to contrast either: there is no better attempt to push towards. The information “this task is easy” or “this task is hard” is present, but it is not information about how to do the task better, and that is the only kind of information a policy gradient can use.

You could, in principle, add a learned value model back in, as PPO does, and then an all-pass group would have a non-zero advantage if the value model under-predicted it. But that reintroduces the cost GRPO was designed to remove, and it does not create information that was not there. If the model already passes a prompt every time, there is no behaviour to change; if it fails every time, there is no success to learn from. The trap is structural.

How much of a real corpus is wasted?

MagicBeanStalk measured it. The authors characterise a dataset’s usefulness as a function of three things: its problems, the target policy, and the training rollout budget (MagicBeanStalk, 2026). The same corpus can be mostly useful for one model and mostly useless for another, so “how good is this dataset” is not a well-formed question without saying for whom.

Under that characterisation they pooled a large open-source maths corpus from widely used RLVR datasets and found that, for a fixed target policy and rollout budget, roughly 88% of its problems were solved in every rollout or failed in every rollout (MagicBeanStalk, 2026). The corpus had 78K problems. Fewer than one in eight of them could move the policy.

The downstream effect was measurable. Continual RLVR on the full 78K corpus improved the model by only +1.1 points averaged across eight maths benchmarks (MagicBeanStalk, 2026). The corpus was large. It was also, for that model at that budget, almost entirely dead weight.

A point that is easy to miss: the 88% is not a property of the problems alone. The abstract is explicit that it holds for a fixed target policy and rollout budget. A weaker model would find more of the hard problems hopeless and more of the easy ones still informative; a stronger model would find more of them trivial. The trap moves with the model.

Why is the trap worse for agents than for maths?

Everything in the MagicBeanStalk study concerns maths problems. The mechanism applies to any binary-reward, group-based setup, and agent training makes each ingredient worse.

Rollouts cost more. A maths rollout is a few hundred tokens. An agent rollout is a sandbox, a sequence of tool calls, a test run, and minutes of wall-clock time. Spending 88% of that on prompts that carry no signal is a different order of waste.

Difficulty is harder to set in advance. A maths problem’s difficulty is roughly stable. An agent task’s difficulty depends on the environment’s state, the tools available, the model’s familiarity with the stack, and chance. A task that looks medium on paper may be trivial once the model learns one trick, or hopeless because of one missing permission.

Rewards are more binary. Many agent checkers answer a single yes-or-no question: did the end state meet the specification. There is no partial credit to supply spread within a group.

The trap and the hack overlap. An all-pass group on a task the model should not be able to pass is a warning sign for reward hacking, not a sign of competence. A task suite with many all-pass groups hides that warning among the genuine easy tasks.

So a team training an agent with GRPO should assume, until measured otherwise, that most of its task suite is in the trap for its current model, and that the share will change as the model learns.

Does filtering fix it?

The obvious response is to drop the prompts that teach nothing. There are two places to do it.

Before training, by running the current model on the corpus, measuring pass rates, and keeping only the prompts with a pass rate strictly between 0 and 1. This saves compute and concentrates the batch on useful prompts.

During training, by oversampling and discarding. DAPO, the Decoupled Clip and Dynamic Sampling Policy Optimization algorithm, is the best-known example. Its authors open-source a large-scale RL system that reaches 50 points on AIME 2024 from a Qwen2.5-32B base model, and present four key techniques that they argue make large-scale LLM RL work (DAPO, 2025). The “dynamic sampling” in the name is the relevant one: prompts whose group all passes or all fails are dropped and resampled, so that each batch is filled with prompts that carry signal (DAPO, 2025). The system, its training code on the verl framework, and a curated dataset are all released, which is part of why the technique spread (DAPO, 2025).

Filtering of either kind works, up to a point. The MagicBeanStalk authors report that the best filtered subset of their real corpus improved the model by +5.8 points, against +1.1 for the full corpus (MagicBeanStalk, 2026). That is a large gain from simply not training on the dead weight.

But filtering has a ceiling that the same paper names. Filtering concentrates what remains, but it can only select; it discards the hard problems (MagicBeanStalk, 2026). The all-fail prompts are exactly the ones that would teach the model the most if it could learn from them, and filtering throws them away along with the trivial ones. Nothing in a filter can turn a hopeless problem into a learnable one. It can only find the small fraction of the corpus that was learnable already.

Does synthesising calibrated tasks fix it?

If filtering can only select, the alternative is to make new problems whose difficulty sits in the learnable band. MagicBeanStalk sets out what such problems must satisfy. They should require the reasoning operations and compositions needed to solve the hard problems, but their difficulty must be calibrated so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget (MagicBeanStalk, 2026).

That last clause deserves emphasis. A problem that the model can solve in principle, but only with a solution longer than the rollout budget allows, is in the trap just as surely as an impossible one. The budget is part of the difficulty.

The authors also report what does not work: prompting a teacher model to vary the hard problems directly does not produce calibrated problems (MagicBeanStalk, 2026). Their method instead is a top-down decomposition followed by bottom-up recursive synthesis, which grows seed problems into families of new problems calibrated to the policy and the budget, while preserving the seeds’ reasoning blueprints (MagicBeanStalk, 2026).

The results, on a 16K-token budget across eight maths benchmarks (MagicBeanStalk, 2026):

The comparison that matters is +8.6 against +5.8: calibrated synthetic problems beat the best filtered subset of real problems by nearly three points, on the same model and the same budget. The paper also reports the method transferring to a different domain, synthesising over 14K verifiable finance problems from only 340 solutions and improving the base model by 6.9 points on SecQue and 2.4 on FinanceBench (MagicBeanStalk, 2026).

What the abstract does not settle is how far this generalises beyond maths and finance, and the +5.4 for DeepSeek v4Flash is a smaller gain than for gpt-oss-120b. One study, two domains, two models. The direction is clear; the magnitude elsewhere is not yet known.

How does the trap move during training?

Because the trap is defined relative to the policy, it is not a fixed set of prompts. It is a moving window.

At the start, a task suite for a given model splits three ways: tasks the model always solves, tasks it sometimes solves, and tasks it never solves. Only the middle band teaches. As training proceeds, the middle band’s tasks are learnt and migrate into the always-solved band. Some never-solved tasks become sometimes-solved, if the skills the model gained on the middle band transfer to them. The learnable band drifts upwards in difficulty.

Three things follow for anyone running a long training job.

1. A filter computed once goes stale. A pass-rate measurement taken before training describes the model before training. After a few hundred steps the always-solved band has grown and the filter is letting dead weight through. Dynamic sampling, as in DAPO, avoids this by filtering against the current policy on every batch (DAPO, 2025), at the cost of generating rollouts that are then discarded.
2. The suite runs out. If the middle band is learnt and the never-solved band does not open up, training stalls with no signal left, however much data remains in the corpus. The 88% figure describes one snapshot; by the end of a long run the share in the trap can be higher.
3. Calibration has to be repeated. A synthesis method calibrated to the policy at step zero is calibrated to a model that no longer exists by step one thousand. MagicBeanStalk’s framing of effectiveness as a function of the target policy makes this explicit (MagicBeanStalk, 2026): the data has to track the model.

How do you know whether your own data is in the trap?

None of this requires a paper to diagnose. The measurement is cheap relative to the training it protects.

• Log the group split rate. For every batch, record the fraction of groups whose rewards were all identical. That number is the share of your compute that taught nothing. If it is above half, the data is the bottleneck.
• Histogram pass rates per task. Run the current model on each task in the suite several times and bin by pass rate. The bins at exactly 0 and exactly 1 are the trap. The bins in between are your training set.
• Track the histogram over time. Repeat the measurement at intervals. Watch the middle bins empty and decide, before they do, where the next tasks are coming from.
• Treat unexpected all-pass as a flag. A task that was in the never-solved bin last week and the always-solved bin this week may reflect learning, or it may reflect the agent having found a way to satisfy the checker without doing the task. Check a sample by hand, against something the agent did not write.
• Keep a held-out split. A held-out evaluation set that never enters training is the only way to tell whether the middle-band gains are generalising or being memorised.

The cost of these measurements is a few rollouts per task. The cost of not making them is a training run whose reward curve looks healthy while most of its budget is spent on groups with zero spread.

What this means for teams training agents

The zero-advantage trap is not a bug in GRPO; it follows from using the group as its own baseline, which is the thing that made the method affordable. What it means in practice is that a task suite is only as valuable as the share of it that the current model can sometimes, but not always, solve.

• Measure before you train. The share of a corpus in the trap can be most of it. 88% of a 78K-problem maths corpus was, for one model and one budget (MagicBeanStalk, 2026).
• Filter, but know its ceiling. Dropping dead-weight prompts, before training or dynamically as DAPO does (DAPO, 2025), concentrates the signal that exists. It does not create signal, and it discards the hard tasks along with the trivial ones (MagicBeanStalk, 2026).
• Calibrate difficulty to the model, and recalibrate as it learns. Problems built to sit in the learnable band, with a solution that fits the budget, beat the best filtered real data in the one study that has measured it, +8.6 against +5.8 (MagicBeanStalk, 2026).
• Count the budget as part of difficulty. A task the agent could solve with more steps than the rollout allows is, for training purposes, impossible.
• Be suspicious of easy wins. For agents, an all-pass group on a task that should be hard is a reward-hacking signal first and a competence signal second.

Those principles are why we calibrate the task environments we are building to the model they are meant to train, and refresh them as the model improves. The algorithm that makes calibration necessary, and the open question of what RLVR teaches when the data does carry signal, are covered in the guide to reinforcement learning for AI agents.