Does RLVR create new skills or sharpen old ones?

Nobody knows for certain, and the two best-argued positions reach opposite conclusions by different routes. A theoretical analysis of GRPO under verifiable rewards finds that it amplifies the probability of successes the base model could already produce (Mroueh, 2025); an empirical study of prolonged RL reports models solving tasks that the base model failed regardless of how many attempts it was given (ProRL, 2025).

This post lays out the debate as it stands: what each side claims, where the claims actually conflict, and what would have to be true for both to hold. It does not pick a winner, because the evidence does not. The background on reinforcement learning with verifiable rewards (RLVR) and the GRPO algorithm is in the guide to reinforcement learning for AI agents.

What is the debate actually about?

When a model trained with RLVR solves a problem the untrained model got wrong, one of two things has happened. Either the model has learnt to do something it could not do before, or it has learnt to do reliably something it could already do occasionally. From the outside, on a single attempt, the two look identical. The difference only shows up when you ask how the base model would have fared given many attempts.

ProRL’s abstract states the question in exactly these terms: it remains contentious whether RL truly expands a model’s reasoning capabilities or merely amplifies high-reward outputs already latent in the base model’s distribution (ProRL, 2025). The two sides are usually given shorthand names.

The phrase base-model barrier is sometimes used for the sharpening view’s strongest form: the idea that the base model’s support, the set of outputs it can produce with non-negligible probability, is a ceiling that RL cannot lift. The expansion view is the claim that the ceiling can be lifted, under conditions.

Why is pass@k the battleground?

The measurement that separates the positions is pass@k: the probability that at least one of k independent attempts solves the problem. Pass@1 is ordinary accuracy. Pass@64, pass@256 and beyond ask what the model can do if you let it keep trying.

The sharpening view predicts a specific pattern. RL training should raise pass@1 a lot, because it concentrates probability on the solutions the model used to find only occasionally. But at large k, the base model should catch up, because it was already finding those solutions sometimes and k attempts give it enough chances. On a pass@k curve, the trained model starts above the base model and the gap closes as k grows. If the curves cross, so that the base model overtakes at large k, that is evidence that RL narrowed the model’s output distribution: it got better at what it does and lost some of what it occasionally did.

The expansion view predicts a different pattern. If RL has reached solutions the base model does not produce, then no value of k lets the base model catch up on those problems. The trained model’s curve stays above the base model’s at every k, and for some problems the base model’s curve is flat at zero.

ProRL’s abstract reports results of the second kind: RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts (ProRL, 2025). That last clause is the direct challenge to the barrier.

Two caveats about the measurement itself are worth stating, because they apply to any paper on either side. First, “regardless of the number of attempts” always means “regardless of the number we tried”. A problem the base model solves once in ten thousand attempts looks the same as one it never solves if you stopped at a thousand. Second, pass@k on a public benchmark inherits that benchmark’s exposure to training data; if the base model has seen the problems, benchmark contamination muddies both curves.

What does the sharpening position say?

Mroueh’s analysis is of GRPO as actually used: binary verifiable rewards, group-relative normalisation, and regularisation towards a reference model. It shows that normalising the rewards by their mean and variance induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy (Mroueh, 2025). The model learns from its own earlier successes and failures on each prompt.

The analysis then follows what happens over repeated updates. For each variant considered, which differ in whether rewards are normalised by mean only or mean and variance, and whether the KL regularisation is towards the previous model, a fixed reference model, or both, the optimal policy at each step has an explicit form, and iterating gives a sequence of policies whose probability of success obeys a simple recurrence (Mroueh, 2025). That recurrence converges to a fixed point determined by the reference model’s probability of success and the regularisation strength, and the fixed point exceeds the reference (Mroueh, 2025).

The paper’s own summary is that GRPO amplifies the policy’s probability of success (Mroueh, 2025). Read carefully, the result has a precise shape.

• The thing being amplified is a per-prompt probability of success that the reference model already has. The recurrence starts from the reference’s probability and climbs from there.
• Where that probability is zero, the recurrence has nothing to climb from. A prompt the reference never solves is not a prompt the analysis predicts GRPO will learn.
• The ceiling is set by the regularisation. A stronger pull towards the reference gives a lower fixed point; a weaker pull gives a higher one.

This is why the sharpening view is sometimes called the theoretical position. It is not an empirical claim that RL never produces new behaviour. It is a derivation of what the GRPO update does under its stated assumptions, and under those assumptions the update is an amplifier of what the reference can already do.

What does the expansion position say?

ProRL is an empirical study with a method. The authors introduce a training recipe that combines KL divergence control, reference policy resetting, and a diverse suite of tasks, and train for much longer than is usual (ProRL, 2025). They report that prolonged RL training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling (ProRL, 2025).

The specific findings in the abstract are these. RL-trained models outperform base models across a wide range of pass@k evaluations, including cases where the base model fails at every attempt. The improvement in what the authors call the reasoning boundary correlates strongly with the base model’s competence on the task and with training duration, which they take to suggest that RL can explore and populate new regions of the solution space over time (ProRL, 2025). The model weights are released for others to check.

Two of the recipe’s ingredients are worth connecting to the theory. Reference policy resetting means periodically replacing the reference model that the KL term pulls towards with the current policy. KL divergence control means managing how hard that pull is. Both act directly on the quantity that, in Mroueh’s analysis, sets the ceiling on amplification.

Where do the two positions actually disagree?

Less than the headlines suggest. It helps to separate three claims.

Claim A: a single GRPO run with a fixed reference amplifies what the reference can do, towards a ceiling set by the regularisation. Mroueh derives this (Mroueh, 2025). Nothing in ProRL’s abstract contradicts it; ProRL does not use a fixed reference.

Claim B: prolonged training with reference resets and diverse tasks produces models that solve problems the base model fails at every attempt tried. ProRL reports this (ProRL, 2025). Nothing in Mroueh’s abstract contradicts it; the analysis does not cover reference resets or transfer across tasks.

Claim C: the base model’s support is a hard ceiling that no amount of RL can lift. This is the strong barrier claim. ProRL’s abstract argues against it. Mroueh’s abstract does not assert it; the fixed point exceeding the reference is a statement about one prompt’s probability of success, not about the support of the whole distribution.

The real disagreement, then, is over claim C, and the sharpest version of each position does not quite engage the other. The theoretical result is about one prompt under one reference. The empirical result is about a recipe designed to move the reference and to let gains on some tasks reach others.

How could both be right?

A reconciliation that fits both abstracts, without being proven by either, runs as follows.

Within a single prompt, GRPO is an amplifier: it raises the probability of a success the model already sometimes has, and does nothing for a prompt it never solves. That is claim A. But a model is trained on many prompts at once, and what it learns on one can change its probability of success on another. A prompt that started at zero can become non-zero not because GRPO learnt it directly, but because the skills amplified on neighbouring prompts transferred. Once it is non-zero, it has something to amplify.

Reference resets then matter, because they move the ceiling. If the reference is updated to the current policy, the fixed point is recomputed from a higher starting probability, and the next phase of training can climb further. Repeat that enough times, with enough task diversity for transfer, and a model can end up solving problems the original base never did at any tested k, while every individual update was an amplification of what the then-current reference could do.

This story is consistent with ProRL’s observation that the gains correlate with the base model’s competence and with training duration (ProRL, 2025). A base model with more competence has more non-zero probabilities to amplify and more skills to transfer; a longer run has more reset-and-amplify cycles. It is also consistent with Mroueh’s fixed point depending on the reference (Mroueh, 2025). But it is a story, not a result. Neither abstract demonstrates the transfer step, and that step is where the real question lives.

What would settle it?

The debate is partly about measurement, and the measurements that would narrow it are known.

• Larger k, on problems chosen in advance. Pick problems the base model fails at k attempts, train, and then test the base model at ten or a hundred times k. If it still fails, the “never” claim is stronger. If it eventually succeeds, the problem was in the amplifiable set after all.
• Tasks with no training-data exposure. A pass@k curve on a problem the base model has seen in pre-training measures recall, not capability. Held-out tasks, created after the base model’s data cut-off, remove that confound. A held-out evaluation is the standard tool.
• Ablations on the recipe. ProRL combines KL control, reference resets and task diversity (ProRL, 2025). Removing each in turn would show which one, if any, is doing the lifting that a fixed-reference analysis says cannot happen.
• Per-prompt tracking. Record each prompt’s pass rate throughout training. A prompt that moves from exactly zero to non-zero is a direct observation of the transfer step the reconciliation above relies on. A prompt that never leaves zero is a direct observation of the barrier.
• Theory for the multi-prompt case. Mroueh’s recurrence is per prompt (Mroueh, 2025). An analysis that lets success on one prompt change the reference probability on another would say whether transfer can be derived, not just observed.

Until results of this kind accumulate, the honest summary is that single-run amplification is established in theory, boundary expansion is reported in practice under a specific recipe, and the mechanism connecting them is a hypothesis.

What does this mean for agents in particular?

Both papers concern reasoning and maths. Agent tasks differ in ways that cut both ways.

On the sharpening side, agent tasks make zero-reference-success prompts more common. A task that requires a tool the model has never used, or a sequence of steps it has never composed, may have a base pass rate of exactly zero, and that is the case where the amplifier has nothing to work with. It is also the zero-advantage problem seen from a different angle: a prompt the group always fails gives zero gradient, and a prompt the reference never solves gives zero amplification. They are the same prompts.

On the expansion side, agent tasks are composed of sub-skills, which is exactly the setting where transfer could lift a zero. An agent that learns to read error output on one family of tasks may find its pass rate on a previously hopeless family move off zero, not because that family was trained, but because one of its sub-skills was. If the reconciliation above is right, task diversity is not a nice-to-have for agent training but the mechanism by which the boundary moves.

Either way, one practical warning applies with extra force to agents. A pass rate that jumps from zero to high on a task the model has no business solving is not evidence for expansion. It is a flag for reward hacking. The amplifier does not know the difference between a solution and a shortcut to the checker, and an agent has far more ways to reach the checker than a maths model does. Before concluding that the boundary moved, confirm the task was done by a check the agent could not influence.

What this means for teams training agents

The question in the title is open. What a team can take from the two positions is not an answer but a set of expectations.

• Expect reliable gains on what the model can sometimes do. That is what the theory says GRPO does, and it is well established (Mroueh, 2025). If a task family has a non-trivial base pass rate, RLVR should raise it.
• Do not expect a single run with a fixed reference to teach a task the model never solves. The analysis gives it nothing to amplify (Mroueh, 2025).
• If you need the boundary to move, the evidence that it can comes with a recipe attached: long training, KL control, reference resets and diverse tasks (ProRL, 2025). Budget for that, and measure pass@k at large k on held-out tasks to see whether it worked.
• Build task suites with a spread of difficulty and a spread of sub-skills. Difficulty keeps the amplifier fed; sub-skill diversity is the best current hypothesis for how a zero becomes a non-zero.
• Treat sudden success on a hopeless task as suspect until an independent check confirms it.

The debate will move as more results come in, and this post will be updated when it does. The surrounding material, on the algorithm, the data problem and the checker the whole method depends on, is in the guide to reinforcement learning for AI agents.