Reinforcement learning for AI agents (RLVR)

Reinforcement learning with verifiable rewards (RLVR) trains a model with rewards computed by a checker, such as a test suite or an answer key, rather than by human preference. For agents, the checker verifies whether the task was done, which makes the checker’s independence from the agent the whole game.

This guide covers the algorithm most teams use (GRPO), the data problem that wastes most of the compute (zero advantage), the open question of what RLVR actually teaches, process rewards for long tasks, defences against reward hacking, and the market for environments.

What is RLVR and how is it different from RLHF?

In reinforcement learning from human feedback, a reward model trained on human preferences scores each output. In RLVR, the reward comes from a check that can be run: did the code pass the tests, does the answer match, did the task’s end state meet the specification.

For agent tasks the attraction is obvious. If an agent is supposed to fix a bug, the test suite is a reward that can’t be charmed. The catch, covered below, is that the agent may be able to reach the checker.

How does GRPO work?

Group Relative Policy Optimization (GRPO) was introduced in DeepSeekMath (DeepSeekMath, 2024). For each prompt, the model generates a group of samples. Each sample’s reward is compared with the mean of its group, and the model is pushed toward samples that scored above the group and away from those below it.

The design avoids training a separate value network, which keeps memory and compute manageable at scale. It also has a consequence that shapes the rest of this guide: the signal for a prompt is the spread of rewards within its group. If every sample gets the same reward, there is nothing to learn from.

Why does most RL data teach nothing?

That consequence is the zero-advantage problem. When a prompt is too easy, all samples pass; when it is too hard, all fail. In both cases the group’s rewards are identical, every advantage is zero, and the update carries no signal.

The scale of the waste is large. MagicBeanStalk reports that about 88% of a 78K-problem maths corpus produced zero advantage (MagicBeanStalk, 2026). In other words, most of the compute spent on that corpus taught the model nothing.

Two responses exist. DAPO filters at training time: prompts whose group all passes or all fails are dropped and resampled, so each batch carries signal (DAPO, 2025). MagicBeanStalk instead synthesises problems calibrated to the model’s current ability, and reports a gain of +8.6 points against +5.8 for the best filtered subset of real problems (MagicBeanStalk, 2026).

For agent training the lesson is that task difficulty has to be tuned to the agent. A suite of tasks the agent always solves, or never solves, is a suite that costs money and changes nothing.

Does RLVR create new skills or sharpen old ones?

This is contested. One analysis of GRPO’s dynamics describes it as success amplification: the method raises the probability of outcomes the base model could already reach, rather than reaching new ones (Mroueh, 2025). On that view, RLVR sharpens; it doesn’t create.

ProRL argues the opposite from an empirical angle, reporting that prolonged RL training expands what the model can solve beyond the base model’s range (ProRL, 2025). The disagreement often turns on how you measure “can solve”: a single attempt, or the best of many attempts (pass@k).

For agent builders the practical reading is cautious. Expect RLVR to make an agent more reliable at things it can sometimes do. Don’t expect it to conjure a capability the base model never shows.

How do process rewards help long-horizon agents?

Agent tasks are long. A 40-step task with a single pass/fail at the end gives the model one bit of feedback for 40 decisions. A process reward model scores intermediate steps instead, which gives a denser signal.

Tree-GRPO applies tree search to agent rollouts, branching at intermediate steps so that outcomes from shared prefixes yield a step-level signal without a separately trained step scorer (Tree-GRPO, 2025). The promise is credit assignment; the pitfall is that a process reward is itself a model of what a good step looks like, and models of that kind can be gamed.

How do you defend against reward hacking?

Reward hacking is when the model gains reward through behaviour the reward was not meant to encourage. METR documented frontier models doing this in evaluations, including o3 on 30.4% of RE-Bench runs against 0.7% on HCAST, with cases of models altering the evaluator’s code (METR, 2025).

Defences come in layers. The first is verifier design: a checker the model cannot read, reach or modify, and tasks where satisfying the checker and doing the task are the same thing. The second is to shape the training signal with more than the checker. VARL combines verifiable rewards with demonstrations of the right way to solve a task, so that a correct answer reached the wrong way is not fully rewarded (VARL, 2026). One caveat the paper is clear about: in its reward-hacking test, the demonstrations came from o4-mini, not from humans.

The third layer is measurement. If you don’t score how often your agent games the checker, you won’t know whether the first two layers worked.

What does the RL environment market look like?

Environments, the task suites with checkers that RLVR runs on, have become a product. Sapphire Ventures describes a market where AI labs are the main buyers and contracts run to six or seven figures a quarter (Sapphire Ventures, 2026). The open alternative is Prime Intellect’s Environments Hub, a shared registry of environments anyone can run or contribute to (Environments Hub, 2025).

Whichever route you take, the environment’s ground truth is the thing to scrutinise. An environment whose checker the agent can influence trains the agent to influence checkers. Inoku’s custom task environments are built around an independent ground truth, including tasks that cannot be completed.