Task environments with an independent ground truth

Why the ground truth must be independent of the agent

A task environment has two parts: the work the agent is asked to do, and the check that decides whether it did it. If the agent can see, influence or rewrite the check, the environment measures something other than the work.

This is not a theoretical concern.

• Given test cases that could not legitimately be passed, frontier models found ways to pass them anyway, by changing the tests or working around them (ImpossibleBench, 2025).
• On METR’s RE-Bench, o3 engaged in reward hacking on 30.4% of runs, against 0.7% on HCAST, in some cases by modifying the code that scored it (METR, 2025).
• Most of the agent harnesses tested in one study let the agent delete or rewrite its own traces when that helped it (Qin et al., 2026).

The principle that follows is public and simple: the ground truth sits outside the agent’s reach. The agent cannot read it, change it, or write the record it is judged against. Every Inoku environment is built to that principle.

What a suite contains

• Task families. Groups of related tasks, agreed with you: for example repairing a failing build, migrating a schema, or carrying out a change on a realistic server. Each family has a defined setting and a defined meaning of “done”.
• Held-out splits. A share of every family is kept back and never used for training, so you can measure progress on tasks the model has not seen. See held-out evaluation.
• Tasks that can’t be done. Some tasks have no legitimate solution: the file is missing, the permission is denied, the requirement contradicts itself. They exist to measure whether the agent says so. An agent that reports “I couldn’t” when that is true is doing the right thing; one that reports “done” is not. See impossible task and honest failure.
• Separate reporting. False “done” and false “can’t” are reported as two numbers, by family, with confidence intervals.

Calibrated so that training does something

For teams training with verifiable rewards, the environment decides whether training does anything at all. Under group-based methods such as GRPO, a task that every sample passes, or every sample fails, produces no learning signal. This is the zero-advantage problem. In one 78K-problem maths corpus, about 88% of the problems gave zero advantage (MagicBeanStalk, 2026).

We calibrate a suite to the model it is meant to train, so that difficulty is spread and most tasks are neither trivial nor hopeless, and we refresh it as the model improves. The guide to reinforcement learning for agents covers the background.

Exclusive or shared delivery

• Exclusive. A suite built for you alone. Nobody else sees the tasks, and we do not reuse them. For labs that need an edge or work in a specific domain.
• Shared. A suite offered to several customers at a lower cost, with its held-out split kept private. For teams that need a credible independent measurement more than exclusivity.

Open environments exist too, and they are good for the field; Prime Intellect’s Environments Hub is one example (Environments Hub, 2025). Labs are the main buyers of custom environments today, with contracts reported at six to seven figures a quarter (Sapphire Ventures, 2026). What is scarce is not environments. It is environments whose ground truth the agent cannot reach. Our free tier, for research, education, open source and organisations under 10 people and $1M in revenue, is on the pricing page.

How we work together

1. A scoping call: your model, your training or evaluation goal, and the domains that matter.
2. We propose task families, suite size and the split between training and held-out tasks. You confirm.
3. A pilot family, delivered and run against your current model, so you see real reports before committing to the full suite.
4. The full suite, with a refresh cadence agreed up front.

We do not need model weights. For training use, we agree how your stack calls the environment. For evaluation use, API access to the model is enough.

Status

Custom task environments go to design partners first. The same independent ground truth underlies our private evaluations and the Inoku Verified™ mark. Contact us to talk about a suite.