Same image, same behaviour? Not for agent runs

Pinning the container image fixes the environment an agent runs in, but it does not fix what the agent does there. Two runs of the same agent, on the same image, with the same task and temperature set to zero, can and regularly do end in different places, because the model’s own output is not deterministic, because the arithmetic beneath it depends on precision and batch size, and because the clock never reads the same twice.

This post explains each of those three sources, puts numbers on the first two from the published research, and draws the conclusion for anyone who wants to know what an agent did. It belongs to the guide to sandboxing AI agents on Linux, which covers the isolation options and a practical Incus setup alongside this question.

Why isn’t temperature 0 deterministic?

Temperature 0 means greedy decoding: at each step the model picks the single most probable next token. In theory that removes randomness from sampling, so the same prompt should always produce the same completion. In practice it does not, and the reason sits below the sampler.

Thinking Machines’ analysis starts from this puzzle. Hosted APIs and self-hosted engines such as vLLM and SGLang are not deterministic at temperature 0, even though the forward pass of the model is, for a fixed input, deterministic from one run to the next. The authors show that a single matrix multiplication repeated a thousand times gives bitwise-identical results, so neither floating-point arithmetic nor GPU concurrency explains the variation on its own (Thinking Machines, 2025).

The ingredient that does explain it is that the model’s output for one request depends on what else is in the batch with it. The forward pass is deterministic for a given batch, but a different batch, meaning a different number of concurrent requests, changes how the GPU kernels group their arithmetic, and that changes the result. Since server load is, from any one user’s point of view, random, the output is random too. The post’s summary is that the load, and therefore the batch size, varies nondeterministically, which makes the endpoint nondeterministic for every user even though the server itself is not (Thinking Machines, 2025).

So “temperature 0” is a promise about sampling, not about the logits being sampled from. The logits move with the batch, and the batch is outside your control.

What does floating-point precision have to do with it?

Everything, at the bottom. Floating-point addition is not associative: (a + b) + c is not always equal to a + (b + c), because each operation rounds to the precision available, and adding numbers with very different magnitudes drops low-order digits. Thinking Machines illustrate this with a shuffle experiment in which summing the same set of values in different orders gave 102 distinct results (Thinking Machines, 2025).

A transformer’s forward pass is a long chain of such sums. Any change to the order in which a reduction is carried out, or to the precision in which intermediate values are held, can flip a rounding decision. Most of the time the flip is invisible. Occasionally it changes which token is the argmax, and from that point the completion is on a different path.

Yuan et al. examine the precision side directly. Their paper traces the root cause of reproducibility failures to the non-associative nature of floating-point arithmetic under limited numerical precision, and finds the problem especially pronounced in reasoning models, where minor rounding differences in early tokens cascade into divergent chains of thought and ultimately change accuracy (Yuan et al., 2025). Their mitigation, LayerCast, stores the weights in 16-bit precision to save memory but performs all computation in FP32, trading some speed for numerical stability.

The lower the precision, the more rounding decisions there are to flip. BF16, which is the default for most inference today, has far fewer mantissa bits than FP32, so the same reordering that is harmless in FP32 can be visible in BF16.

Why does server load change the answer?

Because kernels are tuned for throughput, and throughput means choosing a different strategy for a small batch than for a large one.

Thinking Machines call the property that is missing “batch invariance”: a kernel is batch-invariant if the result for one row does not depend on how many other rows are being processed. They walk through the three reduction-based operations in a transformer and show how each loses that property in a standard serving stack (Thinking Machines, 2025).

The post also addresses a popular explanation, that nondeterminism comes from concurrent threads doing atomic adds in an unpredictable order. The authors say that account is not wrong but incomplete: atomic adds are unnecessary for the vast majority of kernels, and a typical inference forward pass contains essentially none. The one well-known exception they name, the FlashAttention backward pass, is not part of inference at all (Thinking Machines, 2025).

The point for anyone running agents is that the variation is not a bug in one library that will be patched. It is a consequence of serving many users efficiently on shared hardware. Any endpoint you do not control will show it, and so will any endpoint you do control unless you choose kernels for invariance rather than speed.

How much does the output actually vary?

Enough to change results, not just wording.

Yuan et al. report that under bfloat16 with greedy decoding, DeepSeek-R1-Distill-Qwen-7B can show up to 9% variation in accuracy and a difference of 9,000 tokens in response length depending only on the GPU count, GPU type and evaluation batch size. The paper’s broader claim is that changing system configuration in these ways introduces significant differences in generated responses, and that floating-point precision is a neglected variable in evaluation practice (Yuan et al., 2025).

Thinking Machines’ experiment is smaller and more concrete. They sent the prompt “Tell me about Richard Feynman” 1,000 times at temperature 0 to Qwen3-235B-A22B-Instruct-2507, each completion up to 1,000 tokens. Without batch-invariant kernels they got 80 distinct completions, the most common appearing 78 times. Every completion agreed for the first 102 tokens; at token 103, 992 of them continued “Queens, New York” and 8 continued “New York City”. With their batch-invariant kernels installed, all 1,000 completions were identical (Thinking Machines, 2025).

That experiment is a clean picture of how the divergence works. Nothing differs for a hundred tokens, then one near-tie at the argmax goes the other way in a handful of runs, and the two populations are on different paths from then on. In a chat answer, the difference between the two phrasings is trivial. In an agent, the token where the paths split might be the one that decides which file to open.

What does timing noise add?

The two papers above are about the model. An agent run also contains a machine, and the machine does not repeat itself either. This section is general reasoning rather than a cited finding.

An agent reads the clock, directly or indirectly, many times in a run. Commands have timeouts. Package installs depend on a mirror’s response time. Test suites include tests that race. Build tools cache by timestamp. Log lines carry times, and the agent reads logs. Any of these can differ between two runs on the same image, and any of them can change what the agent sees next.

Two concrete patterns recur. First, a command that completes in one run and times out in the next, so one agent proceeds and the other retries, backs off, or takes a different approach. Second, a flaky test that passes in one run and fails in the other, so one agent reports done and the other starts debugging a problem that is not there. Neither is the agent’s fault, and neither is reproducible from the image.

Scheduling adds a third layer. Processes the agent starts run concurrently, and their interleaving is decided by the kernel. Output order from parallel jobs, the content of a directory listing taken mid-build, and which of two racing writes lands last all vary.

The last two rows are why even a fully deterministic model would not give you a deterministic agent run.

Why does the difference compound in an agent run?

A single completion diverges once and then runs to the end. An agent run is a loop: the model produces a command, the environment runs it, the output goes back into the model’s context, and the next command depends on everything so far.

That loop turns a one-token difference into a different run. If two runs choose different first commands, they read different outputs, so their contexts differ from the second step onward, and the chance that they later converge is small. If they choose the same commands but one hits a timeout, the same thing happens one step later. The variation does not average out over a long run; it accumulates.

Yuan et al. observe this within a single completion for reasoning models, where early rounding differences cascade into divergent chains of thought (Yuan et al., 2025). An agent adds the environment to the chain, so the cascade has more places to start and more ways to grow.

The practical consequence is that “run it again and see” is not a diagnostic for agent behaviour. A second run is a second sample from a distribution, not a replay of the first. If the first run did something you did not expect, the second run tells you how often that happens, not what happened the first time.

Can inference be made deterministic?

Partly, and at a cost, and only on hardware you control.

Thinking Machines’ answer is batch-invariant kernels for RMSNorm, matrix multiplication and attention. With those in place their 1,000 completions were identical. The cost on a single GPU running Qwen-3-8B over 1,000 sequences was a rise from 26 seconds with default vLLM to 55 seconds with an unoptimised deterministic build, and 42 seconds after improving the attention kernel. They attribute much of the gap to an attention integration that had not yet been optimised and describe the result as not disastrous (Thinking Machines, 2025).

Yuan et al.’s answer for the precision side is LayerCast: 16-bit weights, FP32 computation (Yuan et al., 2025). It trades compute for stability rather than changing kernel strategy, and the two approaches address different parts of the same problem.

Both require control of the serving stack. If your agent calls a hosted API, the batch is shared with strangers, the hardware is whatever the provider has, and the precision is the provider’s choice. Nothing you set on the client side changes that. Temperature 0 narrows the distribution; it does not collapse it.

Thinking Machines give one further reason to care, from training rather than serving. When the numerics of the sampler and the trainer differ, on-policy reinforcement learning becomes, in effect, slightly off-policy. In their experiment, training without correction saw the reward collapse partway through, with a loss spike and a matching jump in KL divergence near step 318; importance weighting kept training smooth with the KL around 0.001; and a bitwise-identical sampler and trainer kept it flat at zero (Thinking Machines, 2025). For anyone training agents on their own rollouts, that is a reason to want determinism that has nothing to do with reproducibility for its own sake.

What does this mean for records of agent runs?

If a run cannot be replayed, the only account of it is the one made while it happened.

That changes what a record is for. In a deterministic system, a log is a convenience: you could in principle reconstruct events from the inputs. In an agent run, the agent trace, the sequence of prompts, outputs, commands and results, is the only copy of what happened. Lose it, or let it be altered, and the run is gone.

Which makes the question of who controls the record sharp. Qin et al. tested a range of agent harnesses and found that most of them let the agent delete its own trace; they recommend logging outside the agent’s control (Qin et al., 2026). Combine that with nondeterminism and the position is this: the run cannot be re-created, and the default record of it can be edited by the thing being recorded. Whatever you keep has to be made during the run, by something other than the agent, and stored where the agent cannot reach it. That is a principle of general practice, and it is the same one that separates a backup from a copy the user can delete.

What that record should contain, and how to check an agent’s claims against it, is the subject of the guide to verifying what an AI agent actually did.

What this means for teams running agents

Stop treating a pinned image as a reproducibility guarantee for agent runs. It guarantees the environment, and the environment is one of three inputs; the model’s numerics and the machine’s timing are the other two, and neither is fixed by the image. Measure agent behaviour across many runs rather than inferring it from one, and report rates rather than outcomes. If you control the serving stack and need runs to repeat, batch-invariant kernels and higher-precision computation are documented paths with documented costs; if you call a hosted API, accept that two runs are two samples. And since a run cannot be replayed, keep the record of it outside the agent’s reach and make it during the run, not after.

The guide to sandboxing AI agents on Linux takes the next step: given that each run is unique, how to give each one a clean, bounded instance of its own, with a snapshot of its starting state and no route to anything it does not need.