Temperature 0 was never deterministic

A language model does not write words. At every step it produces a score for every token in its vocabulary — a hundred thousand or so numbers — and something downstream picks one.

Temperature controls that pick. At temperature 1 the model samples in proportion to the scores, so a token the model rates at 20% gets chosen about a fifth of the time. Turn temperature down and the distribution sharpens. At temperature 0 it collapses completely: always take the highest-scoring token. No dice, no randomness. That is called greedy decoding, and it is why every tutorial tells you to set temperature to 0 when you want a reproducible result.

The reasoning is sound. Greedy decoding is deterministic — given the same scores. This page is about that last clause.

The experiment is as plain as it sounds. One prompt. Temperature 0. On OpenAI, seed pinned to 42 as the documentation recommends. Send it 30 times and count how many distinct answers come back.

If greedy decoding did what it says, the answer would be 1 every time. Pick a model and a prompt:

The short arithmetic prompt is the control, and it matters: it comes back identical every time. Determinism is reachable. So the question is not whether these systems can be reproducible — it is what you lose it to.

The divergence is not noise smeared across the output. Every run produces the same opening, character for character, and then splits at one exact point — and from there the copies drift apart.

Which means the length of your output is a risk budget. Every token is another chance to land on a near-tie, so a fork is not a fixed hazard — it accumulates. Same prompt, same model, only the token limit changing:

So why there? Ask the model. The OpenAI API will return the top five candidate tokens and their probabilities at every position, which lets us look at exactly how close the race was at each step.

Most steps are not close at all. Tap any token to see what it beat:

The typical step is a landslide — the leading token beats the runner-up by a wide margin, and no plausible perturbation would change the winner. But a handful of positions are exact ties, where the two candidates come back at the same probability to every digit the API reports.

At a position like that, "take the highest-scoring token" is not a decision the model is making. It is a coin balanced on its edge, and something has to tip it.

Computers do not do arithmetic on real numbers. They use floating point, which keeps a fixed number of digits and rounds away the rest at every single step. One consequence is that floating-point addition is not associative: (a+b)+c and a+(b+c) can differ, because the two roundings happen in different places.

Here is that, run on this machine. The same 4,096 numbers, added up four ways:

Every score a model produces is the end of a chain of millions of such additions. The order they happen in is not fixed: it depends on which GPU kernel was selected, how the work was split across cores, and how many other requests were being processed alongside yours. Change any of that and the last bits of the score change with it.

On the vast majority of tokens that is invisible — a landslide stays a landslide. On a coin balanced on its edge, it is the whole decision. That is the mechanism: a difference too small to see, arriving exactly where the model had no preference.

I started this page with a mechanism already in mind, and the measurements refused it. Both corrections are worth more than the original guess.

Wrong the first time: "it's your concurrent requests." The story I expected was that firing 30 requests at once lands them in differently-shaped server batches, and batch shape changes the arithmetic. It predicts something testable — 30 requests sent one after another should be far more stable than 30 sent at once. So I ran both:

The prediction half-landed, which is worse for it than failing outright. On Claude the two columns are identical — 3 and 3, 7 and 7 — so concurrency explains nothing at all there. On GPT-4.1 mini sending them at once did produce more variants, 7 against 4 and 5 against 3, so the effect is not zero.

But look at what the sequential column actually says: 3 and 4 distinct answers, not 1. Serialising your own requests moved the number a little and did not come close to buying determinism. From the client there is no quiet moment to wait for — the server is batching your request with strangers' traffic either way, and you cannot see or control that queue.

Wrong the second time: I could not reproduce the mechanism I was going to assert. The published explanation for this is batch invariance — that the same row of a matrix multiply gives different bits depending on the shape of the batch it travels in. So I wrote the test: put an identical vector through a matrix multiply inside batches of different sizes, and compare the results bit for bit.

That does not make the explanation wrong — it is a property of GPU kernels, which choose different reduction strategies for different shapes, and a CPU matrix library is not obliged to behave the same way. It makes it something I am citing rather than demonstrating. The float experiment in Section 4 is the part I can show you on hardware I control; the step from "float addition is order-sensitive" to "your server changes the order between requests" rests on Thinking Machines' work, not on mine.

While measuring the above, a request failed in a way I did not expect. Setting temperature: 0 on the newest models does not give you an unreliable result — it gives you an error.

Both vendors are withdrawing it, and neither announced it. But the shape is not what I expected, and it is worth being exact about: this is not a simple "newer models dropped it" line.

Anthropic's boundary is clean — everything up to Sonnet 4.6 accepts it, everything from Opus 4.7 on rejects it. OpenAI's zig-zags: GPT-5 rejects temperature, then GPT-5.2 and GPT-5.4 accept it again, then GPT-5.5 and GPT-5.6 reject it. A version number will not tell you which behaviour you get, and neither will release order.

Which leaves one reliable method, and it is the one that produced this table: ask the API. A single call with the parameter set tells you whether your assumptions still hold, and it is the only answer that cannot go stale on you.

One knob survives the cut on OpenAI: seed is still accepted on models that reject temperature. It is documented as best-effort, and it is worth knowing exactly how much effort that is — so, same test, seed pinned, nothing else available:

The parameter is accepted, and the outputs still differ. It is not lying to you. OpenAI's own documentation says outputs will be "mostly identical" and that "determinism is not guaranteed" — which is exactly what we measured, and almost nobody reads.

But a knob that is accepted and does not deliver is worse than one that was removed. Your code looks like it asked for something.

Everything so far could be waved away as cosmetic. "Waves crash" or "waves lap" — nobody's decision turns on that.

It would stop being cosmetic the moment the output is a verdict. So: six yes/no questions, from one with an obvious answer to several genuinely contested, each asked 50 times at temperature 0 with the reply held to a single word. 300 runs. Beside each one, the model's own margin on the verdict token, so how often it flipped and how close it was can be read off the same row.

That is the honest result, and it is the more useful one. A fork needs a tie, and on all six questions the model had a clear favourite — the narrowest margin here still had the winner ahead by around three to one. Nothing was balanced on its edge, so nothing tipped.

Put that beside Section 2 and the rule falls out: the risk lives in length, not in difficulty. A long answer makes hundreds of decisions and only needs one of them to be tied. A one-word answer makes a single decision, and a single decision the model is confident about is stable — which is why the same models that produced 26 different essays produced the same word 50 times running.

None of this means the systems are unusable. It means reproducibility has to come from your side of the wire, and there are four things that actually work.

Cache the output, not the request. If a result has to be stable — a benchmark answer, anything a user will see twice, anything another team will cite — store the text you got and reuse it. This is the only method on the list that gives an actual guarantee, and it costs nothing.

Stop reporting single runs. If you evaluate on one sample per item, part of your score is a coin flip. Run each item several times and report the spread with the mean. A one-point difference between two models is not a difference if re-running either one moves it by two.

Ask for less text. This is the lever the measurements actually support, and it is the one nobody reaches for. Twenty tokens came back identical 30 times out of 30; the same model writing a few hundred came back almost entirely unique. If a step in your pipeline needs to be stable, make it emit a label, a number or a small JSON object — not a paragraph that happens to contain one.

Pin the model version, and know that isn't enough. claude-haiku-4-5-20251001 is a promise about weights, not about the serving stack — kernels, hardware and batching all change underneath a fixed version string. Pinning removes the largest source of drift; it does not remove this one.