Why Your Model Never Answers the Same Way Twice

The Same Prompt, Two Answers

Send an LLM the identical prompt twice and you often get two different answers. Most people file this under "AI is non-deterministic" and move on. That label is true but useless — it explains nothing and tells you nothing about which knob to turn.

The non-determinism isn't spread mysteriously across the model. It enters at exactly one step: the moment the model converts its output into a single next token. Everything before that step is a fixed, repeatable matrix computation. A model given the same input computes the same raw scores every time. The variability is introduced after those scores exist, by the sampling procedure you chose — usually without choosing it.

This is a field guide to that procedure. By the end you will understand the softmax, temperature, and top-p mechanically, and — the part most explanations skip — why the two ends of the temperature dial behave the way they do. Not by analogy to "creativity," but by what actually happens to the probability distribution.

From Logits to a Distribution: The Softmax Is the Whole Game

At each step, the model produces one raw score — a logit — for every token in its vocabulary. For a 100,000-token vocabulary, that's 100,000 numbers. Logits are unbounded and uncalibrated: a token might score +14.2, another −3.7. They aren't probabilities yet.

The softmax function turns them into one. For each logit it computes an exponential, then divides by the sum of all the exponentials:

p_i = exp(z_i) / Σ_j exp(z_j)

Two properties of this step drive everything that follows:

• Every token gets a non-zero probability. Exponentials are always positive, so even absurd tokens retain a tiny sliver of probability mass. The tail is never empty — it's just small. This is exactly why an unlucky draw can produce a bizarre continuation.
• The exponential exaggerates gaps. A logit lead of a few points becomes a large probability lead after exponentiation. The top token often holds most of the mass while thousands of others split the remainder.

The output of the softmax is a full probability distribution: a ranked list of every possible next token with a probability attached. This distribution is deterministic. What you do with it is not.

Greedy vs. Sampling: The Fork Where Determinism Dies

There are two families of ways to pick a token from that distribution.

Greedy decoding takes the single highest-probability token every time — the argmax. Given identical numerics, it is repeatable: same distribution, same top token, same output. It is also rigid. It can only ever produce one continuation, and it is prone to getting stuck in loops and flat, generic phrasing because it always reaches for the safest next word.

Sampling draws a token randomly, in proportion to the probabilities. A token with probability 0.6 is chosen ~60% of the time; a token at 0.02 is chosen ~2% of the time. Run it twice and the draws differ. This is where "the same prompt gives different answers" is born — not a bug, a design choice about how to consume the distribution.

Everything interesting lives in how you sample. Temperature and truncation are the two controls that shape it.

Temperature Reshapes the Distribution — and That's the Whole "Why"

Temperature is a single number applied before the softmax. You divide every logit by the temperature T, then run the softmax on the scaled logits:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

That one division is the entire mechanism. It doesn't add randomness — it reshapes the distribution that randomness draws from. The behavior at each end follows directly from what division does to the gaps between logits.

Low temperature (T < 1): why it's focused and repeatable. Dividing by a number below 1 magnifies the differences between logits. The gap between the top token and the rest grows, so after the softmax even more of the probability mass piles onto the leading token(s). The tail is crushed toward zero. Sampling from this sharpened distribution almost always returns the top token — so outputs become consistent, on-topic, and reproducible. Push T toward 0 and the top token's probability approaches 1: sampling becomes argmax, i.e. greedy decoding. The cost of that focus is real — the model reaches for the same high-probability phrasings repeatedly, produces blander text, and is more likely to loop, because you've told it to distrust every alternative.

High temperature (T > 1): why it's diverse and creative. Dividing by a number above 1 compresses the differences between logits. The gaps shrink, the distribution flattens toward uniform, and probability mass flows from the leading token out into the tail. Now the second-, tenth-, and hundredth-ranked tokens have a genuine chance of being drawn. That's what produces variety and surprising word choices. The cost is the mirror image of the benefit: as the tail rises, so does the chance of drawing a token that's grammatically fine but semantically wrong, and outputs drift, lose coherence, and hallucinate more. Past roughly T = 1.3–1.5, most models begin to derail.

The dial is not a "creativity slider." It's a control on the entropy of the distribution you sample from. Low temperature = low entropy = predictable. High temperature = high entropy = varied. Every downstream behavior is a consequence of that.

”

"The higher the temperature, the less likely it is that the model is going to pick the most obvious value (the value with the highest logit), making the model's outputs more creative but potentially less coherent. The lower the temperature, the more likely it is that the model is going to pick the most obvious value, making the model's output more consistent but potentially more boring." — Chip Huyen, AI Engineering (p. 193)

Truncation: Top-k, Top-p, and Min-p Keep the Tail From Poisoning the Draw

There's a problem with raising temperature alone. Flattening the distribution lifts thousands of junk tokens off the floor simultaneously. Individually each is unlikely, but collectively their probability mass is large — so at high temperature you draw genuine garbage more often than the good tail justifies. Truncation fixes this by cutting the tail off before sampling.

The three common truncation methods differ in how they decide where to cut:

• Top-k keeps the k highest-probability tokens (e.g. the top 40) and zeroes out the rest. Simple, but blind to shape: it keeps 40 tokens whether the model is highly confident or wildly unsure.
• Top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to p (e.g. 0.9), then renormalizes. It adapts to the distribution: when the model is confident, the nucleus is tiny; when it's uncertain, the nucleus widens. This is why top-p is the more common default.
• Min-p keeps tokens whose probability is at least min_p times the top token's probability. It scales the cutoff to the model's confidence in a slightly different way and has gained traction for staying coherent at higher temperatures.

Why Temperature 0 Still Isn't Deterministic

A natural conclusion: set temperature to 0, get greedy decoding, get identical outputs forever. In practice, "T = 0" gets you mostly reproducible, not guaranteed reproducible — and the reasons are instructive.

”

"It's common practice to set the temperature to 0 for the model's outputs to be more consistent. Technically, temperature can never be 0—logits can't be divided by 0. In practice, when we set the temperature to 0, the model just picks the token with the largest logit, without doing logit adjustment and softmax calculation." — Chip Huyen, AI Engineering (p. 194)

• Floating-point addition isn't associative. (a + b) + c can differ from a + (b + c) in the last bits. GPUs sum the logits and the softmax denominator in parallel, in an order that can vary run to run. Those last-bit differences occasionally flip which token is the argmax when the top two are nearly tied.
• Batching changes the numerics. Your request is processed alongside whatever else is in the batch. Different batch composition means different kernel shapes and reduction orders — so the same prompt can produce subtly different logits depending on who else is being served at that instant.
• Mixture-of-Experts routing can be batch-dependent. In MoE models, which experts a token is routed to can depend on the composition of the batch, introducing another source of run-to-run variation invisible to you.
• Some GPU kernels are non-deterministic by default, using atomic operations whose accumulation order isn't fixed.

Precision Feeds the Distribution, Too

Everything above assumes the logits themselves are fixed. Model quantization changes that. Running the weights — or the KV cache — at reduced precision (int8, int4) perturbs every logit slightly before the softmax ever sees it. The perturbation is systematic, not random: a given quantized model produces the same shifted distribution every time. But it is a different distribution than the full-precision model would have produced.

Where this bites is exactly where temperature already made things fragile: the near-ties. When the top two tokens sit close together — routine in long, precise outputs like code — a small quantization nudge to their logits can flip which one wins. One flipped token early in a long derivation changes everything after it. That is the mechanical reason aggressive KV-cache quantization degrades reasoning-heavy generation, and why the damage concentrates on the long-context work that leans hardest on precise recall.

Choosing the Knob on Purpose

Most people inherit whatever temperature the SDK defaults to and never touch it. That's a mistake in both directions: too high for tasks with one correct answer, too low for tasks that need range. Match the setting to the shape of the task.

The setting also has an evaluation consequence people miss: temperature sets the noise floor your measurements have to clear. At T = 0.8, the same prompt scored twice can disagree purely from sampling variance. If your eval runs each task once, you're reading noise as signal. This is why serious evaluation samples each task multiple times and reports the spread — the discipline covered in how to know if your agent actually works and the broader probabilistic stack. It's also why a higher temperature widens the gap between a confident answer and a confident hallucination: you've given the low-probability-but-wrong tokens more room to be drawn.

The Dial Is a Tradeoff, Not a Personality

"Non-deterministic" was never a good explanation. The precise version: a language model emits a probability distribution over tokens, and sampling collapses that distribution into text. Temperature reshapes the distribution before the collapse — sharpening it concentrates mass on the likeliest tokens (focused, repeatable, occasionally stuck), flattening it lifts the tail (varied, creative, occasionally incoherent). Truncation decides how much of the tail is even eligible. And temperature 0 buys you deliberate determinism, not freedom from the incidental non-determinism of parallel floating-point hardware.

None of this is randomness for its own sake. It's a controllable bias/variance tradeoff over the next-token distribution — and once you can see the distribution, the dial stops being mysterious and starts being a tool.

See also: The Probabilistic Stack for engineering patterns around non-determinism, KV-Cache Quantization for how run-to-run variance breaks naive A/B tests, and How to Know If Your AI Agent Actually Works for measuring through the noise.