KV-Cache Quantization and Model Quality: What's Real and What's Overclaimed

The Claim Bundles Two Very Different Assertions

Here is a claim someone on your team might plausibly make after a week of running local inference for coding:

”

"I had no idea how much quantizing the KV cache affects model quality. From now on I'm running my coding models without quantization. Especially at 27B — the difference is one-shot vs two-shot in most cases. Way better thinker too — produces better outputs overall."

It is a useful claim to dissect because it bundles two assertions that deserve very different levels of trust. One is a claim about a mechanism: does reduced-precision KV cache have a quality cost? The other is a claim about effect size and generality: how big, how often, how broadly? The first is defensible. The second is where careful teams get burned.

The Mechanism Is Real: Reduced Precision Can Corrupt Recall

The KV cache stores the attention keys and values for tokens already generated. It exists only at inference, and its cost is exactly why anyone quantizes it in the first place.

”

"A KV cache is used only during inference, not training… Because generating a token requires computing the attention scores with all previous tokens, the number of attention computations grows exponentially with sequence length. The KV cache size, on the other hand, grows linear[ly]…" — Chip Huyen, AI Engineering (p. 822)

That linear-in-context growth is what makes the cache the memory bottleneck for long inputs — and the natural thing to compress.

Compression by precision reduction is not free. Huyen is explicit that the failure is not only gentle rounding — the sharp edge is outliers:

”

"There are downsides to reduced precision. Each conversion often causes a small value change, and many small changes can cause a big performance change… How to reduce precision with minimal impact on model performance is an active area of research." — Chip Huyen, AI Engineering (p. 636)

”

"If a value is outside the range the reduced precision format can represent, it might be converted to infinity or an arbitrary value, causing the model's quality to further degrade." — Chip Huyen, AI Engineering (p. 636)

Why this bites coding specifically: code generation and long reasoning chains depend on precise, long-range recall — the exact variable name defined 500 tokens back, the open brace, the import. That recall is what the KV cache holds. Corrupt a few high-magnitude entries and attention can land on a similar-but-wrong prior token. In a long derivation, one wrong step is the whole answer. The direction of the claim — "un-quantizing the cache can help a coding model" — has a real mechanism behind it.

But keep the mechanism honest in both directions. Quantization is usually deployed precisely because it is often nearly lossless:

”

"Weight-only quantization is by far the most popular approach since it's easy to use, works out of the box for many models, and is extremely effective." — Chip Huyen, AI Engineering (p. 719)

"Can degrade" is not "does degrade a lot, everywhere." How much depends on what you quantize and how far. That distinction is the whole ballgame — and it is exactly what a felt impression cannot measure. This is the probabilistic stack in miniature: a non-deterministic system where the effect of a config change is a distribution, not a data point.

Where the Claim Outruns Its Evidence

"One-shot vs two-shot in most cases" and "a better thinker overall" are not mechanism claims. They are inferences about how big the effect is and how widely it holds, drawn from lived observation. This is precisely the terrain the decision-science canon exists to police — and it names the exact moves at work.

The law of small numbers. The intuition that a few runs faithfully mirror the true difference is a documented error, not a shortcut:

”

"…a lingering belief in what may be called the 'law of small numbers,' according to which even small samples are highly representative of the populations from which they are drawn." — Daniel Kahneman, Thinking, Fast and Slow

The precise tell. The sharpest line describes the observer who mistakes noise for signal — and the giveaway is that they find a cause for every wobble:

”

"He rarely attributes a deviation of results from expectations to sampling variability, because he finds a causal 'explanation' for any discrepancy. Thus, he has little opportunity to recognize sampling variation in action." — Judgment Under Uncertainty: Heuristics and Biases (p. 42)

Why the confidence feels earned. "Way better thinker… overall" feels solid from the inside because subjective confidence tracks the coherence of the story, not the amount of evidence:

”

"The confidence that individuals have in their beliefs depends mostly on the quality of the story they can tell about what they see, even if they see little." — Daniel Kahneman, Thinking, Fast and Slow

Capitalization on chance. And the shape of the reasoning — a fixed suspect, evidence sifted until it fits — is itself a named anti-pattern:

”

"…the data are fixed and undeniable, while the set of possible explanations is relatively unbounded; one hunts until one finds an explanation that fits." — Judgment Under Uncertainty: Heuristics and Biases (p. 370)

None of this says the observation is wrong. It says the leap from the observation to "in most cases" and "overall" is not licensed by what was actually seen. Distinguishing signal from run-to-run noise is the entire discipline of knowing whether your system actually works.

The Confounds: What Else Explains "One-Shot vs Two-Shot"

Before the KV cache gets the credit, these have to be off the table. Each is a live alternative:

Confounding is not a rhetorical dodge — removing it is the entire job of controlled evaluation:

”

"The process of computing adjusted treatment effects is called adjusting (or controlling) for confounding. The selection of what confounders Z to measure and control for requires scientific insight." — Larry Wasserman, All of Statistics (§ 16.3)

Don't Discard the Hunch — Promote It

The correct response is not "so you saw nothing." Dismissing the observation is its own error. Hubbard's whole thesis is that people systematically undervalue small samples taken from a state of near-ignorance:

”

"…small samples can be informative, especially when you start from a position of minimal information. In fact, mathematically speaking, when you know almost nothing, almost anything will tell you something. Kahneman and Tversky showed that the error of a sample is often overestimated, which results in an underestimation of the value of a sample." — Douglas Hubbard, How to Measure Anything

The apparent contradiction — Kahneman says don't trust small samples, Hubbard says small samples inform — resolves in one move: a measurement is uncertainty reduction, not elimination.

”

"For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty. A mere reduction, not necessarily elimination, of uncertainty will suffice for a measurement." — Douglas Hubbard, How to Measure Anything

The Afternoon That Settles It

The gap between "I noticed something" and "un-quantized KV cache is worth the extra VRAM for our coding work" is closed by one small, boring experiment. The canon supplies the skeleton:

”

"The gold standard for measuring the impact of a product is to employ randomized controlled experiments, otherwise known as A/B tests or split tests." — Stephen Wendel, Designing for Behavior Change

”

"Every experiment involves a sequence of activities: 1. Conjecture… 2. Experiment… 3. Analysis… 4. Conclusion…" — Montgomery & Runger, Applied Statistics and Probability for Engineers (p. 563)

A concrete recipe for this exact claim:

Run that, and the claim graduates from plausible anecdote to "int4 cache costs us X points of pass@1 on our suite; int8 costs ~0" — a number you can provision hardware against, scoped to your model and your workload, which is the only scope the data supports. That is the same move that separates hobbyist from production in building agent evals: the metric is defined before the run, not narrated after it.

Bottom Line

• The physics is on your side. Reduced-precision KV cache genuinely can degrade output, most sharply at aggressive quantization on long, reasoning-heavy context — exactly coding's profile. Turning it off can help. As context windows grow, the cache is the first thing under memory pressure, which is why the temptation to compress it is permanent.
• The generalization is ahead of the data. "One-shot vs two-shot in most cases" and "a better thinker overall" are effect-size and scope claims a small, unblinded, un-A/B'd sample cannot carry. The tell — a causal story attached to every discrepancy — is textbook law-of-small-numbers.
• Don't throw the hunch away — measure it. A small sample from near-ignorance is worth a lot. It buys a hypothesis, not a law.
• Next move: the A/B above. It is an afternoon, and it turns "27B feels way better un-quantized" into a defensible number.

This is not a KV-cache problem. It is the general shape of every "the model got smarter when I flipped a setting" claim in a non-deterministic system — and the discipline of the third era of coding is treating those claims as experiments, not conclusions.

Scope note: this piece deliberately stays inside primary-research grounding. Model-specific KV-quant benchmarks and the current low-bit-cache literature (K-vs-V asymmetry, FP8 calibration, and so on) live outside this note's evidentiary standard; cite the primary sources directly before standardizing on any of it.