Why Does Post-Training Quantization Work?

2609.11716 ↗ · 2026-09-10 · cs.LG, cs.CL

Yuxiang Chen, Michael Beyer, Jun Zhu, Jianfei Chen

45 pages, 26 figures, including appendices

The finding

Headline claim

Post-training 4-bit quantization works because pretrained models acquire two mechanisms: layer-introduced error partially cancels inherited error (counteraction), and the LM head's geometry preferentially preserves top-token scores, so large internal errors yield small output changes.

The test

Block-by-block comparison of full-precision vs NVFP4-quantized (RTN, weight-only) forward passes on the same inputs, across Qwen3-32B (64 layers), OLMo3-7B, Gemma3, and OLMoE, on C4 and six zero-shot benchmarks, with theoretical decompositions (error recurrence theorems) verified experimentally.

Effect size & scope

Quantized Qwen3-32B loses only 0.43 percentage points average across six zero-shot benchmarks; final relative hidden error ~0.15 with cosine ~0.98; counteraction cancels 50.2% of accumulated block-update error over layers 1-48, versus at most 0.16% per layer at random initialization; random-init models show 5.5x larger final absolute hidden error (6.7x relative) for Qwen3-32B and 20.2x (3.7x relative) for OLMo3-7B despite matched weight reconstruction error (~0.9955 cosine). Findings hold across model families and PTQ settings per the authors.

Limitations

Authors note a limitations section and rare failures in hidden-error propagation (abnormal trajectories filtered); analysis covers weight-only RTN NVFP4 primarily (calibrated algorithms and INT4 in appendices), LM head kept full precision, evaluation on C4 and standard benchmarks rather than long-horizon agentic workloads, and counteraction is descriptive of correlation across layers, not proven to be a training-induced adaptation causally.

Counterpoint

The 5.5x gap could partly reflect activation-norm scale differences between random and pretrained models rather than a learned mechanism; results shown mainly on strong dense and MoE families with no small or very old architectures; benchmark accuracy averaging over six zero-shot tasks may hide task-specific degradation; mechanism identification relies on aggregate cosine/inner-product statistics that could conflate normalization effects with true error cancellation.

Economy implication

If quantization robustness is a general property of pretrained transformers, low-precision inference (including 4-bit hardware formats) can serve models with minimal quality loss, supporting falling inference cost per unit of capability and cheaper local deployment over the next 12 to 24 months; this paper itself demonstrates the mechanism but does not measure costs, so the economic reading is indirect.

Who this matters for

Technical Capability: Explains mechanistically why 4-bit serving barely degrades accuracy, which matters for anyone compressing models or designing low-precision inference kernels.

Market & Valuation: If cheap quantized inference is near-lossless, it erodes any pricing premium for full-precision API serving and reinforces inference cost deflation.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.