Hao Yu, Zheng Li, Dayiheng Liu, Jianwei Zhang
Replacing the naive RTN scale-selection step in NVFP4 quantization with a Hessian-weighted per-group scale search (H-Scale) improves accuracy of 4-bit quantized LLMs at zero inference-time cost.
Post-training quantization of mainstream LLMs (including Qwen3-4B-Instruct and Qwen3-30A3-Thinking) to NVFP4 on Blackwell-style micro-block formats (g=16, E2M1 values, E4M3 scales). Layer-wise validation on 36 gate_proj layers of Qwen3-4B-Instruct using 4096 FineWeb calibration samples at sequence length 8192, comparing H-Scale against Min-Max RTN, an unweighted weight-guided scale search, and a Best Scale oracle, plus end-task benchmark comparisons across NVFP4 baselines (RTN, 4over6, ArcQuant, MR-GPTQ-style pipelines).
On the layer-wise validation, H-Scale reduces normalized layer output (activation) error by 32.5% on average versus Min-Max, capturing about 80% of the oracle's 40.8% reduction, while the unweighted weight-guided search achieves only 12.6% reduction despite the lowest weight MSE. End-to-end benchmark gains over various NVFP4 baselines are described qualitatively ('generally improves', 'closer to BF16') with no headline aggregate number in the provided text. Results hold for weight-only NVFP4 post-training quantization with offline calibration; scope is Blackwell-native FP4 inference, not training or other formats.
The authors note the method targets only the final scale-selection step and leaves quantized weight codes fixed, so gains are bounded by what scale choice alone can recover; the diagonal Hessian drops cross-channel correlation terms; calibration-set dependence is acknowledged (they run calibration-size and run-to-run variability studies); the method requires modest offline calibration compute. Unstated: results shown in detail on small models (Qwen3-4B layer-wise), benchmark coverage and statistical significance of end-task gains are not evidenced in the provided excerpt, and no latency/throughput measurements of actual Blackwell kernels are shown.
A skeptic would note the headline layer-wise evidence is from one small model's gate_proj layers only, that end-to-end benchmark improvements are asserted rather than quantified in the provided text, that calibration-set choice can be tuned toward the eval set, that the comparison baselines are the authors' own pipelines on their own models, and that 'zero inference overhead' claims are untested without kernel benchmarks. The gain may also shrink as NVFP4 pipelines mature and error budgets tighten.
Modest but real: cheaper 4-bit inference with less accuracy loss lowers the cost per token of deploying open and proprietary models on Blackwell hardware over the next 12 to 24 months, reinforcing the falling inference cost curve. It does not by itself change capex, labor, or revenue economics.
Technical Capability: Quantization researchers and inference engineers get a drop-in, zero-cost accuracy refinement for FP4 deployment, showing that per-group scale choice, not just weight reconstruction, is a first-order error source.
Market & Valuation: Cheaper near-lossless FP4 inference strengthens the case that inference prices per unit of capability keep falling, pressuring vendor pricing power at the model layer.