LaMoC: Loss-Aware Modular Compression for LLMs

2608.30226 ↗ · 2026-08-31 · cs.AI, cs.CL, cs.PF

Mohanad Odema, Jacob Song

EMNLP 2026 Findings

The finding

Headline claim

Blending Empirical Fisher gradient statistics with activation statistics in modular (training-free) LLM compression reduces perplexity and improves downstream task accuracy relative to activation-only SOTA modular compression, without any fine-tuning.

The test

Compression of 8 models across 4 model families (including 4-8B Llama-family models) using LaMoC applied on top of modular compression pipelines like MoDeGPT, evaluated on WikiText-2 perplexity and downstream task benchmarks, plus latency benchmarking and ablations on calibration, aggressive compression, and gamma selection.

Effect size & scope

Small and incremental: average 2.5% reduction in perplexity and 1% relative improvement in task accuracy versus the strongest baseline (MoDeGPT) on 4-8B models; the full text provided is truncated mid-method so extreme-compression-regime results and latency numbers are not visible.

Limitations

Gains are averages over a modest model set and are small relative to baseline; only 8 models from 4 families, all apparently small-to-mid scale (1B-8B range), so nothing near frontier scale is tested; Fisher approximation is diagonal (ignores cross-channel gradient correlations); the method adds gradient computation cost per module that the paper must offset; per-module gamma selection requires estimating expected cross-entropy change, which itself depends on a first-order proxy whose Pearson correlation to true loss change (their own Figure 3) is imperfect; truncation of the provided text prevents verifying latency and aggressive-compression claims.

Counterpoint

A 1-2.5% improvement over the immediately preceding SOTA is the typical shape of an incremental compression paper: possible sensitivity to calibration data choice, cherry-pickable blending rates, and baselines limited to one or two prior methods rather than the full pruning/low-rank landscape. EMNLP Findings (not main conference) suggests the contribution was judged incremental. No frontier-scale model is evaluated, and whether gradient statistics justify their extra compute cost at scale is unproven.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months. At most this is one more data point that training-free parameter reduction keeps inching toward acceptable quality loss, which, if the trend continued at much larger compression ratios, would cheapen small-model deployment at the edge. It does not move inference cost curves or capex economics on its own.

Who this matters for

Technical Capability: Compression researchers and edge-deployment engineers get a concrete, training-free technique that adds loss-sensitivity (Fisher) information to modular compression and modestly beats activation-only baselines.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.