Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
Off-the-shelf language models can, via a zero-shot prompting protocol (Declarative Attention), declare which context regions they need to attend to, letting the inference engine skip most KV-cache reads during decoding with only modest accuracy loss.
Zero-shot evaluation of two mid-size models (Gemma-4-31B, Qwen-3.6-27B) on 15 long-context tasks with contexts on the order of 100K tokens, using a custom vLLM integration with block-aligned masking compatible with FlashAttention, plus a roofline-based wall-time analysis rather than measured end-to-end production latency.
DA reduces total attended tokens during decoding by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, with accuracy drops of 1.27pp and 2.75pp that shrink with model scale; projected decode wall-clock cost falls to 0.71x and 0.77x of vanilla; token savings grow with context length (up to 21M tokens saved per response); the dynamic mask cuts attended tokens up to 71.1% vs the maskless ablation. Results are limited to two models, static benchmark contexts (not real agentic traces), and global-attention layers only.
Authors concede: zero-shot is a lower bound with headroom via post-training; DA trades more decode steps for lower per-step cost, so savings depend on deployment regime (only favorable in large-batch, well-utilized serving); wall-time figures are projected roofline estimates, not measured production latencies; protocol adherence is imperfect and analyzed as a failure mode; DA leaves efficient layers (SWA, GDN) untouched; segmentation is heuristic. Unstated: prompt sensitivity of the protocol, benchmark tasks may not reflect agentic workloads where interleaved tool calls change attention patterns, and no end-to-end system benchmark at production batch sizes.
A skeptic would note the wall-time gains are analytical projections, not measured serving benchmarks; the protocol requires heavy prompt engineering whose gains may not transfer to unseen tasks or different prompt formats; only two models are tested at moderate scale; accuracy drops on long-context benchmarks may hide selective degradation on harder subsets; and the approach competes with already-deployed proxy-score sparse attention methods whose O(N) scan cost is shrinking, narrowing DA's advantage.
In the next 12 to 24 months this suggests a mechanism-level lever for cutting long-context inference costs, relevant if long-context and agentic workloads dominate serving spend, but the evidence here is a projection from roofline math on two models, so it signals a direction rather than a deployable saving; little or nothing yet can be concluded about actual serving economics.
Technical Capability: Researchers and inference engineers get evidence that attention sparsity can be elicited from the model itself rather than approximated externally, a new axis of sparse attention with positive scaling in model capability.
Market & Valuation: Hyperscalers and inference providers watching long-context serving costs should track whether model-controlled sparsity compounds with existing KV-cache techniques to shift the decode cost curve at scale.