Compile, Don't Memorize: A Context Compilation Architecture (CCA) for In-Context Learning

2609.00759 ↗ · 2026-09-01 · cs.CL

Jinhu Qi, Minda Hu, Wentao Zhang, Weiqiang Jin, Yanyu Chen, Junli Wang, Irwin King

Accepted to EMNLP 2026 (Findings). Code, data, and cached completions available at https://github.com/TonyQJH/cca-emnlp2026

The finding

Headline claim

Compiling a long context once into a typed JSON intermediate representation with generated Python verifiers and a gated correction loop materially raises pass rates on rubric-graded in-context learning tasks versus vanilla prompting and two long-context baselines.

The test

CL-bench: 1,899 rubric-graded tasks (4 domains, 18 sub-categories), contexts with median 20K characters (max 247K), across 4 open base models, compared against vanilla prompting, ReadAgent-P, and Ctx2Skill, with ablations over pipeline components F1-F7 and McNemar paired significance tests.

Effect size & scope

Vanilla open-weight models pass only 12-16% of tasks; CCA lifts Kimi K2.5 from 15.4% to 21.4% (about 6 points absolute, roughly 40% relative), with gains reported on every base model and concentrated on rule-dense sub-categories. The lift is large in relative terms but absolute pass rates remain low, and the benefit is domain-selective, holding mainly on rule-dense open-ended tasks.

Limitations

Authors concede the harness is domain-selective on open-ended long-context tasks and that scoring uses a single judge (no multi-judge robustness check). They also show cost is nontrivial: roughly 34K tokens per task budget, with the correction loop firing only on some variants. Unstated: results are limited to open base models (no frontier closed models), the benchmark explicitly grades counterfactual/rubric-heavy contexts that may overfit to a rule-extraction design, and prompt-engineering effort (the IR schema and stage prompts) is a confound versus baselines ported by the authors.

Counterpoint

A skeptic would note the benchmark rewards exactly the artifact CCA builds (rubrics with must/never rules that map cleanly onto codeable checks), so part of the gain could be benchmark-alignment rather than general capability; baselines were re-implemented by the authors (porting fairness argued in Appendix E but still author-controlled); single-judge grading is prompt-sensitive; and per-task gains of a few points on a specialist benchmark with ~34K token overhead may not survive contact with messier production contexts where rules are implicit rather than enumerated.

Economy implication

Little direct economic evidence yet: this is a methods paper showing that harness engineering, not model scale, is currently the binding constraint on long-context compliance tasks. If the pattern generalizes, it implies value is migrating to the orchestration/verification layer around frozen models rather than to the model itself, which modestly supports app-layer margin durability and model commoditization over the next 12 to 24 months, but the paper itself measures none of that.

Who this matters for

Technical Capability: Shows that explicit typed representations plus executable verifiers beat raw forward-pass reasoning on rule-dense long-context tasks, a concrete datapoint for the claim that reliability comes from the harness, not bigger models.

Labor & Knowledge Work: If context-compilation harnesses reliably close the gap between strong and reliable model behavior, fewer human review/editors are needed for long-context compliance work, an early mechanistic hint for the agentic-reliability-to-labor channel.

Bears on (advisory · papers never write evidence)

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

Large language models (LLMs) increasingly handle in-context learning (ICL) tasks where a long, novel context defines the rules, knowledge, and output schema for a series of questions. On benchmarks that grade against every detail of the context, even strong open-weights models pass only 12-16% of tasks: a single overlooked rule fails the whole response. We argue this brittleness is structural: the dominant "read-and-reason" paradigm asks the model to extract, plan, generate, and self-verify in one forward pass. We therefore ask whether explicit context compilation can fix it, how it compares to existing long-context strategies (gist retrieval, multi-agent self-play), and where the resulting harness benefit holds across task structure and model scale. We propose the Context Compilation Architecture (CCA), whose central novelty is a typed intermediate representation (IR) with fixed slots (rules.{must_do, must_not, conditional}, output_spec, available_tools, data_profile) into which any prose context is compiled once; executable verifiers and a violation-gated correction loop follow as downstream consequences. On CL-bench (1,899 tasks across 4 open base models), CCA outperforms vanilla prompting and two long-context baselines (ReadAgent-P, Ctx2Skill) on every base model, lifting Kimi K2.5 from 15.4% to 21.4% with gains concentrated on rule-dense sub-categories. Code and cached completions are available at https://github.com/TonyQJH/cca-emnlp2026.