CurveShift: Is Agent Progress Scalar? Separating Level from Shape

2608.00355 ↗ · 2026-07-31 · cs.CL, cs.LG · h-max 111 · 1 citations

Hanwen Xing, Pengyun Wang, BingXu Meng, Kumail Alhamoud, Xiang Li, Jicheng Wang, Xin Yu, Xinyang Han, Xiaomin Li, Philip Torr, Yuexing Hao

25 pages, 4 figures, 7 tables. Data and code: https://github.com/harvenstar/CurveShift

The finding

Headline claim

Most of the apparent shift in AI model gains toward harder tasks is a measurement artifact of overall rising ability, not a genuine change in the difficulty-response curve, but a small real hard-task effect of about +0.40 logits does survive after controlling for this artifact on a no-scaffold coding benchmark.

The test

Item response theory (Rasch and era-differential item functioning models) applied to three datasets: METR time-horizon agentic data (~33 models), SWE-bench Verified (134 model submissions on 500 tasks), and LiveCodeBench (66 dated models across 1,055 competitive programming problems labeled easy, medium, and hard by human contest organizers). The key identifying benchmark is LiveCodeBench, which has no agentic scaffold and uses exogenous human-assigned difficulty. A post-September 2024 era indicator is used to test whether hard-item solve rates exceed what overall easy and medium performance would predict.

Effect size & scope

On LiveCodeBench, models released after September 2024 show a hard-item era effect of approximately +0.40 logits under the most conservative assumption (hard items no more discriminating than easy or medium items), corresponding to a rise in hard-problem solve rate from roughly 18% to 25%. On METR and SWE-bench, model era and scaffold era are collinear, so the shape effect cannot be identified there. The hard-task gain on LiveCodeBench holds across 11 model families and is led by the strongest reasoning models, but it is scoped to tasks requiring short reasoning paths, not long-horizon autonomous agent tasks, and to competitive programming specifically.

Limitations

Authors explicitly note: (1) the clean identification rests on a single coding benchmark, so the result is specific to competitive programming and cannot be generalized to agentic or long-horizon tasks; (2) the reasoning era indicator bundles multiple concurrent changes (training regime, inference-time compute, data curation), so reasoning training per se is not isolated as a causal factor; (3) post-era models are mostly sampled once per problem (n=1) versus 10 times for pre-era models, introducing differential noise correlated with era; (4) OpenAI is the largest model family (18 of 66 models), creating sensitivity to family clustering; (5) math domain analysis failed on preconditions (Appendix F). Unstated: results may not hold for modalities beyond code, and the September 2024 breakpoint is somewhat arbitrary.

Counterpoint

A fair skeptic would note that the entire positive finding rests on a single benchmark (LiveCodeBench), which is itself a competitive programming leaderboard potentially subject to targeted optimization by post-era model developers. The +0.40 logit effect is modest and estimated under a pinned-discrimination sensitivity analysis rather than a freely identified model (the authors concede the free 2PL is not stably identified on these data). The 18%-to-25% solve rate shift, while statistically controlled, is small in absolute terms. The paper's primary contribution may be the deflation result (most claimed hard-task gains are artifacts), which is actually a negative finding about the reliability of existing capability claims rather than evidence of genuine qualitative progress.

Economy implication

In the next 12 to 24 months, this finding has modest but directional economic relevance: it suggests that aggregate benchmark scores and scalar capability metrics used to justify model pricing, procurement decisions, and capability forecasts are partially misleading, overstating qualitative progress on hard tasks. The residual genuine hard-task gain, confined to competitive programming, does not yet license broad claims about autonomous agent capability that would underpin enterprise deployment at scale. Little or nothing in the near term changes because of this paper alone, but it provides methodological grounds for skepticism toward vendor capability claims that rely on scalar leaderboard improvements.

Who this matters for

Technical Capability: Researchers and evaluators building capability forecasts or deciding which benchmarks to trust should care: the paper shows that standard scalar summaries conflate level and shape, and that scaffold confounding makes agentic benchmarks unable to attribute gains to models versus harnesses.

Market & Valuation: Investors and enterprise buyers using benchmark trajectories to assess capability acceleration should note that most of the apparent hard-task gain is a measurement artifact, which complicates valuations premised on qualitative capability shifts.

Bears on (advisory · papers never write evidence)

[1.1] Frontier benchmark scores kept rising gen-over-gen through 2025–26.

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score. These summaries capture the overall performance, but they do not test whether progress is distributed differently across task difficulty. We find that most of the apparent shift in gains toward harder tasks does not reflect a change in the shape of the difficulty-response curve. On METR time-horizon data, a single Rasch model with rising ability reproduces this pattern, so it is largely explained by ceiling effects rather than a qualitative change in capability. This echoes how the choice of metric can make claimed emergent abilities look like a property of the models themselves. We then identify a smaller hard-task effect that survives this control. Isolating it is difficult on agentic benchmarks, because newer models are usually run with newer agentic harnesses, so a gain on hard tasks cannot be assigned to the model or its scaffold. We break the confound with LiveCodeBench, a public competitive programming benchmark that runs no agentic scaffold while pairing dated models with an exogenous difficulty ordering. After accounting for the rise in overall ability, models released after September 2024 still gain on the hardest problems beyond what their easy and medium performance predicts, by about +0.40 logits under our most conservative assumption, raising the hard-problem solve rate from roughly 18% to 25%. The effect is led by the strongest reasoning models and holds for hard tasks that need only short reasoning, not autonomy over long horizons. We present this as a result specific to competitive programming, since our clean identification rests on a single coding benchmark. We release the LiveCodeBench Difficulty Panel (66 dated models x 1,055 problems) and our analysis code.