Stable Answers, Unfinished Reasoning: Why Self-Consensus Is Not a Safe Early-Exit Signal

2609.09989 ↗ · 2026-09-09 · cs.CL

Yunxiang Mo, Donghao Zhao, Hejia Geng

21 pages, 9 figures, 10 tables. Yunxiang Mo and Donghao Zhao contributed equally. Code and data will be released at https://github.com/Antony-zdh/stable-answers-unfinished-reasoning

The finding

Headline claim

Self-consensus early exit (stopping a reasoning trajectory once repeated probes agree on an answer) is unsafe and cannot be made both safe and token-saving: repeated agreement reflects a probe-elicited placeholder, not reasoning termination, while a non-consensus confidence signal (DEER) clears the same safety gates.

The test

A preregistered sweep of 3,520 windowed-consensus rules replayed offline on frozen trajectories from two models (DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B) on MATH 500, AMC23 and AIME24 across three seeds (18 environments), with three acceptance gates fixed in advance and problem-grouped 60/20/20 splits; confirmation on unseen models (Llama-8B, Qwen-32B distills) at 4x scale, plus hand-labelling of 134 wrong stops and paired re-probes with two probe wordings.

Effect size & scope

No rule of 3,520 clears any gate; 478 clear in-sample on train only, so single-split studies could conclude otherwise. First actionable consensus commits answers correct only 50.5% vs 90.7% run to completion (40.2pp loss) on MATH 500, and 27.5% vs 85.2% across 18 environments when fired early. At 32% token saving, one stop in nine commits an answer the trajectory later abandons (155 of 216 pre-empt corrections). Re-probing with different wording changes the answer 54% early vs 16% late. Widening windows levels non-terminal stops near 7% but savings collapse to 8%. At matched savings, consensus costs 6.0pp (28% saving) and 8.3pp (33%) vs 0.5pp and 2.0pp for DEER; DEER clears all gates, losing as little as 0.3pp at 28% saving. Scope: small math benchmarks, 7-32B-class reasoning models, one main trajectory per problem.

Limitations

Authors concede: scope limited to consensus used alone (agreement gated behind reasoning-level signals may work, per PUMA); DEER comparison limited to their pipeline; small held-out sets on hard benchmarks; savings axis depends on probe-tax accounting conventions; domain limited to math reasoning. Unstated: single trajectory per problem per seed limits generality; probe prefill tokens uncharged; findings may not transfer to non-math, non-verifiable domains or larger frontier models; the paper is a short-context cut and full text (appendices, generalization section) not fully verifiable here.

Counterpoint

A skeptic could note the negative result is on small math benchmarks with small distill models where trajectories self-correct often; on easier or non-self-correcting settings consensus early exit may be fine, and prior work (Dynasor/CertaIndex, Liu and Wang 2025) reported acceptable aggregate accuracy at large savings, suggesting the strict 1.0pp gate sets an unusually high bar. The conclusion that no consensus rule works anywhere exceeds what 3,520 rules on two small models and three math benchmarks can establish; the finding may be 'consensus fails here under a strict gate', not 'consensus is unsafe'.

Economy implication

Little or nothing yet for the AI economy directly; it bears modestly on inference-cost reduction efforts by warning that one popular cheap early-exit heuristic locks in wrong answers, and that safer confidence-based exits exist. It does not itself move cost curves or enterprise ROI in the next 12 to 24 months.

Who this matters for

Technical Capability: Researchers and engineers building reasoning-inference cost controls learn that answer-agreement early exit measures the wrong object and that boundary-confidence signals like DEER are a safer basis for early exit.

Market & Valuation: Vendors selling token-reduction or inference-optimization features built on consensus-style stopping should note that the technique can silently lock in wrong answers, a correctness liability in pricing efficiency claims.

Bears on (advisory · papers never write evidence)

[1.4] Inference-time compute yields gains that don't saturate quickly on hard reasoning.

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

A natural way to cut reasoning-model inference cost is to repeatedly probe a single partial trajectory for its current answer and stop once probes agree -- self-consensus. We ask whether any such rule is both safe and token-saving, and whether one can be selected once and reused. A preregistered sweep of 3,520 consensus rules, replayed on frozen trajectories from two models and three benchmarks, clears none of three acceptance gates fixed in advance; the frontier reproduces on a held-out split and on two unseen models -- while a boundary-confidence control (DEER) swept through the same pipeline clears all three. The reason lies in the signal: agreement establishes that the current answer persists under a fixed probing procedure, not that the reasoning has terminated -- a consensus-termination gap. Stopping on it commits non-terminal answers. At a rule still saving 32% of the tokens, one stop in nine fires on an answer the trajectory itself later abandons, and most of those stops cut off a correction it would otherwise have made. Widening the agreement window does not remove them: the share levels off near 7%, and by then the saving has fallen to 8%. Probe re-wording and a hand-labelled error taxonomy show the agreed answer is often a placeholder the model had not settled on. Used on its own as the stop signal, agreement fails not because it is insufficiently strict, but because it repeatedly measures the wrong object.