When the Martingale Never Stops Firing: Anytime-Valid Gating on Real Forecast Streams

2608.30502 ↗ · 2026-08-31 · cs.LG, stat.ME, stat.ML

Weijia Han, Lisha Qu

4 pages, 2 figures

The finding

Headline claim

Anytime-valid martingale monitors deployed on real dependent forecast streams lose their false-alarm guarantee: the same gate that fires in at most 1 of 60 exchangeable synthetic runs fired in 135 of 135 clean real-stream runs at alpha=0.05, and the fire-triggered drift response actively amplified the very transient it was meant to catch.

The test

A pre-specified, version-stamped case study: a conformal test martingale gating the online Kalman adapter updates of three frozen time-series foundation models (TiRex, Chronos-2, TimesFM 2.5) across five forecasting datasets, three seeds, clean and contaminated (spike, level-shift, stuck) label streams; with null-calibration controls of the deployed construction on exchangeable synthetic score streams at T=20,000 and 52,117, 60 runs per condition.

Effect size & scope

On exchangeable inputs the deployed construction fired 0 to 1 of 60 runs; on clean real streams all 135 runs fired (expected near 7 under Ville's bound at alpha=0.05). On the weather stream the gated filter amplified a natural transient: per-step MSE 1.155e5 versus 9.8e3 ungated at the event (TimesFM 2.5), and clean-stream divergence in 6 of 6 TiRex/Chronos-2 runs. The Huber-only robust-update half cut isolated-spike degradation by roughly an order of magnitude (e.g., ETTh2 +0.15 versus +3.32 ungated) with no validity claim. Sustained drift (level shift, stuck injectors) defeated all gated arms, with gated often worst.

Limitations

The authors state all claims are about a single gate configuration on five datasets with cached deterministic replay and an optimistic immediate-reveal protocol whose overlapping labels mechanically induce serial dependence; 60-run null controls cannot certify strict conservatism; the synthetic pipeline control does not isolate which loop element breaks exchangeability; Ljung-Box diagnostics are inconclusive under heteroskedasticity; survivor-only cells where runs diverge are read only qualitatively; e-detector swaps did not repair firing.

Counterpoint

This is one deployment and one gate configuration, n=5 datasets from the same benchmark families, so the premise-failure rate on other streams and other score types is unmeasured; the null controls use synthetic i.i.d. scores, not the deployed stream with the loop broken in controlled ways; the catastrophic amplification traces to an identifiable configuration incoherence (forgetting factor pinned below the drift value plus a soft reset), suggesting an engineering bug more than a fundamental indictment of martingale gating; and the headline framing (anytime-validity fails) partly restates the known fact that exchangeability assumptions do not hold on non-exchangeable streams.

Economy implication

Little or nothing yet for the economics of AI. The narrow operational message is that statistically-guaranteed monitors used to gate deployed adaptive systems may carry their nominal error rates only under assumptions real streams violate, a warning for teams automating intervention decisions, but no labor, cost, or revenue effect is established.

Who this matters for

Technical Capability: Engineers deploying self-monitoring or self-correcting online ML systems should care that a monitor's theoretical guarantee did not transfer to the real score stream, and that the monitor-triggered response made outcomes an order of magnitude worse than no gate at all.

Abstract

Machine learning systems are increasingly corrected while they run, and the decision of when to intervene is increasingly delegated to statistical monitors. Anytime-valid inference promises evidence that can be acted on at any moment, exactly the guarantee this setting needs, and it is moving from theory into deployed monitoring. Conformal test martingales are the change-detection instrument, and Ville's inequality caps their false-alarm probability on exchangeable data. The guarantee is conditional. A deployment inherits it only if the stream it monitors behaves exchangeably. The premise is hardest to satisfy where these monitors are most useful, on dependent data and inside loops where the monitor modifies the learner whose scores it reads. It is also rarely measured. We measure it in a pre-specified case study, where such a monitor gates the online updates of a Kalman adapter correcting frozen time-series foundation models on five forecasting streams. On exchangeable synthetic streams, the same implementation fires in at most 1 of 60 runs. On the real streams, at alpha = 0.05, 135 of 135 clean-stream runs fired. The construction does not explain the firing; the failure comes from the deployed score stream itself. Repeated fires hold the gate's drift response active, and the gated filter amplifies the very transient it was designed to prevent. The component worth keeping makes no validity claim. Huber-style gating of the filter's own updates cuts isolated-spike degradation by an order of magnitude with no dataset specific tuning. Anytime-valid methods proposed for dependent data should therefore be accompanied by null-calibration controls and mechanism traces.