A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

2609.10357 ↗ · 2026-09-09 · cs.LG

Mahdi Naser Moghadasi, Faezeh Ghaderi

11 pages, 2 figures, 5 tables. Code, data fetchers and per-series results: https://github.com/mahdinaser/tsfm-bench

The finding

Headline claim

Even when the test window postdates every pretrained time-series model, foundation-model gains concentrate in domains resembling their pretraining corpora: Wikipedia pageviews account for the largest advantage, and elsewhere classical baselines tie or beat the pretrained models.

The test

Thirteen forecasters (four classical, three per-dataset, six pretrained checkpoints) across seven groups in five domains, all scored on observations after 1 January 2026 with no API-key-gated data; roughly 2,800 series total (500 each daily/weekly/monthly Wikipedia, 300 weather, 379 air quality, 46 electricity, 29 exchange rates), compared by MASE/sMAPE, average rank with Nemenyi intervals, and Holm-corrected Wilcoxon tests.

Effect size & scope

Pretrained models win 5 of 7 groups, lose electricity to Theta, and on exchange rates are statistically indistinguishable from seasonal naive. Largest gain: TimesFM family 28% lower MASE than best classical on weekly Wikipedia pageviews. Within-family comparison: TimesFM outranks Chronos by -0.53 average rank on Wikipedia vs -0.09 elsewhere (1,500 vs 754 series, Mann-Whitney p < 10^-5).

Limitations

Authors concede corpora are disclosed only in summary so corpus-familiarity attribution relies on TimesFM authors' own statement about Wikipedia dominance; monthly horizon cut to 8 steps by design; small groups (46 electricity, 29 exchange rate series) limit power; two models' corpora dominate the familiarity analysis. Unstated: 'corpus familiarity' is inferred indirectly, no direct corpus access verifies the mechanism; only six pretrained models, all forecasting-specific.

Counterpoint

The headline result is partly a naming choice: five wins out of seven still favors foundation models, and the Wikipedia effect rests on one model family and an anecdotal corpus disclosure rather than a controlled corpus manipulation. Domain selection is post hoc in spirit (the paper picked domains partly knowing TimesFM's corpus), n=46 and n=29 groups are tiny, and the familiarity-vs-intrinsic-property test covers only two candidate properties.

Economy implication

Little or nothing for the AI-economy debate in the near term, beyond one quiet caution: benchmark-based claims of broad zero-shot superiority can overstate where pretrained models actually add value, so buyers of forecasting tooling should demand domain-specific validation rather than leaderboard numbers.

Who this matters for

Technical Capability: Anyone claiming time-series foundation models generalize should note that a post-release hold-out still leaves domain-corpus familiarity confounding zero-shot scores, so 'zero-shot' claims need domain-level scrutiny.

Market & Valuation: Vendors and buyers pricing forecasting products off public benchmark tables face a measurement problem: the published rankings may reflect pretraining corpus overlap with benchmark domains rather than transferable capability.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.