Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets

2609.11449 ↗ · 2026-09-10 · cs.LG, cs.AI

Jia Huang, Yankai Wan, Yangjun Ou

9 pages

The finding

Headline claim

The precision of a detector-defined dataset is a property of the deployment pool's prevalence, not of the detector, and this was measured end-to-end on real data where naive cross-pool precision transfer erred by +422% while a Bayes expression using prevalence erred by only 3.3%.

The test

A magnitude-threshold scan over E-mini S&P 500 futures (2010-2026, 1-second bid-offer data) deployed on three candidate pools: FOMC meeting calendar (140 windows), first Fridays (200), and business days 6-16 (1569). Every detected item was checked against an independent official index of 555 CPI, nonfarm payroll, and FOMC announcement windows, making phantoms identifiable. Mixture decomposition, estimator-comparison, and non-integrable-estimator analyses were run on the 335-event screened sample.

Effect size & scope

Phantom rates were 81.7% / 9.0% / 0.0% across the three pools for one detector on one price series. Naive transfer from the two high-prevalence pools predicted precision 0.955 against a measured 0.183 (+422% error); Bayes prediction 0.177 vs measured 0.183 (-3.3% error). Phantom items outnumbered true events 473 to 308 (c = 0.606). The common per-item normalised statistic returned 0.40 where the well-defined estimator returned 0.10 on the same events. The mixture identity held to 1.1e-16. Scope: one instrument, one detector family, one class of official index, one financial market.

Limitations

Authors concede one instrument, one detector family, one class of official index, and one asset. Their phantom null reproduced shape but sat systematically low (RMSE 0.092), leaving a residual level they call an open question. The financial-magnitude threshold detector is a stylised stand-in for the ML detectors practitioners actually use, and the official index exists only because central bank releases are unusually verifiable, so the recovery mechanism does not transfer to ordinary detector-defined datasets. The text was truncated mid-Section 5.4 so later results were only partially visible.

Counterpoint

The Bayes mechanism is textbook and the authors concede no novelty for the identity itself; the contribution is a demonstration on one hand-built pipeline. A skeptic could argue the finding is largely an artifact of choosing an extremely low-prevalence pool (CPI) against near-perfect pools to manufacture a dramatic transfer error, that a toy price-threshold detector is not evidence that modern learned detectors or AI evaluation pipelines suffer this at similar magnitude, and that the paper's own pipeline experienced six accidents, which cuts both ways as documentation of care and of fragility.

Economy implication

Little or nothing yet for the AI economy in the next 12 to 24 months as a direct matter. The indirect warning is real but unquantified: a large share of AI training and evaluation datasets are detector-defined (weak supervision, pseudo-labelling, synthetic filtering), and if precision does not transfer across pools, then ROI claims, benchmark results, and labor-effect studies built on such datasets could be silently miscalibrated. No market-level magnitude can be attached from this paper.

Who this matters for

Technical Capability: Anyone building or evaluating models on weakly supervised, pseudo-labelled, or anomaly-filtered data should report prevalence, TPR, FPR, phantom mass fraction, and phantom intensity ratio alongside any detector-defined dataset, since precision alone is uninformative and can be off by 5x at low prevalence.

Market & Valuation: Analysts consuming AI research that relies on event studies or automatically constructed datasets should discount findings where the candidate pool's base rate is unreported, as contamination can inflate or dilute measured effects depending on the estimator.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Many ML datasets are constructed by running a detector, heuristic, or model over candidate pools; accepted items become labels. Dataset precision is then governed by true-positive prevalence in each pool via Bayes, not solely by detector quality. Using one instrument and period, we hold a detector-defined event dataset plus an independent official index labeling every detected item as real or phantom. One detector, three pools yield phantom rates 81.7%, 9.0%, and 0.0%. Transferring precision from the two high-rate pools to the low-rate pool predicts 0.955 versus measured 0.183, a +422% error; the Bayes expression predicts all three within 3.3%. The detected response curve is an exact convex combination of a true-event and a phantom component (residual 1.1e-16), with phantoms outnumbering true events 473 to 308, so contamination is a second signal with detector-inherited shape, not additive noise. Contamination direction depends on the estimator: on identical windows one statistic is diluted and another inflated because its denominator is also contaminated. A common normalization turns the estimator into a mean of ratios whose expectation need not exist; on the same 335 events it returns 0.40 where the well-defined estimator returns 0.10.