Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

2609.04198 ↗ · 2026-09-03 · cs.AI, cs.LG

Haoyaun Zhu, Jie Zhang

The finding

Headline claim

On shared inference endpoints, an LLM used as a measurement instrument is not stable: byte-identical requests to the same model name return inconsistent rankings, enough to fail preregistered instrument-validation gates even when the engineering execution is flawless.

The test

Two fully preregistered campaigns where an external LLM observer ranked candidate solutions to deterministic arithmetic tasks, terminated at instrument validation before any scientific claim was testable. 52,988 audited request attempts across two model generations, four providers in three jurisdictions, self-hosted batch-invariant kernels, and a constructed-error battery, all under an append-only SHA-256 audit chain.

Effect size & scope

Same-window repeat rankings agreed at Spearman 0.400 against a frozen 0.90 requirement; byte-identical next-day replays agreed at 0.78 against a required 0.99. Waiting did not help (0.805 same-day vs 0.800 cross-day). Four providers shared a floor (medians 0.74 to 0.88), uncorrelated with exposed metadata. Self-hosting helped only under low load: concurrent load raised disagreement 8.4-fold. A 748,000-call simulated design passed 0 of 500 times, and a constructed-error battery showed the readout tracks error type, not magnitude.

Limitations

Authors concede the analyses rest on small effective samples (31 valid task groups, 100 replay pairs, ten windows per supplement), a task family that manufactured its own worst case, results scoped to externally measured behavior on shared infrastructure with no bearing on model internals or provider quality, and disclosed preregistration deviations. Unstated: the near-degenerate measurand (candidate gaps seven orders of magnitude below noise) is unusual, so the failure magnitude may not generalize to better-separated tasks.

Counterpoint

A skeptic would note the study's own measurand was poorly conditioned: the observer was asked to rank candidates whose true gaps were tiny relative to noise, which would defeat many instruments. The failure may say more about permutation-style exact-ranking readouts on hard tasks than about LLM judges generally, and the negative result is a research-programme autopsy promoted to a methods paper by two authors with no independent evaluation harness or baseline comparisons against standard soft-judge protocols.

Economy implication

In the next 12 to 24 months the direct economic content is modest: this is a methods and QA discipline paper, not a market signal. Its indirect implication is that any product or benchmark built on LLM-as-judge evaluation carries a hidden measurement-error cost, and that pinning model snapshots or paying for batch-invariant serving becomes a real line item for anyone whose AI quality claims must be auditable.

Who this matters for

Technical Capability: Anyone building evals or agentic pipelines on shared endpoints learns that model-name stability cannot be assumed and must be measured before freezing thresholds.

Regulatory & Legal: Regulators or auditors relying on LLM-judged benchmarks as evidence of model quality should note that fixed model IDs do not guarantee reproducible measurements, complicating certification-style claims.

Abstract

Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, each time with the execution record at ceiling. Three mechanisms explain the gap: a label-to-meaning mapping that biased readouts as strongly as the signal; candidate gaps seven orders of magnitude below the instrument's own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound. Neither metric substitution nor sampling repaired it on the tested grid. Preregistered follow-ups bound the problem: waiting did not help on the days sampled (0.805 versus 0.800, replicated over five further days); switching providers did not help (four providers share the floor, medians 0.74 to 0.88, predicted by none of the metadata fields they expose); self-hosting on batch-invariant kernels helped only while the server was quiet; and on constructed errors with known gaps, the readout's separation tracks error type, not size. We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance. All results concern externally measured behaviour on shared serving infrastructure. On a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.