WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

2608.04008 ↗ · 2026-08-04 · cs.CL · h-max 6 · 0 citations

Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi

Project page: https://co-minder.github.io/worldcup2026

The finding

Headline claim

Six frontier LLMs, evaluated prospectively and leakage-free on all 104 matches of the 2026 World Cup, fail to beat the bookmaker's favourite baseline and cluster within a few points of each other on every market.

The test

Over 39 days (10 June to 19 July 2026), six flagship models (Claude, GPT, Gemini, Kimi, GLM, Seed), each at maximum thinking settings with provider-native web search, filled a seven-market prediction card before kickoff for all 104 matches plus 12 group winners and an outright pool, yielding 4,494 scored predictions settled against official scorelines; three leakage locks (snapshot, kickoff, template) and a frozen archive of prompts enforce the prospective design.

Effect size & scope

Mean 1X2 accuracy is 63.9% versus 64.4% from mechanically backing the bookmaker's favourite; only Claude (66.3%) exceeds it, by two matches. Models predicted 8-14 draws where 27 occurred and placed 28.0% of scoreline picks on 2-1. Models agree on the outcome in 76.0% of matches, so majority vote (65.4%) adds nothing. Leaderboard spread is 84 points (897 to 813), with Claude first in only 53.6% of 10,000 bootstrap resamples and the Claude-GPT difference of +22 points carrying a 95% interval of [-49,+94]; only the Claude-Gemini contrast approaches significance (P=0.964).

Limitations

The authors concede this is a single event, a purposive six-model cohort, one choice per market with flattened weights, and that template refinements complicate cross-time comparison. The text cuts off before a full limitations section. Obvious unstated ones: n=104 matches gives very low statistical power (confidence intervals overlap almost everywhere); football outcome prediction may not generalize to other forecasting domains; provider-side search is a black box whose behavior may differ per vendor; scoring weights were chosen by the authors; the prompt template wording was changed twice mid-run.

Counterpoint

A skeptic would note the headline finding (models match but do not beat the market) is exactly what efficient-market theory predicts for any forecaster, so the benchmark may say little about model capability differences at all; the leaderboard ranking is statistically indistinguishable from random with these confidence intervals; the single-domain, single-event design cannot separate football-specific conservatism from general forecasting weakness; and the framing as a 'benchmark release' may oversell what one tournament can establish.

Economy implication

Little or nothing yet. The paper's economics signal is indirect: frontier models from six organizations perform nearly identically on a real forecasting task and herd heavily, which weakly corroborates commoditization at the frontier, but one football tournament cannot establish that for economic use cases.

Who this matters for

Technical Capability: Researchers studying benchmark contamination get a rare fully prospective evaluation design, plus evidence that frontier models are macro-strong but micro-weak forecasters (20 to 44 points better on tournament-level than single-match questions).

Market & Valuation: Investors wondering whether frontier model differentiation justifies pricing power see six flagships land within a few percentage points on every market, with herding so strong that ensembling adds nothing.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

[5.1] Frontier models cluster tightly (small gaps, fast catch-up).

Abstract

Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.