Guangxiang Zhao, Qilong Shi, Xusen Xiao, Wenpu Liu, Yaoming Li, Linfeng Hao, Shuyang Hou, Zijian Guo, Xinrui Zhang, Yuntian Zhao, Zhengyang Wang, Wenrui Liu +4
18 pages, EMNLP 2026 (Main)
On fresh, post-contamination science olympiad exams graded with official partial-credit rubrics, top frontier LLMs reach medal-equivalent scores on several international physics and chemistry exams, while chemistry and long-horizon consistency remain the bottlenecks.
Thirteen public olympiad competitions (IPhO/IChO 2025-2026, IBO 2023, USAPhO, USNCO, CPhO, CChO, etc.), digitized and audited by olympiad medalists; fourteen recent LLMs evaluated under interleaved prompting; LLM-as-judge calibrated against medalist ground truth on archived answers from five models on IPhO/IChO 2025.
Gemini 3.5 Flash interleaved scores 25.67/30 on IPhO 2025 (gold threshold 23.4) and 48.43/60 on IChO 2025 (gold 36.6); Gemini 3.1 Pro reaches 302.1/320 on CPhO 2025. Interleaved beats one-shot in all 16 displayed model-exam comparisons. Both judge models stay within one point of expert total scores. Biology appears closest to saturation; chemistry (33.98-46.45 of 60 for non-Google models) remains the bottleneck.
Judge calibration rests on only two exams and five archived models; the four 2026 columns use capability-coupled input routing (eight vision routes get raw PNGs, six text-only routes get staticized question renderings produced by Gemini 3.1 Pro), so the modality comparison is not controlled; interleaving is a natural protocol choice but the headline scores depend on it; national exams lack medal reference points; benchmark freshness decays as exam years age.
A fair skeptic would note that public exams are contamination-prone over time, that judge calibration on five models and two exams is a small n, that staticizing question images with a frontier model could leak or distort content, and that LLM-as-judge partial credit on rubric-heavy chemistry drawings is exactly where judge error hides. Correlation between labs' own models and top scores on a lab-adjacent benchmark also deserves scrutiny.
Little or nothing yet for the economics of AI. The paper mostly establishes a cleaner measurement instrument; its main economic signal is indirect: frontier models now score at medal level on elite science exams, which modestly supports claims that advanced technical reasoning is becoming automatable, but it measures no ROI, labor, or revenue outcomes.
Technical Capability: This is the core audience: it offers a fresh, rubric-graded, contamination-resistant yardstick showing where frontier scientific reasoning actually stands and which failure axes (visual grounding, structure fidelity, long-horizon consistency) persist.
Labor & Knowledge Work: Knowledge-work professionals in physics-, chemistry-, and analysis-heavy roles get a concrete reference point for how close top models are to elite human technical performance on derivation-heavy tasks.