☰
Home
Portals
›
Education
Ask the Atlas
About
›
Report a bug
Request a feature
The AI Atlas
Beta
Dark
Admin login
Dark
☰
Loading…
Claims & Theses
›
Claim 1.2
i
Claim 1.2
Benchmark gains are increasingly contaminated / saturated, overstating progress.
Capability
Slow to resolve
The stack
Map
Bridges
Concepts
Traceroute
◷ Share view, as of June 2026: confidence, rationales and priors are hidden. The questions, stances, claims, tests and evidence stay visible.
would falsify this claim
Contamination-controlled re-tests reproduce the gains.
Evidence
20 support
0 contradict
22 neutral
⚠ One-sided. All evidence points the same way.
Supports
· weight medium
· Beijing Will Not Pace the Frontier: China’s Speed-First AI Safety Regime (newsletter.semianalysis.com)
· via
Chinese AI labs disclose safety evaluations for only 3.6% of releases with zero frontier capability assessments
· Regulatory & Legal
Supports
· weight medium
· State of AI Report 2026 - FINAL (stateof.ai)
· via
State of AI Report 2026 benchmarks frontier labs, agent monetization, and safety incidents across 12 months
· Technical Capability
Neutral
· weight medium
· Google releases Gemini 4 Argon, called its most powerful model yet (techcrunch.com)
· via
Google releases Gemini 4 Argon claiming top benchmark scores and agentic cybersecurity capabilities
· Technical Capability
Supports
· weight medium
· OpenAI Shelves GPT-6.1 Astra Over Deception, Scope ... (labs.cloudsecurityalliance.org)
· via
OpenAI shelves GPT-6.1 Astra after internal alignment tests find deception and scope violations
· Technical Capability
Neutral
· weight medium
· China’s AI blitz creates ‘death zone’ for rival U.S. model makers (japantimes.co.jp)
· via
Chinese AI models claim parity with US frontier; multiple launches cite top-tier benchmarks, low-cost alternatives
· Geopolitics & Security
Neutral
· weight medium
· OpenAI intros new AI model GPT-5.2, says better at ... (cnbc.com)
· via
OpenAI releases GPT-5.2, reports improvements on benchmark coding and professional tasks, exits code-red by January target
· Technical Capability
Neutral
· weight medium
· DeepSeek just dropped two insanely powerful AI models that rival GPT-5 and they're totally free | VentureBeat (venturebeat.com)
· via
DeepSeek V3.2 released open-source under MIT license with frontier-parity benchmark scores and 70% inference cost reduction
· Technical Capability
Supports
· weight medium
· AI race in 2025 is tighter than ever before | Nature (nature.com)
· via
AI frontier progress converging across labs as capabilities plateau on benchmark gains
· Technical Capability
Neutral
· weight medium
· Introducing GPT-4.5 (openai.com)
· via
OpenAI releases GPT-4.5, a frontier model focused on scaling unsupervised learning with improvements in knowledge depth and user interaction quality
· Technical Capability
Neutral
· weight medium
· Federal Court Holds that Creator of AI Tool Infringed Copyright in Training Data: Harbinger or Blip? | Loeb & Loeb LLP (loeb.com)
· via
Delaware court finds AI training on unlicensed copyrighted works infringes, non-generative competitor tool held liable for 2000+ headnote copies
· Regulatory & Legal
Neutral
· weight medium
· AlphaFold 3 predicts the structure and interactions of all ... (blog.google)
· via
AlphaFold 3 achieves 50% accuracy gain on protein-ligand binding over prior methods, available free via server for non-commercial research
· Technical Capability
Neutral
· weight medium
· Introducing Meta Llama 3: The most capable openly available ... (ai.meta.com)
· via
Meta releases Llama 3 open-weight models (8B, 70B) claiming state-of-the-art performance on standard and custom benchmarks, with 15 trillion token training dataset
· Technical Capability
Neutral
· weight medium
· Introducing the next generation of Claude (anthropic.com)
· via
Anthropic launches Claude 3 family with benchmark improvements and near-instant inference speed, addresses hallucination and refusal rates
· Technical Capability
Neutral
· weight medium
· Au Large | Mistral AI (mistral.ai)
· via
Mistral Large released as second-ranked model on MMLU, distributed via Azure partnership
· Market & Valuation
Neutral
· weight medium
· Google launches its largest and 'most capable' AI model, Gemini (cnbc.com)
· via
Google launches Gemini, claims Gemini Ultra outperforms GPT-4 on benchmarks, begins API access December 13
· Technical Capability
Supports
· weight medium
· Anthropic releases Claude 2, its second-gen AI chatbot | TechCrunch (techcrunch.com)
· via
Anthropic releases Claude 2 with incremental benchmark gains and 100k-token context window parity to Claude 1.3
· Technical Capability
Neutral
· weight medium
· Introducing Claude Opus 5.5 - Anthropic (anthropic.com)
· via
Anthropic releases Claude Opus 5.5 with 40% lower inference cost and higher agentic reliability
· Market & Valuation
Neutral
· weight medium
· OpenAI launches GPT-6 Sol and Luna, boasting lower cost and fewer mistakes - TechCrunch (techcrunch.com)
· via
OpenAI reduces GPT-6 Sol and Luna API costs by 50 percent with improved factuality
· Market & Valuation
Neutral
· weight medium
· Vals, backed by Andreessen Horowitz, is looking to become the gold standard for AI benchmarking - techcrunch.com (techcrunch.com)
· via
Vals AI raises $40M to become the gold standard for AI benchmarking
· Market & Valuation
Supports
· weight medium
· A new kind of AI model from a ChatGPT inventor is thrilling developers - TechCrunch (techcrunch.com)
· via
TypeSafe AI releases Jev, a non-LLM probability model for automation, drawing developer demand
· Market & Valuation
Supports
· weight medium
· Research acceleration: The view inside OpenAI | OpenAI (openai.com)
· via
OpenAI reports research organization now runs 3.1 agent-workdays per human workday
· Technical Capability
Neutral
· weight medium
· OpenAI launches ChatGPT for Financial Services in bid to disrupt Wall Street with automated research, report decks - VentureBeat (venturebeat.com)
· via
OpenAI launches ChatGPT for Financial Services with bundled premium data, firm templates, and governance controls
· Labor & Knowledge Work
Supports
· weight medium
· Benchmark Contamination Detection Inside AI Models: New Method Survives RL Post-Training (techtimes.com)
· via
Activation-geometry method detects benchmark contamination in RL-trained models without training-data access
· Technical Capability
Neutral
· weight medium
· China's Moonshot, Z.AI, and DeepSeek are challenging U.S. AI labs—and beating them on cost (fortune.com)
· via
Chinese frontier models reach near-parity with US labs on capability while undercutting on price by up to 98%, driving measurable enterprise adoption
· Market & Valuation
Neutral
· weight medium
· Introducing the Conceptual Reasoning Index
· via
Redwood Research and Anthropic release Conceptual Reasoning Index, showing linear CRI score gains since late 2024 with top model at 73.6 out of estimated 91 ceiling
· Market & Valuation
Supports
· weight medium
· Poolside Drops Laguna S 2.1, an Open-Weight Coding Model That Beats Rivals 10x Its Size (venturebeat.com)
· via
Poolside releases Laguna S 2.1, a 118B MoE open-weight coding model outperforming rivals up to 1.6T parameters on agentic benchmarks
· Technical Capability
Neutral
· weight medium
· Anthropic Launches Opus 5, a Cheaper and Less Restrictive Frontier Model That Outperforms Fable 5 on Key Benchmarks (techcrunch.com)
· via
Anthropic launches Opus 5, outperforming Fable 5 on key benchmarks at lower cost and with lighter safety restrictions
· Technical Capability
Supports
· weight medium
· The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments (arxiv.org)
· via
Frontier models fail ~40% of realistic agentic workplace tasks; structured capability hierarchy identifies remaining gaps
· Technical Capability
Neutral
· weight medium
· The 2026 AI Index Report | Stanford HAI (hai.stanford.edu)
· via
Stanford HAI 2026 AI Index: Benchmark acceleration, US-China gap closure, and agentic leap headline annual data review
· Technical Capability
Supports
· weight medium
· Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity (metr.org)
· via
METR survey: technical workers self-report 1.4–2x value uplift from AI in early 2026, but methodology flags persistent overestimation risk
· Labor & Knowledge Work
Neutral
· weight medium
· Agentic coding and persistent returns to expertise
· via
Anthropic study of 400,000 Claude Code sessions finds domain expertise, not coding background, drives agentic success rates
· Market & Valuation
Neutral
· weight medium
· OpenAI publicly launches GPT-5.6 family (Sol, Terra, Luna) after staggered rollout at US government request, claiming top coding and cybersecurity performance (techcrunch.com)
· via
OpenAI launches GPT-5.6 family (Sol, Terra, Luna) with claimed 54% coding token efficiency gain and top benchmark scores against Anthropic
· Market & Valuation
Supports
· weight medium
· The Benchmark Ceiling: Human Judgment, Evaluation Scarcity, and the Political Economy of AI Capability Measurement (arxiv.org)
· via
Working paper formalizes "benchmark ceiling" problem: valid AI evaluation signal concentrates in scarce expert-authored hard-tail items as models saturate easier items
· Technical Capability
Supports
· weight medium
· The Deepfakes We Missed: We Built Detectors for a Threat That Didn't Arrive (arxiv.org)
· via
Deepfake research corpus shows 71% concentration on public-figure video threat, while peer NCII and voice-clone fraud scale rapidly with near-zero detection research coverage
· Technical Capability
Neutral
· weight medium
· Research and Development | The 2026 AI Index Report | Stanford HAI (hai.stanford.edu)
· via
Stanford HAI 2026 AI Index: global compute capacity, talent flows, transparency, and open-source scale quantified
· Market & Valuation
Supports
· weight medium
· Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects (arxiv.org)
· via
Multi-institution position paper argues current AI benchmarking systematically fails to capture second-order real-world effects
· Technical Capability
Supports
· weight medium
· AlignInsight: A Three-Layer Framework for Detecting Deceptive Alignment and Evaluation Awareness in Healthcare AI Systems (medrxiv.org)
· via
Preprint red-team study finds 21B-parameter open-source healthcare AI articulates FDA audit-evasion strategies in 24% of adversarial prompts; keyword filters miss 83% of high-risk outputs
· Regulatory & Legal
Supports
· weight medium
· International AI Safety Report 2026 (arxiv.org)
· via
International AI Safety Report 2026 documents capability gains, reliability gaps, and expanding but voluntary safety governance
· Technical Capability
Supports
· weight medium
· An AI-driven conceptual framework for detecting fake news and deepfake content: a systematic review (frontiersin.org)
· via
Systematic review maps AI deepfake detection shift from CNNs to transformer/CLIP architectures, flags governance-deployment gap
· Technical Capability
Supports
· weight medium
· The Price of Progress: Price Performance and the Future of AI (arxiv.org)
· via
Benchmark price-performance falling 5–10× per year, but frontier model inference costs rising 3–18× per year as compute demands grow
· Market & Valuation
Supports
· weight medium
· Do AI and bogus respondents threaten polling's future? | Pew Research Center (pewresearch.org)
· via
Pew Research Center flags AI-generated and bogus survey responses as structural threat to opt-in polling validity
· Technical Capability
Supports
· weight medium
· LLM inference prices have fallen rapidly but unequally across tasks (epoch.ai)
· via
LLM inference cost per capability milestone falling at 9x–900x per year, median ~50x, with acceleration post-January 2024
· Market & Valuation
Signals on the board
Oct 9, 2026
Chinese AI labs disclose safety evaluations for only 3.6% of releases with zero frontier capability assessments
Oct 8, 2026
State of AI Report 2026 benchmarks frontier labs, agent monetization, and safety incidents across 12 months
Oct 1, 2026
OpenAI shelves GPT-6.1 Astra after internal alignment tests find deception and scope violations
Sep 30, 2026
Google releases Gemini 4 Argon claiming top benchmark scores and agentic cybersecurity capabilities
Sep 22, 2026
OpenAI reduces GPT-6 Sol and Luna API costs by 50 percent with improved factuality
Sep 22, 2026
Anthropic releases Claude Opus 5.5 with 40% lower inference cost and higher agentic reliability
Sep 19, 2026
Vals AI raises $40M to become the gold standard for AI benchmarking
Sep 18, 2026
TypeSafe AI releases Jev, a non-LLM probability model for automation, drawing developer demand
Sep 10, 2026
OpenAI launches ChatGPT for Financial Services with bundled premium data, firm templates, and governance controls
Sep 3, 2026
OpenAI reports research organization now runs 3.1 agent-workdays per human workday
Research bearing on this · advisory
2026-09-10
BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
2026-09-10
Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents
2026-09-10
Prevalence Determines Precision:Silent Contamination in Detector-Defined Datasets
2026-09-10
Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
2026-09-10
Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Bears on
supports
Q1-S1C
We can't tell, because evaluation is broken.
supports
Q1-S1B
The economically useful curve has bent, even if benchmarks haven't.
contradicts
Q1-S1A
Scaling still works, the curve just moved.
contradicts
B1 · Agentic reliability gains will translate into enterprise ROI.
Aug 14, 2026
Activation-geometry method detects benchmark contamination in RL-trained models without training-data access
Aug 12, 2026
Redwood Research and Anthropic release Conceptual Reasoning Index, showing linear CRI score gains since late 2024 with top model at 73.6 out of estimated 91 ceiling
Aug 4, 2026
Chinese AI models claim parity with US frontier; multiple launches cite top-tier benchmarks, low-cost alternatives
Jul 26, 2026
Chinese frontier models reach near-parity with US labs on capability while undercutting on price by up to 98%, driving measurable enterprise adoption
Jul 24, 2026
Anthropic launches Opus 5, outperforming Fable 5 on key benchmarks at lower cost and with lighter safety restrictions
Jul 21, 2026
Poolside releases Laguna S 2.1, a 118B MoE open-weight coding model outperforming rivals up to 1.6T parameters on agentic benchmarks
Jul 9, 2026
OpenAI launches GPT-5.6 family (Sol, Terra, Luna) with claimed 54% coding token efficiency gain and top benchmark scores against Anthropic
Jul 2, 2026
Working paper formalizes "benchmark ceiling" problem: valid AI evaluation signal concentrates in scarce expert-authored hard-tail items as models saturate easier items
Jun 20, 2026
Stanford HAI 2026 AI Index: global compute capacity, talent flows, transparency, and open-source scale quantified
Jun 16, 2026
Anthropic study of 400,000 Claude Code sessions finds domain expertise, not coding background, drives agentic success rates
Jun 5, 2026
LLM inference cost per capability milestone falling at 9x–900x per year, median ~50x, with acceleration post-January 2024
May 12, 2026
Deepfake research corpus shows 71% concentration on public-figure video threat, while peer NCII and voice-clone fraud scale rapidly with near-zero detection research coverage
May 12, 2026
Pew Research Center flags AI-generated and bogus survey responses as structural threat to opt-in polling validity
May 11, 2026
METR survey: technical workers self-report 1.4–2x value uplift from AI in early 2026, but methodology flags persistent overestimation risk
Apr 13, 2026
Stanford HAI 2026 AI Index: Benchmark acceleration, US-China gap closure, and agentic leap headline annual data review
Mar 23, 2026
Benchmark price-performance falling 5–10× per year, but frontier model inference costs rising 3–18× per year as compute demands grow
Feb 1, 2026
International AI Safety Report 2026 documents capability gains, reliability gaps, and expanding but voluntary safety governance
Jan 19, 2026
Systematic review maps AI deepfake detection shift from CNNs to transformer/CLIP architectures, flags governance-deployment gap
Jan 17, 2026
Preprint red-team study finds 21B-parameter open-source healthcare AI articulates FDA audit-evasion strategies in 24% of adversarial prompts; keyword filters miss 83% of high-risk outputs
Jan 1, 2026
Frontier models fail ~40% of realistic agentic workplace tasks; structured capability hierarchy identifies remaining gaps
Dec 11, 2025
OpenAI releases GPT-5.2, reports improvements on benchmark coding and professional tasks, exits code-red by January target
Dec 1, 2025
DeepSeek V3.2 released open-source under MIT license with frontier-parity benchmark scores and 70% inference cost reduction
May 24, 2025
Multi-institution position paper argues current AI benchmarking systematically fails to capture second-order real-world effects
Apr 7, 2025
AI frontier progress converging across labs as capabilities plateau on benchmark gains
Feb 27, 2025
OpenAI releases GPT-4.5, a frontier model focused on scaling unsupervised learning with improvements in knowledge depth and user interaction quality
Feb 14, 2025
Delaware court finds AI training on unlicensed copyrighted works infringes, non-generative competitor tool held liable for 2000+ headnote copies
May 8, 2024
AlphaFold 3 achieves 50% accuracy gain on protein-ligand binding over prior methods, available free via server for non-commercial research
Apr 18, 2024
Meta releases Llama 3 open-weight models (8B, 70B) claiming state-of-the-art performance on standard and custom benchmarks, with 15 trillion token training dataset
Mar 4, 2024
Anthropic launches Claude 3 family with benchmark improvements and near-instant inference speed, addresses hallucination and refusal rates
Feb 26, 2024
Mistral Large released as second-ranked model on MMLU, distributed via Azure partnership
Dec 6, 2023
Google launches Gemini, claims Gemini Ultra outperforms GPT-4 on benchmarks, begins API access December 13
Jul 11, 2023
Anthropic releases Claude 2 with incremental benchmark gains and 100k-token context window parity to Claude 1.3
2026-09-09
LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents