Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

2609.11115 ↗ · 2026-09-10 · cs.AI, cs.IR

Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu

Project site: https://benchmark-radar.org/ Code: https://github.com/ktwu01/benchmark-radar

The finding

Headline claim

A daily-updated, searchable database of 1,283 AI benchmark records with 12,916 score observations can expose how poorly documented and incomparable most published benchmark evidence is, enabling prior-art checks and saturation analysis that leaderboards cannot.

The test

A full-catalog census over every record in the rebuilt v0.11.0 index (no filters), drawing on 4 catalog sources (LLM Stats 687, OpenCompass Hub 461, Artificial Analysis 25, model reports 110), plus daily discovery over 46 snapshots across 37 sources yielding 11,068 observations on 6,546 artifacts.

Effect size & scope

Score coverage is uneven: 790 of 1,283 records have numeric scores, 493 have none; OpenCompass contributes 461 records with zero archived scores. Only 615 of 1,283 records carry a release date. The agentic share of dated releases rises over time, robust to reweighting for source mix (moves at most 1.1 percentage points). 345 records are agentic by facet but 117 of those sit under Coding & Software Engineering as their primary class.

Limitations

The authors concede they do not measure retrieval accuracy or time saved; only 5 of 46 snapshots are real historical backfills (4 simulated); 88% of discovery observations come from five source labels, so coverage is concentrated; identity links across sources are sparse (only 59 of 6,546 artifacts multi-source) and may undercount cross-source matches. Unstated: catalogs differ in scope so the census mixes registry conventions, and the worked example is a single anecdote.

Counterpoint

This is a tool paper with PR structure: no baseline system is compared against, no evaluation of search quality, and the headline statistics largely describe the incompleteness of the ingested catalogs rather than new scientific findings; a careful researcher using Google and Hugging Face could reach similar prior-art conclusions.

Economy implication

Little or nothing directly yet. The indirect value is better meta-evidence infrastructure: it makes benchmark contamination, saturation, and documented evaluation settings more auditable, which over time disciplines the capability claims that drive enterprise buying decisions.

Who this matters for

Technical Capability: Benchmarks researchers and eval builders get one searchable place to find prior art and inspect whether reported scores are on comparable scales, versions, and settings.

Market & Valuation: Investors pricing model capability claims get a tool to check whether a headline benchmark number was measured on a matched setting or is a saturated, unverifiable comparison.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Benchmark researchers and developers of large language models (LLMs) and other AI systems need to find relevant evaluations, locate their benchmark datasets and code, and understand the settings behind reported scores. We present Benchmark Radar, a living database and search engine for retrieval and discovery of AI benchmarks, covering LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety, and domain-specific evaluations. The system combines daily discovery of benchmark papers, repositories, datasets, and releases with a searchable benchmark catalog, mentions in model cards and technical reports, and score histories. It retains source identities and citations so readers can inspect candidate benchmarks and their evaluation evidence. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog contains 1,283 source records drawn from 4 benchmark catalogs and 12,916 numeric observations on 790 records. We describe collection and retrieval, audit the full catalog, and examine benchmark saturation, adoption trends, and the limits of score comparisons. A worked example walks through a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation. We release the web dashboard with a benchmark leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence, a command-line interface (CLI) for offline queries, and reproducible analysis.