What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead

2609.10962 ↗ · 2026-09-10 · cs.SE, cs.AI

Haseeb Mohammed Afsar

10 pages, 3 figures. Seeded, re-runnable pipeline and per-server outcomes: https://github.com/itguruhaseeb/mcp-probe ; archived at doi:10.5281/zenodo.21347997

The finding

Headline claim

An unrepaired random draw of published MCP servers is far less runnable than curated samples suggest (only 48.8% complete an initialize handshake, dominated by servers that never start rather than missing credentials), and BFCL v4's tool corpus is heavily self-repeating across tasks while real deployed MCP tools show zero cross-author near-duplication.

The test

A seeded probability draw (n=400) from a 24,135-server registry census (npm/stdio frame of 7,258), each server probed once over the MCP wire protocol with no repair, retries, or credentials; the resulting 2,766 advertised tool descriptions compared with BFCL v4 (8,726 records) and UltraTool EN (14,084 records) using a single held-constant TF-IDF cosine method after global exact deduplication.

Effect size & scope

48.8% inclusion on the random draw vs 66.7% on a hand-curated 24-server frame; 37.5% of draws never start vs 13.3% credential-gated; zero fatal JSON Schema violations across 2,766 tools; 58.8% of real tools lack safety annotations (17.3 points worse than curated); exact-duplicate rates 0.4% real vs 68.8% BFCL vs 85.6% UltraTool; cross-author near-duplication 0.0% for real tools at all thresholds, BFCL 16.7% at cosine 0.70 (16.4 points cross-task), UltraTool only 0.3%. Holds only for the npm/stdio slice, 30.7% of the population and shrinking, and the annotation result is one-probe, no-retry, single-sample.

Limitations

Authors concede: single probe attempt so 48.8% is a lower bound; stdio-only coverage of a shrinking minority; TF-IDF lexical similarity may miss semantic duplication, downward-biasing all corpora; cross-unit decomposition is not power-matched (BFCL has 1.8 tools per row vs 14.2 per server); clone-filter concern only weakly tested; two census snapshots support difference not trend; registry metadata is self-declared; an earlier run found partial-annotation servers that this run does not reproduce. Unstated: n=400 with heavy per-server skew in tool counts, and curated-frame comparison at n=24 is very small.

Counterpoint

The headline 'benchmarks repeat themselves' may partly be an artifact of non-power-matched cross-unit columns (the author admits the BFCL near-its-all-pairs result is partly construction); exact-duplicate rates are sensitive to which files are counted; semantic-equivalence testing absent; the curated comparison baseline is 24 servers; and the fact that UltraTool is cleaner than real tools shows no general claim about synthetic corpora is licensed, limiting the benchmark-validity punchline to one artifact.

Economy implication

Little or nothing yet: this is an ecosystem-health and benchmark-validity measurement, not an economic one. Its indirect relevance is that if tool-use benchmark scores partly measure corpus repetition rather than tool diversity, then benchmark-based claims of agentic capability gains are weaker evidence than they appear, which matters for anyone pricing agent-driven ROI; but this paper does not itself measure ROI or labor effects.

Who this matters for

Technical Capability: Anyone using tool-use benchmarks (especially BFCL) to score agentic models should deduplicate corpora first or risk measuring repetition, and should not extrapolate curated-server health rates to the deployed MCP population.

Market & Valuation: Builders and investors in MCP-tooling startups get the first measured picture of ecosystem reliability: roughly two in five published npm servers are inert, so production tool-marketplaces must gate on runtime health, not registry listings.

Abstract

Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.