$Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

2609.10226 ↗ · 2026-09-09 · cs.CL

Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang, Xiaomeng Chu, Guoliang You, Wuyang Zhang, Daxin Jiang +1

The finding

Headline claim

Frontier LLM agents currently score only around 36.5 percent on a new benchmark of open-ended, long-horizon LLM infrastructure engineering tasks, showing that even top models cannot yet reliably engineer the systems that run them.

The test

The authors built Phi-Bench, 85 tasks (55 kernel function completion, 20 long-horizon implementation, 10 end-to-end optimization) synthesized from 2,260 systems papers and 1,852 GitHub artifacts, organized by a taxonomy of 410 tags, 62 topics, and 9 categories, then evaluated multiple frontier LLMs with agents on a rubric combining correctness-gated performance ratios and binary implementation checks with cheating detection.

Effect size & scope

The best model, Claude Opus 5, scores 36.53 percent overall, leaving large headroom; performance degrades as tasks grow more open-ended (KFC to LHI to E2EO), and both iteration count and reasoning budget affect scores. Results hold on this benchmark's environments only; no real-world deployment evidence is given.

Limitations

The authors concede scores are far from ceiling and analyze error modes as open challenges. Unstated limitations: 85 tasks is a small sample (only 10 E2EO), tasks are synthesized or reconstructed from public PRs and papers so leakage is possible despite proctoring, scoring depends on reference solutions and log-normalized thresholds, and the agent scaffold and compute budget may understate model capability.

Counterpoint

A skeptic would note the benchmark is constructed largely by LLMs themselves (filtering, taxonomy, test generation), that anti-cheating checks may not catch memorized upstream solutions, that best-of-frontier scores on new benchmarks typically rise quickly, and that a single leaderboard number with tiny open-ended-task counts is weak evidence about autonomous infrastructure engineering in production.

Economy implication

Little or nothing yet for near-term AI economics: the paper suggests LLM-driven self-optimization of inference and training infrastructure remains a research prospect, so it weakly discounts claims that AI will soon compress infrastructure engineering headcount or costs within 12 to 24 months.

Who this matters for

Technical Capability: This gives AI researchers a first systematic read on where agentic long-horizon systems engineering actually breaks down, beyond single-kernel benchmarks.

Market & Valuation: Investors betting on AI self-improvement flywheels should note that the top model completes only about a third of real infrastructure engineering tasks, capping near-term autonomous-optimization narratives.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.