Science sandboxes measure the scientific capability of AI agents

2608.30165 ↗ · 2026-08-31 · q-bio.QM, cs.AI

Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai, Kenneth B. Hsu, Yasha Ektefaie, Shantanu Singh, Sangeeta N. Bhatia, Steven K. Reilly, Ryan Tewhey, Eric S. Lander, Pardis C. Sabeti

72 pages, 5 main figures, 3 tables, and 5 supplementary figures; includes supplementary agent instructions and harness

The finding

Headline claim

Frontier AI agents can optimize quantitative metrics in closed-loop scientific experimentation but their scientific reasoning deteriorates when system rules fall outside familiar biological priors, and sandboxes can measure this gap.

The test

Two sandboxes (MPRAbox using a Malinois-based damp oracle with N=50,000-sequence libraries scored on 14 hidden test sets, and CodonBox for rule discovery) evaluated three frontier agents (Claude Opus 4.7 in Claude Code, GPT-5.5 in Codex, Gemini 3.5 Flash CLI) in single-round and multi-round regimes with five replicates each, compared against 14 human-chosen baseline library strategies.

Effect size & scope

Not quantifiable from the provided text: the full-text excerpt cuts off before reporting comparative scores; the paper states only that human baseline strategies 'showed large differences in performance' and that agents could be seen optimizing without understanding, with reasoning quality dropping on disguised/dry oracles.

Limitations

Authors note sandboxes here use one specimen type, one assay, and small scalar reports (severe simplification of real science); damp oracles (Malinois) are proxies for wet labs so results may not transfer; qualitative notebook judgment introduces subjective or AI-judge evaluation. Unstated: prompt/instruction sensitivity, agent configuration choices, and the possibility that single-round results mostly reflect literature priors rather than scientific skill.

Counterpoint

A skeptic would note this is a framework paper whose headline results are qualitative 'we could see' claims, evaluated partly by AI judges on lab notebooks, on only two biological settings and three vendors' models; the wet-oracle validity of Malinois-based scoring and the generalizability of 'deteriorated reasoning' beyond de-anonymized DNA are unproven.

Economy implication

Little or nothing yet economically; this is a measurement framework, not evidence of scientific AI productivity, though if it matures it could become a procurement-style eval for AI-in-R&D claims in pharma and biotech within a couple of years.

Who this matters for

Technical Capability: Offers a benchmark design that separates score optimization from genuine rule-learning, which is exactly the distinction capability trackers need when agents post high numbers without understanding.

Market & Valuation: If adopted, science sandboxes could become a credibility filter for AI-in-science vendor claims, affecting which lab and pharma AI partnerships are believed.

Abstract

Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.