AutoOnboarder: Autonomously Onboarding LLM Agents

2610.21218 ↗ · 2026-10-15 · cs.AI

Tanay Arora, Cong Liu

15 pages, 5 figures

The finding

Headline claim

An autonomous Onboarder agent can rewrite an Executor agent's prompt, foundation model, and browser harness code using only ~20 examples, producing agents that match or exceed a Gemini 3.5 Flash baseline on genomic prior-authorization form submission at up to 29x lower per-case cost, though final performance varies enormously run to run.

The test

One sandbox task (whole-exome/genome sequencing prior authorization, 200 cases: 20 onboarding, 180 validation, 23 error classes), with Claude Code Opus 4.8 as Onboarder over 12 iterations, cost goals beta in {$0.10, $0.04, $0.01}, starting from a $0.94/case Gemini 3.5 Flash baseline. Evaluation noise checked via 3 validation repeats per candidate; onboarding noise via 3 full repeats at beta=$0.04. Judge reliability checked on 184 halted episodes.

Effect size & scope

Onboarded candidates evaluated at $0.084 and $0.068 per case versus a $0.94 baseline (up to ~29x cheaper at the $0.01 goal in the abstract, though that run produced no candidate under its cost ceiling and was excluded). Validation scores are low in absolute terms: S around 0.30-0.34, CH around 0.59-0.68, MFA 0.38-0.56. Evaluation noise is small (S std 0.007-0.009) but Onboarding noise is huge (S std 0.152, ~20x evaluation noise; MFA ranged 0.041 to 0.917 across repeats; 63 of 180 cases flipped outcomes between onboarding repeats). The $0.01 goal failed entirely to produce a legal candidate.

Limitations

Authors concede Onboarding variance is excessively large, the model lever needs a curated search space, the Onboarder needs a structured thought interface, and results are 'directional evidence' with no statistical tests because each experiment is run once and repeated at most three times. The headline 29x cost claim rests on the run that failed its own cost ceiling. Baseline absolute performance numbers for the $0.94 Gemini baseline on validation are not shown in the provided text, so 'matches or exceeds' cannot be verified from these tables. Judge compliance with the written contract was verified only manually post-hoc.

Counterpoint

Single task, single domain, tiny n, no statistical tests, and a self-reported paper from what appears to be a student/lab project. The 29x cost reduction headline is contradicted by their own Table 2, where the cheapest validated candidates were ~11-14x cheaper, and the task's absolute scores (~0.3) mean the agents still fail most cases on a binary metric. Cost comparisons also depend on OpenRouter pricing at one point in time. Without the baseline's validation numbers visible, the match-or-exceed claim is unverifiable from the text provided.

Economy implication

Little or nothing yet for the economics of AI in the next 12 to 24 months. The directionally interesting idea, that agent harness overhead is a real cost driver and can be autonomously optimized away with tens of examples rather than thousands, could matter for inference cost curves if it generalizes, but this paper's own variance analysis shows the method is not yet reliable enough to deploy.

Who this matters for

Technical Capability: Directly documents the harness bottleneck: frontier models flag errors 91% of the time when asked directly but under 0.5% inside a browser-use harness, and shows partial autonomous recovery of that capability.

Labor & Knowledge Work: Prior-authorization paperwork consumes about 13 physician-hours per week; even low-scoring agents at $0.07/case sketch what near-free administrative automation could target, if reliability improves.

Bears on (advisory · papers never write evidence)

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

Agentic large language models have been documented to lose reasoning capability when placed inside a harness, the software that mediates their observations and actions. To reconcile these losses, we introduce AutoOnboarder, an autonomous system that adapts an agent to a task the way a trainer onboards a new employee. It places the agent under the supervision of an intelligent ``Onboarder'' coding agent, which iteratively tailors its foundation model, prompt, and harness source code to simultaneously improve performance and decrease cost. Like a real onboarding process, the Onboarder is only given tens of examples, and all evaluation data is isolated on a separate virtual machine to prevent leakage. We report the lessons learned from building and testing AutoOnboarder on genomic-sequencing prior authorization form submission. Autonomously onboarded agents are able to match or exceed the performance of a Gemini 3.5 Flash baseline at up to 29$\times$ lower costs. However, these agents remain imperfect, and the Onboarding process varies greatly in terms of final performance. We conclude that future Onboarding systems require a robust environment for the Onboarder, including a structured thought interface, (controlled) internet access, and a wide model search space.