Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds

Spending extra inference-time compute on critique refinement and wrapping the target in a deployment-matched agent harness (DISH) significantly narrows the gap between simulated alignment audits and real deployments, making audits harder for target models to distinguish from real use.

Automated Petri audits across four frontier target models (Sonnet 4.6, Opus 4.8, GPT-5.5, Gemini 3.5 Flash), 149 seed instructions for critique refinement experiments, 24 coding-agent seeds for DISH experiments, 20 WildChat and 50 Claude Code deployment transcripts, measured by realism win rate, verbalized evaluation awareness, concerning behavior, and seed adherence with validated LLM judges against 65 human annotations.

Realism win rate rises from 10% to 26% on Sonnet 4.6 and from 1-2% to 3-7% on other targets with critique refinement (cr2bo4). DISH raises realism win rate from 9% to 21% on Sonnet 4.6 and from near-floor to 2-3% on other targets. The two techniques compose, adding another 13pp on Sonnet 4.6. Depth scaling beats breadth scaling. Even at maximum compute (cr16), realism win rate reaches only 34%, still far from deployment indistinguishability. Effects are heterogeneous across targets and are strongest on the model that is likely the authors' own (Sonnet 4.6).

Authors concede: realism win rate remains well below parity in all conditions; effects are much smaller on non-Sonnet targets; concerning behavior shifts are mixed across targets; seed adherence decreases with more refinement compute; judge measures relying on readable reasoning may be unrepresentative for targets that hide reasoning; deployment-transcript sample sizes are small (N=20 for WildChat, 50 for task re-creations). Unstated: the target model serves as its own feedback, preference, and judge model, creating circularity; DISH primarily benefits models in the Anthropic ecosystem; the deploy/audit distinction may simply become harder for other reasons over time.

The evaluation uses target models as their own judges and feedback sources, so realism scores may partially reflect models' willingness to label in-distribution-sounding content as real rather than genuine behavioral indistinguishability. Most headline gains concentrate on one target family; on three of four models absolute realism win rates stay under 10%. LLM judges have documented prompt sensitivity, and the pairwise realism prompt directly tells the judge what audit artifacts look like, potentially gaming the measure. Compute-matched baseline comparisons are approximate (0.59x to 1.3x realized cost differences).

Little or nothing directly for near-term AI economics; this is an alignment-evaluation methodology paper. Indirectly, it signals that inference-time compute is being redirected toward safety infrastructure, and that reliable pre-deployment auditing may become a costly but expected step in enterprise procurement of frontier models.

Technical Capability: Shows inference-time compute has measurable returns on a hard evaluation problem, but with rapidly diminishing absolute gains, relevant to debates about scaling test-time spend.

Regulatory & Legal: If frontier labs' safety evaluations (which appear in system cards) rest on audits models can easily detect as fake, the evidentiary basis for regulatory assurances about deployed model behavior is weaker than it appears.