Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments

2608.02670 ↗ · 2026-08-02 · cs.CR, cs.AI · h-max 1 · 0 citations

Dotan Davidovich, Yair Amar, Hai Rozencwajg, Or Hiltch

The finding

Headline claim

Enterprise security hardening degrades every coding agent's performance, but non-uniformly: the model that best preserves task success under strict policy suffers the largest cost inflation, making model choice policy-dependent rather than universally rankable.

The test

Twelve coding-agent model-harness bundles were evaluated on all 89 tasks in Terminal-Bench 2.1 across three nested security policy levels (control, non-root, NIST-derived high) enforced via native Linux mechanisms (network egress proxy, read-only filesystem mounts, unprivileged user). Each bundle-task-policy cell received three valid trials, yielding 267 trials per policy per bundle. Success rate and mean cost per task were measured at each policy level, and failure modes were decomposed into timeouts, wrong solutions, and early stops.

Effect size & scope

Under NIST-derived high policy: success losses range from 7.1 pp (Grok 4.5) to 18.3 pp (Claude Sonnet 5); cost inflation ranges from 16.0% (GPT-5.6 Luna) to 167.3% (Grok 4.5). The model with the smallest success loss (Grok 4.5) incurs the largest cost inflation, and the two axes do not co-move across bundles. Non-root policy produces intermediate degradation. Results hold across all 89 tasks, with solvability witnesses verified for 82 tasks under the strictest policy.

Limitations

Authors acknowledge that stepping from non-root to NIST-derived high changes multiple policy surfaces simultaneously, so individual axis effects cannot be attributed cleanly. The benchmark is limited to 89 Terminal-Bench tasks, which may not represent the full distribution of enterprise coding work. Only 12 specific model-harness bundles are tested, all evaluated at a single point in time using frozen configurations. The NIST mapping is stated not to constitute a compliance claim. Provider-side safety interventions (e.g., Claude Fable 5 rerouting to Opus 4.8) are kept as deployed defaults, conflating model and harness effects in some cases.

Counterpoint

The 89-task benchmark is small enough that a few swung tasks could shift headline percentages materially. Policy levels are not standardized across the industry, so the NIST-derived mapping may not reflect the specific controls any given enterprise deploys. The study measures one point in time for models that update frequently, and the rankings could shift with new model versions. Cost figures depend on OpenRouter pricing and harness-internal retry behavior, which vary outside the authors' control. The finding that "model choice is policy-dependent" is plausible but the practical magnitude of the effect, across a broader task distribution, remains untested.

Economy implication

Enterprises purchasing or deploying coding agents based on open leaderboard rankings may be systematically misinformed: unconstrained benchmark scores do not predict performance inside hardened environments, and the cost-success tradeoff reshapes at every policy level. Over the next 12 to 24 months, this implies procurement teams need policy-matched evaluations before committing to a vendor, and that the total cost of ownership for coding agents in regulated or security-conscious industries is materially higher than benchmark-adjacent estimates suggest. For vendors, differentiation on hardened-environment performance becomes a plausible enterprise sales lever.

Who this matters for

Market & Valuation: Enterprise software buyers and AI vendors need policy-graded benchmarks to make accurate procurement decisions, since standard leaderboard rankings predict neither success rate nor cost inside hardened production environments.

Technical Capability: The finding that failure modes (timeouts vs. wrong solutions) differ by model under identical policy conditions reveals meaningful behavioral differences in how agents handle denied actions, which matters for agent design and harness engineering.

Regulatory & Legal: Security and compliance teams enforcing NIST SP 800-53 controls now have empirical evidence that those controls impose non-trivial and model-specific performance costs on coding agents, which must be factored into risk and budget assessments.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[3.3] Enterprises achieve measurable ROI and renew / expand, not just pilot.

[B1] Agentic reliability gains will translate into enterprise ROI.

Abstract

Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.