Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
11 pages, 2 figures, 5 tables
A substantial share of LLM-generated repairs that pass functional tests fail to satisfy review-derived acceptance constraints, so functional-only benchmarks like SWE-bench overstate agents' real repair capability.
SWE-Gate: 303 synthesized repository-level repair instances across 75 Python repositories and six domains, each with separate executable functional and constraint tests; four LLM backends run under a common Mini-SWE-Agent scaffold.
Among 644 repairs that passed functional tests, 221 (roughly 34%) failed the review constraints; the paper reports per-model and per-constraint-category breakdowns but the abstract does not give per-model numbers.
Instances are synthesized via LLM-driven constraint extraction and cross-repository transfer rather than drawn directly from real issue-PR pairs, with LLM screening before human review; the paper concedes transferability must be verified per instance and that contamination is mitigated, not eliminated; the text is truncated so the final human-review results and per-model numbers are not visible here.
Benchmark instances are LLM-generated and possibly harder or more contrived than real review requirements; the constraint guidance framing may bias what agents are expected to do; a 303-instance benchmark with synthetic mutants may not reflect on-the-job code review rejection rates; per-model baselines and significance are unclear from the text shown.
If agent deployments are judged by whether patches are accepted by human reviewers rather than by test pass rates, realized productivity gains from coding agents may be materially lower than SWE-bench-style scores suggest; this is a qualification on ROI claims, not yet direct evidence about revenue or headcount.
Technical Capability: Shows that agentic coding progress measured on functional tests only may systematically overstate real-world task completion, a directly relevant correction to how capability is tracked.
Labor & Knowledge Work: Enterprises counting on agents to reduce code-review or engineering headcount should discount test-pass-based performance claims until acceptance-rate evidence exists.