Qibai Chen, Zeming Liu
Accepted at the 38th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2026). 8 pages, 6 tables
Frontier LLMs systematically misapply specific documented version-constraint rules (e.g., partial-comparator carry, PEP 440 zero-pad prefix matching) despite handling their basic forms at ceiling, and the failures look like activation gaps fixable by delegating to a deterministic resolver, which reaches ~100% accuracy.
SemVerBench: 240 Boolean membership items across npm, PEP 440, and Cargo grammars, built author-neutrally from official resolver test suites plus three LLM proposers, labeled by a non-circular two-implementation code oracle (no human labels). Six frontier models (Claude-Opus/Sonnet, GPT-5.1/4.1/4o, Gemini-2.5-Pro) at temperature 0, three runs each, scored with Wilson CIs and McNemar paired tests; plus a 67-item prefix-match validation set and hint/rule-injection ablations.
Overall accuracy 80.1-90.6% vs a 57.1% majority-class floor. Blind spots: Cargo partial-comparator carry drops all six models to ~59-63% (at or below Cargo's 60% majority rate, i.e., no signal beyond guessing); PEP 440 prefix corner cases: GPT-5.1 scores 19% and GPT-4.1 38% on the 7-item bucket while Opus is 95%; on the separate 67-item zero-pad set GPT-5.1 is 0/26 on zero-pad cases while Claude stays at 97-100%. Rule injection recovers most errors and light correct hints fix previously-failed items; tool delegation reaches ~100%. Findings hold on official-suite and cross-family items.
Authors concede the design cannot fully separate activating latent knowledge from supplying missing knowledge; models are closed so the cross-vendor gap cannot be attributed to training causes; the proposers are the panel models themselves (mitigated by discarding suggested answers and an author-stratified analysis, which found no significant self-favoritism though cells are small); several headline buckets have tiny n (7 items for prefix match, 27 for Cargo carry); run-to-run spread (0.4-3.3 points) is treated as understating true uncertainty; a light hint effect may partly reflect prompt sensitivity rather than latent competence.
The benchmark's 240 items come half from the same frontier models being tested, and the sharpest headline claims rest on buckets of 7-35 items, so per-mechanism conclusions are fragile to sampling; hint-injection results invite the standard objection that any benchmark score is prompt-sensitive; 'delegate to the resolver' is a known engineering pattern, so the practical contribution is small; and n=240 across 9 mechanisms means per-cell power is thin even with McNemar on pairs.
Little or nothing for AI economics at the macro level; the near-term relevance is narrow engineering guidance for coding-agent builders (delegate version resolution to tooling, do not trust in-head reasoning) and a caution that newer-model recency does not predict competence on such narrow tasks, with no bearing on revenue, capex, or labor outcomes.
Technical Capability: Shows that frontier-model competence is mechanism-specific and predictable rather than diffuse, and that a vendor-specific blind spot (GPT-5.1 zero-padding at 0/26) coexists with overall strength, which matters for anyone building agents that touch dependency files.
Labor & Knowledge Work: Engineers evaluating coding agents for dependency-sensitive work should assume silent wrong-version decisions are possible even from top models and require resolver-based tooling in the loop.