Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

2608.04828 ↗ · 2026-08-05 · cs.CL · h-max 5 · 0 citations

Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang, Zishang Jiang, Zixiang Di, Fanyang Lu, Zhichao Hu, Yanghua Xiao

The finding

Headline claim

Reliable use of progressively disclosed skills by LLM agents remains out of reach: the best model-harness configuration scores only 0.613 on the Skill-Use benchmark, with skill triggering and procedural compliance failing as separate bottlenecks, and skill use behaving as a harness-conditioned capability rather than a fixed model property.

The test

A new benchmark pairing 79 real skills from community repositories with 177 executable tasks across nine domains, each run in an isolated Docker sandbox with full tool access and scored by trajectory-based rubrics (1,314 rubric items, averaging 7.4 per task). Eight frontier LLMs (Claude Opus 4.7 and 4.8, GPT-5.5, DeepSeek-V4-Pro, GLM-5.1, MiniMax-M3, Kimi-K2.6, Qwen3.6-Max) were evaluated under two agent harnesses (Claude Code and Codex), with metrics decomposed into Trigger, Compliance, Boundary, and a gated SU score.

Effect size & scope

Best SU is 0.613 (GPT-5.5 under Claude Code); Trigger is near-ceiling for closed models (0.94-0.97 under CC) but drops to roughly a third for several open-weight models (DeepSeek-V4-Pro 0.324, Kimi-K2.6 0.337), while even on triggered traces the best Compliance is only 0.638. Boundary consistently exceeds Compliance across all configurations. Rankings shift with harness: GPT-5.5 falls from SU 0.613 under CC to 0.503 under Codex, and several mid-tier models rise. Effects hold across nine domains but the sample is small: one task per skill, 177 total.

Limitations

The authors concede results are harness-dependent, and they fix the Compliance-Boundary weighting (alpha 0.7) with sensitivity analysis deferred to an appendix. Unstated limitations: one task per skill limits statistical power and domain generalization; scores depend on LLM-as-judge (GPT-5.4) for items not expressible as deterministic checks; tasks and rubrics are author-constructed despite adversarial review; the alpha=0.7 weighting is a modeling choice that shapes the headline number; the 'tempting forbidden shortcut' design may or may not reflect real deployment distributions.

Counterpoint

A skeptic would note the benchmark is small (177 tasks, one per skill), judge-dependent where deterministic checks were impossible, and potentially prompt- or harness-sensitive by design, since the headline finding is itself that rankings flip with harness. The near-ceiling Trigger for top closed models suggests the interesting signal is concentrated in a narrow regime, and 0.613 versus, say, 0.5 has unclear practical meaning without a human or baseline-agent reference point, which the visible text does not provide. Benchmark authors running their own multi-agent construction pipeline have an incentive to show models failing the new capability their benchmark defines.

Economy implication

Little or nothing yet for the economics of AI directly. The relevant signal is indirect: if agents cannot reliably trigger and follow packaged procedures, the emerging skills ecosystem (Claude Code, Cursor, Codex) overstates near-term agentic productivity gains, and enterprise agent deployments will need human oversight or harness engineering, raising effective deployment costs. This is an early capability diagnostic, not a measured economic effect.

Who this matters for

Technical Capability: Agent builders get a concrete decomposition showing that retrieval/recognition (Trigger) and execution fidelity (Compliance) fail independently, and that harness choice moves scores as much as model choice, so harness engineering is a first-class lever.

Labor & Knowledge Work: If skills are the interface through which agents absorb workplace procedures, unreliable skill use implies agents cannot yet autonomously execute codified professional workflows without supervision, limiting near-term task substitution claims.

Market & Valuation: The finding that skill use is harness-conditioned rather than model-fixed weakens the moat story around any single frontier model, since packaged knowledge plus a good harness can close gaps that model rankings suggest exist.

In the Atlas

thread Agent reliability vs. the eval-real-work gap concept Agent concept Benchmark concept Tool use

Abstract

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.