Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

2609.00549 ↗ · 2026-09-01 · cs.CL · h-max 14 · 0 citations

Seonghyeon Cho, Chanjun Park

Accepted to Findings of EMNLP 2026

The finding

Headline claim

Standard aggregate retrieval-lift metrics systematically overstate whether retrieved skills actually help LLM agents: many models show positive aggregate lift while their same-task paired effect on retrieved tasks (RAE) is negative.

The test

Paired skill-enabled vs skill-disabled executions of 17 LLMs on 80-task samples of MBPP+ (seeds 42/43/44), HumanEval+, and Math500, with a BM25 skill-retrieval tool (9 coding skills, 8 math skills, top-3 returned); sign agreement between aggregate retrieval lift and RAE compared per model, plus perturbed-skill controls and GPT-5.5-based adherence annotation on reversal models (DeepSeek-V3.2, Gemini-2.5-Flash-lite).

Effect size & scope

Sign reversals recur across MBPP+ partitions and on 3 of 13 models on HumanEval+. Gemini-2.5-Flash-lite shows positive aggregate lift on all three MBPP+ partitions (e.g., +6.2, +16.2, +4.4) but strictly negative RAE (e.g., -15.6, -15.9, -22.2), and -0.1 aggregate with -21.1 RAE on HumanEval+. Qwen3.5-9B on MBPP+ s44 shows -36.4 aggregate lift with -4.1 RAE. Effects are confined to small procedural skill libraries on short coding/math tasks.

Limitations

Authors concede RAE is conditional on each system's self-selected retrieval subset, making cross-model RAE values non-comparable; adherence labels come from an LLM annotator with only a stratified human audit; small task samples (80 per partition); fixed tiny skill pools and a single BM25 retrieval configuration (they add sensitivity checks in an appendix). Unstated: results may not transfer to richer skill libraries, larger task suites, or agentic settings where retrieval is more central; paired same-seed runs add noise from decoding nondeterminism in API models.

Counterpoint

A skeptic would note n=80 tasks per benchmark with 8-9 hand-written skills is a thin basis for 'pervasive' metric reversal claims; the negative-RAE phenomenon may be an artifact of injecting three full skill documents into contexts tuned for short MBPP-style problems rather than a general skill-following failure; no comparison to alternative remedies (e.g., shorter skill summaries, better retrieval ranking); and aggregate lift plus RAE measure different populations, so 'sign disagreement' partly reflects that the retrieved subset is small and easier, which the authors themselves demonstrate.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months. At most, it is a caution that agent-product benchmarks reporting aggregate tool-use gains may not predict realized reliability of retrieval-augmented features, which matters only if validated at production skill-library scale.

Who this matters for

Technical Capability: Evaluation builders and agent developers get a cheap paired protocol that can reveal when a retrieval pipeline is hurting exactly the tasks it triggers on.

Market & Valuation: Investors pricing agent-platform claims should discount benchmark-level tool-use lift until vendors report same-task conditioned effects.

Abstract

Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.