Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

2609.05385 ↗ · 2026-09-04 · cs.AI

Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri

The finding

Headline claim

LLM self-reported top-three explanations only partially agree with the model's actual decision behavior: cited-feature rankings correlate weakly with intervention-based necessity and sufficiency scores, and in advisor recommendation an uncited feature often beats the weakest cited one.

The test

Black-box interventions on 100 synthetic client profiles (18 features, 13 advisors) and 100 synthetic risk-monitoring prompts across eight models from Claude, GPT, and Gemini families at temperature 0. Necessity score: how often changing one feature flips the output (3 trials). Sufficiency score: how often retaining only that feature preserves the output. Spearman correlation of stated ranking vs. intervention scores, plus comparisons of highest-scoring uncited feature against lowest-scoring cited feature; 13 prompt-elicitation formats compared first, fixed feature names selected for stability.

Effect size & scope

Mean Spearman correlations across models: 0.349 (necessity) and 0.354 (sufficiency) for advisor recommendation; 0.431 and 0.580 for prompt monitoring; model-level means span 0.014 to 0.755. Uncited features outrank the weakest cited feature in 57.6% of advisor responses under necessity and 58.1% under sufficiency, but only 25.8% and 8.9% in prompt monitoring. Citation frequency correlates strongly with measured influence in advisor recommendation (rho 0.903 necessity, 0.808 sufficiency) but less so in prompt monitoring (0.690, 0.500).

Limitations

Authors scope the work explicitly to individual LLM decisions, not multi-step agent behavior, and note the elicitation format choice is for reproducibility, not evidence of faithfulness. Synthetic datasets only (both generated partly by Claude Opus 4.6), 100 samples per use case, three intervention trials per feature, only two narrowly defined use cases, fixed-vocabulary feature names that may not reflect how models explain freely, and results are exploratory for full rankings. Unstated: findings may not generalize to realistic inputs, and intervention definitions themselves change absolute scores substantially.

Counterpoint

Synthetic, self-generated data with small n; the choice of necessity intervention (contrast substitution vs. blank removal vs. similar-value substitution) materially changes scores, so the ground truth being compared against is itself intervention-dependent. Prompt sensitivity literature the authors cite cuts both ways: a different elicitation format could show much better or worse calibration. A correlation of 0.35 to 0.58 is partial but nonzero signal; the negative framing assumes operators interpret the top three as exact causal ranks, which they may not.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months directly, though it complicates one premise of agent-driven enterprise ROI: if operators use model self-explanations to monitor and escalate, unreliable citations could raise the human-oversight cost of agentic deployments rather than lower it.

Who this matters for

Technical Capability: Shows that even frontier models' self-explanations diverge measurably from their causal decision drivers, a concrete faithfulness gap for anyone building oversight or evaluation on stated rationales.

Societal & Cultural: Directly relevant to auditability: regulators and reviewers who trust cited risk factors (e.g., jailbreak over harmful_request) may be misdirecting attention, which matters for any governance regime relying on model-provided justifications.

Abstract

LLM decision components that can operate within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use the named factors to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes that the explanations agree with the component's observable decision behaviour. We test two interpretations of the named factors: necessity, meaning that changing a factor would change the output, and sufficiency, meaning that retaining it while removing other changeable information would preserve the output. We evaluate these interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Models return an output and the top three factors that most influenced it. Controlled black-box interventions estimate a necessity score for each factor by measuring how often changing it changes the output, and a sufficiency score by measuring how often retaining it preserves the output. Across eight models from the Claude, GPT, and Gemini families, the mean Spearman correlations between the cited ranking and the necessity and sufficiency scores are 0.349 and 0.354 for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. Furthermore, an uncited factor scores above the lowest-scoring cited factor in 57.6% of advisor responses under necessity and 58.1% under sufficiency; the corresponding prompt-monitoring rates are 25.8% and 8.9%. The cited top three contain useful information but do not reliably identify the three factors with the strongest measured influence under necessity or sufficiency. The framework provides a black-box reliability check for explanations used in agent oversight while remaining scoped to individual LLM decisions.