Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

2609.08765 ↗ · 2026-09-08 · cs.CR, cs.AI, cs.CL

Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf

The finding

Headline claim

Cybersecurity LLM benchmark scores are properties of the evaluation pipeline as much as of the models: single pipeline choices can shift a model's score by more than 80 percentage points and reorder rankings.

The test

A systematic audit of eight cybersecurity benchmarks (48,662 questions, 23 tasks) run against 10 LLMs (proprietary, open-weight, and security-specialized), modeling each benchmark as a five-stage measurement pipeline (dataset, prompt, inference, extraction/scoring, aggregation) and applying controlled perturbations to nine pipeline configuration fields plus a standardizing harness and cross-benchmark rank-agreement analysis (PCA, Kendall's tau-b).

Effect size & scope

15 failure modes documented across all pipeline stages; maximum effects include format-token leakage of 90.9 pp, stop-sequence mismatch recovering 85.9 pp for Qwen3.6 on RedSage-Bench, denominator inflation up to 99.8 pp, and task-level metric drift with median 33.5 pp. Semantically similar task pairs in CTI-Bench and AthenaBench disagree on rankings (Kendall's tau-b 0.29 and 0.24), and under a standardized harness nine of 10 models shift by at least three ranks on at least one benchmark. The first PCA component explains 95.25% of variance, indicating heavy task redundancy. Scope: failure rates are specific to these cybersecurity benchmarks, per the authors.

Limitations

Authors concede the perturbation isolation is opportunistic (some fields cannot be varied independently), the harness is a diagnostic tool not a universally correct evaluator, and observed failure rates may not generalize beyond cybersecurity. Only one subset of SecBench (661 of 47,910 reported questions) could actually be scored, and gold-label correctness was spot-checked on only 998 flags. Unstated: 10 models is a small model sample, and the standardized harness embeds the authors' own choices, which could introduce their own biases.

Counterpoint

A skeptic would note the audit mostly shows that badly specified open benchmarks and reasoning models with verbose preambles interact poorly, which prior work already anticipated; the headline 80-plus-point shifts come largely from pathological single cases (one model, one benchmark) and may say little about typical frontier model comparisons. The standardized harness is the authors' preferred configuration, so rank shifts under it partly measure their own choices. Domain-specific and vendor-reported benchmarks may be more careful than the eight audited ones.

Economy implication

Little or nothing for the AI economy in the next 12 to 24 months directly. The indirect relevance is real but modest: enterprise model-selection and procurement decisions that lean on domain leaderboards, especially in security, may be buying pipeline artifacts rather than capability differences, which argues for in-house evals over leaderboard trust. It does not by itself establish anything about ROI, revenue, or labor effects.

Who this matters for

Technical Capability: Benchmark-reported model rankings, especially for specialized fine-tunes, should be read as pipeline-conditional, so capability claims need standardized-harness replication before being trusted.

Market & Valuation: Buyers pricing cybersecurity AI tools off leaderboard positions may be systematically mispricing model quality, since rankings can flip three or more places under equivalent evaluation conventions.

Regulatory & Legal: If regulators or standards bodies reference public benchmark scores as evidence of model fitness, this audit shows those numbers lack the measurement reliability such decisions assume.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.