Auditing MCQA Benchmarks through Probability Landscapes

2608.30372 ↗ · 2026-08-31 · cs.CL

Minsoo Song, Chanjun Park

Accepted to EMNLP 2026

The finding

Headline claim

A lightweight probability-landscape framework using model output distributions (P_top1, normalized residual entropy, MPD) plus noise injection can characterize MCQA benchmark structure and flag candidate flawed items for targeted human review, with signals that align with expert error annotations from MMLU-Redux.

The test

Four MCQA benchmarks (MMLU, ARC, HellaSwag, CommonsenseQA) with 1,000 randomly sampled items each, evaluated with three open-weight sub-10B instruction-tuned models (Qwen-2.5 7B, Gemma-2 9B, Llama-3 8B), plus scaling checks on Qwen-2.5 32B/72B and Llama-3.1 70B. Perturbations included token masking, no-answer injection, score bias, and distractor replacement with city names; flagged items were validated against MMLU-Redux expert labels.

Effect size & scope

Benchmark-level differences were modest and descriptive: mean H_norm ranged from 0.488 (CSQA) to 0.572 (ARC) and MPD from 0.404 (ARC) to 0.479 (MMLU). MPD shifted only slightly under positional-bias conditions (largest change about 0.047 on ARC), suggesting robustness to answer-position perturbations. Item-level noise-injection flags showed qualitative alignment with MMLU-Redux expert annotations; no precision/recall numbers are given in the provided text.

Limitations

The authors themselves concede the signals are not definitive evidence of benchmark flaws: probability metrics characterize structure, and noise-injection failures only prioritize candidates for human review rather than proving item defects. They also note failures may reflect limitations of the perturbation itself, and MPD is explicitly 'not a direct measure of benchmark quality'. Unstated limitations: only three small-scale models for the main analysis, 1,000-item samples per benchmark, no quantitative agreement statistics shown in the provided text, distractor replacement by city names is a crude heuristic, and the frameworks' value depends on frontiers models whose probability distributions may differ from the sub-10B models used.

Counterpoint

The method audits benchmarks using models known to be poorly calibrated, so 'residual competition' may reflect model confusion rather than item quality. Alignment with MMLU-Redux is shown but without effect sizes or agreement statistics in the main text, so the validation could be weak. The framework was tested only on saturated, well-studied benchmarks with small open-weight models, leaving open whether it scales to frontier models and harder benchmarks like GPQA. It is a triage tool, not an audit: it still requires human review of flagged items.

Economy implication

Little or nothing yet for the economics of AI in the next 12 to 24 months. If benchmark contamination and flawed items are as widespread as prior work suggests, cheap automated auditing could modestly improve the credibility of capability claims used in enterprise procurement, but this paper does not measure any economic outcome.

Who this matters for

Technical Capability: Researchers and labs relying on MCQA benchmarks get a cheap diagnostic for distinguishing genuine distractor competition from artifact-driven scores, which matters as benchmarks saturate.

Market & Valuation: Investors discounting benchmark-driven capability narratives get one more signal that headline MCQA scores can partly reflect dataset artifacts rather than reasoning gains.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While the community has responded by developing increasingly difficult datasets, validating question quality and filtering flawed items remains a labor-intensive process. To provide a scalable diagnostic approach, we propose a two-component probabilistic framework for auditing MCQA benchmarks using model output distributions. First, for benchmark-level analysis, we characterize the probability landscape using the top prediction probability ($P_{top1}$) and normalized residual entropy ($H_{norm}$), summarized globally by Mean Pairwise Distance (MPD). Second, for item-level diagnostics, we introduce noise injection to reduce meaningful distractor competition, enabling us to flag candidate items for targeted human review and categorize residual failure patterns. Across four MCQA benchmarks, our landscape analysis reveals benchmark-level differences in model confidence and residual option competition. Concurrently, our noise-injection method flags potentially actionable item-level issues, showing alignment with expert error annotations from MMLU-Redux. These results suggest that our probability-based framework provides a lightweight audit lens for comparing macro-level benchmark structure and prioritizing individual items for targeted human review.