What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities

2610.19476 ↗ · 2026-10-14 · cs.SD, cs.CL, cs.MM, eess.AS

Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu, Chih-Kai Yang, Yu-Xiang Lin, Sung-Feng Huang, Haibin Wu, Hung-yi Lee

Work in progress

The finding

Headline claim

Post-training gains on audio-reasoning benchmarks (MMAU, MMAR) mostly reflect learned multiple-choice format shortcuts and text priors rather than improved audio understanding: gains persist under silent audio and collapse when answer options are removed.

The test

Controlled evaluation of two base models (Qwen2.5-Omni-3B, Qwen2-Audio-7B-Instruct) after four post-training methods (SFT, RFT, GRPO, VCRL) on a 12,267-item audio MCQ pool, evaluated on MMAU and MMAR (1,000 items each) under real audio vs silence and multiple-choice vs open-ended formats; results compared against five open-source post-trained LALMs.

Effect size & scope

GRPO on Qwen2-Audio gives +14.8 on MMAU with audio but +11.2 under silence; R1-AQA shows +23.4 audio gain with +17.2 silence gain; AudioMCQ's silence gain (+6.4) exceeds its audio gain (+5.0). Models trained entirely on silence retain about half the MMAU gain (7.6 of 14.8). Open-ended evaluation nearly erases gains: Qwen2.5-Omni-3B GRPO drops from +8.0 (MCQ) to roughly +1-2 points open-ended.

Limitations

Small models (3B/7B), LoRA-only training (200 steps), two benchmarks (test-mini, 1,000 items each), LLM-judged open-ended scoring introduces its own noise, and one judge model (Qwen3.6-27B); authors label the work 'work in progress'. Unstated: results may not extend to larger frontier LALMs or speech-only production systems, and open-ended degradation partly reflects format mismatch rather than absence of audio skill.

Counterpoint

Open-ended rewriting of questions may make some items genuinely harder or unanswerable, so format-change collapse could overstate shortcut reliance; the judge-based scoring is itself sensitive to prompting; silence input is out-of-distribution for these models and could degrade audio-encoder behavior in ways that mask true audio use; the training pool is small and generic relative to full published recipes.

Economy implication

Little or nothing yet for the AI economy directly; the relevance is as a benchmark-validity caution: if multimodal post-training gains are partly format artifacts, reported capability improvements feeding vendor marketing and valuation narratives overstate deployable multimodal skill.

Who this matters for

Technical Capability: Practitioners deploying audio models should discount headline MMAU/MMAR gains and demand silence-control and open-ended ablations before trusting audio-reasoning improvements.

Market & Valuation: Investors relying on benchmark-score jumps as evidence of multimodal differentiation should note that a large share of these gains may be test-format gaming rather than capability moats.

Bears on (advisory · papers never write evidence)

[1.2] Benchmark gains are increasingly contaminated / saturated, overstating progress.

Abstract

Post-training often improves the performance of large audio-language models on audio understanding and reasoning benchmarks. However, these improvements may reflect adaptation to specific question formats, such as multiple-choice questions, rather than genuine advances in audio understanding and reasoning capabilities. To investigate what LALMs actually learn from post-training, we conduct controlled experiments that disentangle improvements in audio understanding from adaptation to specific question formats. Across different models and post-training methods, we find that much of the reported improvement persists even without informative audio, but largely disappears when the question format changes. These findings suggest that post-training often teaches models to exploit task-specific textual cues and answer patterns rather than developing generalizable audio understanding and reasoning capabilities.