VoiceLongMemEval: Do Assistants Remember How You Sounded?

2609.00570 ↗ · 2026-09-01 · cs.AI, cs.LG · h-max 7 · 0 citations

Ramit Pahwa, Parivesh Priye, Apoorva Beedu

The finding

Headline claim

Current LLM assistants cannot remember how something was said: long-term conversational memory benchmarks and pipelines discard paralinguistic information, and supplying acoustic metadata improves accuracy on delivery-dependent memory questions by +0.09 to +0.38 across eight models.

The test

A new benchmark (VoiceLongMemEval, 523 adversarially gated items built on LongMemEval) where each answer is recoverable only from emotion, prosody, and voice-event metadata attached to conversational turns, embedded in roughly 100k-token haystacks. Eight models (three proprietary, five open-weight) were evaluated blind (transcript only) versus descriptive (transcript plus acoustic stage directions) across three question families and three seeds, with paired bootstrap and McNemar significance tests; audio-native models were evaluated in a separate condition.

Effect size & scope

The affect gap is positive and significant (p < 0.001) for all eight models: Claude Opus 4.8 +0.383, GPT-5.5 +0.351, Claude Sonnet 4.6 +0.239, down to Gemma 3-12B +0.089, and the gap grows with model capability. Blind accuracy is uniformly low (0.09-0.18), indicating the adversarial gates work; a 72B gate confirmed 0 of 175 core items are blind-solvable. Audio-native models extract cues from speech directly (0.354-0.412 vs 0.325 blind), and hint prompting yields +0.61-0.69. The gap is largest for delivery-direct question types (affective-preference +0.613 for Opus) and smallest for cross-session integration.

Limitations

The authors concede: all audio is machine-generated (Dia TTS with acted RAVDESS voice references, no real recordings), the SER quality gate was demoted to advisory after ~30% accuracy on RAVDESS and only 91/104 (87.5%) clips passed human listen-through, and derived families (nuanced, indirect) were not separately blind-attacked. Unstated: the 'descriptive' condition hands models human-written natural-language stage directions, so the affect gap measures how well models use a clean annotation, not whether any production system could produce or retain such metadata; the adversarial gate used Qwen2.5-72B as both adversary and judge, so blind-unsolvability is relative to one model's ability; synthetic acted emotion may be far easier to recover than natural speech; results section in the provided text is truncated so audio-native and prompting-recovery analyses are only partially visible.

Counterpoint

This is a benchmark-construction paper evaluating models on a task the benchmark authors designed and synthesized: the metadata channel is a bespoke text annotation format no deployed assistant actually stores, so the 'gap' partly measures prompt-format sensitivity (structured tags revealing answers) rather than a real product deficit. Five question types per family with ~25-36 items each is small; LLM-judge scoring of free-text affective answers is itself noisy; and the corpus is synthetic dialogue with acted emotions, which may not transfer to real user speech. Code and dataset are 'available upon acceptance,' i.e., not yet independently verifiable.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months. At most it flags a design constraint for voice-assistant products: cascaded ASR-to-LLM pipelines silently lose paralinguistic signal, so vendors wanting affect-aware long-term memory must either adopt audio-native models or build annotation-retention layers, both of which add cost. No revenue, adoption, or labor implications can be drawn from a synthetic benchmark.

Who this matters for

Technical Capability: Establishes that paralinguistic long-term memory is a measurable, currently unsolved axis, and that audio-native models only partially close it (0.354-0.412 vs a +0.61-0.69 hint ceiling), giving speech-model builders a concrete target.

Societal & Cultural: If assistants are to interpret user emotion across sessions, benchmark evidence that current systems fail at this (blind accuracy 0.09-0.18) matters for expectations about emotionally aware AI companions and their failure modes.

Abstract

With the growing scale of multi-agent architectures and large language models, deployed AI assistants are increasingly tasked with reasoning over long, continuous, multi-session conversation histories. Current benchmarks evaluate this dialogue history as information retrieval over long horizon, temporal reasoning, or knowledge updates, while crucially ignoring the fundamental dynamics of human-agent interaction, i.e. how they said it. To address this gap, we present VoiceLongMemEval (VLME) benchmark, where every answer depends on paralinguistic metadata (emotion labels, prosody descriptors, and voice events) attached to conversational turns, which is otherwise unrecoverable from the words alone. Every item passes a three-stage adversarial gate, ensuring that a strong language model fails when given only the transcript. Evaluating leading frontier and open-weight models reveals a pervasive affect gap; providing text-track paralinguistic metadata yields a 0.09 to 0.38 accuracy boost (0.61 to 0.69 when prompted with evidence hints), while standard ASR pipelines systematically discard this signal. Additionally, audio-native models successfully extract these cues directly from speech (0.354 to 0.412 vs. 0.325 blind). Code and dataset will be made available upon acceptance.