DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

2609.03423 ↗ · 2026-09-03 · cs.AI

Puneet Mathur, Dinesh Manocha

Under Submission

The finding

Headline claim

Full-duplex voice agents follow explicit floor-management instructions better than behavior inferred from a persona, and content-level persona adherence is a separate capability from real-time floor control, with safety conflicts systematically un-resolved.

The test

A controlled benchmark, DSB-IFEval, of 1,038 test cases from 240 scripted conversations across eight behaviorally contrastive assistant roles (ER nurse, 911 dispatcher, grief counselor, etc.) and six probes, applied to six real-time speech systems (GPT-Realtime, PersonaPlex-7B, F-Actor, Moshi, MiniCPM-o-4.5, Fun-Audio-Chat-8B) under five conditioning levels (default, explicit instruction, persona-only, persona-plus-rule, conflict). Floor behavior scored by a deterministic Instruction Adherence Score; persona-consistent content by an LLM-judged Persona Adherence Score; synthetic user speech with programmatically timed barge-ins, pauses, and overlaps.

Effect size & scope

Under persona-only vs explicit-instruction conditioning, adherence drops by 9.7% for F-Actor and 4.5% for PersonaPlex, while GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat show strong persona content adherence but floor behavior that barely adapts between conditioning levels and stays constrained on proactive actions (backchanneling, interrupting). Under L4b safety conflicts, all six systems struggle to override explicit directives even when the persona-implied safety behavior should win. The paper gives headline deltas but the excerpt does not state IAS/PAS point values per system, so absolute performance levels cannot be verified from the provided text.

Limitations

Authors note in Limitations (section content not fully provided) and concede via design choices: synthetic TTS user speech, only eight personas and six probe structures generated from an LLM pipeline, deterministic verifiers with threshold sensitivity discussed in Appendix E.7, PAS relies on an LLM judge, only six systems evaluated. Obvious unstated ones: synthetic conversations may not capture real caller behavior, the persona-to-behavior mapping is author-designed so 'entailment' of floor policy from a persona is an assumption, and GPT-Realtime is a commercial system evaluated via API so internal architecture is unknown, weakening the architecture-dependent framing.

Counterpoint

This is a lab-authored benchmark under submission with no external validation; 'under submission' per the comment field, so no peer review signal yet. The Entailment Gap may partly reflect prompt-engineering artifacts: personas were constructed with explicit leakage controls (imperative lint, lexical-overlap gates), meaning any behavior that does transfer through a persona does so only via vibes the authors did not lint out. Six systems is a tiny n, per-condition samples are small (240 conversations split five ways), and LLM-judged PAS is prompt-sensitive. The finding that 'inferring, executing, and resolving are distinct' could also read as a definitional consequence of the benchmark's separation of scoring axes rather than an empirical discovery.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months. At most it flags a deployment risk for enterprise voice-agent products: agents configured by role/persona rather than explicit rules may not reliably enact role-appropriate conversational behavior or safety overrides, which matters for voice-agent vendors in support, dispatch, and healthcare triage. But the paper is a capability diagnostic, not an ROI, adoption, or revenue study.

Who this matters for

Technical Capability: Shows that instruction-hierarchy and implicit-instruction-following capabilities documented in text models do not automatically carry into full-duplex speech agents, and that architecture choices trade persona-content fidelity against floor-control flexibility.

Societal & Cultural: Safety-conflict failures, where systems follow explicit directives over role-implied safety behavior in roles like 911 dispatcher and ER triage, are a concrete red flag for deploying voice agents in high-stakes spoken interactions.

Abstract

Full-duplex voice agents must continuously decide when to listen, backchannel, interrupt, handle speech overlaps, take the floor, and yield. Existing benchmarks largely test these behaviors through explicit turn-management instructions, while deployed agents are often configured through roles or personas from which the appropriate conversational behavior must be inferred. We introduce DuplexSpeechBench-IFEval (DSB-IFEval) for evaluating implicit instruction-following in real-time spoken interaction. (DSB-IFEval) comprises 1,038 test cases spanning eight diverse assistant roles and evaluates five conditioning protocols for instruction-following: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona--rule conditioning, and instruction conflict. We measure real-time floor management using a deterministic Instruction Adherence Score (IAS) and persona-consistent content using LLM-judged Persona Adherence Score (PAS). Across six real-time speech systems, we find architecture-dependent trade-offs. Full duplex models like F-Actor and PersonaPlex are more sensitive to whether conversational behavior is stated explicitly or must be inferred from a persona, with adherence dropping by 9.7% and 4.5%, respectively, under persona-only conditioning. In contrast, GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly adhere to persona-consistent content, but their floor behavior does not adapt across explicit and persona-only instructions and remains constrained on several proactive actions. We further find that even if systems reliably follow conflicting directives to their prescribed persona, they still struggle to override them under safety conflict. These results show that inferring the behavior implied by a role, executing it at the appropriate conversational moment, and resolving competing instructions remain distinct challenges for full-duplex voice agents.