Phillip Schneider, Alexandre Mercier, Joshua Oehms, Kristiina Jokinen, Florian Matthes
Accepted to EMNLP 2026
In a deployed voice assistant, lightweight embedding classifiers route fallback utterances to recovery actions as well as or better than small instruction-tuned LLMs while being 4x to 40x faster, and 70.6% of fallbacks are unintended activations best handled by suppression rather than generative recovery.
Six months of production logs from a smartwatch health assistant serving 500+ mostly elderly German users yielded 7,857 fallback utterances (about 6-7% of all requests), deduplicated to 3,030 anonymized utterances, annotated with a three-level taxonomy by four researchers (Fleiss Kappa 0.759 on a 350-utterance subset). They evaluated a two-stage pipeline: embedding models (E5, BGE-M3) plus LR/SVM/random forest versus Gemma 4 E4B, Qwen 3.5 4B, and GPT-4.1 nano in zero-shot and few-shot, with 5-fold cross-validation, reporting accuracy, F1, AUC, and per-utterance latency.
Stage 1 binary classification: E5+LR reached F1 0.852, accuracy 0.908, at 0.043 ms per utterance (plus ~40 ms embedding), versus Gemma 4 few-shot F1 0.818 at 229 ms, GPT-4.1 nano zero-shot F1 0.780 at 637 ms. The effect is on a single deployed German-language system with one 5-class label distribution dominated by unintended input; Stage 2 has tiny classes (6 multi-intent, 52 unspecific examples), so multi-class results are fragile.
The authors concede they did not evaluate the DIET retriever in isolation, that labels come from a majority vote of only four annotators on a 350-item subset with the full set labeled by one research assistant plus discussion, and that the deduplicated dataset understates unintended input relative to raw production frequency (Jensen-Shannon distance 0.119). Unstated: all utterances are single-turn, German, elderly-domain, from one product's skill set, so taxonomy and routing gains may not transfer; no end-to-end recovery outcome is measured, only classification.
The LLM baselines are small (4B) and evaluated zero-shot with temperature 0 and reasoning disabled, an intentionally handicap-prone setting, while the embedding classifiers benefit from a trained pipeline on the same data the deployed system generated; 'lightweight beats LLM' may say more about the task being near-duplicate/noise filtering than about LLMs generally. Prompt sensitivity is unquantified, and no fine-tuned LLM baseline is tried.
Little or nothing for the macro AI-economy debate. The narrow reading: for high-volume, latency-constrained edge deployments, targeted small models plus selective LLM use can beat blanket LLM adoption on both cost and quality, a small counterpoint to 'replace every NLU component with an LLM' procurement defaults.
Technical Capability: Production evidence that failure handling in deployed agents is heterogeneous and that small embedding models remain state of practice for intent routing, with LLMs reserved for generation.
Societal & Cultural: Documents the interaction failure burden for elderly and chronically ill voice-assistant users, where unsuppressed accidental activations directly drive frustration and churn.