Loom: Weaving Diagnostic Strands into Free-Text Consensus via Embedding-Space Reweighting

2609.02649 ↗ · 2026-09-02 · cs.AI, cs.CL, cs.LG

Ron Begleiter, Katya Egert Berg, Gilad Saban, Gil Shabat

Accepted to EMNLP 2026

The finding

Headline claim

A deterministic embedding-space consensus over templated heuristic outputs, followed by a single LLM synthesis call, achieves accuracy comparable to an iterative autonomous RCA agent on 2 of 4 OpenRCA datasets while using one LLM call instead of roughly 62 and being about 26x faster.

The test

Evaluation on the OpenRCA benchmark (Bank n=136, Market-1 n=70, Market-2 n=78, Telecom n=51) comparing Loom plus Claude 4.6 against the RCA-Agent baseline on strict and partial accuracy, LLM calls, and wall-clock time per incident; plus qualitative case studies from deployment in NVIDIA production datacenters and an ablation on the Bank dataset including an 8B synthesizer.

Effect size & scope

Loom matches the agent on Bank (38.97% vs 40.44% strict) and Market-2 (35.90% vs 35.90% strict) but trails on Market-1 (28.57% vs 40.00%) and Telecom (29.41% vs 41.18%); efficiency is 1 LLM call and about 22s per incident versus about 62 calls and about 567s, a roughly 26x speedup (about 33x with an 8B synthesizer, which drops Bank strict to 35.29% and hard-item accuracy to 23.53% vs 41.18%); oracle scores on the Loom slate reach 70% on Market-1 and Market-2, showing the consensus often surfaces the right candidate but the single-shot synthesis step fails to pick it.

Limitations

Authors concede accuracy gaps on Market-1 and Telecom, failure of static redundancy detection (negative result), dependence on embedding expressiveness, a cold-start problem requiring DS authoring, limits of single-shot synthesis, and partial reproducibility since DS catalogs include proprietary production heuristics. Unstated: tiny per-dataset n (51 to 136), accuracy measured with an LLM judge of unknown reliability, single benchmark, and DS catalogs that may be tuned to benchmark failure modes.

Counterpoint

With n as low as 51, the 10-point gaps on Market-1 and Telecom could be noise; the headline efficiency win partly reflects trading authoring cost upfront (expert-written or LLM-extracted heuristics) for inference savings, which is never costed; oracle numbers suggest the pipeline still needs a better selector, and deployment case studies are unverifiable anecdotes inside the vendor's own datacenter; the benchmark itself is from a related line of work and LLM-judge scoring adds prompt sensitivity.

Economy implication

Little or nothing yet for AI economics broadly; the narrow signal is that for well-scoped, high-frequency operational tasks, replacing agentic LLM loops with pre-compiled heuristics plus one cheap synthesis call can cut inference cost by an order of magnitude at modest accuracy cost, which matters for anyone pricing agentic incident-response tooling.

Who this matters for

Technical Capability: Direct evidence that much of an agent's measured value can be replaced by deterministic consensus plus a single LLM call, questioning where agentic depth is actually needed.

Market & Valuation: Relevant to AIOps and observability vendors: if a single inference call plus heuristics rivals a 62-call agent, inference cost is not the moat, and authoring expertise is.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

Aggregating noisy, conflicting textual hypotheses into a reliable consensus is a fundamental challenge when deploying NLP systems in real-world industrial settings. While monolithic Large Language Model (LLM) agents offer unbounded expressivity for tasks like Root Cause Analysis (RCA), they suffer from context limits, compounding hallucinations, and prohibitive inference latency. Traditional weak supervision offers statistical rigor but is mathematically restricted to discrete classes. We present Loom, a generative consensus framework deployed for real-world RCA that bridges these paradigms. Loom aggregates open-form hypotheses emitted by modular heuristics (diagnostic templates dynamically populated with episode-specific entities, times, and metrics) by projecting them into a continuous embedding space, and resolves conflicting signals with an iterative centroid-based reweighting algorithm. The resulting consensus weights ground a single lightweight LLM synthesis step. Evaluated on the OpenRCA benchmark, Loom occupies the accuracy--efficiency Pareto frontier: it matches a state-of-the-art autonomous agent on Bank and Market-2 and trails on Market-1 and Telecom, while using a single LLM call per incident on all four datasets ($\sim$26$\times$ faster; $\sim$33$\times$ with an 8B-parameter synthesizer). We discuss our deployment experience, highlighting lessons learned regarding the trade-offs between agentic depth and inference latency, negative results in redundancy detection, and how deterministic consensus fosters trust among Subject Matter Experts~(SMEs).