Generating Workflow DAGs from Natural Language with Non-Reasoning LLMs

2608.30250 ↗ · 2026-08-31 · cs.AI

Anand Iyer, Bhanu Khetharpal, Srinivas Upadhya, Ramkumar Rajagopal

The finding

Headline claim

A neuro-symbolic pipeline (LLM emits a compact IR, a deterministic compiler builds the JSON DAG, with a learned registry selector) lets cheap non-reasoning LLMs produce production-quality contact-center workflow graphs, statistically matching a reasoning model's out-of-the-box quality on the semantic metric.

The test

A 635-rule synthetic benchmark (programmatically templated natural-language rules, seven structural families, 1-15 branches, injected typos and non-English values) across four models (GPT-4.1, GPT-5.3-chat non-reasoning; GPT-5.2, Opus 4.6 reasoning) in four configurations (monolithic, selector, IR-only, selector+IR), all greedy decoding, judged by LLM-as-judge (Opus 4.6) plus judge-independent exact-match condition accuracy computed programmatically against ground truth.

Effect size & scope

GPT-5.3-chat judge validity rises 56.4 to 80.6 AI% and exact condition accuracy 56.5 to 82.2 Cond% under Selector+IR, a +24-point paired lift (p<10^-3) sufficient to statistically match GPT-5.2's monolithic quality within a pre-specified +/-5-point equivalence margin; JSON validity reaches 99-100% with 0% compile failures; prompt tokens roughly halved. The lift is conditional: it is small or negative for models that already emit dense structure well (monolithic Opus already 84.1 AI% / 87.2 Cond%), and an ~8-point frontier gap to the best reasoning model persists.

Limitations

The authors concede the benchmark is manufactured synthetic data from templates (proprietary, not real admin rules), that templated prompts do not isolate free-form paraphrase robustness, that an Opus judge scoring Opus generations raises self-preference bias (they argue it cancels or is conservative), and that the equivalence match is to a reasoning model's out-of-the-box monolithic quality, not to a reasoning model running the same pipeline (which reaches 88-88.8). An obvious unstated one: since ground truth comes from the same templates that generate the prompts, exact-match metrics may flatter systems tuned to this distribution.

Counterpoint

Single domain, single benchmark, vendor-affiliated authors (Microsoft) with a proprietary, non-released dataset, synthetic templated inputs that may understate real-world linguistic messiness, and the headline comparison pits a non-reasoning model with the new architecture against a reasoning model without it rather than applying the pipeline to both equally (and where both get the pipeline, the reasoning model still wins by ~6-8 points). The core finding is also well-anticipated by grammar-constrained decoding and IR-compiler prior work, so novelty rests mainly on the emission-density diagnostic.

Economy implication

For enterprises doing structured-generation tasks (workflow authoring, config, form-to-schema), cheaper non-reasoning models plus task-appropriate scaffolding can substitute for expensive reasoning models at production quality, directly lowering per-task inference cost; the paper's own numbers show this is domain-conditional, not a general capability result.

Who this matters for

Technical Capability: Localizes structured-generation failure to emission rather than interpretation (node-set accuracy 98-100% vs much lower node-exact), a transferable diagnostic for anyone building NL-to-structured-output systems on commodity models.

Labor & Knowledge Work: If cheap models can author complex routing workflows, the platform-expertise bottleneck for business-rule authoring narrows, shifting admin work from specialist configuration toward rule writing and review.

Market & Valuation: Microsoft shipping this in a commercial contact-center product is an early concrete instance of cheaper-tier models doing production-grade structured work, relevant to per-token pricing pressure on reasoning-model providers.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

This paper addresses the problem of translating natural-language routing rules written by business administrators into executable workflow graphs for enterprise contact centers. Each target is a directed acyclic graph (DAG) of conditional actions with parallel branches, hit-first fallback chains, and per-branch Boolean predicates, encoded in the JSON dialect of a commercial routing platform. We show that neuro-symbolic decomposition enables lower-cost, non-reasoning large language models to generate complex workflow DAGs at production-relevant quality without expensive extended-reasoning models. Our central diagnostic is an emission-density bottleneck: on a 635-rule benchmark of manufactured synthetic data, models select the correct graph nodes with high accuracy but increasingly misconfigure attributes and Boolean grouping as the number of interdependent nodes emitted in one pass grows. We therefore move combinatorial graph construction from the model into a deterministic compiler driven by a compact intermediate representation, with a learned registry-selection front end that focuses generation on relevant vocabulary. Across four models, the full system reaches approximately 89% LLM-judge validity, approximately 90% exact-match condition accuracy, and 99-100% valid JSON while using roughly half the per-rule prompt tokens of a monolithic prompt. On GPT-5.3-chat, the method improves judge validity by 24 percentage points and achieves statistical equivalence to a reasoning model's out-of-the-box quality, although an approximately 8-point frontier gap remains. We also present a deployment path and transferable lessons for structured-generation applications.