Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

2608.28432 ↗ · 2026-08-28 · cs.CL, cs.AI, cs.DB

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, Feiran Huang

The finding

Headline claim

In ICL text-to-SQL pipelines, execution-feedback refinement is the only module whose accuracy gain holds universally across backbones at low cost, and a fixed inference budget is usually better spent building a richer pipeline on a mid-tier backbone than upgrading to a frontier model with a lean pipeline.

The test

Controlled single-swap ablations of 17 paradigm-level configurations across five modules on BIRD dev (1,534 questions, 11 databases), with four primary backbones (GPT-4o-mini, Gemini-2.5-Flash, DeepSeek-V4-Flash reasoning, GPT-5.4), plus Qwen3-235B and o3-mini for reasoning-behavior analysis, Spider (1,034 samples) for transfer, and five additional backbones for guideline validation; costs measured as USD per accuracy point (CPP) plus input/output token counts, with McNemar's tests at alpha 0.05.

Effect size & scope

The paper's headline qualitative result is that most modules help only under backbone-dependent conditions while R1 execution-feedback refinement Pareto-dominates or helps universally; leaderboard accuracy reportedly rises as modules stack while marginal cost per point grows disproportionately, with agentic refinement and multi-candidate sampling consuming several times up to an order of magnitude more tokens than a single pass. The excerpt gives no exact EX deltas or CPP dollar figures, so effect sizes cannot be quantified from the provided text.

Limitations

Authors concede aggregate-only accuracy reporting obscures module attribution and that gains compose differently by backbone capability; the design-space is bounded by one task (text-to-SQL), benchmark dev splits, and their own re-implementations rather than the original published systems, so numbers may not match cited leaderboard results; API pricing varies and token counts may not transfer as deployments scale or prices change.

Counterpoint

Controlled re-implementations of paradigms can under- or over-state the performance of the actual published systems they represent (paradigm != method); BIRD and Spider are saturated benchmarks where leaderboard chasing is a known distortion; 'frontier vs. pipeline' conclusions depend heavily on the specific model snapshot and API prices at test time; costs measured as API tokens ignore engineering, latency, and failure-handling costs that dominate real deployments.

Economy implication

Little or nothing yet for aggregate AI economics: this is a narrow-benchmark engineering study. Its mild relevance is directional evidence for the inference-cost-curves story, that accuracy bought via inference-time scaffolding carries steeply rising marginal token cost, implying enterprises should weigh pipeline engineering against model-tier spend rather than default to frontier models.

Who this matters for

Technical Capability: Practitioners building LLM pipelines get an evidence-backed ordering of which inference-time modules (refinement, selection, linking) pay for themselves versus which are backbone-dependent padding.

Labor & Knowledge Work: Weakly speaks to knowledge-work tooling economics: it suggests cheaper models plus better orchestration can substitute for premium model spend in structured data-query tasks.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

[1.4] Inference-time compute yields gains that don't saturate quickly on hard reasoning.

Abstract

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.