MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

2609.10016 ↗ · 2026-09-09 · cs.LG, cs.AI, cs.CL

Remco Hendriks

23 pages, 5 figures, 10 tables. Code and data at https://github.com/continker/metrollm-bench (tag paper-v1.2); DOI 10.5281/zenodo.21893944

The finding

Headline claim

A 2.6 GB, 4B-parameter open-weight model fine-tuned on a narrow transit-kiosk task set outperforms both GPT-5.6 tiers on deterministic task scoring (91.3 vs 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), and fine-tuning gains shrink monotonically as base model size grows, turning negative at 27B.

The test

A 955-case benchmark (238 held-out) for LLMs acting as transit-kiosk policy agents across six metro systems and eleven categories, with a ReAct tool loop and machine-checkable terminal states; 26 models from 6 vendors evaluated, 23 ranked; a QLoRA PEFT sweep at 2B/4B/9B/27B with 2-3 seeds each (9.4 GPU-hours total); scoring calibrated against 100 blind human case-rubric ratings.

Effect size & scope

Held-out Tier 1: 4B PEFT student 91.32 vs GPT-5.6 sol 90.00 and luna 90.63, GPT-5.4 full xhigh 91.37; PEFT delta over base falls from +7.03 at 2B to -0.91 at 27B, same direction across every seed; top eleven rows span only 3.18 composite points; a deterministic rule-based baseline reaches 84.6 Tier 1, leaving LLM advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning; serving configuration alone moves one comparison by 2.7 points.

Limitations

Authors concede: 40 percent of held-out cases have a same-origin-destination training neighbour, and held-out cases share structural templates with training cases; per-system pretraining familiarity is unmeasured; 100-pair judge calibration with a skewed-marginal second annotator; cross-judge calibration with a non-Anthropic judge left as future work; cases are designed scenarios, not exhaustive of real operations; composite ranking mixes tiers and includes judge-scored Tier 2. Unstated: single-author paper from a company (Continker) with a transit-kiosk product angle, tiny held-out n=238, and the narrowness of a single vertical limits generalization of the capacity-ceiling finding.

Counterpoint

The 91.3-vs-91.4 gap is inside serving-configuration noise (2.7 points from configuration alone) and bootstrap spread, so the 'beats GPT-5.6' headline is thin; PEFT students may exploit template overlap between train and held-out splits that frontier models, evaluated cold, cannot use; the negative PEFT delta at 27B may reflect training-pipeline saturation (600 examples, 3 epochs) rather than a fundamental ceiling; no non-Anthropic judge on Tier 2 raises vendor-dependence concerns; the benchmark is one synthetic domain with generator-derived ground truth.

Economy implication

Little for the macro AI-economy debate on its own, but it is a concrete data point for the narrowing-app-vs-frontier thesis: in a rule-heavy, tool-bound vertical, small fine-tuned open models at 2.6 GB match frontier API performance, which if replicated across other narrow domains would shift inference spend away from frontier APIs toward fine-tuned local runtimes and pressure frontier-API pricing in commodity verticals.

Who this matters for

Technical Capability: Direct evidence that PEFT on a narrow agentic task makes small open-weight models competitive with frontier APIs, and that fine-tuning utility declines with base capability, a constraint on the 'just fine-tune the frontier model' playbook.

Market & Valuation: Supports the commoditization-in-narrow-verticals thesis: an Apache-2.0 4B student matching top-tier API models at kiosk-scale tasks weakens frontier-API pricing power in rule-bound enterprise workloads.

Bears on (advisory · papers never write evidence)

[5.2] Open-weight models reach "good enough" parity for most economic use cases.

[3.1] Inference cost per unit of capability is falling rapidly.

[5.1] Frontier models cluster tightly (small gaps, fast catch-up).

Abstract

We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at https://github.com/continker/metrollm-bench.