SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology

2609.02292 ↗ · 2026-09-02 · cs.AI, cs.CL · h-max 4 · 0 citations

Ihor Stepanov, Aleksandr Smechov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov

20 pages, 10 tables, 6 figures

The finding

Headline claim

A lightweight 0.6B zero-shot router with a streaming decoder-KV classifier can select the best LLM per request across a changing roster and beat even the strongest fixed model on LiveBench subsets.

The test

Trained on an ontology of 23 task families, 115 task types, 345 subtypes, plus 150,000 verifier-scored and 15,000 judge-scored synthetic tasks; evaluated end-to-end on six LiveBench subsets and a selected 1,000-task subset with an 11-endpoint roster, using outcome-masked loss for unequal coverage.

Effect size & scope

Aggregate top-1 score of 0.707 versus 0.696 for the strongest fixed model, about 1.1 points, with benchmark-dependent gains; the margin over fixed models is thin and gains do not hold uniformly across benchmarks.

Limitations

The authors concede that semantic zero-shot transfer does not calibrate new endpoints without outcome data, that they avoid 11-way comparative accuracy due to unequal coverage, that LLM-judge scoring of open-ended tasks carries known bias, and that a rerouted destination may still need to replay conversation history so cache reuse does not make switching free. Unstated: the 1,000-task subset was 'selected', raising cherry-picking risk, and the margin (0.707 vs 0.696) is small enough that prompt or seed variance could erase it.

Counterpoint

A 1.1-point aggregate edge over the best single model is within the noise band for many benchmark comparisons, and the evaluation rests on synthetic, self-generated training tasks scored by verifiers and by gpt5.6-sol as judge, which invites benchmark gaming and circularity. Cost and latency gains from routing are asserted via architecture and policy design, not demonstrated with measured dollar savings or end-to-end latency numbers in the provided text.

Economy implication

If routing margins hold in production, the economic story is that buyers can mix cheaper open-weight and specialized endpoints per task, compressing the premium any single frontier vendor can charge. But with a ~1-point quality edge on a selected benchmark subset and no reported cost savings, the honest answer is: little or nothing yet for the economics of AI in the next 12 to 24 months; it is a plausible efficiency lever, not demonstrated ROI.

Who this matters for

Technical Capability: Shows model selection can be done with a 0.6B non-generative classifier in a streaming session, which is a real architectural contribution to inference-time orchestration.

Market & Valuation: Routers that make per-task model swapping cheap and calibrated would erode single-vendor pricing power, but this paper does not yet quantify cost or margin effects.

Bears on (advisory · papers never write evidence)

[5.3] API switching costs are low (customers multi-home and swap easily).

[3.1] Inference cost per unit of capability is falling rapidly.

Abstract

The rapid proliferation of large language models (LLMs) and the growing diversity of their applications presents a unique optimization opportunity: selecting the right model for the task, while optimizing for speed, cost, and quality at a per-task level. However, inference endpoints can vary widely in quality, price, latency, context support, tool use, domain expertise, and reasoning behavior. This heterogeneity makes manual heuristics difficult to maintain and unlikely to achieve consistently favorable speed--cost--quality trade-offs on their own. We introduce \router{}, a lightweight GLiClass-based router that assigns a suitability score to each inference-time model label without autoregressive generation. The released 0.6B-parameter checkpoint combines a Qwen3 decoder with a shallow bidirectional scorer. Its decoder-KV execution path preserves a text-only key--value cache across a session, encodes only new dialogue turns, and evaluates transient candidate-label tokens without adding them to the persistent cache. The same checkpoint also predicts task type, difficulty, reasoning mode, and expected output length, and supports custom zero-shot labels. For task generation, we construct a task ontology with 23 families, 115 task types, 345 routable subtypes, 1,173 synthetic examples, and an orthogonal axis of 30 domains. Using this structure, we generate 150,000 verifier-scored tasks and 15,000 open-ended tasks. We then train the Qwen3 decoder on these tasks, while explicitly separating learned request prediction from per-task policies for attributes such as eligibility, cost, cache reuse, safety, and sovereignty. Across six LiveBench subsets, the router outperforms the mean candidate; on the selected 1,000-task subset, it achieves an aggregate top-1 score of 0.707 versus 0.696 for the strongest fixed model, with benchmark-dependent gains.