Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe

Fine-tuning a 9B open-source model on execution-verified synthetic tool-calling data from live Korean public APIs raises multi-step tool-calling accuracy by about 10 to 13 points, letting the 9B model nearly match its untuned 27B sibling, with gains generalizing to the held-out BFCL benchmark.

145 real-world multi-step tasks over 10 live Korean public API platforms and 2,318 MCP tools, evaluated on pass@1/pass@4 plus tool-call accuracy across 12 models, with EDGE synthetic data (1,781 trajectories) used to GRPO-train Qwen3.5-4B and Qwen3.5-9B on 8x H100s; statistical significance via 8 re-evaluation seeds.

KOPA-Bench pass@1: 4B 0.1758 to 0.3094 (+13pp), 9B 0.3275 to 0.4310 (+10pp), versus Qwen3.5-27B at 0.4482; 95% CIs from 8 seeds do not overlap between tuned and base models; held-out platform tasks show +22.6pp pass@4 for the 4B model; BFCL multi-turn gains of +4 to +6pp. All effects confined to small open-source models in this domain.

Authors concede reliance on live endpoints (availability and nondeterminism), scope limited to Korean public APIs, and limited scope of baseline comparison. Unstated: benchmark of only 145 tasks authored and audited by the same organization that produced the training method, evaluation dimensions partly scored by LLM judgment, tuning and benchmark drawn from the same API ecosystem, and base pass rates so low that gains partly reflect fixing brittle baseline behavior.

A 145-task benchmark with frontier models scoring only 0.37 to 0.47 pass@1 suggests the tasks may be idiosyncratic or harshly graded; the headline '9B matches 27B' holds only on their own benchmark, where the tuned 9B still trails the 27B on pass@1; training data and benchmark both derive from the same tool graph, so generalization to real government deployments is untested; proprietary-model verification in the validation loop adds a subtle distribution dependency.

Little or nothing for aggregate AI economics yet, but a concrete data point that cheap domain-specific synthetic-data pipelines can close capability gaps between open-source model sizes, which supports the data-sovereignty niche where regulated institutions must run on-premise open models instead of commercial APIs.

Technical Capability: Shows execution-grounded data synthesis plus GRPO on verifiable outcomes reliably lifts small open-weight models on genuine multi-step tool use, not just static benchmarks.

Regulatory & Legal: Demonstrates that data-sovereignty mandates pushing agencies to on-premise open-source agents are becoming technically feasible, at least for a narrow domain.

Labor & Knowledge Work: Weak but real signal that niche government integrator work (building and verifying tool chains) is now partially automatable by a 9B model.