Calibrating Small Language Models for Claim Check-Worthiness Detection

2608.30731 ↗ · 2026-08-31 · cs.CL, cs.AI

Pratuat Amatya, Venktesh Viswanathan, Vinay Setty

The finding

Headline claim

A post-hoc nearest-neighbor residual calibration layer (NN-PPI) using only a small human-labeled calibration set lifts small language models' claim check-worthiness F1 to near frontier-LLM levels without retraining the model.

The test

Two datasets (ClaimBuster 2016 debates, n=2,740 test; CLEF 2024 CheckThat English, n=317 test) with calibration sets of 1,314 and 2,406 labeled claims; models span Gemma 3 270M/1B/4B served locally, a production fine-tuned XLM-RoBERTa-Large, and GPT-5.2 and Claude Opus 4.6 via API; NN-PPI (k in 3, 5, 10 neighbors, MiniLM embeddings, ChromaDB) compared against uncalibrated few-shot baseline and plain KNN averaging using weighted and classwise F1.

Effect size & scope

Largest gains on the weakest models: Gemma 3 270M goes from 0.114 to 0.721 weighted F1 on ClaimBuster and 0.179 to 0.827 on CLEF; Gemma 3 4B gains +33.8% (0.568 to 0.760) and +20% (0.688 to 0.827); the fine-tuned XLM-R gains about +11% on CLEF. Gains shrink or vanish where baselines are strong: frontier models gain at most ~5% on CLEF and near zero or slightly negative on ClaimBuster. NN-PPI beats plain KNN in most but not all conditions (Gemma 1B on CLEF at k=10 is an exception). CI coverage is under nominal 95% overall (46.6% to 68.5% depending on k), with check-worthy class under-covered.

Limitations

Authors concede the residual correction is unconstrained so scores can fall outside [0,1], that KNN alone can match NN-PPI when the base model is already well-calibrated, and that confidence intervals systematically under-cover, especially for the check-worthy class; test sets are small (CLEF test is only 317 items, 107 check-worthy) and English-only, datasets are debate/social-media style rather than modern web claims, and results depend on fixed few-shot prompts and a fixed threshold of 0.5. Unstated: two datasets only, no significance testing reported, frontier-model comparisons at possibly differing prompt quality, and the calibration set itself requires human labeling effort that the cost claim does not price.

Counterpoint

A skeptic would note the headline gains come almost entirely from correcting severely miscalibrated or degenerate baselines (a 270M model predicting one class, a 4B model over-predicting positives), which a simple threshold shift or plain KNN could partially fix; NN-PPI adds little over KNN in several cells and nothing over strong frontier baselines, so the real finding is that weak SLM baselines were badly biased, not that residual calibration recovers frontier capability generally. Small n (317 CLEF test items) makes the fine-grained comparisons fragile, and reported CI coverage failure undercuts the paper's own uncertainty guarantees.

Economy implication

For the next 12 to 24 months the practical implication is modest but real: for narrow, well-instrumented classification tasks in production (fact-checking triage here), cheap small open-weight models plus a small labeled calibration set and a lightweight post-hoc layer can approach frontier API quality at a fraction of serving cost, which supports selectively substituting small local models for API calls in cost-sensitive pipelines. It says little about general agentic or reasoning workloads, and the authors provide no serving-cost or latency measurements, so the 'order of magnitude cheaper' framing is asserted, not measured.

Who this matters for

Technical Capability: Shows that per-instance residual calibration from a few hundred labeled examples is a cheap complement to fine-tuning, shifting where capability-per-dollar comes from: model scale versus calibration data and retrieval plumbing.

Societal & Cultural: If small calibrated models match frontier LLMs on check-worthiness triage, fact-checking and content-moderation pipelines become viable for low-budget organizations and non-English-scale newsrooms without API dependence.

Market & Valuation: A concrete data point that narrow app-layer workloads can substitute small open-weight models for frontier API spend, one mechanism behind pressure on per-token API pricing.

Bears on (advisory · papers never write evidence)

[3.1] Inference cost per unit of capability is falling rapidly.

[5.2] Open-weight models reach "good enough" parity for most economic use cases.

Abstract

Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.