A Human-in-the-Loop Framework for AI-Assisted Scoring in Large-Scale Writing Assessment

An LLM with rubric-guided prompting can score Spanish argumentative writing in a high-stakes national exam with moderate-to-high agreement with human raters, and a human-in-the-loop workflow that routes only critical cases to experts can preserve pass/fail decision quality while cutting human grading workload.

Two annual editions (2024 and 2025) of Uruguay's Acredita EB national test, roughly 5,000 to 6,000 responses each (6,204 in 2024). 2024 data used to develop the prompt-based scoring model, 2025 used for validation. Short argumentative texts of about 150-200 words scored against a detailed analytic rubric; agreement measured against trained human raters at rubric-item level, plus impact on proficiency-level classification and pass/fail outcomes.

Agreement is described only qualitatively in the provided text as 'moderate to high... in most dimensions'; no specific kappa, QWK, or accuracy numbers appear in the excerpt. The workload-reduction estimate is a simulation under 'realistic deployment assumptions', not a deployed result. The framework is validated on one language, one exam format, one country, and short structured texts, the favorable case per the literature the authors themselves cite.

The authors concede results generalize poorly from English to other languages, that the HITL workflow is simulated rather than operated, and they flag unexamined cognitive bias introduced by AI-supported review. Unstated issues: the pass/fail decision-flow analysis depends on the multiple-choice sections that anchor final outcomes, so AI errors may be diluted rather than harmless; no results yet on longitudinal drift of model-human alignment; single-model, single-vendor dependence is not disclosed in the excerpt; no adversarial or prompt-injection testing reported.

This is an operational case study by the exam's own administrators (Ceibal and ANEP), not an independent evaluation; prompt-engineering results are notoriously sensitive to the exact prompt and model version, and both are not fully specified; the workload savings are projections from simulations; agreement metrics can look good while systematic under-grading of atypical writers, the equity risk the authors themselves cite, may still be buried in 'moderate' dimensions.

Little or nothing for the broad AI-economy debate in the next 12 to 24 months; the narrow implication is that public-sector assessment bodies in non-English markets can plausibly substitute LLM-assisted scoring plus selective human review for a large share of rater labor, a small but concrete example of task-level cost compression in knowledge work.

Labor & Knowledge Work: Shows a specific rater/teacher workload pool where AI assistance can genuinely displace human hours, while human review shifts toward exception handling rather than disappearing.

Societal & Cultural: Directly bears on fairness for high-stakes credentialing of adult learners, since under-grading by a model in a certification pathway is an equity problem, not just a metric.

Technical Capability: Evidence that current LLMs reach usable rater-agreement on short structured writing in Spanish via prompting alone, no fine-tuning or labeled training data, though only in that favorable regime.