When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

2610.19345 ↗ · 2026-10-14 · cs.AI, cs.MA

Sajid Siraj, Maria Barbati

The finding

Headline claim

For agentic tasks, comparing models per call is misleading: measured per successfully completed task, the greenest (and cheapest/fastest) model changes with task length, from 1-3B parameter models for short tasks to 8-27B models for long ones, and crossover lengths predicted from per-step measurements match observed ones for most model pairs.

The test

A decision model taking the completed task (not the call) as the functional unit, with a Pareto screen then weighted choice; pilot study with 15 open-weight models from five developers (0.6B to 27B parameters) on two kinds of multi-step agent task, with directly measured GPU energy per attempt.

Effect size & scope

Per-call advantages shrink sharply with task length: in the motivating example a model with 95% per-step reliability vs 99% needs ~2.8 vs 1.2 attempts for a 20-step task, and a 10x per-call energy advantage falls to ~4x, reversing by 60 steps. The pilot reports the predicted three-regime structure and crossover agreement for most model pairs, but only on two task types and open-weight models up to 27B.

Limitations

The authors themselves note the pilot is small (two task families), the general model assumes a single per-step reliability, and results are from a pilot with an evaluation design for agentic benchmarks left as future work. Unstated limits: binary independence of step failures is likely false (correlated errors compound), energy measured on own GPU serving not commercial APIs, only open-weight models, no commercial frontier models tested, and success probability assumptions may not hold for retries on irreversible actions.

Counterpoint

This is a framework paper with a small pilot dressed as a validated result: two task types and 15 small open-weight models cannot establish that crossover predictions generalize, per-step reliability is probably not constant across steps or context states, and the motivating arithmetic (independent 95%^20) assumes exactly the structure the model asserts. No commercial APIs, no code-level baseline comparison against simple routing methods.

Economy implication

Little or nothing yet for the macro economics of AI; the useful near-term implication is micro-level: inference energy and cost per completed agent task can invert small-versus-large model rankings, which matters for inference cost curves at deployment scale if the framework holds up at frontier scale, which this paper has not shown.

Who this matters for

Technical Capability: Gives agent builders a concrete, testable rule for when a small open-weight model beats a larger one once retries and failed attempts are counted, directly relevant to right-sizing deployed agents.

Societal & Cultural: Reframes sustainable-AI comparisons by showing the per-call energy measure can invert the small-is-greener story, which matters for claims about the energy footprint of agentic AI.

Bears on (advisory · papers never write evidence)

[1.3] Agentic reliability (multi-step completion without human correction) is improving materially, not just peak single-shot.

[3.1] Inference cost per unit of capability is falling rapidly.

[5.2] Open-weight models reach "good enough" parity for most economic use cases.

Abstract

AI agents use large language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing language models in agentic AI.