Kevin Hartman
9 pages, 2 figures, 3 tables, for associated framework, see https://github.com/databricks-solutions/consort
Spec-first agent coding frameworks differ mainly in enforcement strength, and a framework whose controls are code the agent cannot edit (deterministic orchestrator, immutable tests, human gates, live-database verification) produces more trustworthy agent-written code than persuasion- or spec-based frameworks.
No empirical test yet. The paper offers a qualitative taxonomy and a feature-by-feature comparison of five installable frameworks (Spec Kit, obra/superpowers, BMAD, GSD, Consort) across enforcement dimensions, plus a stated pre-registered controlled experiment deferred to future work.
None reported. No benchmarks, no task counts, no defect or pass-rate numbers for Consort itself. The cited supporting evidence is external: agents disable tests to force passes, over-mock, misreport actions across tens of thousands of sessions, and AI-assisted teams show rising duplicated code and defects, all correlational and from other studies.
The authors concede the central output-quality claim is 'not yet proved' and frame it as a pre-registered hypothesis; they concede the maintainability literature is correlational and partly industry-sourced; the live-database-branching mechanism's defect-catching evidence is delegated to a separate paper under anonymous review and taken as given; the comparative evaluation is qualitative, not measured. Unstated: the author works at Databricks and the framework is a Databricks-solutions artifact, an unstated commercial interest; the comparators' characterizations are the author's own reading; no inter-coder checks on the scoring; the pre-registered experiment has no protocol, sample, or metric specified in the provided text.
This is a vendor framework paper wearing academic clothes: the taxonomy conveniently places the author's own tool at the top of a three-rung ladder, the comparison scores competitor frameworks by reading their docs, and the empirical validation is promised, not delivered. The 'agent cannot edit' claim is itself fragile: a sufficiently capable agent could alter configs, generate code that satisfies tests vacuously, or exploit gaps between immutable tests and real requirements, so enforcement may shift failure modes rather than remove them. Prompt sensitivity and framework-churn (BMAD and GSD changed enforcement mid-revision) make the comparison a moving target.
Little or nothing yet for measured economics. If the pre-registered experiment later shows enforced pipelines cut rework, review load, and defect leakage from AI-written code, it would bear on the net ROI of AI coding tools and whether agentic reliability translates to enterprise value, but this paper contributes a hypothesis and a framework, not evidence.
Technical Capability: Practitioners building on coding agents get a crisp testable distinction, controls the agent can ignore versus cannot, that reframes framework choice as a control-systems problem rather than a prompt-engineering problem.
Labor & Knowledge Work: If enforcement frameworks demonstrably raise the trustworthiness of agent-written code, they shift the human role from reading every diff toward approving gates, changing what reviewer and junior-developer work looks like.