Building a research-software catalog with a coding agent: from hackathon prototype to public deployment

2609.04711 ↗ · 2026-09-04 · cs.SE, cs.AI, cs.CY, physics.ed-ph

Kazuyoshi Yoshimi, Satoshi Terasaki, Gotai Yamada

23 pages, 6 figures, 2 tables

The finding

Headline claim

Turning a coding-agent-built research-software catalog from a hackathon prototype into a deployable public system required substantial additional engineering, and the most consequential failures were silent ones: plausible-looking but incorrect outputs from incomplete data acquisition, misleading assessments, and retrieval errors rather than crashes.

The test

An experience report from a three-day hackathon (June 2026) building a catalog of 68 research-software records via Claude Code, with adversarial review by Codex in fresh contexts (54 commits, 25 issues), plus an exploratory retrieval-augmented case study on the MateriApps portal (336 applications, ~49,500 indexed passages, crawl completed 3 August 2026).

Effect size & scope

Small scale and mostly qualitative: 68 catalog entries with readiness scores across seven binary signals; one concrete retrieval failure example (variational Monte Carlo queries also returning ordinary Monte Carlo software); no quantitative benchmark of agent performance, error rates, or time savings is reported.

Limitations

Authors concede the MateriApps work is exploratory and preliminary, the QA feature is disabled in public builds, the taxonomy is institution-specific, and their review process does not establish formal reviewer independence; unstated: n=1 project, no baseline comparison against purely human curation, no measurement of curation-effort reduction, and the corpus snapshot is tiny.

Counterpoint

This is an anecdotal experience report with no baselines, no metrics, and a 68-record sample; the claim that silent failures dominate is plausible but demonstrated through a handful of examples, and prompt/agent-specific choices (Claude Code plus Codex review) may not generalize.

Economy implication

Little or nothing yet for the AI economy in aggregate; the only signal is a suggestive one: agent speed in building software is real, but the verification and maintenance labor it forces may offset part of the claimed savings, a cost most ROI estimates ignore.

Who this matters for

Technical Capability: Documents that agentic coding failures cluster in silent, plausible-but-wrong outputs rather than exceptions, which matters for anyone assuming agent task completion equals task correctness.

Labor & Knowledge Work: Suggests AI shifts software-portal curation effort from writing code toward adversarial verification and data-quality review, though the paper quantifies none of it.

Bears on (advisory · papers never write evidence)

[3.7] AI coding tools reduce enterprise software headcount or external vendor spend faster than new AI-related roles or tooling costs offset the savings, producing net cost compression.

Abstract

Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackathon and subsequently examined the engineering required to make it suitable for public deployment, including adversarial review, data-quality checks, browser-level validation, and publication safeguards. We then explored whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal, through a retrieval agent under development for MateriApps that combines curated portal metadata, external documentation, vector search, and local language-model generation. Implementation with coding agents was rapid, but achieving reliable operation required substantial additional engineering: the most consequential problems were not crashes but silent failures that produced plausible yet incomplete or incorrect outputs, arising from incomplete data acquisition, misleading assessments, and retrieval or preprocessing failures. These observations suggest that AI-assisted software portals require explicit validation, monitoring, and repeated review, and that curated metadata and maintained documentation remain essential. The MateriApps work is exploratory and remains under active development, so the observations reported for it are preliminary; a comparable combination of curated metadata, automatically collected documentation, and retrieval-based assistance may nevertheless be useful for extending other research-software portals.