A May 2026 working paper from Esposito and Zhang (arXiv) formally models benchmark signal depreciation, arguing that as frontier AI models saturate easier evaluation items, discriminating signal concentrates in a hard tail of items that require elite expert judgment to construct, judgment that is structurally scarce and convexly expensive to reproduce. The paper documents that contamination accelerates this process by disproportionately inflating scores on easy items, leaving the hard tail as the primary source of valid capability discrimination. The paper further argues that private benchmark producers systematically underinvest in validity relative to the social optimum because they cannot appropriate the full governance value of rigorous evaluation infrastructure. For financial institutions, this structural argument implies that frontier benchmark scores are increasingly unreliable as signals of latent model quality, complicating any due-diligence or procurement process that relies on published benchmark rankings to compare AI vendors or assess capability claims.
469 other tracked developments touching the same claims.