Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models

2609.03422 ↗ · 2026-09-03 · cs.LG, cs.MA

Ross Tieman, Evan Markou

31 pages, 13 figures

The finding

Headline claim

A compression-based measure of generative-process diversity between model pairs, residualised against a permutation control, predicts chance-corrected correlated wrong-answer agreement across ten benchmark families even after controlling for semantic similarity and capability.

The test

38 language models produced repeated responses to 100 open-ended Infinity-Chat prompts; pairwise Normalised Compression Distances (permutation-residualised, PPMd compressor) were computed for all 703 pairs and regressed against chance-corrected correlated wrong-answer kappa on ten disjoint closed-form benchmarks (TruthfulQA, MMLU-Pro, GSM8K, HLE, etc.), each with five sampled epochs per model, using partial Spearman correlations with a model-node bootstrap (2,000 replicates).

Effect size & scope

Cross-benchmark mean partial rank association of -0.216, 95% interval [-0.309, -0.122], negative on all ten benchmarks. The effect is a moderate conditional association within this specific 38-model population; it is a pairwise, cross-task prediction within evaluated models, not a general law, and the measure itself is signed and not a true metric.

Limitations

The authors concede the residual does not remove all low-order statistics or identify any mechanism, that results are conditional on the sampled prompts, benchmarks, and model population, and that the analysis covers text-only input-output behaviour without agentic scaffolding or tools. Unstated: the open-ended prompts and benchmarks come from overlapping research lineages (Artificial Hivemind taxonomy), the effect size is modest, and out-of-population generalisation is untested.

Counterpoint

A skeptic would note the association is small (explaining only a few percent of variance), the model-node bootstrap quantifies uncertainty only within the evaluated population, compression distance on raw bytes is a crude and potentially prompt-format-sensitive proxy, the disjointness of estimation and outcome texts does not guarantee the models' shared training provenance isn't the real driver, and correlated failure could partly reflect shared benchmark distractor artifacts rather than genuine generative-process similarity.

Economy implication

Little or nothing yet for AI economics in the next 12 to 24 months: this is a safety and reliability-measurement result about multi-model ensembles and oversight, not a cost, adoption, or labor finding. At most it suggests that multi-vendor redundancy as a risk-management strategy may be less effective than purchasers assume, which could matter eventually for enterprise risk assessments of AI stack design.

Who this matters for

Technical Capability: Researchers and engineers building ensembles or cross-model oversight should care because it gives an output-only tool for predicting which model pairs actually fail independently, showing provider diversity does not imply behavioural independence.

Regulatory & Legal: Auditors and standards bodies assessing AI system robustness could use generative-process diversity as a measurable criterion for whether nominal multi-model redundancy constitutes real defence in depth.

Abstract

Diversity is a widely observed factor in the resilient function of collective systems, yet the type of diversity that matters depends on the properties and failure modes of the system. This distinction is important for systems composed of multiple language models. Different models may be treated as independent components even when their behaviour and failures remain strongly correlated. Assessments of language-model populations using semantic similarity demonstrate limited semantic diversity, but this captures only differences in the meaning of observed outputs. We argue that a more fundamental notion of model diversity is generative-process diversity, the differences between processes capable of generating the observed outputs. Drawing from Algorithmic Information Theory, we use Normalised Compression Distance between raw model outputs, residualised against a permutation control, as a measure of inferred generative-process diversity. Across 38 language models, this measure identifies population structure missed by semantic similarity and predicts cross-task variation in chance-corrected correlated failure among model pairs across ten disjoint benchmark families, beyond semantic similarity and model-pair capability. The cross-benchmark partial rank association is $-0.216$ with a 95% interval of $[-0.309,-0.122]$, and the estimate is negative on all ten benchmarks. These results indicate that increased generative-process diversity is associated with reduced correlated failure in model pairs that is not attributable to semantic similarity or capability. Inferred generative-process diversity offers a novel and practical approach for investigating diversity of multi-model systems in safety-relevant contexts.