A robust association between LLM use and scientific productivity: Assessing stopping-time selection

LLM use by scientists is robustly associated with increased publication output even after controlling for the stopping-time selection artifact that can mechanically inflate event-study estimates, with a positive productivity association of roughly 0.07 to 0.18 log-points above a rate-matched placebo baseline.

Four complementary identification strategies applied to arXiv paper output data: (1) a before-and-after comparison dating adoption in 2023 and measuring output in separated windows (2022 vs. 2024), finding +17.3% to +25.1% output gains at two LLM-detection thresholds; (2) a conservative DiD using not-yet-treated authors as controls, yielding a pooled estimate of 0.068 log-points (95% CI [0.053, 0.083]); (3) an intensity specification with no adoption date, using lagged detector scores and within-author fixed effects, all positive and significant (p less than 0.001); and (4) a rank-based design holding the flag rate fixed, finding top-alpha vs. bottom-alpha rules straddle the random artifact baseline by 15.3% at p=0.2%. All designs replicated on pre-ChatGPT placebo data return null or negative estimates.

Average post-treatment excess over rate-matched placebo: 0.119 log-points (95% CI [0.068, 0.169]) at alpha greater than 0.1 threshold; 0.183 log-points (95% CI [0.121, 0.244]) at stricter alpha greater than 0.5 threshold. Before-and-after: +17.3% to +25.1% output gain for adopters vs. pre-LLM period. Conservative DiD lower bound: 0.068 log-points. Effect is largest at the stricter detection threshold where LLM use is most cleanly identified. Pre-ChatGPT placebo returns -0.9% to 0.0% on the before-and-after design and individually insignificant coefficients across all other designs.

Authors explicitly caution against causal interpretation: all designs establish association, not causation. The before-and-after design is not immune to secular changes in submission rates between 2022 and 2024. The rate-matched placebo is itself contaminated by output-driven selection, making the excess estimate conservative rather than exact. LLM detection relies on an algorithmic detector with a realized flag rate that may misclassify papers. The intensity specification coefficients are not directly comparable in scale to DiD estimates. Selection into LLM adoption is not random. The paper is a methodological rebuttal, not a new primary study, so the underlying sample size and data coverage are inherited from the original paper being defended.

A fair skeptic notes this is a response paper defending a prior finding, which is an inherently adversarial framing that may emphasize designs favorable to the original result. The LLM detector used to classify adoption is algorithmic and imperfect, and contamination by prolific authors inflating both the real and placebo treated groups is acknowledged but not fully resolved. The before-and-after design explicitly cannot rule out secular trends. The 2022-to-2024 comparison window overlaps with multiple confounders beyond LLM adoption. The claim of convergence across designs is persuasive in direction but the magnitude estimates vary considerably (0.068 to 0.25 log-points), and none of the designs fully satisfies a credible-identification standard for causal inference.

If the association is eventually confirmed as causal, LLMs may be materially raising individual researcher productivity in academic science, which would imply downstream effects on knowledge production rates and potentially on labor demand for research assistants and junior researchers. In the next 12 to 24 months, this evidence is suggestive but not yet sufficient to drive enterprise or institutional staffing decisions. The paper contributes to an emerging body of productivity-measurement evidence but does not by itself shift the economic calculus.

Labor & Knowledge Work: If LLM use raises scientific output by 17-25% per author, research institutions and funders face a genuine question about whether to adjust hiring norms for postdocs and research assistants who historically provided marginal productivity support.

Technical Capability: The methodological battery demonstrates that event-study designs dating treatment from first detected LLM use contain a bounded but real stopping-time artifact, which has implications for how researchers design and interpret future AI-adoption studies.

Regulatory & Legal: Evidence of measurable productivity gains from LLM use in scientific publishing is relevant to policy discussions about AI disclosure requirements, authorship norms, and funding agency guidelines on AI assistance.