A randomized controlled trial (n=52 professional and freelance developers, Anthropic Safety Fellows Program) tested whether AI assistance during a novel-library coding task improved productivity and skill formation. AI assistance produced no statistically significant reduction in task completion time (p=0.391), while quiz scores measuring conceptual understanding, debugging, and code reading fell by 17 percentage points (Cohen's d=0.738, p=0.010). Qualitative analysis identified six AI interaction patterns: three "high-engagement" patterns (conceptual questions only, explanation-accompanied generation, and generation-then-comprehension) preserved learning outcomes and yielded quiz scores of 65-86%, while three delegation-heavy patterns produced scores of 24-39%. For financial institutions assessing AI-driven productivity claims in software engineering workforces, this study adds a causal data point that aggregate productivity metrics can mask a degradation in the supervisory and debugging competencies that underpin code quality and AI-output verification.
295 other tracked developments touching the same claims.