Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccolò Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein
5 pages, 2 figures, workshop
Scaling-law fits matching those from exhaustive training grids can be recovered at 10 to 100 times lower compute cost by combining Bayesian optimization with compute slicing and surrogate fantasization.
Simulated scaling-law construction over existing public scaling datasets (OELLM-English, plus Porian-HP, StepLaw, Warmstart, OELLM-Multilingual in ablations), using BoTorch Gaussian-process BO, comparing Window vs Full search space with and without fantasization across 10 seeds, evaluating coefficient regret, held-out MSE, envelope coverage, and extrapolation error.
Window+fantasization matches the full-grid fit at 10x savings vs Full+fantasization and up to 100x vs Full observed-only; at 1% of total grid compute, predicted losses stay within 1.6% of the full-data fit even at 10^25 to 10^29 FLOPs, i.e., 6 to 7 orders of magnitude beyond the observed grid. All results are retrospective simulations on pre-collected grids, not new large-scale data collection.
The authors concede the framework was not validated on a novel large-scale scaling run (the 'ultimate stress-test' is future work), the compute-slicing restriction is not isolated in its effect, and specialized kernels and fit-variance-aware acquisition are left unexplored. Unstated: all results are replays of existing small grids where the 'ground truth' is itself a parametric fit, GP surrogates may scale poorly to realistic multi-dimensional grids, and extrapolation accuracy to frontier scale is asserted only in low-dimensional parametric forms.
This is a 5-page workshop paper whose headline numbers come from replaying small, curated public grids, so the 10 to 100x figure may not survive contact with real frontier-scale grid design, noise, architecture search, or data-mixture complexity; the claimed extrapolation precision depends on the chosen two- or three-parameter forms rather than demonstrated frontier-relevant runs.
If it generalizes, it modestly lowers the experimental-compute overhead of frontier training planning, a second-order cost reduction that does not change the economics of frontier training runs themselves. Little or nothing yet for the AI-economy debate: this concerns how labs derive scaling laws, not whether scaling pays off commercially.
Technical Capability: Practitioners designing pretraining grids learn that compute-ordered acquisition plus surrogate fantasization can recover the same scaling-law coefficients for a fraction of the experimental compute.
Market & Valuation: At most marginally relevant: cheaper scaling-law studies slightly reduce frontier labs' experimental overhead, but the paper offers no evidence bearing on capex or revenue economics.