Lilian Weng's In-Depth Analysis of Scaling Laws: OpenAI and DeepMind Conclusions Diverge, Chinchilla Formula Has Methodological Flaws

Former OpenAI VP and Peking University alumna Lilian Weng, after a three-year hiatus, published a blog post titled "Scaling Laws, Carefully" on June 24, 2025, systematically reviewing the predictive framework of scaling laws, controversies over compute-optimal allocation, data limitations, and fitting details. The article points out that OpenAI (2020) and DeepMind (2022) reached opposite conclusions regarding compute-optimal allocation (model vs. data scaling ratios of 0.73 and 0.50, respectively), with the divergence stemming from differences in parameter statistical definitions and insufficient experimental scale. More surprisingly, the widely adopted Chinchilla formula itself has methodological flaws: the loss function incorrectly uses the mean instead of the sum, causing the optimizer to converge prematurely; key parameters are retained to only two decimal places, resulting in overly narrow confidence intervals. Additionally, scaling laws assume infinite data supply, but high-quality text data is nearing depletion, prompting the industry to shift toward reinforcement learning, test-time computation, and synthetic data. The blog includes an interactive simulator to visually demonstrate how fitting parameter changes affect conclusions.