Scaling Laws for Neural Language Models: New Evidence Challenges Chinchilla Predictions

Researchers at Epoch AI have published new empirical evidence that challenges the widely-adopted Chinchilla scaling laws, which have guided the training of most frontier language models since 2022. The new research suggests that compute-optimal training requires substantially more training tokens than the Chinchilla predictions indicate, particularly at large compute budgets.

The Chinchilla paper, published by DeepMind in 2022, established that optimal model training requires approximately 20 training tokens per model parameter. This finding led to a shift in the field toward training smaller models on more data, with models like Llama 2 and Mistral following this prescription.

The new Epoch AI research, based on a comprehensive analysis of training runs across multiple organizations, finds that the optimal token-to-parameter ratio increases significantly at larger compute scales. At the compute budgets now used for frontier models, the optimal ratio may be closer to 50-100 tokens per parameter.

If confirmed, these findings would suggest that current frontier models are significantly undertrained relative to their optimal configuration. The implications are significant: achieving optimal performance from a given compute budget may require training for longer on more data rather than scaling model size.