Notes on Scaling Laws

A short set of notes on the empirical scaling laws that guide how we size models and datasets.

Why it connects to attention

Most of the scaling story is really a story about transformers, so it helps to first understand Attention Is All You Need. I also touched on the basics in my first article.

The rough shape

Loss tends to fall as a power law in parameters, data, and compute — until one of them becomes the bottleneck.