Notes on Scaling Laws

A short set of notes on the empirical scaling laws that guide how we size models and datasets.

Why it connects to attentionss

Whyyy??

Why?

Most of the scaling story is really a story about transformers, so it helps to first understand Attention Is All You Need. I also touched on the basics in my first article.

The rough shape

Loss tends to fall as a power law in parameters, data, and compute — until one of them becomes the bottleneck.