Skip to content
Kaevu
Home
Projects
Baseball
Bookmarks
More
Reading
Chess
Blog
Bookmarks: cs/hardware
All bookmarks
· shelf cs/hardware
1 bookmark tagged cs/hardware
article
·
horace.io
·
17 min
Making Deep Learning go Brrrr From First Principles
An in-depth guide to optimizing deep learning performance by reasoning from first principles about hardware bottlenecks. It breaks down GPU utilization into three regimes—compute, memory bandwidth, and overhead—explaining how FLOPS growth has outpaced memory bandwidth, why matmuls dominate workloads, and how understanding these constraints guides whether techniques like operator fusion or kernel rewriting will actually yield speedups. Aimed at ML systems engineers, it uses PyTorch and NVIDIA hardware as concrete examples.
ml
systems
hardware
2026-09-04
