A GPU is not "a fast CPU". It is a machine built around one bet: that you have thousands of identical pieces of arithmetic to do, and that you can keep it fed with data. This page walks down the hardware — die, SM, warp, memory hierarchy — and then back up to the two questions that decide how fast your model trains: how much arithmetic per byte, and where does the data live.
Every structural number comes from the vendor datasheet; every derived number is computed live by gpu.mjs and checked by 30 tests. Nothing here is a plausible-looking figure typed in by hand. Switch the GPU in the header and the whole page recomputes.
Almost the whole die is the same unit repeated: the Streaming Multiprocessor. Each SM is an independent scheduler with its own registers and scratchpad memory. They share a big L2 cache and a memory system, and otherwise ignore each other. A CPU spends its transistors on making one instruction stream fast — branch predictors, out-of-order windows, deep caches. A GPU spends them on having more copies of the SM.
Zoom into one square. An SM is divided into four processing blocks, each with its own warp scheduler, its own slice of the register file, and its own arithmetic lanes — including one tensor core. Two things here are unusual and both matter enormously for machine learning: the register file is larger than the L1 cache, and the L1 is software-managed — you decide what to put in shared memory, which is what makes tiling possible.
You launch a grid of blocks. The hardware assigns whole blocks to SMs; a block never migrates and never splits. Inside an SM the block is chopped into warps of 32 threads, and the warp — not the thread — is the unit the scheduler actually issues. The number of warps an SM can hold at once is its occupancy, and occupancy is not a vanity metric: it is the SM's supply of other work to run while some warp sits waiting several hundred cycles for HBM.
A warp's 32 lanes execute the same instruction each cycle. That is the whole trick — one fetch and one decode amortized over 32 pieces of arithmetic. The price is that when lanes disagree about a branch, the hardware runs both sides in sequence with the non-participating lanes switched off. The cost is the sum of the paths, not the longer of them, and a single disagreeing lane is enough to trigger it.
if (cond) { heavy(); // 40 cycles } else { light(); // 10 cycles }
Getting a number out of HBM costs a few hundred cycles; getting it out of a register costs about one. Between those sits a scratchpad you control by hand. But there is a second, less-taught rule: the memory system does not serve threads, it serves 32-byte sectors. A warp asking for 32 contiguous floats is one tidy 128-byte request. The same 32 floats, strided, become up to 32 separate sector fetches — the same useful bytes, eight times the traffic.
A naive matmul has every thread stream a whole row of A and a whole column of B from global memory to produce one output value — 2N reads for 1 FLOP-pair. The tiled version stages a small patch of A and B into shared memory and has the whole block reuse it. The arithmetic is identical. The traffic drops by exactly the tile size. This one transformation is the difference between a matmul at a few percent of peak and one near it.
An FP32 lane does one multiply-add per cycle. A tensor core consumes an entire small matrix multiply per instruction — a 16×8×16 tile in one go — because the operands are fed through a fixed systolic arrangement instead of one register pair at a time. This is where the order-of-magnitude jump in the specs comes from, and it is why mixed precision is not an optimization you add later: on modern hardware the FP32 path is the slow path.
Divide a kernel's FLOPs by the bytes it must move, and you get its arithmetic intensity. Plot it against what the hardware could deliver at that intensity and you get the roofline: a diagonal where bandwidth is the limit, a flat ceiling where arithmetic is. The intensity where they meet — the ridge point — is the bar your kernel has to clear to be worth the hardware. Drag the shapes and watch a transformer layer's kernels slide across it.