Reading a roofline chart

Inference and Serving, lesson 11

Whether a workload is limited by compute or by memory bandwidth.

You know that prefill is compute-bound and decode is memory-bound ("Reading the prompt, writing the reply"), and how a GPU's arithmetic speed sets prefill time ("Prompt length on different GPUs").

Part 1. Two clocks for every step

Every step on a GPU runs two clocks at once. One times the reading: weights have to come in from memory before they can be used. The other times the arithmetic. The step takes as long as the slower of the two. Put numbers on an 8B model stored in 16-bit numbers (2 bytes each) on an H100, which reads about 3.35 TB a second from memory and does about 990 trillion operations a second.

Example 1.1

One decode step for one user, two clocks

8 billion weights×2 bytes=16 GB to read
To make one token, every weight is read from memory once.
16 GB÷3.35 TB/s=4.78 ms
Reading 16 GB takes at least 4.78 ms.
2 × 8 billion ops÷990 trillion/s=0.016 ms
The arithmetic for one token is 16 billion operations. The cores could do it in 0.016 ms.
4.78 ms÷0.016 ms≈296×
The cores are idle for over 99% of the step, waiting for weights to arrive.

Your turn

Decode on this setup is slow. Which change speeds it up the most?

The step is waiting on 16 GB of reading. Halve the bytes and the step is close to half as long. Faster arithmetic speeds up the 0.016 ms part, which was never the problem. "Quantizing weights", later in this module, shows how the 8-bit version is made.

Show the answer

A. The step is waiting on 16 GB of reading. Halve the bytes and the step is close to half as long. Faster arithmetic speeds up the 0.016 ms part, which was never the problem. "Quantizing weights", later in this module, shows how the 8-bit version is made.

Now prefill an 8,000-token prompt on the same GPU. The weights are still read once, in 4.78 ms. But each one is used for all 8,000 tokens, so the arithmetic is 128 trillion operations: 129 ms even at full peak, about 27 times as long as the reading.

Which clock wins, in Python

BANDWIDTH = 3.35e12   # bytes per second, H100 SXM (approx)
PEAK = 990e12         # operations per second, dense BF16 (approx)
weights = 8e9

def step(tokens):
    read = weights * 2 / BANDWIDTH
    math = 2 * weights * tokens / PEAK
    kind = "memory-bound" if read > math else "compute-bound"
    return f"{max(read, math) * 1000:.2f} ms, {kind}"

print("decode, 1 token: ", step(1))
print("prefill, 8000:   ", step(8000))
# decode, 1 token:  4.78 ms, memory-bound
# prefill, 8000:    129.29 ms, compute-bound

Your turn

In that code, how many tokens per step would it take for the arithmetic time to catch up with the reading time?

One token's arithmetic was 296 times shorter than the reading, so you need about 296 tokens per step to even them out. That number comes from the GPU alone: 990 trillion ÷ 3.35 trillion. Keep it in mind for the next section.

Show the answer

A. One token's arithmetic was 296 times shorter than the reading, so you need about 296 tokens per step to even them out. That number comes from the GPU alone: 990 trillion ÷ 3.35 trillion. Keep it in mind for the next section.

Part 2. Two ceilings

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Two ceilings
Drawing it
Where your job really sits

See Pro plans I have Pro: sign in

Next lessonArithmetic intensityOperations per byte moved, and why decode is starved for bandwidth. Start lesson 12

Inference and Serving overviewSpotted a mistake? Tell us.

11 · Reading a roofline chart