insideout Get Pro
Inference Engineer

Inference Engineer

The night before

Read it tonight, then once more on the way in.

Back to the rounds

1The day

  1. What fits on a GPUHow much memory the model and its cache take, and how many requests fit on one GPU at once.
  2. Serving and batchingHow a request is served, reading the prompt and then writing tokens one by one, and how serving many requests together changes speed and waiting time.
  3. What limits speedWhat really limits speed: memory bandwidth or raw compute, how attention cost grows with length, and what smaller number formats buy.
  4. CodingGeneral coding, and for some teams reading or writing GPU code.
  5. Serving designPut it all together into one serving system: reusing shared prompts, drafting ahead with a small model, scaling up and down, and cost per request.
  6. Your work and the teamPast projects and how you work: something you made faster or cheaper, how you measured it, and a time it went wrong.

2How to answer

Design a serving stack45 min

  1. Targets5 min
  2. Memory per request8 min
  3. Batching8 min
  4. Caching and speculation8 min
  5. Cost per request8 min
  6. What breaks first8 min

Tell a story from your work2 min

  1. The situation in one line15 sec
  2. What you did60 sec
  3. The result, with a number30 sec
  4. What you'd change15 sec

Walk through your project10 min

  1. The problem2 min
  2. Your choice and its alternative3 min
  3. What broke2 min
  4. The result2 min
  5. What's next1 min

3Numbers to have ready

Say each answer out loud, then open the fold.

  • The cache per token: 32 layers, 32 key and value heads, head size 128, 16-bit.

    2 × 32 × 128 × 32 × 2 = 524,288 bytes, 512 KiB. How big the cache gets

  • The same model shape with 8 key and value heads.

    131,072 bytes, 128 KiB: four times smaller. How big the cache gets

  • A 70B model's weights in BF16, and in INT4.

    140 GB, and 35 GB. Number formats

  • 16 GB of weights at 3.35 TB/s, one user decoding. The time per token?

    16 / 3,350 s, about 4.78 ms: at most about 209 tokens a second. Reading a roofline chart

  • 4 requests arrive a second, and each spends 6 seconds inside. How many are in flight?

    L = λ × W = 4 × 6 = 24. Throughput and Little's law

  • A draft model guesses 4 tokens and each is accepted 80% of the time. Tokens per big step?

    1 + 0.8 + 0.64 + 0.51 + 0.41, about 3.4. Drafting with a small model

4Traps

  • Divide the GPU's memory by the model size to get the users.The weights are only part of it. The cache left after them, and how long each user's context runs, set the number.
  • We decided, we built it, and we fixed it.They're hiring you. With "we" all the way through, they can't tell what you did.
  • The stack was Python, Postgres, Airflow and Docker.A list of tools isn't a decision. Say what you chose, what you turned down, and why.
  • Set the batch size where throughput peaks.Throughput can keep rising while every request misses its latency target. The targets set the cap.

5On the day

  • Ask which AI tools, if any, you may use in each round.
  • Say your plan before you solve, so they can steer you early.
  • Stuck for a minute? Say what you've tried and ask for a hint. Taking one well counts.
  • Check units and sizes out loud before you give a number.
  • Work out the bytes before the speed: the weights, then the cache, then what fits.