insideout Get Pro
Learn
AI engineering · Module 06

Inference and Serving

Make a model answer fast and cheaply under real traffic.

Start with: Reading the prompt, writing the reply
Start here · Lesson 01

Reading the prompt, writing the reply

Prefill reads the whole prompt, then decode writes one token at a time.

Start lesson

LessonsAll lessons

  1. 01 Reading the prompt, writing the replyPrefill reads the whole prompt, then decode writes one token at a time. Free
  2. 02 Caching keys and valuesKeep each token's keys and values so decode never recomputes the past. Free
  3. 03 How big the cache getsWork out KV cache memory from layers, heads, length and batch size. Free
  4. 04 The price of a long contextWhat longer prompts cost in memory, compute and attention. Pro
  5. 05 First token and every token afterTime to first token, time per token, and how streaming changes the wait. Pro
  6. 06 Percentiles and the slow tailWhy p99 matters more than the average once traffic is real. Pro
  7. 07 Prompt length on different GPUsHow prompt length and hardware set prefill time. Pro
  8. 08 Throughput and Little's lawRequests per second, requests in flight, and the time each one takes. Pro
  9. 09 Goodput under errors and timeoutsCount only the requests that succeed within their deadline. Pro
  10. 10 Time until the answer is usableEnd-to-end time from request to a result someone can act on. Pro
  11. 11 Reading a roofline chartWhether a workload is limited by compute or by memory bandwidth. Pro
  12. 12 Arithmetic intensityOperations per byte moved, and why decode is starved for bandwidth. Pro
  13. 13 Continuous batchingStatic, dynamic and continuous batching, and what each does to latency. Pro
  14. 14 Number formatsWhat FP32, BF16, FP8, INT8 and INT4 each keep and lose. Pro
  15. 15 Quantizing weightsMap weights to small integers with a scale and a zero point. Pro
  16. 16 Layers that resist quantizationOutliers and sensitive layers, and how to find them. Pro
  17. 17 GPTQ and AWQTwo methods that quantize to four bits with little loss. Pro
  18. 18 Prefix cachingReuse the work for a shared system prompt or document. Pro
  19. 19 Drafting with a small modelSpeculative decoding: a small model drafts and the large one checks. Pro
  20. 20 Pricing a single requestTurn tokens, hardware and utilisation into a cost per request. Pro
  21. 21 Designing a fair benchmarkWarm-up, fixed inputs, enough runs, and one change at a time. Pro
  22. 22 Serving a mixture of expertsEvery expert sits in memory, but each token reads only a few of them. Pro
  23. 23 Contexts that don't fitChoose which keys and values to keep when a chat outgrows memory. Pro
  24. 24 Running a model serverRead what the engine reports, and add servers on the waiting queue. Pro
  25. 25 Capstone: profile and fix a slow modelFind the slowest part of a serving setup and fix it. Pro

Build it yourself · on your own machine

A serving benchmark on your laptop

A benchmark of one small model at three precisions on your own machine, with the numbers you'd need to pick one for real traffic.
  1. Download Qwen2.5-0.5B-Instruct as an F16 GGUF file, and make Q8_0 and Q4_K_M copies with llama.cpp's llama-quantize.
  2. Score each file's quality with llama-perplexity on the WikiText-2 test set, and its prefill and decode speed with llama-bench.
  3. Work out the KV cache size by hand from the model's config, then start llama-server with 8 slots of 4,096 tokens and compare with the size it prints.
  4. Write a load script that streams from llama-server's OpenAI-compatible endpoint and records time to first token, time per output token and total time for every request.
  5. Send the same 100 prompts at 1, 2, 4 and 8 requests at once, and work out p50 and p99 time to first token and total tokens per second at each level.
  6. Give every prompt the same 1,500-token system prompt, and time the first token with prompt caching on and off.

How you'll know it worksA table giving each file's size, perplexity, prefill and decode speed, and p50 and p99 time to first token at every load level (medians of three runs), with your KV cache figure within 5% of llama-server's.

llama.cpp (Metal on a Mac, CPU elsewhere), Python with the openai package pointed at your own server, and matplotlib. No GPU or API key needed. Uses lessons 01 to 25.