Logs, metrics and traces

Production AI, lesson 5

What to record so you can see what the system is doing.

You know a call is priced by input and output tokens ("Routing requests between models"). "Percentiles and the slow tail" in Inference and Serving explains p50 and p95, which come back here.

Part 1. One call, one record

A customer writes in: "Your assistant told me my refund was approved. It wasn't." To find out what happened you need that exact call: which model answered, with which instructions, what it was given to read, and what it wrote. If you didn't record it at the time, it's gone.

Example 1.1

What to record for every model call

request iduser / tenantfeature: support-chat
Who and where. The id lets you find this call again and join it to everything else that happened in the same request.
model: large-2025-06prompt: support-v14temperature 0.2
What produced the answer. Pin the exact model snapshot and the prompt version, or you can't tell which change caused a problem.
in 1,240 tokensout 310 tokensfirst token 420 mstotal 2,100 ms
Size and speed. Time to first token and total time are separate numbers, and customers feel both.
1,240 × $3/M+310 × $15/M=$0.00837
status: okcache: missfallback: no
Cost (at example prices) worked out when the call happens, plus how it ended. Now cost per feature and per tenant is just adding up records.

Your turn

Thumbs-down feedback doubled starting Tuesday afternoon. Which recorded field most likely finds the cause first?

A sudden quality drop usually follows a change: a new prompt, a new model snapshot, new retrieval settings. Group feedback by prompt version and model and the culprit often jumps out. Without versions in the record, every change since Tuesday is a suspect.

Show the answer

D. A sudden quality drop usually follows a change: a new prompt, a new model snapshot, new retrieval settings. Group feedback by prompt version and model and the culprit often jumps out. Without versions in the record, every change since Tuesday is a suspect.

Record the model name the provider sends back, not only the one you asked for. Many providers accept an alias that points at the newest snapshot, and their reply names the exact snapshot that answered. If your records only hold the alias, a silent upgrade is invisible. With both, a change in what answered shows up the day it happens.

In practice

Prompts and answers contain personal data. Mask emails, card numbers and names before they're stored, limit who can read raw text, and delete it on a schedule. Token counts, cost and latency can be kept far longer than the text itself.

Part 2. Logs, metrics and traces

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Logs, metrics and traces
A trace through one request
200 OK is not the same as right

See Pro plans I have Pro: sign in

Next lessonWhen to page someoneAlerts worth waking up for, and how to run an incident. Start lesson 6

Production AI overviewSpotted a mistake? Tell us.

05 · Logs, metrics and traces