Exact and semantic caches

Production AI, lesson 4

Reuse answers safely, and know when not to.

It helps to know how cosine similarity compares two embeddings ("Cosine similarity and the dot product" in Tokens and Embeddings). The provider side of caching is in "Prefix caching" in Inference and Serving; this lesson stays in your app.

Part 1. The same question, asked again

A shop's assistant gets "What are your opening hours?" thousands of times a day. Each time, the model reads the same instructions, sees the same question, and writes nearly the same answer. You pay for it every time.

The cheapest model call is the one you skip. Store the answer the first time, keyed by the request, and hand it back when the identical request comes in again.

Your turn

What should go into the cache key, so a stored answer is only reused when it's really the same request?

Change the model, the instructions or the settings, and the right answer may change too. A key built from the question alone would keep serving answers from last month's prompt. The user's name only belongs in the key if the answer is personal.

Show the answer

D. Change the model, the instructions or the settings, and the right answer may change too. A key built from the question alone would keep serving answers from last month's prompt. The user's name only belongs in the key if the answer is personal.

Exact cache

A lookup table from a hash of the full request to the stored response. A hit needs every character to match, so it's safe, but "opening hours?" and "opening hours" miss each other. Trimming spaces and lowercasing before hashing helps a little.

In practice

Give every entry an expiry time. Opening hours change, prices change, policies change. A cached answer with no expiry will happily outlive the fact it was based on.

Part 2. Same meaning, different words

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Same meaning, different words
The false hit
Prompt cache: reuse the part that never changes

See Pro plans I have Pro: sign in

Next lessonLogs, metrics and tracesWhat to record so you can see what the system is doing. Start lesson 5

Production AI overviewSpotted a mistake? Tell us.

04 · Exact and semantic caches