Fitting more into the window

Prompting, lesson 9

Summarise, extract or retrieve instead of pasting everything.

You've read "A budget for the context window" and "Order effects in long prompts". This lesson is about making what you keep smaller.

Part 1. Three ways to shrink

A support chat has run 60 exchanges, about 9,000 tokens of history. The user's newest question needs some of it. There are three ways to make it smaller.

Example 1.1

One long chat, three ways

full history9,000 tokens
60 exchanges of about 150 tokens each, word for word.
summarize→"Customer on Android 5.2. Export fails since the update. Cache cleared, reinstall tried."about 400 tokens
Summarize: ask a model to rewrite the history as a short account. It keeps the story and loses the exact wording.
extract→{"order": "A-30418", "device": "Android 5.2", "tried": ["cache", "reinstall"]}about 60 tokens
Extract: pull out only the fields you decided matter, into a fixed shape. Exact values kept, everything else gone.
embed→60 stored exchanges→2 fetched backabout 300 tokens
Embed: store the old exchanges outside the prompt and fetch back only the few that match the current question. Nothing in the prompt until it's needed.

Your turn

The user asks, "What was my order number again?" It was in exchange 4. Which method most reliably gives it back exactly?

Extraction copies the exact value into a field made for it. A summary might keep it, garble it or drop it, since the summarizer decides what matters. Embedding would probably find it, if the search matches "order number" to that exchange, but that's a bet on retrieval. The last 10 exchanges lost it long ago.

Show the answer

A. Extraction copies the exact value into a field made for it. A summary might keep it, garble it or drop it, since the summarizer decides what matters. Embedding would probably find it, if the search matches "order number" to that exchange, but that's a bet on retrieval. The last 10 exchanges lost it long ago.

Context compression

Making the context smaller while keeping what the model needs. Summarizing, extracting and embedding are the three main kinds.

Part 2. Keep recent turns, summarize the rest

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Keep recent turns, summarize the rest
Extract what you'll need
Embed now, fetch later

See Pro plans I have Pro: sign in

Next lessonCapstone: a prompt pipeline for a support deskTemplates, examples, structured output and tool calls working together. Start lesson 10

Prompting overviewSpotted a mistake? Tell us.

09 · Fitting more into the window