insideout Get Pro

03 · Few-shot examples

Free

Prompting, lesson 3

Few-shot examples

How examples in the prompt steer format and behaviour.

An instruction describes the answer you want, and an example shows it. This lesson is about putting solved examples in the prompt: what they fix, what they cost, what they quietly teach and how to choose them. You need to know that a chat request is a list of system, user and assistant messages ("How a chat request is built").

By the end, you can:

  1. Turn solved examples into a few-shot prompt.
  2. Work out what examples cost when they're sent on every call.
  3. Say what examples teach besides their answers.
  4. Spot the tilts a set of examples adds, and balance them.

Part 1. Show the answer you want

Turn solved examples into a few-shot prompt.

A bike shop wants every support message tagged order, repair or return, so code can route it. The first prompt is one line: "Tag this message as order, repair or return."

Guess first

You run that prompt on 100 messages. What's the most common problem?

The model usually understands the task. What it doesn't know is the exact shape you want, so it improvises, and your code, which expects exactly order, chokes on many of the replies. New labels like "exchange" turn up too. Both problems shrink once the model sees a few solved messages.

The GPT-3 paper made the same point: with no examples, even a person may not know what format an answer should take. Examples carry the format for you.

Example 1.1

Three solved messages as chat turns

You have three tagged messages: an order question, a repair and a return. Build the prompt for a new message: "These gloves are too big. Can I send them back?"

  1. Write the instructionsystem: Tag each message: order, repair or return.

    One short line names the task and the labels.

  2. Add each example as two turnsuser: Where's my order? → assistant: order

    The customer's message goes in a user turn, and its tag in an assistant turn. Do the same for the repair and the return.

  3. Add the real message lastuser: These gloves are too big…

    The model's next assistant turn is the tag.

  4. Count the messages1 + 3 × 2 + 1 = 8

    One instruction, two turns per example, one real message.

Answer8 messages, and the reply is likely return.

Written as fake earlier turns, the examples look like the model's own past answers, and it tends to match them closely: same labels, same lowercase, nothing else. You can also list examples as plain text in the system prompt. Both work, as long as the examples are in the exact shape you want back.

Few-shot prompting

Putting solved examples in the prompt. Each example is a shot: no examples is zero-shot, one is one-shot, and a handful is few-shot.

3 shots as turns: 1 + 3 × 2 + 1 = 8 messages

Your turn

You write 5 solved examples as chat turns, after one system message and before the real input. How many messages is the prompt?

One system message, 5 × 2 = 10 example turns, and the real input: 1 + 10 + 1 = 12. 7 forgets that each example is two turns, a message and its answer.

Part 2. Nothing is learned for keeps

Work out what examples cost when they're sent on every call.

In 2020 the GPT-3 paper, "Language Models are Few-Shot Learners", showed a large model doing new tasks from examples in its prompt, without any gradient updates or fine-tuning. The weights don't change. The examples are only more tokens to read, so every call has to send them again, and pay for them.

Example 2.1

What the examples cost

Three examples take about 40 tokens each, with their role markers. Your app makes 10,000 calls a day, and the model charges $3 per million input tokens. What do the examples cost a day?

  1. Tokens per call3 × 40 = 120

    Every call carries all three examples.

  2. Tokens per day120 × 10,000 = 1,200,000

    Nothing is saved between calls, so each one pays again.

  3. Price them1.2 × $3 = $3.60

    1,200,000 tokens is 1.2 million, at $3 a million.

Answer$3.60 a day.

Nothing carries over between calls. After 10,000 calls the model is the same as before the first one. The pattern lives in the prompt, so when the prompt loses the examples, the model loses the pattern.

In-context learning

Doing a task by following examples read in the prompt. Nothing is trained, so the skill lasts only as long as the examples stay in the prompt.

cost = example tokens × calls × price per token

Cheap here. Twenty longer examples of 500 tokens each make 10,000 tokens a call, and at the same traffic and price that's $300 a day. At that size, check that each example earns its place by testing with and without it.

Your turn

A prompt carries 5 examples of 60 tokens each. The app makes 20,000 calls a day, at $2 per million input tokens. What do the examples cost a day?

Per call: 5 × 60 = 300 tokens. Per day: 300 × 20,000 = 6,000,000, which is 6 million, and at $2 a million that's $12. $2.40 counts one example instead of five. They're never paid for once: every call carries them.

Part 3. What the examples teach

Say what examples teach besides their answers.

Examples carry more than their answers. They show which labels exist, what shape a reply takes and what kind of input to expect. Researchers have tested how much the answers themselves matter by scrambling them.

Example 3.1

What a shuffled prompt still shows

Your three examples get their tags shuffled: the late order is tagged repair, the squeaky brakes return, and the helmet that's too small order. What can the model still learn from them?

  1. The label setonly order, repair, return

    Every tag is still one of the three allowed.

  2. The formatone lowercase word

    Each reply is still just the tag.

  3. The kind of inputshort customer messages

    The inputs are still real messages like the ones to come.

  4. The pairingmessage to right tag: lost

    Only which tag goes with which message is wrong.

AnswerEverything but the pairing.

Three examples, tags shuffledWhere's my order?repairMy brakes squeal.returnThe helmet is too small.orderthe label set: order, repair, returnthe format: one lowercase wordthe kind of input: customer messageswhich tag goes with which message
Figure 1Shuffling the tags breaks the pairing, and nothing else. The label set, the format and the kind of input all still show.

A model already knows from pretraining what late orders and squeaky brakes are. The examples mostly tell it which task this is, which answers are allowed and what shape they take. The pairing still matters, but less than you might think.

Label space

The set of answers the examples allow, such as order, repair and return. With the format and the kind of input, it's much of what a few-shot prompt teaches.

Wrong labels still do harm, more so in big models. Jerry Wei and colleagues (2023) flipped every label in the examples. Small models mostly ignored the flip and kept to what they knew. The largest followed the flipped labels, even against what pretraining had taught them. So check every example's answer.

Your turn

In a 2022 study, researchers swapped the labels in few-shot examples for random ones, so a glowing film review might be labelled negative. On many classification tasks, what happened to the scores?

Sewon Min and colleagues tested 12 models, GPT-3 among them. Random labels from the right set cost only 0 to 5 points, because the label set, the kind of input and the format did much of the work. The largest models do follow wrong labels more, so get them right anyway.

Part 4. Examples that tilt the answer

Spot the tilts a set of examples adds, and balance them.

Examples can also tilt answers in ways you didn't plan. Tony Zhao and colleagues (2021) measured two tilts in GPT-3: one toward the commonest label, one toward the last.

Example 4.1

Reading the tilts

Four examples come in this order: order, order, repair, order. A message arrives that could be about an order or a repair. Which way does the prompt push it?

  1. Count the labelsorder 3, repair 1, return 0

    The model leans toward the label most examples share.

  2. Look at the last examplelast label: order

    The label of the last example gets an extra pull.

  3. Add the pushesmajority and last: both order

    Both tilts point the same way.

AnswerToward order, from both tilts.

Examples, in orderlastorderorderrepairorder3 of 4 say order, and so does the lastan unclear messageleans order
Figure 2Two pulls toward order: it's the majority label, and it's the label of the last example.

The model reads the examples as a hint about which answers are common here. Zhao and colleagues found GPT-3 leaned heavily toward a class when the examples were unbalanced, and toward the class repeated at the end of a balanced prompt.

Majority label bias and recency bias

Two tilts that examples add. Majority label bias pulls unsure answers toward the label most examples share. Recency bias pulls them toward the label of the last example.

order, order, repair, order → leans order

Order alone matters too. Yao Lu and colleagues found that on some tasks, the same examples scored near the best published results in one order and near random guessing in another. So balance the labels, vary which one comes last, and test the exact set and order you ship.

Your turn

Six examples come in this order: repair, order, repair, return, order, repair. Which label do the tilts favour?

repair has 3 of the 6 examples, the most, and it's also the last label. Both tilts point the same way. Having all three labels present doesn't make a set balanced: here it's 3, 2 and 1.

Where you'll meet it

In interviews

Asked why a few-shot prompt gives unstable results? Name the causes: unbalanced labels, the pull of the last example, the order of the examples, and examples that don't look like real inputs. Then the fixes: balance the labels, vary the order, pick examples like the input, and test the exact prompt on a fixed set before you ship it.

On the job

Many systems pick examples per request: they keep a bank of solved examples and put the few most like the new input in the prompt. Jiachang Liu and colleagues found picking by similarity beat random picks for GPT-3 on several tasks. Real systems compare embeddings; the code below counts shared words.

Picking similar examples, in Python
import re

BANK = [("I was sent the wrong size chain.", "return"),
        ("My chain slips since the service.", "repair"),
        ("Where is the chain I ordered?", "order"),
        ("My order still says processing.", "order")]

def similarity(a, b):   # shared words over all words
    a, b = (set(re.findall(r"[a-z']+", t.lower())) for t in (a, b))
    return len(a & b) / len(a | b)

msg = "The chain you sent is the wrong length."
for text, tag in sorted(BANK, key=lambda ex: -similarity(msg, ex[0]))[:3]:
    print(f"{similarity(msg, text):.2f}  {tag:7} {text}")
# 0.40  return  I was sent the wrong size chain.
# 0.30  order   Where is the chain I ordered?
# 0.18  repair  My chain slips since the service.

All three are about chains and carry three different labels, which shows the model where the lines fall. The second matched only on "the", "chain" and "is", though. Embeddings match meaning instead ("Cosine similarity and the dot product" in Tokens and Embeddings).

Check yourself

Four questions from across the lesson. Try each one before you look back.

1

You've sent 10,000 calls with the three examples. On call 10,001 you leave them out. How does the model behave?

Nothing carries over between calls. The model's weights are the same as before call one. The pattern lived in the prompt, so when the prompt loses the examples, the model loses the pattern.

2

You write few-shot examples as user and assistant turns. What are the assistant turns to the model?

After the chat template, they look exactly like replies the model gave earlier, so it copies their shape. They carry no authority, since earlier replies sit below the user, and they change no weights.

3

You have 200 tagged messages and room for three examples. A new message comes in: "The chain you sent is the wrong length." Which three should go in the prompt?

Examples close to the real input show the model how this kind of case gets handled, and Jiachang Liu and colleagues found similarity beat random picks on several tasks. Keep an eye on the labels, though: if all three similar messages are returns, you've also added a majority-label tilt.

4

You show a very large model and a small one the same examples with every label flipped, so positive reviews are labelled negative. What happens?

Jerry Wei and colleagues found small models stick to what pretraining taught them, while the largest can override it to follow the examples. So a wrong label misleads a big model more.

What to remember

  1. 1Show the answer you want

    Solved examples fix the format and the label set: 3 shots as chat turns make 1 + 3 × 2 + 1 = 8 messages.

  2. 2Nothing is learned for keeps

    Nothing is learned between calls, so the examples are paid for every time: 120 tokens × 10,000 calls at $3 a million is $3.60 a day.

  3. 3What the examples teach

    Examples teach the label set, the format and the kind of input: random labels from the right set cost only 0 to 5 points, but the largest models do follow wrong labels.

  4. 4Examples that tilt the answer

    Examples pull unsure answers toward the majority label and the last label, and order alone can swing accuracy: balance the labels, vary the last one, and test the set you ship.

Recap and going further

The lesson on one page

Show, don't describe
  • Examples fix the format and the label set
  • As fake earlier turns, or listed in the system prompt
Nothing sticks
  • The weights don't change
  • The examples are paid for on every call
What they teach
  • Label set, format and kind of input
  • Random labels cost only 0 to 5 points
  • Big models do follow wrong labels
Steering with solved examples
Tilts
  • The majority label pulls unsure cases
  • So does the last example's label
  • Order alone can swing accuracy
Picking examples
  • The most similar ones, per request
  • Keep the labels balanced
  • Test the exact set you ship

Try this

Measure what your examples do

Take a labelling task with 50 inputs whose right labels you know, such as tagging your own emails or product reviews. Run it with no examples, then with 3 examples, then with the same 3 in a different order, then with the 3 most similar examples picked for each input. Log the exact reply each time, not only whether it was right.

You need: Any chat model API, and a short Python script or a spreadsheet.

Look for

  • How many replies with no examples come back in a shape your code can't use.
  • Whether accuracy moves when only the order changes.
  • Which label the model leans to when it's unsure, and whether that's your examples' majority.
  • How much picking similar examples helps, against the tokens it adds.

The words, in plain language

Few-shot prompting
Putting solved examples in the prompt. Each example is a shot: no examples is zero-shot, one is one-shot, and a handful is few-shot.
In-context learning
Doing a task by following examples read in the prompt. Nothing is trained, so the skill lasts only as long as the examples stay in the prompt.
Label space
The set of answers the examples allow, such as order, repair and return. With the format and the kind of input, it's much of what a few-shot prompt teaches.
Majority label bias and recency bias
Two tilts that examples add. Majority label bias pulls unsure answers toward the label most examples share. Recency bias pulls them toward the label of the last example.

Watch

Read next

Next lessonJSON that always parsesNext, getting replies your code can parse every time: JSON, schemas, and what to do when a reply doesn't fit. Start lesson 4

Prompting overviewSpotted a mistake? Tell us.