A prompt pipeline that reads customer messages and returns valid JSON with the intent, the order number and a reply, calling a lookup tool when it needs one.
Take 100 messages from the Bitext customer support dataset on Hugging Face (27 intents), swap each {{Order Number}} placeholder for a made-up number, and keep 50 for tuning and 50 for the final score.
Write a system prompt with your rules, and a schema with the intent (one of the 27), the order number or null, and a one-line reply.
Run a small model in Ollama three ways (asking for JSON in words, JSON mode, and your schema passed as format), and validate every reply with Pydantic.
Add three examples, then the same three in another order, and record how intent accuracy moves.
Add one read-only tool, get_order(order_id), backed by a Python dict, and run the tool loop so order questions are answered from its result.
Count the tokens in each part of the prompt, and cut the part that earns least.
How you'll know it worksA table of parse rate and intent accuracy on the 50 held-out messages for each setup, with 100% of schema runs parsing, and the share of order questions whose tool call got the right number.
Python, Pydantic, and Ollama with a small tool-calling model such as qwen2.5:3b or llama3.2:3b (runs on a laptop, no API key). Your own API key works too. Uses lessons 01 to 10.
Project: A support desk pipeline on a local model
From insideout, Prompting: A prompt pipeline that reads customer messages and returns valid JSON with the intent, the order number and a reply, calling a lookup tool when it needs one.
Steps:
1. Take 100 messages from the Bitext customer support dataset on Hugging Face (27 intents), swap each {{Order Number}} placeholder for a made-up number, and keep 50 for tuning and 50 for the final score.
2. Write a system prompt with your rules, and a schema with the intent (one of the 27), the order number or null, and a one-line reply.
3. Run a small model in Ollama three ways (asking for JSON in words, JSON mode, and your schema passed as format), and validate every reply with Pydantic.
4. Add three examples, then the same three in another order, and record how intent accuracy moves.
5. Add one read-only tool, get_order(order_id), backed by a Python dict, and run the tool loop so order questions are answered from its result.
6. Count the tokens in each part of the prompt, and cut the part that earns least.
How you'll know it works: A table of parse rate and intent accuracy on the 50 held-out messages for each setup, with 100% of schema runs parsing, and the share of order questions whose tool call got the right number.
Runs on: Python, Pydantic, and Ollama with a small tool-calling model such as qwen2.5:3b or llama3.2:3b (runs on a laptop, no API key). Your own API key works too.
Help me build this on my machine, one step at a time. Ask what I've tried first, give hints before full answers, and check my work against "How you'll know it works".