A small HTTP service in front of two local models that routes, caches, limits spending, logs every request and keeps answering when its main model goes down.
Run two model servers on different ports (llama-server or Ollama), a 3B model as the main one and a 1B model as the cheap one, and put a FastAPI endpoint in front of them.
Route short, simple requests to the cheap model, and write one JSON log line per request: id, user, route, model, tokens in and out, time to first token, total time and status.
Add a timeout that falls back to the other model, a token bucket per user, and an exact cache keyed on the normalised prompt.
Replay 300 instructions from databricks-dolly-15k on Hugging Face with a load script, and stop the main server for one minute in the middle.
From the logs alone, chart p95 latency and error rate per minute, and write the alert rule that should have fired.
Put a canary string in the system prompt, attack the service with the injection rows of deepset/prompt-injections, then add an output check that blocks any reply holding the canary.
How you'll know it worksAt least 99% of requests succeed during the outage minute, the canary reaches no user once your check is in, and your per-minute chart comes from your own logs alone.
Python with FastAPI and httpx, llama.cpp or Ollama for the two models, and free datasets from Hugging Face. It all runs on a laptop, with no API key. Uses lessons 01 to 14.
Project: A model service with a fallback
From insideout, Production AI: A small HTTP service in front of two local models that routes, caches, limits spending, logs every request and keeps answering when its main model goes down.
Steps:
1. Run two model servers on different ports (llama-server or Ollama), a 3B model as the main one and a 1B model as the cheap one, and put a FastAPI endpoint in front of them.
2. Route short, simple requests to the cheap model, and write one JSON log line per request: id, user, route, model, tokens in and out, time to first token, total time and status.
3. Add a timeout that falls back to the other model, a token bucket per user, and an exact cache keyed on the normalised prompt.
4. Replay 300 instructions from databricks-dolly-15k on Hugging Face with a load script, and stop the main server for one minute in the middle.
5. From the logs alone, chart p95 latency and error rate per minute, and write the alert rule that should have fired.
6. Put a canary string in the system prompt, attack the service with the injection rows of deepset/prompt-injections, then add an output check that blocks any reply holding the canary.
How you'll know it works: At least 99% of requests succeed during the outage minute, the canary reaches no user once your check is in, and your per-minute chart comes from your own logs alone.
Runs on: Python with FastAPI and httpx, llama.cpp or Ollama for the two models, and free datasets from Hugging Face. It all runs on a laptop, with no API key.
Help me build this on my machine, one step at a time. Ask what I've tried first, give hints before full answers, and check my work against "How you'll know it works".