insideout Get Pro

14 · Capstone: a service with routing, fallbacks and monitoring

Pro

Production AI, lesson 14

Capstone: a service with routing, fallbacks and monitoring

Run an AI service that stays up and within budget.

This walkthrough uses ideas from across the module: routing, fallbacks, spending limits, caching, filters, monitoring and alerting. Each part has its own lesson, so if a step feels quick, the lesson with that name has the detail.

Part 1. The system on a normal day

Here's a support assistant for an online shop, about 3,000 questions an hour. About 70% are simple (opening hours, delivery times, how returns work) and go to a small model. The rest, mostly billing and order problems, go to a large one. Prices below are examples.

Example 1.1

The layers every question passes through

  1. gatewaysign-inper-user token bucketmonthly quota

    The front door. Nobody can flood the system, and every request gets an id that follows it everywhere.

  2. input checkmask card numbersmoderationtopic check

    The safety boundary on the way in.

  3. cachesexact→semantic (general questions only)

    Repeated general questions are answered here, with no model call. Questions about a customer's own order skip the shared cache.

  4. routersmall: 70%large: 30%
    fallback chainretriescircuit breakersecond provider

    Pick the model, and know where to go if it fails.

  5. output check→customer
    recordslogmetricstrace

    Check the answer before it's shown, and record everything about the call.

Your turn

Why does the cache sit before the router and the model, not after?

Every layer after a cache hit is skipped: no routing, no model call, no cost. The cache still sits after the input check, though, so a blocked message never gets a cached answer it shouldn't.

The injection lessons shaped it too. The model reads customer messages, which anyone can write, and order data, which is private. So its one tool, lookup_order, only returns orders on the signed-in customer's account, refunds go to staff for approval, and the chat window loads no outside images or links. There's no way out for stolen data, whatever text gets in.

Part 2. Following one question through

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Following one question through
  2. A bad day: the main provider goes down
  3. What the records show afterwards
See Pro plansI have Pro: sign in
The last lesson in Production AIBack to the overviewRunning AI for real users Back to Production AI

Production AI overviewSpotted a mistake? Tell us.