insideout Get Pro
AI Engineer

AI Engineer

The night before

Read it tonight, then once more on the way in.

Back to the rounds

1The day

  1. Build roundBuild something small that works with a model, like answering from documents or calling a tool, at home or on a call.
  2. EvalsHow you'd know it works: what good means, a test set, models as graders, and comparing two versions fairly.
  3. System designDesign a product on top of a model: which model gets which request, caching, what happens when a model is slow or down, cost, and keeping attacks out.
  4. How models workEnough of how models work to explain their failures: tokens, how much text fits, how the next word is picked, and what long inputs cost.
  5. CodingPractical code more often than puzzles: retrying a failed call, splitting text into chunks, finding the closest matches.
  6. Your work and the teamStories from your own work: a hard call, something that failed, a disagreement, and why you want this job.

2How to answer

Design a product on a model45 min

  1. Scope the user and the job5 min
  2. What good means and how to measure it5 min
  3. Sketch the request path5 min
  4. Retrieval, tools and prompts10 min
  5. Evals and failure cases8 min
  6. Cost, latency, fallbacks and injection8 min
  7. Wrap up4 min

Tell a story from your work2 min

  1. The situation in one line15 sec
  2. What you did60 sec
  3. The result, with a number30 sec
  4. What you'd change15 sec

Walk through your project10 min

  1. The problem2 min
  2. Your choice and its alternative3 min
  3. What broke2 min
  4. The result2 min
  5. What's next1 min

3Numbers to have ready

Say each answer out loud, then open the fold.

  • One request: 3,000 tokens in and 250 out, at an assumed $1 and $4 per million. The cost?

    $0.003 + $0.001 = $0.004. At 20,000 a day, $80. Pricing a single request

  • 80% of traffic moves to a model at a twentieth of the price. The saving?

    0.8 / 20 + 0.2 = 0.24 of the bill, a 76% saving. Routing requests between models

  • A cascade: $0.0004 for the small model, $0.008 for the large one, a quarter escalate.

    $0.0004 + 0.25 × $0.008 = $0.0024 a request. Routing requests between models

  • 2% of calls fail at random. Out of a million, how many fail after one retry?

    0.02² × 1,000,000 = 400, down from 20,000. When the main model is down

  • 5% of tickets are urgent. The accuracy of a model that never flags one?

    95%, and it catches none of them. The confusion matrix

  • The wobble on a 90% pass rate over 400 test items.

    √(0.9 × 0.1 / 400) = 0.015, so about ±3 points at 95%. Is the new version really better

4Traps

  • I'd start with the biggest model and a vector database.Tools before the user and the job. Nothing after it has a reason the interviewer can check.
  • We decided, we built it, and we fixed it.They're hiring you. With "we" all the way through, they can't tell what you did.
  • The stack was Python, Postgres, Airflow and Docker.A list of tools isn't a decision. Say what you chose, what you turned down, and why.
  • We'll know it works when users are happy with it.That's a hope with no measure. Name a test set and the two or three numbers you'd track before you build.

5On the day

  • Ask which AI tools, if any, you may use in each round.
  • Say your plan before you solve, so they can steer you early.
  • Stuck for a minute? Say what you've tried and ask for a hint. Taking one well counts.
  • Check units and sizes out loud before you give a number.
  • Put a cost per request on a design before anyone asks.