Inference Engineer

Stage by stage

Close each stage out loud, so they always know where you are.

1. 1Targets5 minThose are the targets I'll design to. Next, the memory one request needs.
2. 2Memory per request8 minThat's bytes per request and how many fit. Next, how we batch them.
3. 3Batching8 minThat's how many run at once and the speed each gets. Now caching and drafting ahead.
4. 4Caching and speculation8 minThose are the savings and when they hold. Now the cost of one request.
5. 5Cost per request8 minThat's the cost per request. Last, what breaks first.
6. 6What breaks first8 minThat's the first failure and the guard for it. Where would you like more detail?

How people lose this round

• Divide the GPU's memory by the model size to get the users.The weights are only part of it. The cache left after them, and how long each user's context runs, set the number.
• Set the batch size where throughput peaks.Throughput can keep rising while every request misses its latency target. The targets set the cap.
• Speculative decoding always makes it faster.It uses spare compute, which a large batch doesn't have. Measure it on real prompts at real load.
• The prefix cache is on, so the system prompt is free.Only if the shared text comes first and is identical every time. One timestamp at the top and nothing matches.
• Report the average latency.A few long waits hide in an average. Give the median and the 99th percentile.

Stage by stage

Close each stage out loud, so they always know where you are.

1. 1The situation in one line15 secThat's the situation. Here's what I did.
2. 2What you did60 secThat was my part. Here's how it turned out.
3. 3The result, with a number30 secThat's the result. And here's what I'd change.
4. 4What you'd change15 secThat's what I'd do differently. Happy to go into any part.

How people lose this round

• We decided, we built it, and we fixed it.They're hiring you. With "we" all the way through, they can't tell what you did.
• I can't think of a real failure.Everyone has one. It sounds like you don't notice your mistakes, or won't own them.
• In the end, they saw I was right.A story you win outright sounds one-sided. Show what you took from the other side.
• So to give you some background, the company had three teams...Long setup uses up your time and their attention. One line of situation, then what you did.
• It worked out, so it was the right call.A good result can come from luck. Say why the decision made sense with what you knew then.

Stage by stage

Close each stage out loud, so they always know where you are.

1. 1The problem2 minThat's the problem and why it mattered. Here's the main choice I made.
2. 2Your choice and its alternative3 minThat's what I chose and what I turned down. Now what broke.
3. 3What broke2 minThat's what went wrong and how I found it. Here's where it ended up.
4. 4The result2 minThat's the result, against where we started. Last, what I'd do next.
5. 5What's next1 minThat's what I'd do next. Happy to go further on any part.

How people lose this round

• The stack was Python, Postgres, Airflow and Docker.A list of tools isn't a decision. Say what you chose, what you turned down, and why.
• Nothing really went wrong.Every real project breaks somewhere. It sounds like you didn't build much of it, or didn't watch it run.
• We got 95 percent accuracy.Without the baseline or how rare the positives are, the number means nothing. Say what the simple approach scored.
• I tried logistic regression, then trees, then a neural net.A list of tries isn't a story. Say what each one taught you and why you moved on.
• The team built it, and I helped out.Too modest to score. Name the parts that were yours.