What the words mean

A live experiment where users are randomly split between two versions, and you compare an outcome you chose in advance. Offline tests ask "can it do the job?". An A/B test asks "does it help real users?".

Removing (or switching off) one part of a system and measuring how much the score changes. The set of runs is an ablation study. The word comes from surgery, where it means removing tissue.

The chance that the target model accepts a drafted token. It's high when the draft model is trained on similar data and the text is predictable (code, boilerplate, low temperature), and lower for creative, high-temperature text.

An activation function is the fixed rule a neuron applies to its total before passing it on. ReLU, short for rectified linear unit, is the one above: keep a positive total, and send out 0 for anything else.

Total parameters are every weight the model stores, all experts included, and set the memory. Active parameters are the weights one token uses, and set the compute per token. Qwen3-235B-A22B names both: 235B total, 22B active.

Total parameters count every weight a model stores, and set its memory. Active parameters count the weights one token uses, the shared parts plus k experts per layer, and set its work.

Two running averages per weight, m of the slope and v of the squared slope, both corrected for starting at 0. Kingma and Ba's defaults: a rate of 0.001, β₁ = 0.9, β₂ = 0.999, and a tiny ε = 10⁻⁸ that avoids dividing by 0.

An update rule that keeps two running averages for each weight: of its recent gradients, and of their squares. Each step moves the weight by the learning rate times the first average divided by the square root of the second, so weights with steady gradients move steadily and noisy ones move less. Most language models train with Adam or its variant AdamW.

Adam with weight decay kept out of the division: each weight first shrinks by rate × decay × w, so all weights decay at the same rate. Loshchilov and Hutter proposed it in 2017. PyTorch's AdamW uses a decay of 0.01 unless you set one.

An answer's reward minus the group average (here also divided by the spread) is its advantage. Group relative policy optimisation, GRPO, is the method above. DeepSeek introduced it in 2024. It needs no value model, which makes it lighter than PPO.

A workflow strings model calls and tools along a path written in your code. An agent lets the model choose each next step in a loop. Anthropic's "Building effective agents" (2024) calls both agentic systems.

The agent loop calls the model, runs any tool it asks for, appends the call and the result, and calls again. A model running in it with tools is an agent. Each reply's stop reason says why it stopped: tool_use, end_turn or max_tokens.

The set of tools an agent works through, and how each is named, described and shaped. It's Anthropic's term, by analogy with the human-computer interface: design tools for the jobs the agent does, not to mirror your API's endpoints.

A function that turns many rows into one value, such as COUNT, SUM, AVG, MIN and MAX. With GROUP BY, it gives one value per group.

An aligned instruction from a lower rung fits the higher rules, so the model follows it. A misaligned one works against them, so the model declines that part and carries on helping with the rest.

Data parallel training gives every GPU a full copy of the model and a different slice of each batch. An all-reduce sums a value across GPUs and hands every GPU the result. Gradient accumulation adds up gradients over several micro-batches before one update.

Expert parallelism (EP) puts different experts on different GPUs, so each stores and reads only its share. Every MoE layer then runs an all-to-all: each GPU sends tokens to their experts' GPUs (the dispatch), and the results come back (the combine).

The written description of one level on a scale. Anchors give every grader the same picture of a 4, so their marks match more often. The share of replies two graders mark the same is their agreement.

Whether the answer responds to the question that was asked, regardless of where its facts came from.

Finding passages that are very probably the closest without checking them all, ANN for short. The bucket method is called IVF. In the Faiss library, nlist is the number of buckets and nprobe the number searched.

How many operations a piece of work does for each byte it moves in or out of memory. It's the number along the bottom of the roofline: left of the ridge point the reading sets the pace, right of it the arithmetic does.

Storing pairs, such as names and phone numbers, and later returning the value for one key. Fixed-state layers fall behind attention here. In a phone-book test by Jelassi and colleagues (2024), a 410M transformer beat a 2.8B Mamba model once the book reached about 70 entries.

The zero point is the integer that stands for real 0. Storing a scale and a zero point maps the real minimum and maximum onto the two ends of the integer range, so no codes go to waste. That's asymmetric quantization. Real 0 still lands exactly on an integer, so zeros come back exact.

If you retry until you get a reply, every action happens at least once, and sometimes more. Getting exactly-once delivery over a network isn't possible in general, so the usual fix is to make doing it twice harmless.

Softmax turns scores into weights: raise e to each score, then divide by the total. Attention is the whole move: score the words, softmax the scores, then mix the words' vectors with the weights. A transformer does it for every word at once.

The grid of one head's weights over a piece of text. Reading patterns is the most common first step when people try to work out what a head does.

A token, usually the first, that soaks up a large share of attention weight without the head reading much from it. It's a side effect of softmax, not a sign that the token matters.

Area under the ROC curve: the chance that a randomly chosen positive gets a higher score than a randomly chosen negative, with ties counted as half. 0.5 is a coin toss and 1.0 a perfect ranking. It doesn't depend on any threshold.

Short for automatic differentiation: values that record how they were made, plus a backward() that applies each operation's rule in turn. PyTorch's autograd works this way, on whole arrays of numbers at a time instead of single values.

Producing text one token at a time, where each new token is chosen from a forward pass over everything so far, including the tokens the model itself just wrote.

Activation-aware weight quantization. It measures input channel sizes on calibration data, searches for a scale per channel, multiplies salient weight channels up before rounding and divides their inputs down. Every weight stays in INT4.

The bookkeeping that works out every weight's slope in one pass backwards through the layers, using the chain rule from calculus. A later lesson, "The chain rule by hand", walks one small network through it number by number.

Training many models, each on its own bootstrap sample of the rows (drawn at random with replacement, as many rows as the table has), and averaging their predictions. Short for bootstrap aggregating.

The model you have after pretraining. It continues text very well, but it doesn't answer you: ask it a question and it may carry on with more questions, as if it were writing a quiz. Teaching it to answer is the job of "Instruction tuning", later in this module.

The simplest thing you could do without a model, like flagging nobody or flagging everyone, scored the same way as the model. A model earns its place only by beating the best baseline on the number you care about.

A baseline b is the reward you'd expect before seeing which answer came out. Use the advantage R − b in place of R. Any b that doesn't depend on the sampled answer keeps the average update, and a good one shrinks the noise.

The simplest reasonable system you compare against, run on the same test set with the same metric. The gain you report is the gap between your system and the baseline, not your system's raw score.

A property of the maths kernels: giving the same result for a request whatever else is in its batch. Most fast kernels aren't batch-invariant. Some inference engines now offer batch-invariant modes, at some cost in speed.

Putting several requests into one forward pass, so they all share a single read of the weights. For memory-bound decode it multiplies the tokens a second at almost no cost in speed for each user.

Running several requests through the model in the same step so they share the cost of streaming the weights. The number of requests in a step is the batch size.

Keeping the few best partial paths at each step (say 4), extending each one, and keeping the best 4 again. It finds higher-probability text than greedy and is common in translation. For open-ended writing it tends to make the repetition problem worse, so chat models usually sample instead.

A reference-based score that matches words by the similarity of their embeddings (from a BERT-style model) instead of exact spelling. It gives paraphrases the credit that BLEU and ROUGE miss.

Brain float 16, from Google Brain. FP32's 8 exponent bits with a 7-bit mantissa, so it's really the top half of an FP32 number. Most open models are trained and released in BF16, because training needs range more than it needs digits.

An embedding model used for search. It encodes the question and each passage separately, then compares the two vectors. Fast, because passages are encoded once, ahead of time.

Bias is error that stays however often you redraw the data: the model's average answer is off, because it's too simple for the pattern. Variance is how much a model's answers change from one sample of rows to another.

A weight sets how much one input counts, and in which direction. The bias is added at the end, so the neuron can lean one way even when every input is 0. Training is the search for good values of both.

Two neighbouring tokens, here two letters. A bigram model predicts the next token from the one before it, and nothing else.

Units of surprise. −ln p is in nats. −log₂ p, read "minus log base 2 of p", is in bits. To turn nats into bits, divide by ln 2, about 0.693.

A model's total loss on a text, in bits, divided by the text's length in bytes. Lower is better, and it means the same whatever tokenizer the model uses.

The average cost per request across your models, each model's cost weighted by its share of traffic. It's the number your monthly bill is made of.

A translation score: clipped precision on runs of 1 to 4 words, combined with a geometric mean, times a penalty for being too short. Reported from 0 to 100. Use a standard library (sacrebleu) to compute it, because small changes in how words are split change the score.

A keyword score that adds, for each search word, its IDF times a capped count. k1 sets how fast repeats stop paying; b sets how much length matters, from 0 (not at all) to 1 (fully). L is the length factor, 1 − b + b × length ÷ average.

Testing each of m results at α ÷ m, so the chance of even one false winner across all m stays at α or below. It's strict, so it also misses some real gains. A gentler rule, the false discovery rate of Benjamini and Hochberg, limits the share of flagged results that are false instead.

Estimating how much a number would wobble by resampling your own data with replacement, many times, and recomputing it. The middle 95% of the results is a bootstrap confidence interval. Resample whole items (both models' results together), so the pairing is kept.

A name that points at a commit and moves forward each time you commit on it. HEAD points at the branch you're on. git switch -c name makes one and switches to it.

The current error rate divided by the error rate the SLO allows. A burn rate of 1 uses the budget up in exactly the SLO window. A burn rate of 10 uses it ten times faster.

A byte is a whole number from 0 to 255. UTF-8 is the standard rule for writing each character as one to four bytes, and for reading them back. Nearly every web page is stored this way.

Training a tokenizer by merging. Start from the 256 byte tokens, count every pair of neighbouring tokens, merge the most common pair into a new token, and repeat for as many merges as you chose. GPT-2's and GPT-4's tokenizers were trained this way.

A tokenizer whose starting vocabulary is the 256 byte values, so it can encode any text and never needs an unknown token. The tokenizers of GPT-2, GPT-4 and GPT-4o all start this way.

How well predicted probabilities match how often things really happen. Well calibrated: of the customers given 0.3, about 30% churn. A model can rank customers well and still be badly calibrated.

A model is calibrated when its stated confidence matches how often it's right: of all the answers it gives at 90%, about 90% are correct. Expected calibration error (ECE) is the weighted average gap you just worked out. 0 is perfect.

How well a gate's confidence matches how often it's right. A calibrated gate's answers at about 0.9 are right about 9 times in 10. You check it on held-out labels, band by band. "What a model's probabilities are worth" in Transformers shows how to measure the gap and close it with temperature scaling.

A small set of representative inputs (often a few hundred text samples) run through a model to collect activation statistics before quantizing. If your traffic is code, calibrate on code.

Sending a small share of traffic to a new version and comparing it against the rest (the control). Named after the canaries miners carried to warn of bad air. When a gate fails, an automated rollback sends all traffic back to the old version without waiting for a person.

A random string planted in the system prompt that has no reason to ever appear in a reply. If it shows up in the output, the prompt leaked, and your code can block the reply and log the attempt.

Cheap model first, escalating to a stronger one only when the first answer fails a check. The share that escalates, the escalation rate, sets both the cost and the wait.

Losing old skills while learning a new one, because the same weights get pushed hard toward the new data.

The rule that each position may only attend to itself and earlier positions, done by adding minus infinity to the later scores before softmax. Drawn as a grid, the allowed cells form a triangle: the lower-left half, including the diagonal.

The average of a group of vectors: the middle point of the group. Classifying by nearest centroid is the simplest way to turn a handful of labelled examples into a classifier.

When a change passes through several steps, the overall rate is the product of each step's rate. In calculus these rates are called derivatives.

The exact layout of roles and special tokens a model was tuned with. Each model family has its own, and you have to use the same one in training and in serving.

A chat template is the rule for flattening messages into one run of tokens: which markers open and close each message, and where the role name goes. The markers are special tokens: single tokens meant to come only from the template.

A chunk is one piece of a document: the unit that gets indexed, retrieved and handed on. Chunk size is how long each chunk is, usually counted in tokens. Overlap is how much text neighbouring chunks share, so a fact near a cut still lands whole in at least one chunk. Overlap stores some text twice.

A chunk is one piece of a document: the unit that gets indexed, retrieved and handed on. Chunk size is how long each chunk is, usually counted in tokens. Overlap is how much text neighbouring chunks share, so a fact near a cut still lands whole in at least one chunk. Overlap stores some text twice.

Splitting a long prompt's prefill into pieces that ride along with other users' decode steps, so no single step gets huge. The chunk size comes from the step's token budget.

A guard your code puts around a tool that counts recent failures. Past a threshold it trips, and for a set period calls are refused on the spot instead of being sent. When the period ends a single test request goes out: success resets the guard, failure trips it again.

A marker in the answer that points to the source passage a claim came from. Attribution is the general idea: every claim can be traced to a source.

Whether the passage a sentence cites actually backs it up. The next lesson, "Checking answers against sources", asks the wider question: is each claim backed by anything that was retrieved at all?

Regression predicts a number: a rent, a delivery time, next month's sales. Classification predicts a category from a fixed list: spam or not, one of five teams. Both are supervised learning: every training example comes with its right answer, the label, attached.

The host is the app you use: a chat app, an IDE, an agent. It holds the conversation, calls the model, and runs one MCP client per server it connects to. A server offers tools, resources and prompts. The model only talks to the host.

Least squares picks the parameters with the smallest sum of squared residuals, which is also the smallest MSE. For a line the answer has a closed form: a formula you compute directly, with no steps.

A coefficient is the weight on one feature: how much the prediction changes when that feature rises by 1 and the others stay the same. R² is the share of the label's spread around its average that the model explains: 1 is perfect, 0 is no better than the average.

A measure of agreement between two graders that subtracts the agreement expected by chance: (observed − chance) ÷ (1 − chance). 1 is perfect, 0 is no better than luck, and below 0 is worse. A common rough guide calls 0.41 to 0.60 moderate and above 0.80 strong. Agreement among graders in general is called inter-annotator agreement.

Fluency: does each sentence read as natural language? Coherence: do the sentences fit together in a sensible order? Factuality (or faithfulness, when judged against a source): is every claim supported? Relevance: does it cover what matters for the task? SummEval, a widely used study of summary scoring, had expert raters score news summaries on these four, calling factuality "consistency".

A repository is a project whose history git keeps, in a .git folder inside it. A commit is a saved snapshot of the files you added, with an author, a time, a message and a link to the commit before it.

A group of entities linked to each other more densely than to the rest of the graph, found by a community detection algorithm such as Leiden. Splitting each group again gives levels, from a few broad communities to many narrow ones.

Replacing older parts of the history with a shorter summary once it passes a size limit, while keeping the task and recent turns intact.

When a run succeeds only if every step does, its chance of success is the product of the steps' chances. Small error rates per step grow into large failure rates over a long run.

The record of how a result was made. Each value is a node, linked to the values it was made from. Following the links back from the output reaches every input.

A FLOP is one floating-point operation, an add or a multiply. A run's total is its compute. MFU, model FLOPs utilisation, is the share of the hardware's peak speed a run actually gets. Careful: FLOP/s, per second, is a speed, not an amount.

A pass is compute-bound when its arithmetic takes longer than reading its data, like prefill. It's memory-bound when reading takes longer, like decode. To tell which, compare its operations per byte with the GPU's own ratio.

A model trained at the best split for its budget, about 20 tokens per parameter, is compute-optimal, or Chinchilla-optimal: the lowest loss for that much training compute.

Measure a direction's steepness by how fast its slope grows: 1 per unit of x, 100 per unit of y. The condition number is the steepest over the gentlest, 100 here. Plain steps need a rate below 2 ÷ the steepest, so the gentlest shrinks by at most 2% a step.

A range around a measured score that would contain the true value most of the time (95% of the time, for a 95% interval). Report it next to every score. A score without one hides how little 100 items can tell you.

The confidence threshold is the score a gate's answer must reach before the gate acts on it. Coverage is the share of requests that clear it and get handled by the gate. Raise the threshold, and coverage falls along with the gate's error rate.

The table of a classifier's four counts: true positives, false positives, false negatives and true negatives. scikit-learn's confusion_matrix puts the real classes in rows and the predictions in columns, so for 0/1 labels it reads [[TN, FP], [FN, TP]].

The table of counts of true class against predicted class. For a yes/no task the four boxes are true positives (flagged and urgent, 40), false positives (flagged but not urgent, 30), false negatives (urgent but missed, 10) and true negatives (920).

Steering a model with a written list of principles (the constitution), using a model to critique and revise answers against it instead of asking people to label every example. Anthropic introduced it, and the same pattern scales down to one product's rules.

Masking the scores of rule-breaking tokens at every step, so the model can only produce output that fits a grammar: a JSON Schema, a regular expression, or a fixed list of choices. Also called constrained sampling or guided decoding. The edit itself is a logit mask.

A running instance of an image: a process that sees only the image's files and gets its own network. Delete it and anything it wrote is gone; the image is unchanged.

Everything the model reads in one request: the system prompt, the tool definitions and the history so far. In an agent it grows each turn by that turn's tool call and tool result, and it has to fit in the model's context window.

The number of tokens you allow yourself per call, usually far below the window. Deciding what fills it is often called context engineering.

Making the context smaller while keeping what the model needs. Summarizing, extracting and embedding are the three main kinds.

A wrong answer caused by the other passages in the prompt: two versions of one rule, near-identical rules for different cases, or so much text that the passage that matters gets lost.

Deciding what goes into the context on every turn: what to keep, trim, summarize, store outside, or leave out. The term took hold in 2025, as agents started running for hundreds of steps.

A markdown file in the repo, such as AGENTS.md or CLAUDE.md, that a coding agent loads at the start of every session: commands, layout, conventions and what not to touch. It's sent with every request, so every line has a cost.

How far either side of a word you look for neighbours. Here it's 1. Real systems use somewhere from 2 to 10. A small window tends to group words that do the same job (cat, dog); a wide one also pulls in words from the same topic (cat, fish).

The most tokens a model can read in one call, counting the system prompt, tools, history and the reply it's writing. For an agent, it's a budget that every step spends from.

A vector for one use of a word, built from the word and the words around it. A static embedding gives one vector per word. A contextual embedding gives one per use, so the same word gets a new vector in every sentence.

Deciding who is in the batch again at every decode step, so a finished request's slot goes to a waiting one straight away. Also called iteration-level batching, from the Orca paper (2022). vLLM, SGLang and TensorRT-LLM all work this way by default.

In-batch negatives are the other passages in a batch, used as each query's wrong answers. The contrastive loss is the cross-entropy of picking each query's own passage (score s⁺) from its row, where Σ adds up e^s over the row. van den Oord and colleagues named it InfoNCE in 2018.

An open-loop test sends requests at a set arrival rate whether or not earlier ones have finished, the way real traffic does. The looping kind is closed-loop. The under-counting it causes during slowdowns is called coordinated omission.

The whole pile of text a model trains on, also called its training data or pretraining mix. It's measured in tokens. In English a token is about 4 characters: Llama 3's tokenizer averages 3.94.

Warmup raises the rate from 0 to its peak over the first steps. Cosine decay then lowers it along half a cosine wave to a floor, as progress runs from 0 to 1. The cosine shape comes from Loshchilov and Hutter's SGDR paper (2016).

The dot product of two vectors once both their lengths are divided out. It's the cosine of the angle between them, so it runs from 1 (same direction) through 0 (right angles) to −1 (opposite). Length plays no part.

An embedding is the list of numbers a model gives a text. Cosine similarity scores two embeddings by direction only: 1 for the same direction, 0 for unrelated, −1 for opposite.

A gauge is a reading of right now that can go up or down, like requests waiting. A counter only goes up, like tokens written, and what you use is its rate. A latency histogram is a set of counters: one per bucket, plus a sum and a count.

The confidence threshold is the score a gate's answer must reach before the gate acts on it. Coverage is the share of requests that clear it and get handled by the gate. Raise the threshold, and coverage falls along with the gate's error rate.

A model that scores one (question, passage) pair at a time by reading both together. More accurate than a bi-encoder, and much slower, since nothing can be computed ahead of time.

The loss −ln(p), where p is the probability the network gave the right answer. Also called log loss. When the right answer is no, p is the probability of no: 1 minus the probability of yes.

The cross-entropy loss is −ln of the probability given to the right token, averaged over every position. The perplexity is e raised to that loss.

Cut the data into k folds; train on k − 1 of them and validate on the one left out, k times over, then average. Every row is validated once. Five and ten folds are the usual choices. Also called k-fold cross-validation.

Information reaching the model during training that won't be there, or won't be the same, when it makes real predictions. It inflates every score you compute and disappears in production.

The data mix is the share of training tokens each source gets. An epoch is one full pass over a source. Labs tune the mix by training small models on different mixes and comparing them.

Data parallel training gives every GPU a full copy of the model and a different slice of each batch. An all-reduce sums a value across GPUs and hands every GPU the result. Gradient accumulation adds up gradients over several micro-batches before one update.

Silence on a call that nobody explains. Fill it while a tool runs with a status line: a recording your code plays, or a sentence the model writes before its tool call. Keep it neutral, "Let me check", never "Done", because the tool hasn't answered yet.

A ReLU neuron whose total is negative for every input, so it always outputs 0 and never learns again. One big update that knocks its bias far below zero is a common cause.

The probability at or above which a classifier says yes. Most libraries use 0.5 unless told otherwise. The right value comes from what each kind of mistake costs.

A model that sends each row through a series of yes-or-no questions about its features, where each question depends on the answers before it. It predicts from the training rows that ended up in the same place.

Prefill is the first pass: it reads the whole prompt at once and produces the first reply token. Decode is everything after: one token per pass, each pass taking the token just made as its new input.

Encoding turns text into token IDs: write it as bytes, then apply the merges in the order they were learned. Decoding turns IDs back into text: join each token's bytes and read them as UTF-8. Decoding what you encoded gives back the original text.

Search by comparing embeddings. "Dense" because every number in an embedding carries something. BM25's view of a passage is sparse: a count for every word in the vocabulary, almost all of them 0.

A package your code needs. Direct dependencies are the ones you list; transitive ones are what those need in turn, like numpy for pandas.

A dev set (or validation set) is the data you check against while making choices. A test set is kept back and scored only for a final decision. Once choices are made by looking at a set, it has quietly become a dev set.

A fixed rule for cutting numbers before any merges apply. Llama 1 and 2 split every number into single digits. The tokenizers of GPT-4 and GPT-4o cut runs of digits into chunks of up to three, starting from the left.

The KL divergence KL(p ‖ q) measures how far a distribution q is from p: 0 when they match. Distillation lowers it between the teacher's softened p and the student's softened q, times T², usually plus a small weight w on the usual loss for the true label, at T = 1.

Text in the prompt that looks relevant but doesn't help with the question. A 2024 study of retrieval found that passages related to the question but lacking the answer hurt accuracy more than random passages did.

The idea that words used in the same contexts tend to mean similar things. The linguist J. R. Firth put it this way in 1957: "You shall know a word by the company it keeps." Every method in this lesson leans on it.

A word's document frequency, n, is the number of passages that contain it. Its inverse document frequency, IDF, turns that into a weight that grows as the word gets rarer. With N passages in all:

Multiply two vectors position by position and add up the products. Positive means they point roughly the same way, zero means they're at right angles (unrelated), negative means they point apart. Written a · b, read "a dot b".

Direct preference optimisation: training straight on preference pairs with the loss above, with no reward model and no RL loop. Only two models sit in memory: the one being trained and the frozen reference.

Running a workflow or agent so each step's result is saved as it finishes, and a crashed run resumes from its log instead of starting over. Tools like Temporal are built around this idea.

Batch on a size limit or a time limit, whichever hits first. It suits work where every request is about the same size, like embedding text or classifying it. For chat it still has the run-to-the-end problem.

Setting aside part of the training rows, scoring the model on them after each round, and stopping when that score hasn't improved for a set number of rounds. scikit-learn's HistGradientBoosting holds back 10% and waits 10 rounds, and switches this on by itself when there are more than 10,000 rows.

A model is calibrated when its stated confidence matches how often it's right: of all the answers it gives at 90%, about 90% are correct. Expected calibration error (ECE) is the weighted average gap you just worked out. 0 is perfect.

Installing a project with pip install -e ., so the environment imports it straight from your folder. Edits apply without reinstalling.

A penalty that mixes L1 and L2, with a setting for the share of each (l1_ratio in scikit-learn). It zeroes weak features like lasso and shares weight among related ones like ridge.

A server asking the user for input partway through a call. Form mode shows a small form built from a JSON Schema. URL mode opens a web page, for anything secret such as a sign-in. The user can accept, decline or cancel.

The row of numbers for a token is its embedding, and the table is the embedding table (or embedding layer). It has one row per token in the vocabulary.

An embedding is the list of numbers a model gives a text. Cosine similarity scores two embeddings by direction only: 1 for the same direction, 0 for unrelated, −1 for opposite.

Encoding turns text into token IDs: write it as bytes, then apply the merges in the order they were learned. Decoding turns IDs back into text: join each token's bytes and read them as UTF-8. Decoding what you encoded gives back the original text.

Deciding that the caller has finished, so the agent may reply. A silence threshold ends the turn after a set pause: shorter ones cut people off, longer ones slow every turn. A detector that also reads the words can wait through a pause and still answer a finished sentence quickly.

The time from sending a request to receiving its last token, roughly TTFT + (output tokens − 1) × time per token. For output a program consumes whole, add the parsing and checking after it: that sum is the time until the answer is usable.

One method and path an API answers, such as POST /runs, and the function behind it.

Deciding which names refer to the same real thing, and merging them into one node. Cheap checks go first: exact match after lowercasing and dropping words like "Ltd", then embedding similarity above a threshold, then a model call for the doubtful pairs. Merging too eagerly is the opposite mistake: "Norfield Metals" and "Northfield Metal Works" may be two companies.

A named setting, like HOME or DATA_DIR, that each process gets a copy of when it starts. export adds a shell variable to the list that gets copied, and Python reads the list through os.environ.

The data mix is the share of training tokens each source gets. An epoch is one full pass over a source. Labs tune the mix by training small models on different mixes and comparing them.

Error analysis is reading outputs one by one, noting what went wrong, then grouping and counting the notes. Each pile is a failure mode: one way the system keeps going wrong, such as stale station data.

Reading a sample of real traces, noting the first thing that went wrong in each, and counting the notes by kind, so you fix the most common failure first.

The range a score could move if you reran it on fresh cases of the same kind: about two wobbles either way. "Is the new version really better" turns it into a proper confidence interval.

The SLI (service level indicator) is the measured number. The SLO (service level objective) is the target for it. The error budget is how much failure the target leaves room for. Teams often slow down releases when the budget runs low.

Passing a case the gate isn't sure about to something stronger: the large model, or a person. A gate that's unsure escalates instead of guessing, so every route needs somewhere to escalate to.

A lookup table from a hash of the full request to the stored response. A hit needs every character to match, so it's safe, but "opening hours?" and "opening hours" miss each other. Trimming spaces and lowercasing before hashing helps a little.

Comparing the query with every stored vector (also called brute force or flat search). Always finds the true nearest neighbours. The baseline every faster index is measured against.

The number a program returns when it ends: 0 for success, anything else for failure. The shell keeps the last one in $?, and &&, || and set -e act on it.

Expert parallelism (EP) puts different experts on different GPUs, so each stores and reads only its share. Every MoE layer then runs an all-to-all: each GPU sends tokens to their experts' GPUs (the dispatch), and the results come back (the combine).

The average number of a layer's experts that some token in the step picks, when B tokens each pick k of E at random. It starts at k and nears E once B is a few times E ÷ k. ^B means multiplied by itself B times.

The share of the total spread that a principal component keeps, here 90% and 10%. scikit-learn reports it as explained_variance_ratio_, and you usually keep enough components to reach a share you choose, such as 90%.

Gradients that shrink (vanish) or grow (explode) as they pass back through many layers, because each layer scales them by a bit less, or a bit more, than 1.

The exponent bits pick a power of two, so they set the range: how big or how small a number can get. The mantissa bits are the digits, so they set the precision: how finely you can tell nearby numbers apart. Floating point formats differ only in how many bits go to each.

With exponential backoff each wait is double the last, up to a ceiling. Jitter picks each wait at random below that amount ("full jitter" draws anywhere from zero up to it), so clients that failed at the same moment come back at different moments.

Doubling the longest wait before each retry (exponential backoff) and drawing the actual wait at random below it (jitter). Drawing it between 0 and the limit is full jitter, and a cap stops the limit growing forever.

The harmonic mean of precision and recall, 2 × P × R ÷ (P + R), which is the same as 2 × TP ÷ (2 × TP + FP + FN). It's high only when both are high. It ignores true negatives and treats both kinds of mistake as equally bad.

Precision is true positives ÷ everything flagged (how much to trust a flag). Recall is true positives ÷ everything really positive (how much it catches). F1 combines them as 2 × P × R ÷ (P + R), which here is 2 × 0.57 × 0.8 ÷ 1.37 = 0.67. F1 is low if either one is low.

Fluency: does each sentence read as natural language? Coherence: do the sentences fit together in a sensible order? Factuality (or faithfulness, when judged against a source): is every claim supported? Relevance: does it cover what matters for the task? SummEval, a widely used study of summary scoring, had expert raters score news summaries on these four, calling factuality "consistency".

Error analysis is reading outputs one by one, noting what went wrong, then grouping and counting the notes. Each pile is a failure mode: one way the system keeps going wrong, such as stale station data.

Whether every claim in the answer is backed by the retrieved passages. Also called groundedness. An answer can be true in the world and still unfaithful, if nothing retrieved said it.

The ordered list of places a request goes when the one before it fails, all inside one deadline. Gateways, the proxies teams put in front of their models, let you write it as settings.

A false positive is a yes that should have been a no: a false alarm. A false negative is a no that should have been a yes: a miss. Their costs are rarely equal.

A false positive is a yes that should have been a no: a false alarm. A false negative is a no that should have been a yes: a miss. Their costs are rarely equal.

The number of inputs a neuron reads. Starting-weight rules are built on it: about 1/√(fan-in) for tanh, known as Xavier or Glorot initialisation (the original averages fan-in and fan-out), and √2/√(fan-in) for ReLU, known as He initialisation, because ReLU zeroes about half its outputs.

A merge where the branch being merged into hasn't moved, so git only moves its label forward. No new commit is made.

A feature is a column you know at the moment you need a prediction, used as an input. The label is the column you want to predict: known for past rows, blank for new ones. Each row, features and label together, is one example.

The average number of tokens a tokenizer uses per word. The lower it is for a language, the cheaper, faster and roomier a model is in that language. Measure it on real text, with the tokenizer of the model you'll use.

Putting solved examples in the prompt. Each example is a shot: no examples is zero-shot, one is one-shot, and a handful is few-shot.

Fine-grained experts are many narrow ones in place of a few wide ones, with more picked per token at the same compute. Shared experts run for every token, with no router.

Continuing to train an existing model on your own examples (usually input and ideal output pairs) so its behaviour changes. Often done with adapters such as LoRA, which train a small add-on instead of every weight.

Training a ready-made model further on your own pairs, with the same contrastive loss, so it ranks your passages better. It counts only if held-out questions, split by passage, score better than before.

A model is a function that turns features into a prediction. Its parameters are the numbers left open in it: one weight per feature, plus one on its own. Fitting, or training, sets them so the predictions on known examples come as close as they can to the labels.

Prompt chaining feeds each call's output to the next. Routing labels the input and sends it to one handler. Parallelization runs calls at once, on slices or as votes. Orchestrator-workers lets one call plan the subtasks. Evaluator-optimizer loops a writer and a grader.

Comparing training ideas by giving each the same wall-clock time on the same machine, then scoring the result. Whatever a change does to model size, batch size or speed, it has to pay for it inside the budget.

An option after a command's name that changes what it does, starting with a dash, like -n 3 or -l. Many also have a long form with two dashes, like --lines=3.

The best-known tiled method (Dao and colleagues, 2022). It keeps each tile in the GPU's small fast on-chip memory instead of writing the grid out to main memory. Most serving and training code uses it or something like it.

One floating-point operation: a single multiply or add on decimal numbers. Work is counted in FLOPs. Hardware speed is quoted in FLOP per second, usually written FLOPS, and modern GPUs reach hundreds of trillions (teraFLOPS, TFLOPS).

A FLOP is one floating-point operation, an add or a multiply. A run's total is its compute. MFU, model FLOPs utilisation, is the share of the hardware's peak speed a run actually gets. Careful: FLOP/s, per second, is a speed, not an amount.

Fluency: does each sentence read as natural language? Coherence: do the sentences fit together in a sensible order? Factuality (or faithfulness, when judged against a source): is every claim supported? Relevance: does it cover what matters for the task? SummEval, a widely used study of summary scoring, had expert raters score news summaries on these four, calling factuality "consistency".

A push that replaces the server's branch even though it isn't an ancestor of yours. Needed after a rebase. Use --force-with-lease, which refuses if the server has commits you haven't seen.

A column that holds another table's primary key, like orders.customer_id. It's how one row points at another.

A hidden layer sits between the inputs and the output. Its answers never appear in the data: the network makes up its own in-between questions, here "is either on?" and "are both on?". A forward pass is one run from inputs to output, a layer at a time.

ZeRO (Rajbhandari and others, Microsoft, 2019) shards training state across data-parallel GPUs in three stages: Adam's state, then gradients, then weights. PyTorch's FSDP, fully sharded data parallel, is built on the same idea.

Training every parameter of a pretrained model. It's the most flexible option and the most expensive one.

A rubric part that must pass for the reply to score at all. Fail it and the score is 0. Safety and privacy are the usual gates. The other parts are combined with weights.

A cheap decision step in front of an expensive one. It answers one question with a label from a fixed set plus a confidence, acts when the confidence is high enough, and passes everything else on. It's a fast path, never the only path.

A layer that carries a fixed-size state from token to token and, before each update, scales it by a gate between 0 and 1 set by the current token. Mamba (Gu and Dao, 2023), a state-space model, has this shape, and so does Gated DeltaNet.

A gauge is a reading of right now that can go up or down, like requests waiting. A counter only goes up, like tokens written, and what you use is its rate. A latency histogram is a set of counters: one per bucket, plus a sum and a count.

How well a model does on new examples, not the ones it was fitted on. The only honest way to measure it is on rows the model never saw.

1 − Σ (share of each class)², summed over the classes. It's the chance that two rows drawn at random from the group, with replacement, have different labels: 0 for a pure group, 0.5 for an even mix of two classes.

A file of patterns, one per line, for paths git should leave out. It only affects files git isn't tracking yet. A file that's already committed stays tracked until git rm --cached takes it out of the index.

Your saved collection of test inputs and their expected results, covering the jobs the prompt must keep doing. Every prompt version runs against it. Add a case each time a bug reaches users, so that bug can't come back unnoticed.

Reward hacking is a model scoring well on the reward without doing what the reward was meant to measure. Goodhart's law is the older idea from economics: when a measure becomes a target, it stops being a good measure.

Requests per second that succeeded and met every latency target. Some dashboards give it as a share instead: 80% of requests were good. Multiply the share by the request rate to get the rate.

A post-training method that quantizes a layer one column at a time and updates the remaining weights to cancel each rounding error, measured on calibration inputs. The paper's maths uses the inputs' second-order statistics (a Hessian) to size the nudges.

Running with fewer features instead of failing outright, decided in advance, feature by feature: a weaker model where its scores hold up, a hand-off to a person where they don't, and a clear message about what's off.

The slopes of the loss, one per weight, collected into a list. For one weight it's the single slope you just used.

Data parallel training gives every GPU a full copy of the model and a different slice of each batch. An all-reduce sums a value across GPUs and hands every GPU the result. Gradient accumulation adds up gradients over several micro-batches before one update.

Adding small trees one after another, each fitted to what the model so far still gets wrong, and each scaled down by a learning rate before it's added. For squared error, what's left is the residuals. For other losses it's the slope (gradient) of the loss, which gives the method its name.

Comparing a gradient from the backward pass with an estimate from nudging, called a finite difference. They should agree to several decimal places. If they don't, the backward pass has a bug.

Gradient descent fits a model by repeated small steps: work out the gradient of the loss for every parameter, move each one a little against it, and repeat until the loss stops falling. The learning rate scales each step.

The loop: work out the gradient, move every weight a small step against its own slope, repeat. The size of the step is set by the learning rate, 0.075 above.

Retrieval over a graph built from the documents. In Microsoft Research's version, a model extracts the graph, every community gets a written summary, and questions about the whole collection (global questions) are answered from the summaries, batch by batch, then combined. Questions about particular entities (local questions) start from the matching entities and walk out from them.

Taking the top-ranked token at each step, with no dice roll. In code it's an argmax: max over the probabilities, or over the logits, since softmax keeps their order.

Two ways to pick hyperparameters. Grid search tries every combination from lists you give. Random search draws combinations at random from ranges you give, which covers each important setting more finely for the same number of fits.

An answer built only from the passages in the prompt, so every claim can be traced back to one of them. The rest of this module is about making answers grounded, and checking that they are.

Rows from the same person (or patient, device or store) landing in both the training and the test rows, so the model is scored partly on people it has already seen. Split by group, for example with GroupKFold, to stop it.

A group split keeps all the rows of one customer, patient or house on the same side. A time split trains on older rows and tests on later ones, the way the model will be used: on a future it hasn't seen.

One scale per output row is per-channel. One scale per fixed-size run of weights (often 128) is group-wise. Smaller groups mean finer rulers and more scales to store.

Query heads split into groups, with each group sharing one key head and one value head. With a single shared pair it's called multi-query attention (MQA), and one pair per head is multi-head attention (MHA). In the cache formula, count the key-and-value heads.

An answer's reward minus the group average (here also divided by the spread) is its advantage. Group relative policy optimisation, GRPO, is the method above. DeepSeek introduced it in 2024. It needs no value model, which makes it lighter than PPO.

A criterion that must hold whatever happens to the main metric, such as "at least 90% of questions get an answer". Failing a guardrail blocks a change on its own. Some allow no failures at all: "never promise a refund the policy doesn't allow".

Any check your app runs around the model to keep it inside the lines: input filters, output filters, topic limits. Guardrails sit in code you control, so they keep working when the model is swapped.

A wrong passage that looks right: it shares the query's words, or the model already scores it high. Hard negatives are mined with BM25 or with the model itself, and added to the batch.

Everything around a model that turns its text into actions: the loop, the tool definitions, the system prompt, the permission checks and the choice of what goes into the context. Swap it and the same model behaves like a different agent.

High-bandwidth memory: the stacked memory chips next to the GPU that hold the weights and KV cache. Its speed is the memory bandwidth on spec sheets, about 3.35 TB/s on an H100 SXM. The GPU also has much smaller, much faster memory on the chip itself, which good kernels use to avoid trips to HBM.

Each slice with its own attention is a head. Running many heads in parallel and joining their outputs is multi-head attention. The final grid that mixes them is the output projection, often written W_o.

Git's name for the commit you're on now, usually the newest commit of your branch. HEAD~1 is its parent, and HEAD~2 the commit before that.

The length of each head's query, key and value vectors. It's the model's width divided by its heads: 4,096 ÷ 32 = 128 for Llama 2 7B. Config files call the width hidden_size and the heads num_attention_heads, and some list head_dim directly.

A cheap endpoint, such as GET /health, that answers 200 when the app is up. Hosts call it to decide when a new version is ready and whether a running one needs restarting.

The tokens that have received the most attention, summed over every query so far. H2O, the Heavy-Hitter Oracle, splits each head's budget of B entries evenly between them and the most recent tokens.

Examples kept out of training and used only to measure the model. A loss on held-out data tells you how well the model does on text it hasn't seen, which is the only kind it will meet in use.

A hidden layer sits between the inputs and the output. Its answers never appear in the data: the network makes up its own in-between questions, here "is either on?" and "are both on?". A forward pass is one run from inputs to output, a layer at a time.

Trying one change at a time and keeping it only if the score beats the best so far. Each kept change becomes the starting point for the next.

Hierarchical navigable small world: the layered neighbour graph above. Three settings matter. M is how many links each point keeps (often 16). ef_construction is how hard the build looks for good links. ef_search is how many candidates the bottom-layer search keeps in its list.

Your own command that a coding agent runs on an event, such as just before a tool call. A hook that exits with code 2 blocks the call and tells the model why.

The host is the app you use: a chat app, an IDE, an agent. It holds the conversation, calls the model, and runs one MCP client per server it connects to. A server offers tools, resources and prompts. The model only talks to the host.

The verb at the start of a request. GET reads, POST creates, PUT replaces, PATCH changes part of something and DELETE removes it.

A person who approves a model's output before it takes effect. Here, a support agent reads every draft before it goes to a customer.

A stack that mixes linear layers (linear attention, gated recurrences or state-space layers) with a few full-attention layers in a fixed pattern. Qwen chose 3 to 1, writing that linear attention is fast but weak at recall.

Running a keyword retriever, like BM25, and a dense retriever on the same question, then merging their ranked lists into one.

Query rewriting has a model restate the user's question as one or more searches, phrased the way the documents are written. HyDE (hypothetical document embeddings) has a model draft an answer first, then searches with the draft's embedding instead of the question's. Each adds a model call before the search.

A setting chosen before training rather than learned by it: C in logistic regression, the depth of a tree, k in k-means. Picked by comparing validation or cross-validation scores.

An operation is idempotent if doing it twice leaves things the same as doing it once. An idempotency key is the label that lets a server turn a non-idempotent action, like a charge, into one.

A word's document frequency, n, is the number of passages that contain it. Its inverse document frequency, IDF, turns that into a weight that grows as the word gets rarer. With N passages in all:

A packaged, read-only filesystem (an operating system's basic files, Python, your packages and code) plus the command to start. Built from a Dockerfile, stored in a registry.

Filling missing values with a stand-in, such as the column's median from the training rows. scikit-learn's SimpleImputer does it, and add_indicator=True adds a 0/1 column marking which rows were filled.

In-batch negatives are the other passages in a batch, used as each query's wrong answers. The contrastive loss is the cross-entropy of picking each query's own passage (score s⁺) from its row, where Σ adds up e^s over the row. van den Oord and colleagues named it InfoNCE in 2018.

Doing a task by following examples read in the prompt. Nothing is trained, so the skill lasts only as long as the examples stay in the prompt.

The one person in charge of an incident while it lasts. They keep the picture of what's happening, hand out the work, and decide; they don't do the hands-on fixing themselves.

Decoding that runs only the newest token through the model at each step and reads everything earlier from the KV cache. Serving engines decode this way, and so does Hugging Face transformers unless you pass use_cache=False.

A sorted copy of some columns that points back at the rows, so the database can find matching rows without reading the whole table. It speeds up reads and slows down writes.

Injection that reaches the model through material it pulls in as part of its task: a page it browses, an email it summarises, a retrieved chunk, a tool's reply. The person using the app never wrote it, and usually never sees it.

A head that finds an earlier copy of the current token and copies forward whatever came after it: "A B ... A" → predict "B". It needs a previous-token head in an earlier layer to work, so a model needs at least two layers to build one.

The total squared distance from each row to its centre, k-means' measure of how tight the groups are. Lower is tighter. scikit-learn reports it as inertia_.

Input drift is a change in what comes in: topics, length, language, kinds of user. Output drift is a change in what the model sends back: length, tone, refusals, format. Either can come with quality drift, a change in how often answers are right. That's the one that matters, and the hardest to measure.

A tool definition is all the model knows of a tool: a name, a description and an input schema, the JSON Schema for its arguments. Anthropic's API calls the schema input_schema, OpenAI's parameters and MCP's inputSchema.

Writing the prompt so injected text has a harder time winning: fencing untrusted input, saying plainly that it's data, and restating the task after it. Worth doing, never enough on its own.

The order a model gives to instructions when they clash: the system prompt first, then the user. Text from tools, files and web pages has no authority at all. The higher one wins, on the point of the clash only.

Training a base model on request and answer pairs so it follows requests. It's also called supervised fine-tuning, or SFT. The result is an instruct model, or chat model.

The gap between two tokens in a stream. You'll also see time per output token (TPOT), which is usually the average gap over a whole reply. TPS is one second divided by that average.

When the effect of removing two parts together differs from the sum of removing each alone. Test pairs of parts that do similar jobs, because they're the ones most likely to hide behind each other.

Shared runs divided by all the different runs is the Jaccard similarity. Comparing every pair among billions of documents is impossible, so MinHash squeezes each document's runs into a short fingerprint, and only documents whose fingerprints partly match get compared.

With exponential backoff each wait is double the last, up to a ceiling. Jitter picks each wait at random below that amount ("full jitter" draws anywhere from zero up to it), so clients that failed at the same moment come back at different moments.

Combining rows of two tables where a condition matches, usually a foreign key against a primary key. An inner join keeps matched rows only; a left join keeps every row of the first table.

A text format for data: objects in {}, lists in [], strings in double quotes, numbers, true, false and null. Python's json module reads and writes it, and most APIs speak it.

A standard way to describe the shape of JSON: which keys, which types, which keys are required, which values are allowed (enum), and whether extra keys are forbidden (additionalProperties: false). Most model APIs accept it directly.

Clustering that repeats two steps until nothing changes: assign each row to the nearest of k centres, then move each centre to the mean of its rows. You choose k.

This method. k is the number of groups you ask for. It finds groups but doesn't name them; you read a few items from each to see what it found.

The three roles in a soft lookup. The query is what you're asking for. Each key is what an entry offers to be matched against. Each value is what that entry hands back. Q, K and V for short.

The KL divergence KL(p ‖ q) measures how far a distribution q is from p: 0 when they match. Distillation lowers it between the teacher's softened p and the student's softened q, times T², usually plus a small weight w on the usual loss for the true label, at T = 1.

Reinforcement learning from human feedback: a reward model trained on preferences, then RL against it. The drift penalty is the KL penalty; KL divergence measures how far apart two models' probabilities are. The RL method used most often is PPO.

A triple is one fact written as (subject, relation, object), such as (Norfield Metals, supplies, brake disc). A knowledge graph is a set of triples stored as a graph: each thing (an entity) is a node, and each triple is an edge from subject to object that carries its relation, so "supplies" and "located in" are different kinds of edge.

The stored keys and values of every earlier token, for every layer and every head. Prefill fills it for the prompt, and each decode step adds the newest token's keys and values.

The rule that decides which keys and values leave when a cache is full. Every policy loses something, since an evicted entry is gone for good. The budget sets the memory, and the policy sets what's lost.

What one token leaves behind, times the tokens in each request, times the number of requests. Every factor is a dial: double any one and the cache doubles.

The share of the cache's blocks that hold some request's tokens, from 0 to 1 (vLLM's vllm:kv_cache_usage_perc). Near 1 with the preemption counter climbing means the engine admitted requests the cache can't carry to the end.

A feature is a column you know at the moment you need a prediction, used as an input. The label is the column you want to predict: known for past rows, blank for new ones. Each row, features and label together, is one example.

The set of answers the examples allow, such as order, repair and return. With the format and the kind of input, it's much of what a few-shot prompt teaches.

Linear regression with an L1 penalty, α times the sum of the weights' sizes. It pushes every weight towards 0 by a similar amount and sets the weakest to exactly 0, so it also picks features.

Counts of requests in fixed time buckets, kept by each server. You can add two servers' histograms bucket by bucket and read percentiles from the sum. The percentile is only as precise as the bucket it lands in.

The result of one Dockerfile instruction, stored as a set of file changes. An image is a stack of layers, and unchanged layers are reused from the build cache.

That rescaling step, done on each token's vector separately. Llama models use a lighter version called RMSNorm, which skips subtracting the mean and only divides by the size.

A group at the bottom of a tree, where the questions stop. A classification leaf predicts its majority class, with its class shares as probabilities. A regression leaf predicts the average of its rows.

Gradient descent fits a model by repeated small steps: work out the gradient of the loss for every parameter, move each one a little against it, and repeat until the loss stops falling. The learning rate scales each step.

How the step size changes over a run. Warmup followed by cosine decay is the most common shape. With progress running from 0 to 1 over the decay:

Give the agent only the tools, and each tool only the permissions, that this task needs. The list of permitted tools is its allowlist; anything not on it is refused.

Give each tool, and the model behind it, only the access its job needs, and nothing more. When an injection succeeds, least privilege decides how small the damage stays.

Least squares picks the parameters with the smallest sum of squared residuals, which is also the smallest MSE. For a line the answer has a closed form: a formula you compute directly, with no steps.

A judge's tendency to score longer answers higher whatever they say. Also called verbosity bias. Check for it by padding answers, as above, or by comparing scores for answers of different lengths that people rated equally.

Simon Willison's term for one model having access to private data, exposure to text an outsider could have written, and any channel that can carry data out. When a design has all three, assume injected text can steal the data. Cut one, or put a person in front of the channel out.

A function is linear when its graph is a straight line, like w × t + b. Add linear functions, or feed one into another, and you get another linear one. An activation between layers stops the folding.

Attention whose score splits into a query part times a key part, φ(q) · φ(k), so it runs as a running sum. The softmax's e^(q · k) can't split. The state is a d × d grid per head. Katharopoulos and colleagues (2020) used φ(x) = elu(x) + 1, which keeps scores positive.

When the batch grows k times, multiply the learning rate by k and leave everything else alone. Reach the new rate with a gradual warmup, not a jump.

A yes-or-no job is linearly separable when one straight line (or, with more inputs, one flat cut) can put every yes on one side and every no on the other. AND and OR are. XOR isn't.

Requests in flight = arrival rate × time each one spends inside. Written L = λ × W: L is the average number inside, λ (the Greek letter lambda) the average arrival rate, and W the average time each one spends inside. It works with averages, so the count at any one moment wobbles around L.

A language model used to grade another model's output against a rubric. Also called LLM-as-a-judge. It's the usual way to score open-ended answers at scale, once you've checked it agrees with people.

An extra term in the training loss, smallest when tokens and router probability spread evenly over the experts. Also called the auxiliary loss. The Switch Transformer used α = 0.01.

A file listing the exact version of every package, direct and transitive, so an environment can be rebuilt identically. Written by a tool, committed with the code.

The average of −ln(the probability the model gave to what actually happened). 0 for a model that's always sure and always right, with no ceiling for one that's sure and wrong. Also called cross-entropy or logistic loss.

The natural log of a token's probability, often called a logprob. 0.5 becomes about −0.69 and 0.01 becomes about −4.6. Many APIs can return logprobs for each generated token, and adding logprobs is the same as multiplying probabilities.

ln of the total of e^score over all scores, worked out as m + ln(total of e^(score − m)), with m the largest score. Same answer, no overflow. It's the second piece of the cross-entropy loss.

A weighted sum of the features passed through the sigmoid, read as the probability of the yes class. Despite the name, it's used to classify, not to predict a number.

The raw scores a network gives before they're turned into probabilities. Any size, either sign.

The raw scores, one per vocabulary token, before they're turned into probabilities. The grid that produces them is the output layer (also called the unembedding or LM head).

The history in the context window is often called working memory: fast, visible to the model, small. A store that lasts across runs and is searched on demand is long-term memory.

LoRA (low-rank adaptation) trains two thin matrices, B and A, beside a frozen weight matrix. The rank r is their thin side. Ranks from 8 to 64 are common.

A loss is one number for how wrong a network is on some examples. Lower is better, and training searches for weights that make it small. Mean squared error, or MSE, is the loss you just worked out: square each error, then average.

The rule for which positions count toward the loss. In instruction tuning it covers the assistant's tokens only.

The loss for every possible setting of the weights. With one weight it's a curve, with two it's a surface like a bowl, and with millions nobody can draw it. Training looks for a low spot on it.

The drop in accuracy when the needed information sits in the middle of a long prompt instead of near the start or end. Named after that 2023 paper. The pull toward the start is called primacy bias, and the pull toward the end recency bias.

The plain average of a metric across classes, so each class counts equally however rare it is. Accuracy (and the micro average) weight each ticket equally instead, so big classes dominate. scikit-learn's classification_report prints exactly this table, with the macro average and a weighted average underneath.

Two tilts that examples add. Majority label bias pulls unsure answers toward the label most examples share. Recency bias pulls them toward the label of the last example.

The exponent bits pick a power of two, so they set the range: how big or how small a number can get. The mantissa bits are the digits, so they set the precision: how finely you can tell nearby numbers apart. Floating point formats differ only in how many bits go to each.

A program or web service that offers tools, resources and prompts to an agent over the Model Context Protocol. A local one runs on your machine with your rights. Treat its descriptions and results as untrusted text.

Turning many token vectors into one by averaging them, number by number. It's cheap and often decent, and it throws away word order completely.

A residual is actual minus predicted, for one row. The mean squared error, MSE, is the average of the squared residuals. Its square root is back in the label's units: a typical miss.

A loss is one number for how wrong a network is on some examples. Lower is better, and training searches for weights that make it small. Mean squared error, or MSE, is the loss you just worked out: square each error, then average.

A pass is compute-bound when its arithmetic takes longer than reading its data, like prefill. It's memory-bound when reading takes longer, like decode. To tell which, compare its operations per byte with the GPU's own ratio.

A commit with two parents, made when two branches have both moved on. It joins their histories without changing any commit that already existed.

What happens when two branches changed the same lines: git stops the merge, writes both versions into the file between <<<<<<<, ======= and >>>>>>> markers, and waits for you to choose.

Adding B·A into W after training, so the tuned model is one ordinary matrix per layer with no extra cost when it runs.

Fields stored next to each chunk (country, team, document type, date, who may see it), and a condition on those fields that a chunk must meet to be returned.

A metric is the number you compute, here the share of replies that pass. A threshold is the value it must reach. Both belong to a named test set: the fixed list of inputs you score on.

A FLOP is one floating-point operation, an add or a multiply. A run's total is its compute. MFU, model FLOPs utilisation, is the share of the hardware's peak speed a run actually gets. Careful: FLOP/s, per second, is a speed, not an amount.

Shared runs divided by all the different runs is the Jaccard similarity. Comparing every pair among billions of documents is impossible, so MinHash squeezes each document's runs into a short fingerprint, and only documents whose fingerprints partly match get compared.

An aligned instruction from a lower rung fits the higher rules, so the model follows it. A misaligned one works against them, so the model declines that part and carries on helping with the rest.

Storing different parts of one model in different formats, chosen by how sensitive each part is. llama.cpp's Q4_K_M files, for example, keep a few tensors at 6 bits and most at 4.

Storing different parts of a model in different number formats. In an MoE, the experts hold most of the memory, so they get the fewest bits, while attention, the router and the embeddings keep more.

Several MLPs, the experts, plus a router that scores them for each token. Only the top k run, blended by their weights renormalised over the kept ones. It takes the place of the block's MLP.

The per-token two-layer network inside each block, short for multilayer perceptron. Papers also call it the feed-forward network, or FFN. It holds most of a block's weights: in GPT-2 small, 2 × 768 × 3,072 ≈ 4.7 million, against about 2.4 million for attention.

A model is a function that turns features into a prediction. Its parameters are the numbers left open in it: one weight per feature, plus one on its own. Fitting, or training, sets them so the predictions on known examples come as close as they can to the labels.

The bytes a decode step really reads per second (weights plus KV cache, times steps per second), divided by the GPU's peak memory bandwidth. It's the decode version of how close you are to the roof.

The code that looks at each request and picks the model that answers it. Your app asks for an answer, and the router decides who gives it. Its quality is each model's accuracy on the requests it's sent, weighted by traffic.

A velocity for each weight: β times the old velocity, plus the new slope. The weight moves by the learning rate times the velocity, so steady slopes count up to 1 ÷ (1 − β) times: 10 for β = 0.9. In PyTorch: torch.optim.SGD with momentum=0.9.

Mean reciprocal rank: the average of 1 / (rank of the first relevant result) across a set of questions.

Each slice with its own attention is a head. Running many heads in parallel and joining their outputs is multi-head attention. The final grid that mixes them is the output projection, often written W_o.

A question whose answer joins facts from two or more passages, where each fact leads to the next. Each step along a link is a hop. "Which products depend on Norfield Metals?" takes two: supplier to part, then part to product.

Running many tests and paying attention to the ones that pass. The more you run, the more false winners you get. The chance of at least one across the batch is the family-wise error rate: 1 − (1 − α)ᵐ for m independent tests at level α (read "alpha", one test's false-alarm rate).

Short for "not a number": what floating-point maths gives for things like 0 ÷ 0 or infinity minus infinity. Anything computed from a NaN is NaN too, so one bad value spreads through a whole network within a step or two.

Units of surprise. −ln p is in nats. −log₂ p, read "minus log base 2 of p", is in bits. To turn nats into bits, divide by ln 2, about 0.693.

Normalised discounted cumulative gain. Sum grade ÷ log₂(rank + 1) over the list, then divide by the same sum for the ideal order. log₂(3) is about 1.58, which is where the 0.63 for rank 2 comes from.

A test that hides one fact (the needle) at a chosen depth in a long filler document (the haystack) and asks the model to find it. Run across many depths and lengths, it gives a grid of passes and misses.

Minus the log of the probability a model gave to what actually happened, averaged over the data. It's the same number as the cross-entropy loss a network is trained to lower.

Cross-validation wrapped around a search that itself uses cross-validation. The inner loop chooses the hyperparameters; the outer loop scores each choice on a fold the search never saw. It measures the procedure, not one fixed model.

Guessing each token from the ones before it. It's self-supervised: the labels come from the text itself, so any text at all is training data, with nobody paid to label it.

How much a score moves between runs of the same unchanged setup. A change that moves the score by less than this hasn't shown anything yet.

A customer whose heavy use slows down or blocks everyone else on shared capacity. A cap per team contains it: the bulk job just runs slower.

Dividing a vector by its length so its length becomes 1. The result is a unit vector. It keeps its direction and loses its size.

MCP messages are JSON-RPC 2.0. A request has an id and a method, and gets one response with the same id, holding a result or an error. A notification has no id and gets no reply. Since 2026-07-28 only the client sends requests.

SQL's marker for a missing or unknown value. Any comparison with it is unknown, so test for it with IS NULL or IS NOT NULL.

Read it as "order n": grows in step with n. With a cache, a decode step does fixed weight work plus attention and reads that are O(n) in the n cached tokens. Without one, its weight work is O(n) too, and its attention, redone for every token, is O(n²).

Work that grows with the square of the length n. Read O(n²) aloud as "order n squared": double n and the work goes up about four times. Work that grows in step with n is linear, O(n).

The chance something happens divided by the chance it doesn't, p ÷ (1 − p). Their natural log is the log-odds, which is the weighted sum that logistic regression adds up before the sigmoid.

The right answer written as a list with a 1 in its place and 0 everywhere else, like (1, 0, 0) for tea.

Turning a column of categories into one 0/1 column per category, so each row has a single 1. scikit-learn's OneHotEncoder does it and remembers the categories it saw in training.

A task where you can't list the steps, or say how many there will be, before you see the input. That's the case Anthropic gives for an agent. Anything else gets a workflow, or one well-prompted call.

An open-loop test sends requests at a set arrival rate whether or not earlier ones have finished, the way real traffic does. The looping kind is closed-loop. The under-counting it causes during slowdowns is called coordinated omission.

The lead is the orchestrator (some frameworks say supervisor). Each helper is a subagent: a full agent loop started by the orchestrator, given a narrow task and a fresh context, that returns a condensed result.

A forest's score in which each row is predicted only by the trees that never trained on it. In scikit-learn, pass oob_score=True and read oob_score_ after fitting.

Hidden dimensions whose activations are consistently much larger than the rest, across tokens and layers. They are why naive INT8 activations break large models, and why newer methods look at activations before they quantize weights.

Input drift is a change in what comes in: topics, length, language, kinds of user. Output drift is a change in what the model sends back: length, tone, refusals, format. Either can come with quality drift, a change in how often answers are right. That's the one that matters, and the hardest to measure.

Overfitting is fitting the noise in the training rows along with the pattern: training error keeps falling while error on new data rises. Underfitting is a model too simple for the pattern: error is high on training and new data alike.

Training loss falling while held-out loss rises. The model is storing the examples it has seen rather than rules that work on new ones.

When a model gets better on its training data and worse on data it hasn't seen. The validation set exists to catch it. An epoch is one full pass over the training data.

A chunk is one piece of a document: the unit that gets indexed, retrieved and handed on. Chunk size is how long each chunk is, usually counted in tokens. Overlap is how much text neighbouring chunks share, so a fact near a cut still lands whole in at least one chunk. Overlap stores some text twice.

Training a model on far more than the compute-optimal 20 tokens per parameter. It costs more once, then less on every token the model serves.

Percentiles. p95 is the time that 95% of requests beat. p50 is the median, the typical case. The slow end of the distribution (p95, p99 and beyond) is called the tail, and the times out there are tail latency.

Cache memory cut into fixed-size blocks and handed to a request one block at a time as it grows. A small table per request lists which blocks hold its tokens, so they don't need to sit side by side. vLLM introduced it as PagedAttention (Kwon and colleagues, 2023), with 16 tokens a block as its default.

Comparing two systems item by item on the same test set, so that only the items where they disagree carry information. It's far more sensitive than comparing two separate totals. The coin-flip check above is called a sign test (or McNemar's test).

Pointwise judging scores each answer on its own. Pairwise judging shows two answers side by side and asks for a winner (or a tie). Pairwise results are reported as a win rate: the share of questions where B beat A.

A model is a function that turns features into a prediction. Its parameters are the numbers left open in it: one weight per feature, plus one on its own. Fitting, or training, sets them so the predictions on known examples come as close as they can to the labels.

Every weight and every bias in a network: the numbers training changes. A layer holds its number of neurons times one more than its number of inputs.

pass@k ("pass at k"): the share of tasks where at least one of k runs succeeded. pass^k ("pass hat k", from the τ-bench paper): the share where all k runs succeeded. The first measures what's possible; the second measures reliability.

pass@k ("pass at k"): the share of tasks where at least one of k runs succeeded. pass^k ("pass hat k", from the τ-bench paper): the share where all k runs succeeded. The first measures what's possible; the second measures reliability.

The working directory is the folder the shell is in now, which pwd prints. A path names a file or folder by its route. It's absolute if it starts with /, and relative to the working directory if not. .. means one level up, and ~ your home folder.

One scale per output row is per-channel. One scale per fixed-size run of weights (often 128) is group-wise. Smaller groups mean finer rulers and more scales to store.

This exact machine: weights, a bias and a hard yes/no cut. Frank Rosenblatt built one in 1958. The nudge you just did is the perceptron learning rule.

A transient error may clear up by itself, so waiting and trying again can work: timeouts, 429, and the 5xx family such as 500, 502, 503 and 504. A permanent error comes back the same however often you ask: a malformed request (400), something that doesn't exist (404), or missing rights (401, 403). Retry the first kind automatically; hand the second straight to the model.

How much a model's score on held-out rows drops when you shuffle one column, which breaks its link to the label but keeps its values. A column the model doesn't really use costs nothing to shuffle. scikit-learn has it as permutation_importance.

The number of equally likely choices the model is, on average, picking between at each token. Here it's 4. Lower is better. Formally it's e raised to the average negative log probability of the real tokens.

The cross-entropy loss is −ln of the probability given to the right token, averaged over every position. The perplexity is e raised to that loss.

The | between two commands. The first command's standard output becomes the second one's input. Standard error doesn't go down the pipe: it still goes to the screen.

A chain of preparation steps ending in a model, fitted and used as one object. scikit-learn's Pipeline, with ColumnTransformer for different steps on different columns, makes sure every step learns only from the rows it's fitted on.

Pointwise judging scores each answer on its own. Pairwise judging shows two answers side by side and asks for a winner (or a tie). Pairwise results are reported as a win rate: the share of questions where B beat A.

A policy gradient method changes the weights θ to raise the average reward. The simplest, REINFORCE (Williams, 1992), samples an action, scores it with a reward R, and steps along ∇ ln π(action), read "the gradient of the action's log-probability". α is the step size.

A single number for how far a distribution has moved from a baseline: the sum, over groups, of the change in share times the log of the ratio of shares. Zero means no change.

Connecting a port on your machine to a port inside a container, with -p host:container. Without it, nothing outside the container can reach the app.

A judge's habit of favouring an answer because of where it appears, not what it says. The standard fix is to judge every pair in both orders and count a pair whose winner flips with the order as a tie.

A learned row for each slot in the sequence (slot 0, slot 1, and so on), summed with the token's own vector before the first block. GPT-2's table has 1,024 of these rows.

Post-filtering searches first, then drops results that fail the condition. Pre-filtering applies the condition first, then searches only what's left.

Quantizing a model after training, by rounding the way this lesson does, with no retraining. The alternative, quantization-aware training, lets the model practise with rounding during training, and costs a training run.

PPO, proximal policy optimisation (Schulman and others, 2017), raises this term for each sampled token, with r its new-to-old ratio and A its advantage. The paper used ε = 0.2.

Post-filtering searches first, then drops results that fail the condition. Pre-filtering applies the condition first, then searches only what's left.

Cutting text into chunks before BPE runs: usually a word with the space in front of it, a run of digits, or a run of punctuation. Merges only happen inside a chunk, so no token runs across two words.

Precision is TP ÷ (TP + FP): the share of yes answers that were right. Recall is TP ÷ (TP + FN): the share of real positives that were found. Recall is also called the true positive rate, or sensitivity.

You met them in "Measuring a classifier". For a check graded against your notes, precision is the share of its flags that are real, and recall is the share of real cases it flags.

Precision is true positives ÷ everything flagged (how much to trust a flag). Recall is true positives ÷ everything really positive (how much it catches). F1 combines them as 2 × P × R ÷ (P + R), which here is 2 × 0.57 × 0.8 ÷ 1.37 = 0.67. F1 is low if either one is low.

Recall asks: of the passages you needed, how many made it into your list? Precision asks: of the passages in your list, how many are actually useful? The first stage aims for recall. The reranker aims for precision.

Precision against recall, one point per threshold. More telling than ROC when positives are rare, because false alarms show at full size. Summed up by average precision, where random guessing scores the share of positives.

The share of the top k that is marked relevant. Here, 1 of the top 3: about 0.33. It goes down when you pad the prompt with filler.

Pausing a running request to free its cache blocks for others, then running it again later. vLLM by default recomputes the dropped cache when the request comes back. The user sees their stream stop for a while, and the GPU does the prefill twice.

One prompt with two answers, marked chosen and rejected. Preference data is a big pile of these, labelled by people or by another model acting as judge.

Prefill is the first pass: it reads the whole prompt at once and produces the first reply token. Decode is everything after: one token per pass, each pass taking the token just made as its new input.

Writing the first characters of the assistant's reply yourself, after the open turn, so the model carries on from them. It fixes how a reply starts: a brace for JSON, or the first words of a set answer.

Holding on to the KV cache built for the start of a prompt after its request is done, so a later request that begins with exactly the same tokens skips prefill for that part. Prefill then runs only on what's new.

A column whose value is different in every row, so it names exactly one row, like the id of a customer. The database refuses a second row with the same key.

Finding new columns that are weighted sums of the old ones, chosen in order: the first along the direction of most spread, each next one at right angles to the earlier ones and along the most spread left over. Keep the first few to describe each row with fewer numbers. Short form PCA; the new columns are the principal components.

Multiplying a vector by a grid of weights to get a new vector, as in "the query projection". W_Q, W_K and W_V are learned in training like every other weight, and every word goes through the same three grids.

Instructions planted in text a model reads, such as a web page, an email or a tool result, in the hope that it obeys them. The first defence is the ladder: that text is information, with no right to give orders.

Text that is supposed to be data, written so the model treats it as instructions and drops or overrides the ones you gave it. The name comes from SQL injection, where data typed into a form gets run as a database command.

One fixed copy of a prompt and its settings, with a label you can point to. When an answer looks wrong, the version in the log tells you exactly which text produced it.

A request on a site such as GitHub or GitLab (where it's called a merge request) to merge one branch into another, with a page for the diff, review comments and automated checks.

A Python class, based on BaseModel, whose typed fields describe valid data. Creating one from raw input converts what it can and raises a clear error for what it can't. FastAPI uses them to check requests and shape responses.

LoRA on top of a base model whose frozen weights are stored in 4 bits. Storing numbers in fewer bits like this is called quantisation.

Work that grows with the square of the length n. Read O(n²) aloud as "order n squared": double n and the work goes up about four times. Work that grows in step with n is linear, O(n).

Any step that scores documents and drops the low ones: hand-written rules, or a trained classifier. Each one removes something you didn't want, and a little you did.

Storing vectors with fewer bits per number. It saves memory and speeds up comparisons, and costs some accuracy.

The three roles in a soft lookup. The query is what you're asking for. Each key is what an entry offers to be matched against. Each value is what that entry hands back. Q, K and V for short.

The steps a database chooses to run a query: which tables to scan, which indexes to use, in what order. EXPLAIN QUERY PLAN in SQLite and EXPLAIN in Postgres show it.

Query rewriting has a model restate the user's question as one or more searches, phrased the way the documents are written. HyDE (hypothetical document embeddings) has a model draft an answer first, then searches with the draft's embedding instead of the question's. Each adds a model call before the search.

Adding servers when the average number of requests waiting per server passes a target. The target is the wait you'll accept times what one full server finishes a second, so the rule keeps servers full with a short line.

A coefficient is the weight on one feature: how much the prediction changes when that feature rises by 1 and the others stay the same. R² is the share of the label's spread around its average that the model explains: 1 is perfect, 0 is no better than the average.

SGLang's name for keeping cached prefixes in a radix tree (a tree where each branch holds a run of tokens) and matching new requests against it. vLLM's version is called automatic prefix caching.

Retrieval-augmented generation: answer a question by running a search first, then putting the passages it finds into the prompt beside the question, so the model writes from them. The search side is called the retriever.

Bagged decision trees in which each split may only use a random subset of the columns. The forest's prediction is the average of its trees' predictions (for classes, the average probability).

Two ways to pick hyperparameters. Grid search tries every combination from lists you give. Random search draws combinations at random from ranges you give, which covers each important setting more finely for the same number of fits.

LoRA (low-rank adaptation) trains two thin matrices, B and A, beside a frozen weight matrix. The rank r is their thin side. Ranks from 8 to 64 are common.

A reasoning model is trained to write out long working before it answers. Test-time compute is what it spends while running: more thinking tokens per question. OpenAI's o1 (2024) showed accuracy rising steadily as a model was given more of it.

Moving a branch's own commits so they start from a different commit, by replaying them one at a time. The changes are the same; the commits, and their ids, are new.

Precision is TP ÷ (TP + FP): the share of yes answers that were right. Recall is TP ÷ (TP + FN): the share of real positives that were found. Recall is also called the true positive rate, or sensitivity.

You met them in "Measuring a classifier". For a check graded against your notes, precision is the share of its flags that are real, and recall is the share of real cases it flags.

Precision is true positives ÷ everything flagged (how much to trust a flag). Recall is true positives ÷ everything really positive (how much it catches). F1 combines them as 2 × P × R ÷ (P + R), which here is 2 × 0.57 × 0.8 ÷ 1.37 = 0.67. F1 is low if either one is low.

Recall asks: of the passages you needed, how many made it into your list? Precision asks: of the passages in your list, how many are actually useful? The first stage aims for recall. The reranker aims for precision.

The share of the right passages that appear in the top k results, averaged over the test questions. With one right passage per question, it's the share of questions whose passage made the top k.

Two tilts that examples add. Majority label bias pulls unsure answers toward the label most examples share. Recency bias pulls them toward the label of the last example.

Score each passage by adding 1 ÷ (k + rank) over every list it appears in, then sort. k = 60 comes from the 2009 paper that introduced RRF, whose authors found it near the best and the exact choice not critical.

Done to a folder and everything inside it, at every level. cp -R copies a folder this way and rm -r deletes one. cp -r works too, but the Mac's manual for cp advises -R.

Try the biggest natural boundary first (section breaks, then paragraphs, then sentences, then words), and only drop to a smaller one when a piece is still over the size limit. It's the usual default in text-splitting libraries, such as LangChain's RecursiveCharacterTextSplitter.

Scoring an output without a gold answer to compare against, by checking it against the input, the source documents and a rubric instead. Most evaluation of chat assistants and open-ended generation works this way.

Your repository's private log of every commit HEAD has pointed to, newest first. It's the way back from a bad reset, rebase or branch delete. It isn't shared by push, and it can't help with changes that were never committed.

Regression predicts a number: a rent, a delivery time, next month's sales. Classification predicts a category from a fixed list: spam or not, one of five teams. Both are supervised learning: every training example comes with its right answer, the label, attached.

Something that used to work and stops working after a change. A regression test reruns old cases to check that the new version still does everything the old one did.

A test set for abilities you already had and must not lose. Run it on every new model, prompt or fine-tune, beside the eval for whatever you were trying to improve.

Adding a charge for large weights to the score a model is fitted on, so it prefers simpler answers. Its strength, α, is a setting you choose, not a number the fit learns.

A policy gradient method changes the weights θ to raise the average reward. The simplest, REINFORCE (Williams, 1992), samples an action, scores it with a reward R, and steps along ∇ ln π(action), read "the gradient of the action's log-probability". α is the step size.

Accept a sample from one distribution with probability target ÷ proposal (capped at 1), and fall back to the leftover when you reject. Speculative decoding uses it so the output follows the big model's distribution exactly.

Fluency: does each sentence read as natural language? Coherence: do the sentences fit together in a sensible order? Factuality (or faithfulness, when judged against a source): is every claim supported? Relevance: does it cover what matters for the task? SummEval, a widely used study of summary scoring, had expert raters score news summaries on these four, calling factuality "consistency".

The passages a person has marked as right for a question, sometimes with grades (here 2 for "answers it", 1 for "related"). A set of questions with labels is a test set, and every metric in this lesson compares a ranked list against it.

An activation function is the fixed rule a neuron applies to its total before passing it on. ReLU, short for rectified linear unit, is the one above: keep a positive total, and send out 0 for anything else.

Another copy of the repository that you push to and fetch from, known by a short name. origin is the usual name for the one you cloned from or first pushed to.

A label like origin/main that records where a branch on the remote was the last time you fetched. Only fetch, pull and push move it.

A repository is a project whose history git keeps, in a .git folder inside it. A commit is a saved snapshot of the files you added, with an author, a time, a message and a link to the commit before it.

MCP messages are JSON-RPC 2.0. A request has an id and a method, and gets one response with the same id, holding a result or an error. A notification has no id and gets no reply. Since 2026-07-28 only the client sends requests.

Moving the current branch back to an earlier commit. --soft keeps the changes staged, the default (mixed) keeps them in your files, and --hard throws them away.

A residual is actual minus predicted, for one row. The mean squared error, MSE, is the average of the squared residuals. Its square root is back in the label's units: a typical miss.

The "add the input back" path around each sublayer. The running vector that every layer reads from and adds to is called the residual stream.

MCP messages are JSON-RPC 2.0. A request has an id and a method, and gets one response with the same id, holding a result or an error. A notification has no id and gets no reply. Since 2026-07-28 only the client sends requests.

Everything the next step depends on: the weights, the optimizer state, the step count, the data position and the random-number state. Restore all of it and the run continues as if it never stopped. A weights-only checkpoint is for serving, not for resuming.

A new commit that applies the opposite of an earlier one. It undoes the change without rewriting history, so it's the safe undo for shared branches.

Reward hacking is a model scoring well on the reward without doing what the reward was meant to measure. Goodhart's law is the older idea from economics: when a measure becomes a target, it stops being a good measure.

An optimiser raising its score in a way that misses what the score was meant to measure, such as shrinking the test instead of improving the model. DeepMind collects examples of it under the name specification gaming. Agents are strong optimisers, so keep the score's own code out of their reach.

A model that reads a prompt and an answer and outputs one number, trained on preference pairs so preferred answers score higher. It stands in for a person who can't sit and judge millions of answers.

Linear regression with an L2 penalty, α times the sum of the squared weights. It shrinks every weight towards 0 in proportion to its size, and shares weight evenly between near-copy columns.

Where the slope meets the flat part: peak operations per second divided by memory bandwidth. About 296 operations per byte on an H100. Jobs to the left are memory-bound, jobs to the right compute-bound.

Reinforcement learning from human feedback: a reward model trained on preferences, then RL against it. The drift penalty is the KL penalty; KL divergence measures how far apart two models' probabilities are. The RL method used most often is PPO.

True positive rate against false positive rate, one point per threshold. The closer it hugs the top-left corner, the better the model separates the two classes. ROC stands for receiver operating characteristic, a name from early radar work.

A cache of a fixed W slots. The keys and values of the token at position i go in slot i mod W (the remainder after dividing i by W), over the token W places back. The cache stops growing once the text passes W tokens.

A short summary of the older part of a conversation, updated as the chat grows and sent in place of those old turns. Recent turns stay word for word.

That picture: the best possible speed for any job, as the lower of the memory slope and the compute ceiling. It comes from a 2009 paper by Samuel Williams, Andrew Waterman and David Patterson. Any real job sits on or below the roof.

Rotary position embedding. Queries and keys are rotated by an angle proportional to their position, so attention scores depend on the distance between tokens rather than where they sit. Real models rotate many pairs of numbers, each pair at its own speed.

A family of scores from summarization research that measure how much of the reference shows up in the output. ROUGE-1 counts single words, as above. ROUGE-2 counts pairs of words, and ROUGE-L uses the longest run of words that appear in the same order. Libraries usually report recall, precision and F1 for each.

Requests per minute and tokens per minute. Your account has both, often with input and output tokens counted apart. Whichever you run out of first is the one that throttles you.

A scoring guide that splits quality into named parts, often called criteria or dimensions, and says how to mark each. Every grader, person or model, gets the same rubric.

An isolated place to run untrusted actions, such as code the agent wrote, where the worst outcome is contained and thrown away with the sandbox.

A neuron pushed so far that its output sits near its limits (±1 for tanh, 0 or 1 for sigmoid) and barely moves when its total changes. Almost no gradient gets through it.

A softmax is saturated when one weight sits near 1 and the rest near 0, so changing the scores a little barely changes the weights. Almost no gradient gets back to the scores. You met the word for tanh and sigmoid neurons in Neural Networks.

A server is saturated when it already runs as many requests as it can hold, so every extra arrival waits. On a dashboard: running flat at its cap, waiting climbing, TTFT climbing with it, time per output token flat.

The scale is the real value of one integer step. Choosing it from the largest absolute weight, so that 0 stays at integer 0 and the range reaches equally far both ways, is symmetric quantization. Multiplying an integer by the scale to get a real number back is dequantization.

The attention from the 2017 paper "Attention Is All You Need", and PyTorch's default. Q Kᵀ is every query dotted with every key, where Kᵀ is K turned on its side. Divide by √d, softmax each row, then multiply by V.

A rule, fitted from many training runs, for how loss falls as you add parameters, data or compute. Chinchilla's version, with N parameters and D tokens:

A value that grants access, such as an API key, a password or a database address with credentials in it. Kept out of code, images and repositories, and handed to the app at start time by the host.

The starting number for the random number generator. Fix the seed and the stream of u values is fixed, so the draws are too. Numbers made this way are called pseudo-random.

Attention where the queries, keys and values all come from the same sentence, so every word looks at every word, itself included. The weights for every pair of words form a square grid, the attention matrix, whose rows each add up to 1.

Sampling several answers to the same question at temperature above 0 and taking a majority vote over the final answers. It trades more tokens for better accuracy on reasoning tasks.

A model reviewing and revising its own output. With no outside feedback it rarely helps on reasoning and can make answers worse. With feedback from a test, a compiler or another check, it's how agent loops improve.

A judge rating outputs from its own model, or its own model family, higher than equally good outputs from others. Also called self-enhancement bias.

A cache that matches on meaning (embedding similarity above a threshold) instead of exact text. It catches far more repeats than an exact cache, and it can be wrong.

Cutting where the similarity between neighbouring sentences drops, so each chunk stays on one topic. It costs an embedding per sentence at indexing time.

Finding text by closeness of meaning (embedding similarity) instead of by shared words. The stored vectors are often called the index.

Fine-tuning a student on answers the teacher wrote, not on its probabilities. It needs only the teacher's text, so any model you can query can teach. Sampling several answers and keeping only those a checker passes is called rejection sampling.

The host is the app you use: a chat app, an IDE, an agent. It holds the conversation, calls the model, and runs one MCP client per server it connects to. A server offers tools, resources and prompts. The model only talks to the host.

Running a new version on a copy of live traffic without showing its output. It costs you a second model call for every shadowed request, so teams often shadow a sample.

Fine-grained experts are many narrow ones in place of a few wide ones, with more picked per token at the same compute. Shared experts run for every token, with no router.

The S-shaped function 1 ÷ (1 + e^(−z)). It maps any number into the range 0 to 1, and sends 0 to exactly 0.5. Also called the logistic function.

For each row, a is its average distance to the rest of its own cluster and b its average distance to the nearest other cluster; its silhouette is (b − a) ÷ the larger of the two. It runs from −1 to 1, and you average it over all rows. A row 2 from its own group and 8 from the next scores (8 − 2) ÷ 8 = 0.75.

A folder with a SKILL.md file: a name and description the agent always sees, and instructions (plus any scripts or reference files) it loads only when a task matches the description.

This version of word2vec. Skip-gram means guessing the neighbours from the middle word. Negative sampling means adding a few random fake pairs to push against, instead of scoring every word in the vocabulary each time.

The SLI (service level indicator) is the measured number. The SLO (service level objective) is the target for it. The error budget is how much failure the target leaves room for. Teams often slow down releases when the budget runs low.

A subset of the data picked out by something known about each input, such as a language, a kind of question or a type of customer, and scored on its own. A system can get better overall while one slice gets worse.

Each token attends only to a fixed number of recent tokens instead of all of them. Its cache, and each step's attention work, stay a fixed size however long the text gets. Information from further back can still reach a token, but only by hopping through other tokens, layer by layer, so many models mix window layers with full-attention layers to keep a long reach.

Service level objective: the targets you promise, written as numbers. "TTFT under 0.5 s and token gap under 25 ms" is an SLO. Often it's stated for a percentile, such as "p99 TTFT under 0.8 s".

The SLI (service level indicator) is the measured number. The SLO (service level objective) is the target for it. The error budget is how much failure the target leaves room for. Teams often slow down releases when the budget runs low.

Adding a small count to every cell of the table so nothing gets probability 0. The more you add, the closer every row moves to an even spread.

The day a training row pretends to stand on. Features use only what was known by then, and the label comes from a fixed window after it. It's how you make a table that matches what the model will see when it runs.

The big model is the teacher, and the small one being trained is the student. The teacher's probabilities are soft targets, softened by dividing each logit z by a temperature T before the softmax. T = 1 is the usual softmax, and higher T flattens it.

The recipe above: work out e^logit for each logit, then divide each by their total. Bigger logits get more of the probability.

Softmax turns scores into weights: raise e to each score, then divide by the total. Attention is the whole move: score the words, softmax the scores, then mix the words' vectors with the weights. A transformer does it for every word at once.

One timed unit of work inside a trace, with a parent, a start, an end and a set of attributes. OpenTelemetry's GenAI conventions give standard attribute names, such as gen_ai.request.model and gen_ai.usage.input_tokens, so different tools can read the same traces.

A chat template is the rule for flattening messages into one run of tokens: which markers open and close each message, and where the role name goes. The markers are special tokens: single tokens meant to come only from the template.

A small draft model guesses several tokens, and the big target model checks them all in one forward pass, keeping the longest run it agrees with plus one token of its own.

A fixed target on fixed hardware, scored by the time it takes to reach it. Fixed-budget experiments fix the time and score the result. Speedruns fix the result and score the time.

One question at a node of a tree, of the form "feature ≤ value?". For a number column, the candidate values are midpoints between neighbouring values in the training data.

Text from a user that changes the meaning of a query because it was pasted into the SQL. Placeholders prevent it.

The changes that go into the next commit, also called the index. git add puts a file's current version there, and git commit saves it. The files on disk, with all your edits, are the working tree.

The two streams a program writes to: results to standard output (stream 1), warnings and errors to standard error (stream 2). > sends stream 1 to a file, replacing what was in it, >> adds to the end instead, and 2> sends stream 2.

The two streams a program writes to: results to standard output (stream 1), warnings and errors to standard error (stream 2). > sends stream 1 to a file, replacing what was in it, >> adds to the end instead, and 2> sends stream 2.

Rescaling a column to average 0 and spread 1: subtract the column's mean and divide by its standard deviation, both measured on the training rows. A $95 bill, when the average is $65 and the spread is $30, becomes 1.0. scikit-learn calls it StandardScaler.

A shelf for uncommitted changes. git stash saves them and cleans the working tree, git stash list shows what's on the shelf, and git stash pop puts the newest back.

A model call is stateless: it depends only on what you send in that call. All memory in an agent is something your code keeps and decides to send.

Form a batch, run it until every request in it is done, then form the next. Easy to build. Wasteful when output lengths vary, which for chat they always do.

One fixed vector per word, learned once from a big pile of text. word2vec and GloVe both make these. "Static" because the vector stays the same whatever sentence the word is in.

The three-digit number at the top of an HTTP response: 2xx for success, 3xx for look elsewhere, 4xx for a problem with the request, 5xx for a problem on the server.

The most loop iterations your code allows in one run. Its cousins are the token budget and the deadline. Hitting any of them is a normal, planned ending, not a crash: the run stops cleanly and reports where it got to.

The agent loop calls the model, runs any tool it asks for, appends the call and the result, and calls again. A model running in it with tools is an agent. Each reply's stop reason says why it stopped: tool_use, end_turn or max_tokens.

A special vocabulary entry the model can pick to end its output. Also called the end-of-sequence (EOS) token. When it's picked, the loop stops and the token itself isn't shown to you.

The special token that ends a reply: <|im_end|> for Qwen2.5, <|eot_id|> for Llama 3. The server stops generating when the model writes it. Set the wrong one, and the model writes on past the end of its reply.

Folds built so each one holds the same share of every class as the whole dataset. scikit-learn's StratifiedKFold, and the default for classifiers in its search tools.

Sending each token (or small group of tokens) to the client as soon as it's made, instead of waiting for the full reply. HTTP APIs usually do this with server-sent events: one small message per chunk on a connection that stays open.

Feeding the reply to the speech engine in pieces as the model writes it, and playing audio as it comes back. Smaller pieces start sooner but can sound worse: ElevenLabs' docs say more text gives better audio.

A cache that keeps the first few tokens, the attention sinks, plus a rolling window of recent ones (Xiao and colleagues, 2023). Positions are counted inside the cache, not the original text. With no retraining, it kept Llama-2, MPT, Falcon and Pythia stable over 4 million tokens.

An API feature where you pass a JSON Schema and the reply is guaranteed to match it. It usually works by constrained decoding, which the next lesson, "Grammars that constrain decoding", takes apart. Function calling (also called tool use) runs on the same schemas, and "Letting the model call functions" covers the loop around it.

The big model is the teacher, and the small one being trained is the student. The teacher's probabilities are soft targets, softened by dividing each logit z by a temperature T before the softmax. T = 1 is the usual softmax, and higher T flattens it.

The lead is the orchestrator (some frameworks say supervisor). Each helper is a subagent: a full agent loop started by the orchestrator, given a narrow task and a fresh context, that returns a condensed result.

A helper agent with its own instructions, tool list and fresh context window. The main agent sends it a task and gets back only its final report, so the helper's reading never fills the main window.

A token for a piece of a word, like ing or un. With bytes plus subword tokens, common words come out whole, rare words come out in pieces, and nothing is unknown.

A written statement of one thing a good output must do, worded so that two people checking it give the same mark. A system usually needs a few, one for each way it can fail.

Regression predicts a number: a rent, a delivery time, next month's sales. Classification predicts a category from a fixed list: spam or not, one of five teams. Both are supervised learning: every training example comes with its right answer, the label, attached.

The scale is the real value of one integer step. Choosing it from the largest absolute weight, so that 0 stays at integer 0 and the range reaches equally far both ways, is symmetric quantization. Multiplying an integer by the scale to get a real number back is dequantization.

Paging on what users experience (errors, latency, broken answers) rather than on internal causes. Cause signals still matter, for finding the problem once a symptom alert has fired.

The message you, the developer, put at the top of every request: the bot's job, its limits and its style. The customer never sees it. Anthropic's API takes it as a separate system field, and OpenAI's calls the role developer.

Getting a model to reveal its hidden system prompt. Training makes it rarer, but a small chance per try adds up over many tries. Here p is the chance one try works and n the number of tries.

Replacing each category with the average label for that category in the training rows, pulled toward the overall average for small categories, and computed out of fold so no row sees its own label.

Leakage through a feature that is filled in or changed because of the outcome: recorded after it, or by the process that handles it.

Whether the output did the job it was meant to do, checked by the outcome rather than the wording. It's harder to measure than text similarity, and much closer to what users care about.

Whether the run left the world in the state the task asked for, checked by code after the run: a database row, a passing test suite, a file with the right contents.

The big model is the teacher, and the small one being trained is the student. The teacher's probabilities are soft targets, softened by dividing each logit z by a temperature T before the softmax. T = 1 is the usual softmax, and higher T flattens it.

A number τ (tau) that every score is divided by before the softmax. A small τ makes the guess sharp and focuses the loss on near misses. E5 trained with τ = 0.01. The sentence-transformers library defaults to 0.05, set as its inverse, a scale of 20.

Written T. Every logit is divided by T before softmax. At T = 1 the model's own probabilities come through unchanged. A smaller T widens the gaps, so the leaders gain; a bigger T narrows them, so the odds even out.

Fitting one temperature on a held-out set so the model's confidence matches its accuracy, then using that temperature from then on. It's the same division you met in "Turning logits into choices", used to fix calibration rather than to sample.

A word's term frequency, tf, is how many times it appears in one passage. TF-IDF scores a passage by adding up tf × IDF over the search words.

The training set is the rows a model is fitted on. The validation set is rows kept out of fitting and used to choose: which model, which degree, which settings. The test set is kept out of everything until the end, then scored once to estimate how the choice will do on new data.

Examples kept out of training and used only to measure. A network can score perfectly on the examples it trained on and still fail on new ones. Only unseen examples show whether it learned the shape.

A dev set (or validation set) is the data you check against while making choices. A test set is kept back and scored only for a final decision. Once choices are made by looking at a set, it has quietly become a dev set.

A reasoning model is trained to write out long working before it answers. Test-time compute is what it spends while running: more thinking tokens per question. OpenAI's o1 (2024) showed accuracy rising steadily as a model was given more of it.

A word's term frequency, tf, is how many times it appears in one passage. TF-IDF scores a passage by adding up tf × IDF over the search words.

Logs, metrics and traces, the usual name for the three kinds of telemetry. Each piece inside a trace is a span, with a start time, an end time and its own details.

A metric is the number you compute, here the share of replies that pass. A threshold is the value it must reach. Both belong to a named test set: the fixed list of inputs you score on.

How much work the whole server finishes per second, counted across all users. Usually given as requests per second or output tokens per second. TPS is one user's speed. Throughput is everyone's added together.

A group split keeps all the rows of one customer, patient or house on the same side. A time split trains on older rows and tests on later ones, the way the model will be used: on a future it hasn't seen.

The time from the caller's last word to the first sound of the reply. In a pipeline it's the sum of the stages, so log each one, and watch the median and the slowest turns.

The blank-screen wait. The clock starts when the request leaves the client and stops when the reply's first token shows up there. In the stopwatch above it's 0.6 seconds.

A token is one of the pieces a tokenizer cuts text into. The vocabulary is the fixed list of tokens it knows. A token's place on that list is its token ID, the number the model receives.

Where one token ends and the next begins. A model gets each token as one ID, so it can see the boundaries between tokens but not the letters inside one. Tasks about letters work better when each letter is its own token.

A rate limiter with two settings: a capacity, which sets the biggest burst, and a refill rate, which sets the long-run average. Anthropic's API uses one, so its limits refill continuously instead of resetting each minute.

A ceiling on the tokens, or the money, something may spend: one call, one task, one user per day, the whole app per month. A request that would go over is stopped, moved to a cheaper model or held until someone confirms.

How fast the reply comes out once it has started: the tokens after the first, divided by the time they took. Above, 199 tokens in 4.975 seconds is 40 TPS. Some tools quote the same thing the other way up, as milliseconds per token (25 ms here).

The model's reply when it wants a function run: the function's name, its arguments as a JSON object, and an id to match the answer to. Providers call the feature function calling or tool use. They mean one idea.

A tool call is the model's request to run a tool: a name, JSON input and an id. A tool result is your code's answer, sent back with the same id. The model only writes the call. Your code decides whether to run it.

What you send so the model knows a tool exists: a name, a description and a JSON Schema for the arguments. Providers usually turn these into prompt text behind the scenes, so they take up room in the window like anything else.

A tool definition is all the model knows of a tool: a name, a description and an input schema, the JSON Schema for its arguments. Anthropic's API calls the schema input_schema, OpenAI's parameters and MCP's inputSchema.

The plain text that tells the model what a tool does, what goes in, what comes out, and when to use a different tool. The model reads it every time it picks a tool, so it works like a prompt.

A tool that ran, or checked its input, and failed. It comes back as a normal result marked as an error, isError: true in MCP or is_error: true in Anthropic's API, saying what to fix. A call that can't be routed, like an unknown tool, gets a JSON-RPC protocol error instead.

Instructions hidden in a tool's description, written for the model, not the user. A rug pull delivers them later, when a server you approved changes its tools. The host's defence: pin each definition, and ask again when it changes.

Your reply to a tool call: the function's output, marked with the call's id. To the model it's more prompt text, so whatever a result contains gets read, including text someone planted there (Production AI covers that in "Poisoned documents and web pages").

A tool call is the model's request to run a tool: a name, JSON input and an id. A tool result is your code's answer, sent back with the same id. The model only writes the call. Your code decides whether to run it.

Keep only the k most likely tokens, rescale them to add up to 1, and sample from those. k = 1 is greedy decoding. The sample texts in the GPT-2 paper used k = 40, and values in the tens are common.

Sampling from the head of the sorted list: tokens are kept from the top until their running total reaches p, the rest are dropped, and the kept ones are rescaled. The kept group is called the nucleus. It comes from Holtzman et al. (2019), the same paper that showed greedy text looping.

A list of a graph's nodes in which every node comes after all the nodes it was made from. backward() walks it in reverse.

Total parameters are every weight the model stores, all experts included, and set the memory. Active parameters are the weights one token uses, and set the compute per token. Qwen3-235B-A22B names both: 235B total, 22B active.

Total parameters count every weight a model stores, and set its memory. Active parameters count the weights one token uses, the shared parts plus k experts per layer, and set its work.

Requests per minute and tokens per minute. Your account has both, often with input and output tokens counted apart. Whichever you run out of first is the one that throttles you.

The wait before a reply starts is the time to first token (TTFT), mostly prefill. The gap between later tokens is the time per output token (TPOT), one decode step. With one user on one GPU, a step can't beat the weight bytes divided by the memory bandwidth, the bytes a second the GPU can read:

The full record of one run: every model call, tool call, retry and decision, in order and nested, with timings, inputs and outputs. It's the main tool for debugging an agent.

The training set is the rows a model is fitted on. The validation set is rows kept out of fitting and used to choose: which model, which degree, which settings. The test set is kept out of everything until the end, then scored once to estimate how the choice will do on new data.

The sequence of steps a run took: each model decision, tool call and result. Trajectory evaluation scores that path; outcome evaluation scores only where it ended.

A group of changes the database applies all together or not at all. commit makes them permanent; an error or rollback undoes every one.

A failure caused by the provider's state at that moment, not by your request: a timeout, a 429, a 500 or a 529. Retry these. Most other 4xx errors, such as 400 and 401, mean the request or the account needs fixing.

A transient error may clear up by itself, so waiting and trying again can work: timeouts, 429, and the 5xx family such as 500, 502, 503 and 504. A permanent error comes back the same however often you ask: a malformed request (400), something that doesn't exist (404), or missing rights (401, 403). Retry the first kind automatically; hand the second straight to the model.

The largest number of questions any row answers on its way to a leaf. Deeper trees carve finer groups, and overfit sooner. scikit-learn's max_depth caps it.

A triple is one fact written as (subject, relation, object), such as (Norfield Metals, supplies, brake disc). A knowledge graph is a set of triples stored as a graph: each thing (an entity) is a node, and each triple is an edge from subject to object that carries its relation, so "supplies" and "located in" are different kinds of edge.

The wait before a reply starts is the time to first token (TTFT), mostly prefill. The gap between later tokens is the time per output token (TPOT), one decode step. With one user on one GPU, a step can't beat the weight bytes divided by the memory bandwidth, the bytes a second the GPU can read:

Overfitting is fitting the noise in the training rows along with the pattern: training error keeps falling while error on new data rises. Underfitting is a model too simple for the pattern: error is high on training and new data alike.

A small, fast test of one function with a known input and a known answer. pytest runs every function named test_ in files named test_*.py.

Learning from features alone, with no label to predict. The two common jobs are finding groups of similar rows (clustering) and finding a shorter description of the columns, as PCA does.

A byte is a whole number from 0 to 255. UTF-8 is the standard rule for writing each character as one to four bytes, and for reading them back. Nearly every web page is stored this way.

The share of paid GPU time that does useful work. Cost per token on your own hardware is price ÷ (peak throughput × utilisation), so halving utilisation doubles what every token costs.

The training set is the rows a model is fitted on. The validation set is rows kept out of fitting and used to choose: which model, which degree, which settings. The test set is kept out of everything until the end, then scored once to estimate how the choice will do on new data.

The three roles in a soft lookup. The query is what you're asking for. Each key is what an entry offers to be matched against. Each value is what that entry hands back. Q, K and V for short.

Gradients that shrink (vanish) or grow (explode) as they pass back through many layers, because each layer scales them by a bit less, or a bit more, than 1.

Bias is error that stays however often you redraw the data: the model's average answer is off, because it's too simple for the pattern. Variance is how much a model's answers change from one sample of rows to another.

The spread squared. For independent random numbers, variances add when the numbers add. So a score that adds d products of variance 1 has variance d, and a spread of √d. The 2017 paper makes this argument in a footnote.

A reward computed by a program that checks the answer: a string match, a unit test, a proof checker. Reinforcement learning on these is often called RLVR, reinforcement learning with verifiable rewards.

A check in code that must pass before a run may end as a success. It reads the real world (test results, files, database rows), not the model's description of it.

A folder, usually .venv, with its own Python and its own installed packages, for one project. Activating it puts it first on PATH. It's rebuilt from a list, never committed.

A token is one of the pieces a tokenizer cuts text into. The vocabulary is the fixed list of tokens it knows. A token's place on that list is its token ID, the number the model receives.

When a question and the passage that answers it use different words for the same thing. To keyword search the passage has no matching words, so it scores 0 however well it answers.

Shorthand for a quantized setup: W for the bits per weight, A for the bits per activation. W8A8 stores weights and runs the maths in 8 bits. W4A16 stores weights in 4 bits but does the maths in 16, which cuts memory and reading time without needing 4-bit arithmetic.

Shorthand for a quantized setup: W for the bits per weight, A for the bits per activation. W8A8 stores weights and runs the maths in 8 bits. W4A16 stores weights in 4 bits but does the maths in 16, which cuts memory and reading time without needing 4-bit arithmetic.

Requests you send and then throw away before measuring, so one-off start-up costs don't land in your numbers. Send enough to cover the prompt lengths and batch sizes you'll test. Measure cold start separately, if it matters to you, and label it.

Warmup raises the rate from 0 to its peak over the first steps. Cosine decay then lowers it along half a cosine wave to a floor, as progress runs from 0 to 1. The cosine shape comes from Loshchilov and Hutter's SGDR paper (2016).

A weight sets how much one input counts, and in which direction. The bias is added at the end, so the neuron can lean one way even when every input is 0. Training is the search for good values of both.

A * (any characters, or none) or ? (exactly one character) in a file name. The shell replaces the pattern with every name that matches, in alphabetical order, before the command runs.

A function such as SUM, ROW_NUMBER, RANK or LAG followed by OVER (...). It computes across a group of related rows and returns a value for every row, without merging them.

The best of many measured results tends to overstate its true value, because it was picked partly for its luck. The more results you pick from, the bigger the overstatement.

When you pick the best of many noisy scores, the winner's score is usually better than its true value, because it was partly picked for its luck. The more things you try, the bigger the gap.

A puzzle of the form "a is to b as c is to ?", answered by b − a + c and a nearest-neighbour search. It's the classic test of whether directions in an embedding space mean something.

One meaning of a word: bank as a place for money, bank as a river's edge. A static embedding has one row per word, so it holds a blend of the word's senses, weighted by how often each one appears.

A workflow strings model calls and tools along a path written in your code. An agent lets the model choose each next step in a loop. Anthropic's "Building effective agents" (2024) calls both agentic systems.

The working directory is the folder the shell is in now, which pwd prints. A path names a file or folder by its route. It's absolute if it starts with /, and relative to the working directory if not. .. means one level up, and ~ your home folder.

The history in the context window is often called working memory: fast, visible to the model, small. A store that lasts across runs and is searched on demand is long-term memory.

A rubric part asked as one question with a yes or no answer, such as "Does the reply ask for a password?". It gives graders one border to agree on, where a 1 to 5 scale gives four. Also called a binary or pass/fail criterion.

ZeRO (Rajbhandari and others, Microsoft, 2019) shards training state across data-parallel GPUs in three stages: Adam's state, then gradients, then weights. PyTorch's FSDP, fully sharded data parallel, is built on the same idea.

The zero point is the integer that stands for real 0. Storing a scale and a zero point maps the real minimum and maximum onto the two ends of the integer range, so no codes go to waste. That's asymmetric quantization. Real 0 still lands exactly on an integer, so zeros come back exact.

Adverse selection is when the people who choose to trade with you are, on average, the ones who know better. The term comes from insurance. The Glosten-Milgrom model sets the ask to what you'd believe after a buy and the bid to what you'd believe after a sell. Lawrence Glosten and Paul Milgrom published it in 1985, and its spread pays for adverse selection and nothing else.

Matching rows and columns by label before doing the maths is alignment. NaN, short for not a number, marks a missing value. Most pandas maths skips it or passes it through.

Regress a stock's return on the market's: stock = α + β × market + e. Beta, β, is the slope: how far the stock moves on average for each 1% the market moves. Alpha, α, is the intercept: the return left once the market's part is taken out. e is the stock's own move. ρ, read "rho", is the correlation.

The line of expected return against beta is the security market line. A return's distance above it, what's left after paying for market risk, is its alpha. Measured over a few years, alpha is mostly noise, so it needs a standard error next to it.

Profiling is measuring where a program spends its time. Amdahl's law puts a number on what you saw: if a step takes a share s of the run, making it infinitely fast saves at most s. A step that's 5% of the run can save you 5%, never more.

Changing an order that's already in the book. Some exchanges call it a modify, or a cancel/replace. The usual rule: cut the size and keep your place; change the price or add size and go to the back.

An autoregressive model of order 1: xₜ = c + φxₜ₋₁ + εₜ, read "today is a constant, plus phi times yesterday, plus a fresh shock". With |φ| below 1 it's stationary, with average μ = c / (1 − φ).

An array is numpy's block of numbers, all of one type, packed side by side. p[1:] is a slice: a stretch of the array, picked by position. Code that works on a whole array in one call is vectorised. The loop still runs, inside numpy's compiled code.

A join that matches each row to the most recent row at or before it in time, instead of to an equal key. pandas calls it merge_asof.

Every waiting order, sorted by price, is the order book. Waiting buy orders are bids and waiting sell orders are asks. The highest bid is the best bid and the lowest ask is the best ask.

The autocorrelation is the correlation of a series with itself some days earlier. The gap is the lag. ρ̂(k), read "rho hat of k", is the autocorrelation at lag k measured from data. The hat marks it as an estimate.

Quoting around a reservation price, with a spread made of a risk part and a fill part. Marco Avellaneda and Sasha Stoikov published it in 2008, and it's the usual starting point for quoting that accounts for inventory.

An array's shape counts its elements in each direction: (4, 3) is 4 rows and 3 columns. The axis argument says which direction a sum, mean or standard deviation runs along. That direction disappears from the shape.

To value a game with choices, start at the last choice, where carrying on has a known value, and work back. At each choice, take the larger of what you hold and what carrying on is worth, written max.

How common something is before you see any evidence is its base rate, also called the prior: here 1%. The chance after the evidence is the posterior: here about 9%. Answering 90% leaves the base rate out, a mistake called base rate neglect.

The Sharpe ratio is the average daily return over its standard deviation, times √252 to make it yearly: return per unit of risk. A basis point (bp) is a hundredth of a percent, so 5 bps on each unit traded is 0.05%.

To turn P(B | A) into P(A | B), multiply by the prior P(A) and divide by P(B), the overall chance of the evidence. Find P(B) by splitting into cases: A happens, or it doesn't.

Updating a belief after seeing evidence: count how often the evidence turns up in each possible world, then keep only the worlds where it did. Named after Thomas Bayes, from the 1700s.

Regress a stock's return on the market's: stock = α + β × market + e. Beta, β, is the slope: how far the stock moves on average for each 1% the market moves. Alpha, α, is the intercept: the return left once the market's part is taken out. e is the stock's own move. ρ, read "rho", is the correlation.

Bias is how far a model's average fit sits from the truth, and variance is how much its fit moves between samples. More flexibility lowers bias and raises variance. The noise is a floor no model gets below on average.

Every waiting order, sorted by price, is the order book. Waiting buy orders are bids and waiting sell orders are asks. The highest bid is the best bid and the lowest ask is the best ask.

The number of successes in n independent tries, each a success with chance p, has a binomial distribution. Its mean is n × p. Each try is a 1-or-0 score with variance p × (1 − p), and variances add, so the variance is n × p × (1 − p).

The birthday problem asks how many people make a shared birthday more likely than not. With d equally likely values, n draws are all different with the chance below, and a match passes 50% at about 1.2 × √d draws: 23 people for 365 days.

Resampling runs of consecutive points, such as blocks of 20 days, instead of single points, so the links between neighbours survive. Use it for anything in time order: returns, signals and numbers measured over overlapping windows.

Testing each of m results at α / m, so that the chance of even one false positive stays at α or below, is the Bonferroni correction.

Estimating how much a number would vary by recomputing it on many resamples: datasets the same size as yours, drawn from it with replacement, so each pick goes back before the next. The spread of the results is the bootstrap standard error.

Stretching a single number across a whole array is broadcasting. The array of True and False from a comparison is a mask. r[mask] keeps the elements where the mask is True, and mask.mean() is the share that are True.

A cache miss is a read the cache can't serve, so the processor waits on main memory. Pointer chasing is following a chain of addresses, where each read must finish before the next can start.

A call auction collects orders for a while, then matches them all at one price, the clearing price. While orders are being collected, the book can be crossed. That's the one time a crossed book is normal.

The mix of risky assets with the highest Sharpe ratio sits where a line from cash just touches the curve of mixes, so it's called the tangency portfolio. The line itself, that mix blended with cash or with borrowing, is the capital allocation line.

This is the capital asset pricing model, or CAPM. The market's expected return above cash, E[Rm] − r, is the market risk premium. In CAPM, beta is the only thing that earns a premium.

The central limit theorem says the sum or the average of many independent draws, from almost any distribution, is close to normal. The average has mean μ and standard deviation σ / √n, so z-scores and "68, 95, 99.7" apply to it.

A call auction collects orders for a while, then matches them all at one price, the clearing price. While orders are being collected, the book can be crossed. That's the one time a crossed book is normal.

Two series are cointegrated when each has a unit root but some combination A − βB is stationary. β is the hedge ratio, and A − βB is the spread. Correlation is about whether daily moves line up. Cointegration is about whether the levels stay tied.

Tick-to-trade is the time from a market data message arriving to your order leaving in response. Colocation is renting space for your servers inside the exchange's data centre, metres from its matching engine. Kernel bypass lets your program read and write the network card directly, skipping the operating system.

A permutation is an arrangement, where order matters. A combination is a group, where it doesn't. To count combinations, count the permutations, then divide by the number of orders of one group.

The complement of an event is everything else: "no six" is the complement of "at least one six". P(not A), read "the chance A doesn't happen", is one minus the chance it does:

1 + r is the growth factor, the number the price gets multiplied by. Chaining returns by multiplying their growth factors is compounding:

P(A | B), read "the probability of A given B", is the chance of A once you know B happened. Count A inside B. With chances instead of counts, divide. For the dice, (2/36) / (11/36) = 2/11.

A confidence interval is a range built from a sample by a recipe that catches the true value a set share of the time. That share is the confidence level. For an average, at 95%:

The share of samples for which the interval recipe catches the truth. It describes the recipe, not any one interval. "This recipe works 95% of the time" is right. "There's a 95% chance the truth is in this range" is the common misreading.

A confounder drives both who gets the treatment and the outcome, like order size here. Leaving one out shifts the slope by its effect on the outcome times how much it moves with the treatment: omitted-variable bias.

A view is a new array that shares the original's memory. A copy owns memory of its own. Slices give views; lists of positions and masks give copies.

The correlation coefficient r, also called Pearson's r, runs from −1 to +1. +1 means the points sit exactly on a rising line, −1 exactly on a falling one, and 0 means no straight-line relation. Σ means "add up":

A table of every pair's correlation is a correlation matrix, and its entries have to fit together. If corr(X, Y) = ρ₁ and corr(Y, Z) = ρ₂, read "rho one" and "rho two", then corr(X, Z) lies within:

A correlogram is a bar chart of the autocorrelation at lags 1, 2, 3 and on. Desks draw one for returns, for their sizes, and for whatever a model leaves unexplained.

Spreading money across assets that don't move in lockstep, so that part of their swings cancel, is diversification. In the cross term, ρσAσB is the covariance of A and B, with ρ, read "rho", the correlation. A table of every pair's covariance is the covariance matrix. With a share w in A:

All the stocks on one date. A cross-sectional rank orders them against each other on that date: rank(axis=1) on a wide table, or a groupby("date") and rank() on a long one.

Splitting the data into k folds, fitting on k − 1 of them and testing on the one left out, then rotating so each fold is tested once. The model with the lowest average error wins. With data in time order, each test fold must come after the data it was fitted on.

A buy priced at or above the best ask, or a sell at or below the best bid, crosses the spread and trades on arrival. An order that doesn't cross rests in the book.

Trying many ideas on the same data and reporting the best is data snooping, also called p-hacking. The winner's own p-value comes out too small. The chance that the best of m useless tests reaches p or lower is:

A DataFrame is pandas' table: a numpy array with labels on both edges. Its row labels are the index, here dates. One column on its own, or one row, is a Series: values with labels.

To demean is to subtract the cross-section's average, so each day's values add up to zero. A z-score also divides by that day's standard deviation: (x − mean) / std, read "how many standard deviations from the day's average".

Depth is how many shares wait at and near the best prices. Slippage is how much worse your average price was than the best price when you started.

A series with φ = 1 has a unit root. Taking day-to-day changes, Δxₜ = xₜ − xₜ₋₁, is differencing. A series that becomes stationary after differencing once is integrated of order 1, or I(1). Log prices usually behave like I(1) series.

Spreading money across assets that don't move in lockstep, so that part of their swings cancel, is diversification. In the cross term, ρσAσB is the covariance of A and B, with ρ, read "rho", the correlation. A table of every pair's covariance is the covariance matrix. With a share w in A:

The fall from the highest value so far to the current value is the drawdown. The largest over a period is the maximum drawdown, and the time spent below an old peak is the drawdown's duration.

A fair game has an expected profit of zero for both sides. Your edge is your expected profit for each 1 you stake. For a bet that pays b to 1 and wins with chance p:

The upper edge, the most return for each level of risk, is the efficient frontier. Its left tip, the lowest-risk mix of all, is the minimum-variance portfolio. Add cash, and the steepest line from Mean-variance touches the frontier at one point: the tangency portfolio.

Multiply the covariance matrix C by a component's weights v and you get the same weights back, scaled: Cv = λv, read "C times v equals lambda times v". The weights are an eigenvector, and the scale λ is its eigenvalue, the variance along that component. Each row of the three-stock matrix adds to 0.08, so equal weights come back times 0.08.

Multiply the covariance matrix C by a component's weights v and you get the same weights back, scaled: Cv = λv, read "C times v equals lambda times v". The weights are an eigenvector, and the scale λ is its eigenvalue, the variance along that component. Each row of the three-stock matrix adds to 0.08, so equal weights come back times 0.08.

Outcomes are equally likely when none is favoured over another, like the 36 squares of two fair dice. Only then is a chance a count over a count. Otherwise, add up the chance of each outcome in the event:

A number computed from a sample to stand in for a hidden one is an estimate. The sample average x̄ estimates the true average μ ("mu"), and the sample standard deviation s estimates the true one, σ ("sigma"). Σ means "add up":

The gap between an estimate and the truth is estimation error. Pulling noisy estimates towards a common value, such as their average, is shrinkage. It adds a little bias and removes a lot of noise.

The sample space is every outcome that can happen: the 36 squares. An event is a set of outcomes you care about, such as "the total is 7". If every outcome is equally likely, P(A), read "the probability of A", is A's share of them:

An exponentially weighted moving average of squared returns: σ²ₜ = λσ²ₜ₋₁ + (1 − λ)r²ₜ₋₁, read "today's variance is lambda times yesterday's variance, plus one minus lambda times yesterday's squared return". The day k back gets weight (1 − λ)λᵏ⁻¹.

Return above the cash rate is excess return. Excess return divided by volatility is the Sharpe ratio, the return per unit of risk. With r the cash rate:

The loss that only 1% of days go past is the 99% one-day value at risk, or VaR. The average loss on those worst 1% of days is the 99% expected shortfall, or ES. VaR is a quantile; expected shortfall is the average beyond it.

The expected value of a random amount X, written E[X], is each value times its chance, added up. Σ, read "sum", means add this up over every value. For the die, E[X] = 21/6 = 3.5. A bet's fair price is its expected value: pay that, and neither side has an edge.

A regression of returns on several factors, each a column of returns such as the market, or cheap stocks minus expensive ones. Each slope is an exposure, also called a loading. Beta is the exposure to one factor, the market.

A factor model explains a return as the sum of its exposures to a few common drivers, plus a part of its own: R − r = α + β₁F₁ + β₂F₂ + … + ε. Each F is a factor's return, each β is the loading on that factor, and ε, read "epsilon", is the part left over.

n!, read "n factorial", is n × (n − 1) × … × 1, so 3! = 6 and 5! = 120. The number of groups of k things from n is C(n, k), read "n choose k":

A fair game has an expected profit of zero for both sides. Your edge is your expected profit for each 1 you stake. For a bet that pays b to 1 and wins with chance p:

Your best estimate of what something is worth: its average outcome, given what you know. A market maker centres the quote on it. For two dice it's 7.

The expected share of false finds among the results you keep is the false discovery rate. The rule above, from Yoav Benjamini and Yosef Hochberg (1995), holds it at q: sort the m p-values, find the largest k whose p is under its bar, and keep the first k.

Running many tests and paying attention to the ones that pass is multiple testing. The chance of at least one false positive across the whole family of m tests is the family-wise error rate. At level α, read "alpha", for independent tests:

The matching engine is the exchange's program that takes orders one at a time, matches each against the book by its priority rule, and rests what's left. Each match between the arriving order and one waiting order is a fill, with its own size and price.

In first-step analysis, you condition on what the first move does, write one equation for each state you can be in, and solve them together. It answers "how long until" and "which comes first" questions, and it values games with a stopping rule.

ffill() copies the last known value forward into the gaps. Give it a limit, such as ffill(limit=3), so a stock that stopped trading doesn't keep its last price for ever.

Running at a set fraction of the Kelly size, such as half, is fractional Kelly. It gives up a little growth for a large cut in drawdowns, and in the damage done by overestimating your edge.

Running a call auction every fraction of a second instead of matching continuously. Inside one batch, orders compete on price, not arrival time.

Believing that after a run of one outcome, the other is "due". With independent trials it never is: the chance of the next outcome is the same whatever came before.

A walk that starts at k and stops at 0 or N is the gambler's ruin problem. With a fair coin, the chance of reaching N first is k / N, and the chance of ruin is 1 − k / N.

σ²ₜ = ω + αr²ₜ₋₁ + βσ²ₜ₋₁, read "today's variance is a constant, plus alpha times yesterday's squared return, plus beta times yesterday's variance". With α + β below 1, variance is pulled back to a long-run level V = ω / (1 − α − β). α + β is the persistence. EWMA is the special case ω = 0, α = 1 − λ, β = λ, with persistence exactly 1.

The geometric mean is the steady rate that compounds to the same end: the n-th root of the total growth factor, minus 1. Its gap below the plain average is volatility drag. Roughly, with σ the spread of the returns:

Adverse selection is when the people who choose to trade with you are, on average, the ones who know better. The term comes from insurance. The Glosten-Milgrom model sets the ask to what you'd believe after a buy and the bid to what you'd believe after a sell. Lawrence Glosten and Paul Milgrom published it in 1985, and its spread pays for adverse selection and nothing else.

1 + r is the growth factor, the number the price gets multiplied by. Chaining returns by multiplying their growth factors is compounding:

Your growth rate is what a typical bet multiplies your money by: each result's multiplier, raised to its chance, multiplied together. Above 1, a typical player gets richer. Below 1, poorer. Staking half on the 60% coin, 10 typical flips multiply your money by 0.71, which is about 0.967 a flip. For a share f of your money on an even-money bet that wins with chance p:

A series pulled back toward a long-run average is mean-reverting. Its half-life is the number of days it takes, on average, to close half of any gap from that average.

A market maker keeps a bid and an ask up at the same time, so anyone can trade with them in either direction. The pair is a quote. The distance from your guess to either side (0.10 at the start) is the half-spread.

Two series are cointegrated when each has a unit root but some combination A − βB is stationary. β is the hedge ratio, and A − βB is the spread. Correlation is about whether daily moves line up. Cointegration is about whether the levels stay tied.

The number of steps a walk takes to first reach a level is its hitting time. A fair walk that starts at k and stops at 0 or N takes this many steps on average. From the middle, d from each wall, that's d².

The hot path is the code that runs on every message. An object pool is memory set aside at start-up and handed out and taken back by your own code, so the hot path never allocates.

A series with φ = 1 has a unit root. Taking day-to-day changes, Δxₜ = xₜ − xₜ₋₁, is differencing. A series that becomes stationary after differencing once is integrated of order 1, or I(1). Log prices usually behave like I(1) series.

The shared part that diversification can't remove is systematic risk, or market risk. Each stock's own part, which cancels across many stocks, is idiosyncratic risk, or specific risk. For N stocks with equal money:

.loc picks rows and columns by label, and a label slice keeps both ends. .iloc picks by position, counting from 0, and a position slice leaves out its end, like a list.

Data used to fit a model is in-sample. Data the model never saw is out-of-sample. Walk-forward testing refits on a moving window of the past and forecasts the next stretch, again and again, so every forecast is out of sample, as it would have been in real time.

Misses that say nothing about each other. The standard error assumes them. Copied rows break the assumption completely, and rows that share data break it in part. Either way the slope stays put and its standard error comes out too small.

A and B are independent when knowing one happened doesn't change the chance of the other: P(B | A) = P(B). Then the multiplication rule for chances loses its condition. For the dice, 1/2 × 1/6 = 1/12, and 3 of the 36 squares are both.

Repeats that don't affect each other, such as rolls of a die or flips of a coin, are independent trials. If each succeeds with chance p, the chance of at least one success in n trials is one minus the chance of none:

A DataFrame is pandas' table: a numpy array with labels on both edges. Its row labels are the index, here dates. One column on its own, or one row, is a Series: values with labels.

A score that is 1 when something happens and 0 when it doesn't is an indicator. Its average is the chance that the thing happens. So the average number of things that happen is the sum of their chances:

Traders who know something about value that you don't know yet are informed. The rest are uninformed, sometimes called noise traders. An order doesn't say which kind sent it.

An outer join keeps every date from either side and fills the gaps with NaN. An inner join keeps only the dates both sides have. pd.concat([kite, wren], axis=1, join="inner") does the inner join in one step.

The plain matrix you blend towards is the shrinkage target: here, the sample's own variances with every correlation set to their average. The target's share of the blend, δ ("delta") from 0 to 1, is the shrinkage intensity.

Your inventory is the shares you hold right now: positive when long, negative when short. Inventory risk is the chance of losing money on them while value moves. It grows with how much you hold and how far value can move.

Your inventory is the shares you hold right now: positive when long, negative when short. Inventory risk is the chance of losing money on them while value moves. It grows with how much you hold and how far value can move.

The Kelly criterion stakes the share of your money with the highest growth rate. That share is your edge divided by the payout. For a bet that pays b to 1 and wins with chance p, the best share f* (read "f star") is:

The size that makes wealth grow fastest over the long run is the Kelly size. For a strategy with roughly normal returns it's μ / σ², excess return over variance. For a single bet it's the share of your money to stake. Here it means $5 of positions for every $1 you own.

Tick-to-trade is the time from a market data message arriving to your order leaving in response. Colocation is renting space for your servers inside the exchange's data centre, metres from its matching engine. Kernel bypass lets your program read and write the network card directly, skipping the operating system.

One action that cancels every order and blocks new ones. A loss limit can pull it, a health check can pull it, and so can a person. Firms test theirs, because the day they need it is a bad day to find a bug.

Regression with the penalty λΣ|b|. With uncorrelated, standardised signals it takes λ / 2 off each |Σxy|, keeps the sign and stops at zero, so a signal weaker than that drops out. It chooses signals as well as sizing them.

The law of large numbers says the average of many independent draws settles on the expected value. The typical distance of an average of n draws from that value is its standard error, σ / √n, read "sigma over root n".

Split on what you don't know, into cases that can't overlap and that together cover everything. The chance of A is its chance in each case, weighted by how likely that case is:

A point's residual is its actual y minus the line's y: how far it sits above or below the line. For a line a + bx, least squares picks the a and b with the smallest sum of squared residuals. Fitting a line this way is linear regression, often called OLS, for ordinary least squares.

The slope b is the covariance of x and y divided by the variance of x, and the intercept a puts the line through the point of averages. In finance, the slope of a stock's returns on the market's is the stock's beta.

How likely the evidence is under each explanation is its likelihood: 1 for the two-headed coin, 1/8 for the fair one. Odds compare two chances. In odds, Bayes' rule needs no bottom: multiply the odds by the likelihood ratio, one likelihood over the other.

The price on an order is its limit price: the worst price it will trade at. An order that trades only some of its shares gets a partial fill, and the rest waits in the book at its limit.

The expected value of a sum is the sum of the expected values, whether or not the parts are independent.

.loc picks rows and columns by label, and a label slice keeps both ends. .iloc picks by position, counting from 0, and a position slice leaves out its end, like a list.

The log return is ℓ = ln(P₁ / P₀), read "the natural log of the new price over the old". Over several days, log returns add up to the log return of the whole stretch. The plain kind, P₁ / P₀ − 1, is the simple return.

A simple return is p_t / p_{t−1} − 1, read "today's price over yesterday's, minus 1". A log return is log(p_t / p_{t−1}). Log returns add up over time; simple returns add up across the stocks in a portfolio. For small daily moves the two are almost equal.

A long table has one row per observation: date, ticker, value. A wide table has one row per date and one column per ticker. pivot goes from long to wide, and melt goes back.

Look-ahead bias is a test using information before it was available. A point-in-time table stores each value with the date it became known, so a join can ask: what did we know on this day?

The waiting order made the price, so its owner is the maker. The order that arrived and traded took it, so its owner is the taker.

A fee schedule that charges whoever takes liquidity and pays a rebate to whoever posted it. The rebate is the exchange paying people to keep quotes in its book.

Two standard errors either side of an average is its margin of error: about 95% of the time, the average lands within it of the true mean. To see a gap d against a spread σ, you need the margin below d.

Half the width of a confidence interval is its margin of error. At 95% it's 1.96 standard errors, so it shrinks with √n: a third of the width takes nine times the data.

Valuing what you hold at today's value: P&L = cash + shares × value. Trading desks mark to market all day, not only when a position is closed.

A market by price feed (often called L2) gives the total at each price. A market by order feed (often called L3) gives every order. Top of book alone is L1. Vendors use these nicknames loosely, so check what a feed contains.

A market by price feed (often called L2) gives the total at each price. A market by order feed (often called L3) gives every order. Top of book alone is L1. Vendors use these nicknames loosely, so check what a feed contains.

The market data feed is the stream of messages an exchange sends for every change to its book. Each message carries a sequence number, one higher than the last. A skip means you missed something. A repeat means a copy.

A market maker keeps a bid and an ask up at the same time, so anyone can trade with them in either direction. The pair is a quote. The distance from your guess to either side (0.10 at the start) is the half-spread.

Market risk, also called systematic risk, is the part of a stock's variance that comes with the market: β² × σ²(market). Specific risk, or idiosyncratic risk, is the rest, σ²(e), from the stock's own moves. A market hedge removes the first and leaves the second.

This is the capital asset pricing model, or CAPM. The market's expected return above cash, E[Rm] − r, is the market risk premium. In CAPM, beta is the only thing that earns a premium.

Your queue position is how much size sits ahead of your order at its price. A markout scores a fill against value a short time later: for a buy, value later minus your price. Positive markouts mean you earned the spread. Negative ones mean someone knew more.

Stretching a single number across a whole array is broadcasting. The array of True and False from a comparison is a mask. r[mask] keeps the elements where the mask is True, and mask.mean() is the share that are True.

The matching engine is the exchange's program that takes orders one at a time, matches each against the book by its priority rule, and rests what's left. Each match between the arriving order and one waiting order is a fill, with its own size and price.

The biggest fall from a running peak to a later low, as a share of the peak. Risk reports quote it next to volatility: it's the loss of someone who bought at the worst moment.

A series pulled back toward a long-run average is mean-reverting. Its half-life is the number of days it takes, on average, to close half of any gap from that average.

Choosing a portfolio by its expected return and variance alone is mean-variance optimisation. For uncorrelated assets, the best weights are in proportion to excess return over variance. With correlations, the same recipe uses the covariance matrix Σ: weights in proportion to Σ⁻¹(μ − r), read "the inverse of the covariance matrix times the excess returns".

The spread is the best ask minus the best bid. The mid is halfway between them. The mid is a fair middle for valuing what you hold, but trading straight away happens at the bid or the ask.

The upper edge, the most return for each level of risk, is the efficient frontier. Its left tip, the lowest-risk mix of all, is the minimum-variance portfolio. Add cash, and the steepest line from Mean-variance touches the frontier at one point: the tangency portfolio.

Momentum, or trend, is when past returns over some horizon predict future returns of the same sign. Reversal is when they predict the opposite sign. Trading in the direction of momentum is trend following.

The Monty Hall problem is the usual name for the three doors puzzle. Its method reaches far past game shows: before you update on news, ask how the news was chosen. The same open door means different things under different rules.

Multicollinearity means some inputs are close to a mix of the others. An input's variance inflation factor, VIF, is how many times its coefficient's variance grows compared with unrelated inputs. Its R² comes from regressing that input on the others.

When a choice happens in stages, and each stage has the same number of options whatever came before, the number of outcomes is the product of the options at each stage.

The chance of both is the chance of the first, times the chance of the second given the first. It's the formula for P(B | A), turned around:

n!, read "n factorial", is n × (n − 1) × … × 1, so 3! = 6 and 5! = 120. The number of groups of k things from n is C(n, k), read "n choose k":

Matching rows and columns by label before doing the maths is alignment. NaN, short for not a number, marks a missing value. Most pandas maths skips it or passes it through.

With n days of a series that has no memory, the autocorrelation at any lag lands inside this band 95% of the time. A bar outside it is evidence of memory.

Two returns in one row that cover different stretches of time are non-synchronous. They make related stocks look less related. Join the prices by date, drop the days either market was shut, and only then take returns.

The normal distribution is the bell curve, fixed by its mean μ and standard deviation σ. The z-score of a value x counts how many standard deviations it sits from the mean. "68, 95, 99.7" covers most quick estimates.

The dull explanation you test against is the null hypothesis: here, "the coin is fair". The p-value is the chance, if the null hypothesis were true, of a result at least as extreme as yours. A small p means surprising data.

The hot path is the code that runs on every message. An object pool is memory set aside at start-up and handed out and taken back by your own code, so the hot path never allocates.

How likely the evidence is under each explanation is its likelihood: 1 for the two-headed coin, 1/8 for the fair one. Odds compare two chances. In odds, Bayes' rule needs no bottom: multiply the odds by the likelihood ratio, one likelihood over the other.

One row that sums up a stretch of trading: its first price (open), highest (high), lowest (low), last (close), and the total traded (volume).

A confounder drives both who gets the treatment and the outcome, like order size here. Leaving one out shifts the slope by its effect on the outcome times how much it moves with the treatment: omitted-variable bias.

Every waiting order, sorted by price, is the order book. Waiting buy orders are bids and waiting sell orders are asks. The highest bid is the best bid and the lowest ask is the best ask.

Bid size ÷ (bid size + ask size), at the best prices. Above 0.5 the book leans up, below 0.5 it leans down. 900 ÷ 1,000 = 0.90 leans hard up.

Market, limit, IOC, FOK and post-only are order types. IOC and FOK are also called time-in-force instructions, because they say how long an order may wait. Each one is a check before the matching loop or a rule after it. The loop itself doesn't change.

Orders sent divided by trades made. Regulators and exchanges watch it, because a flood of orders that never trade costs everyone bandwidth.

Data used to fit a model is in-sample. Data the model never saw is out-of-sample. Walk-forward testing refits on a moving window of the past and forecasts the next stretch, again and again, so every forecast is out of sample, as it would have been in real time.

An outer join keeps every date from either side and fills the gaps with NaN. An inner join keeps only the dates both sides have. pd.concat([kite, wren], axis=1, join="inner") does the inner join in one step.

A point far from the rest is an outlier. In r, a point counts by the product of its distances from the averages, so a point twice as far out on both counts four times as much. Before you trust an r, look at the scatter and ask which points are doing the work.

Training error is measured on the rows a model was fitted to, and test error on rows held back. A model much better on its training rows than on test rows is overfitting: it has learned the noise. A model too simple to follow the real shape is underfitting.

The dull explanation you test against is the null hypothesis: here, "the coin is fair". The p-value is the chance, if the null hypothesis were true, of a result at least as extreme as yours. A small p means surprising data.

The partial autocorrelation at lag k is the last coefficient, φₖ, when you fit an AR(k). It measures what k days ago adds once the days in between are known. For an AR(p) it drops to zero after lag p.

The price on an order is its limit price: the worst price it will trade at. An order that trades only some of its shares gets a partial fill, and the rest waits in the book at its limit.

p99, the 99th percentile, is the time that 99% of trips beat. The slow end of the spread is the tail. Trading teams watch p99 and p99.9 at least as closely as the median, because bursts are when the market is busiest.

The middle 95% of a number's values across many resamples: sort them and set aside 2.5% at each end. It needs no formula, and it can be lopsided where the number's spread is.

A permutation is an arrangement, where order matters. A combination is a group, where it doesn't. To count combinations, count the permutations, then divide by the number of orders of one group.

σ²ₜ = ω + αr²ₜ₋₁ + βσ²ₜ₋₁, read "today's variance is a constant, plus alpha times yesterday's squared return, plus beta times yesterday's variance". With α + β below 1, variance is pulled back to a long-run level V = ω / (1 − α − β). α + β is the persistence. EWMA is the special case ω = 0, α = 1 − λ, β = λ, with persistence exactly 1.

Look-ahead bias is a test using information before it was available. A point-in-time table stores each value with the date it became known, so a join can ask: what did we know on this day?

A cache miss is a read the cache can't serve, so the processor waits on main memory. Pointer chasing is following a chain of addresses, where each read must finish before the next can start.

A count of independent events that arrive at a steady average rate λ has a Poisson distribution. k! is k × (k − 1) × … × 1, and 0! = 1. Its mean is λ, and so is its variance, so σ = √λ.

The signal is the rule's number at each close. The position is what you hold over the next day. In a daily backtest, today's position is yesterday's signal: pos = signal.shift(1).

How common something is before you see any evidence is its base rate, also called the prior: here 1%. The chance after the evidence is the posterior: here about 9%. Answering 90% leaves the base rate out, a mistake called base rate neglect.

Checks every order must pass before it's sent: size, price, rate and position. They stay simple and separate from the strategy, so a bug in the strategy can't switch them off. The price check is often called a price collar.