insideout Get Pro

05 · Cosine similarity and the dot product

Pro

Tokens and Embeddings, lesson 5

Cosine similarity and the dot product

Two ways to score how close two vectors are, and when they disagree.

You know each token becomes a row of numbers, its embedding ("Looking up a token's vector"). This lesson scores how alike two of those rows are.

Part 1. Multiply and add

Here are three words, each given three numbers by hand: cat is (1, 2, 2), dog is (2, 1, 2) and car is (2, −2, 1). The numbers don't mean anything on their own. What matters is how two lists line up.

The quickest test is this: multiply the two lists position by position, then add up the products. When both lists are big in the same places, the products are big and positive, and so is the total.

Example 1.1

Two scores, by hand

  1. cat (1, 2, 2)dog (2, 1, 2)

    Two words, three numbers each.

  2. 1 × 2+2 × 1+2 × 2
    2+2+4=8

    Multiply in pairs, then add. Every pair agrees in sign, so every product pushes the total up. Cat and dog score 8.

  3. cat (1, 2, 2)car (2, −2, 1)
    1 × 2+2 × −2+2 × 1
    2+−4+2=0

    Cat and car. The middle pair disagrees in sign and cancels the other two. The score is 0: no match at all.

Dot product

Multiply two vectors position by position and add up the products. Positive means they point roughly the same way, zero means they're at right angles (unrelated), negative means they point apart. Written a · b, read "a dot b".

Your turn

A fourth word has the vector (−1, −2, −2). What's its dot product with cat (1, 2, 2)?

−1 × 1 + −2 × 2 + −2 × 2 = −1 − 4 − 4 = −9. It's cat with every sign flipped, so every product is negative. The dot product is as low as it can be for vectors this long: they point in exactly opposite directions.

Part 2. Length gets in the way

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Length gets in the way
  2. Scale to length 1 first
See Pro plansI have Pro: sign in
Next lessonThe distributional hypothesisWords used in the same contexts end up with similar vectors. Start lesson 6

Tokens and Embeddings overviewSpotted a mistake? Tell us.