insideout Get Pro

05 · Chunk size and overlap

Pro

Search and Retrieval, lesson 5

Chunk size and overlap

How you split documents decides what search can find.

"Vector search" and "Reranking": how passages get scored and reordered. This lesson is about how a document becomes passages in the first place.

Part 1. Where the cut falls

Search never runs on whole documents. Before indexing, every document is cut into pieces, and each piece gets its own BM25 entry and its own embedding. Where those cuts fall decides what search can find.

Here is one paragraph from the handbook, 183 characters long: "Annual plans can get a refund within 30 days. Enterprise plans get a refund within 60 days. The refund is pro-rated by unused months. Staff may work from home up to three days a week."

The simplest way to chunk it is to cut every 60 characters, wherever that lands.

Example 1.1

Cutting every 60 characters

  1. Annual plans can get a refund within 30 days. Enterprise pla

    Chunk 1 ends in the middle of a word. It has "Enterprise", "refund" and "30 days".

  2. ns get a refund within 60 days. The refund is pro-rated by u

    Chunk 2 holds the answer for enterprise customers, but the word "Enterprise" was left behind in chunk 1.

  3. nused months. Staff may work from home up to three days a we
    ek.

    Chunks 3 and 4 mix the end of the refund rule with the remote-work rule.

Your turn

Someone asks "How long do enterprise customers have to get a refund?" Which chunk is a retriever most likely to return first, and what goes wrong?

Chunk 1 contains "enterprise", "refund" and a number of days, so it matches the question best. But its 30 days belongs to annual plans. Chunk 2 has the real answer and doesn't say whose rule it is. The cut moved a fact away from its subject.

A common patch is to let neighbouring chunks share some text. Cut every 40 characters but keep each chunk 60 long, and one chunk becomes "days. Enterprise plans get a refund within 60 days. The refu", which holds the whole rule.

Chunk, chunk size and overlap

A chunk is one piece of a document: the unit that gets indexed, retrieved and handed on. Chunk size is how long each chunk is, usually counted in tokens. Overlap is how much text neighbouring chunks share, so a fact near a cut still lands whole in at least one chunk. Overlap stores some text twice.

Part 2. Too small, too big

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Too small, too big
  2. Cut on boundaries
  3. Cut where the topic changes
See Pro plansI have Pro: sign in
Next lessonIndexes at scale with HNSWApproximate nearest neighbours, fast enough for millions of vectors. Start lesson 6

Search and Retrieval overviewSpotted a mistake? Tell us.