RAG & LLMs / 2. TEXT SPLITTING

Stage 2: Text Splitting

Chunking strategy directly impacts retrieval quality

EXPLANATION

Embedding models and LLMs have context limits. You must split documents into chunks. But naive splitting breaks semantic meaning — a sentence cut in half loses context.

LangChain's RecursiveCharacterTextSplitter is the gold standard. It tries to split on paragraph breaks → sentence ends → words, in that order — preserving natural boundaries.

Key params:
• chunk_size → max chars per chunk (typically 500–1000)
• chunk_overlap → shared chars between adjacent chunks (prevents losing context at split boundaries)

DATA FLOW

FULL DOCUMENT (10,000 chars)
  ────────────────────────────────────────────────────────────
  [        chunk 1 (800)        ][        chunk 2 (800)       ]
                            [overlap: 150]
                                       [       chunk 3 (800)  ]

  Overlap ensures no sentence is cut off without context.
  RecursiveCharacterTextSplitter tries splits in this order:
    \n\n  →  \n  →  ". "  →  " "  →  ""

CODE

PYTHON
1from langchain.text_splitter import (
2    RecursiveCharacterTextSplitter,
3    TokenTextSplitter,
4)
5
6# ── RecursiveCharacterTextSplitter (recommended) ──────────────────
7splitter = RecursiveCharacterTextSplitter(
8    chunk_size=800,
9    chunk_overlap=150,
10    length_function=len,
11    separators=[
12        "\n\n",   # paragraph (try first)
13        "\n",      # line break
14        ". ",       # sentence
15        " ",        # word
16        "",         # char (last resort)
17    ],
18)
19
20chunks = splitter.split_documents(docs)
21
22print(f"Original pages : {len(docs)}")
23print(f"Chunks created : {len(chunks)}")
24avg = sum(len(c.page_content) for c in chunks) // len(chunks)
25print(f"Avg chunk size : {avg} chars")
26
27# ── Inspect chunks ────────────────────────────────────────────────
28for i, chunk in enumerate(chunks[:3]):
29    print(f"\nChunk {i + 1}")
30    print(f"  Size    : {len(chunk.page_content)} chars")
31    print(f"  Source  : {chunk.metadata['source']}")
32    print(f"  Content : {chunk.page_content[:120]}...")
33
34# ── Token-based splitting (more precise for LLM context) ──────────
35token_splitter = TokenTextSplitter(chunk_size=256, chunk_overlap=32)
36token_chunks = token_splitter.split_documents(docs)
37
38# ── Pro tip: tag chunks with their index ──────────────────────────
39for i, chunk in enumerate(chunks):
40    chunk.metadata["chunk_id"] = i
41    chunk.metadata["total_chunks"] = len(chunks)
← PREV1. Document LoadingNEXT →3. Embedding (Bi-Encoder)