RAG & LLMs / 3. EMBEDDING (BI-ENCODER)

Stage 3: Embedding — The Bi-Encoder

Converting text into dense vectors using BERT encoder-only models

EXPLANATION

Embedding models convert text into high-dimensional vectors where semantic similarity equals geometric proximity. These are BERT-family encoder-only transformer models.

When you call embed_query(), the text is tokenized, run through all BERT layers with bidirectional self-attention, then mean-pooled into a single dense vector.

Popular choices:
• BAAI/bge-large-en-v1.5 → best open source, 1024 dims
• all-MiniLM-L6-v2 → fast, 384 dims, great for dev
• text-embedding-3-small → OpenAI (API)
• models/text-embedding-004 → Google Gemini (API)

DATA FLOW

"What is machine learning?"
            ↓
     Tokenize: [CLS] What is machine learning ? [SEP]
            ↓
     BERT Encoder (12 layers of self-attention)
     All tokens attend to ALL other tokens ← bidirectional
            ↓
     Mean pooling over all token vectors
            ↓
  [0.23, -0.87, 0.45, 0.12, ...]   ← 1024-dim vector
            ↓
     Stored in ChromaDB

CODE

PYTHON
1from langchain_huggingface import HuggingFaceEmbeddings
2import numpy as np
3
4# ── Local HuggingFace model (free, no API needed) ─────────────────
5embeddings = HuggingFaceEmbeddings(
6    model_name="BAAI/bge-large-en-v1.5",   # best OSS model
7    model_kwargs={"device": "cpu"},          # "cuda" if GPU
8    encode_kwargs={
9        "normalize_embeddings": True,        # unit vectors → cosine sim
10        "batch_size": 32,
11    },
12)
13
14# ── Google Gemini embeddings (API alternative) ────────────────────
15# from langchain_google_genai import GoogleGenerativeAIEmbeddings
16# embeddings = GoogleGenerativeAIEmbeddings(
17#     model="models/text-embedding-004",
18#     google_api_key="YOUR_KEY",
19# )
20
21# ── Sanity check: semantic similarity ────────────────────────────
22v1 = np.array(embeddings.embed_query("Machine learning algorithms"))
23v2 = np.array(embeddings.embed_query("AI and ML techniques"))
24v3 = np.array(embeddings.embed_query("Paris is the capital of France"))
25
26print(f"ML vs AI    : {np.dot(v1, v2):.4f}")   # ~0.88 HIGH
27print(f"ML vs Paris : {np.dot(v1, v3):.4f}")   # ~0.07 LOW
28
29# ── Embed all chunks (LangChain handles batching) ─────────────────
30texts = [chunk.page_content for chunk in chunks]
31vectors = embeddings.embed_documents(texts)
32print(f"Embedded {len(vectors)} chunks → each {len(vectors[0])} dims")
← PREV2. Text SplittingNEXT →4. Vector Store