RAG & LLMs / 7. LLM GENERATION

Stage 7: LLM Generation

Decoder-only LLM grounded by reranked context

EXPLANATION

The LLM (decoder-only transformer) takes the reranked chunks as context and generates an answer token by token. LangChain's LCEL (LangChain Expression Language) chains everything together cleanly using the | pipe operator.

The prompt is critical:
• Inject retrieved context at the top
• Explicitly tell the LLM to answer ONLY from context
• Tell it to say "I don't know" if context is insufficient
• Low temperature (0.1) = factual, consistent answers

The model generates token by token conditioned on (context + question). This is the decoder-only part — pure autoregressive generation.

DATA FLOW

[reranked_chunk_1]
  [reranked_chunk_2]   ← context from retrieval
  [reranked_chunk_3]
          +
  Question: {query}
          ↓
  ChatPromptTemplate
          ↓
  Gemini / GPT / Claude   ← decoder-only LLM
  generates token by token
          ↓
  Grounded answer with source references

CODE

PYTHON
1from langchain_core.prompts import ChatPromptTemplate
2from langchain_core.output_parsers import StrOutputParser
3from langchain_core.runnables import RunnableLambda, RunnablePassthrough
4from langchain_google_genai import ChatGoogleGenerativeAI
5
6llm = ChatGoogleGenerativeAI(
7    model="gemini-1.5-flash",
8    google_api_key="YOUR_GOOGLE_API_KEY",
9    temperature=0.1,   # low = factual and grounded
10)
11
12prompt = ChatPromptTemplate.from_template("""
13You are an expert assistant. Answer the question using ONLY the
14provided context. If the context is insufficient, say:
15"I don't have enough information to answer this."
16
17Context:
18{context}
19
20Question: {question}
21
22Answer (cite relevant sources):
23""")
24
25def format_docs(docs):
26    return "\n\n---\n\n".join([
27        f"[Source: {doc.metadata.get('source','?')}, "
28        f"Page {doc.metadata.get('page','?')}]\n{doc.page_content}"
29        for doc in docs
30    ])
31
32def retrieve_rerank_format(inputs: dict) -> str:
33    q    = inputs["question"]
34    docs = hybrid_retriever.invoke(q)          # step 5: retrieve
35    top  = rerank(q, docs, top_n=3)            # step 6: rerank
36    return format_docs(top)                    # format for prompt
37
38# ── LCEL Chain: retrieval + reranking + LLM ───────────────────────
39rag_chain = (
40    {
41        "context":  RunnableLambda(retrieve_rerank_format),
42        "question": RunnablePassthrough() | (lambda x: x["question"]),
43    }
44    | prompt
45    | llm
46    | StrOutputParser()
47)
48
49# ── Invoke ─────────────────────────────────────────────────────────
50answer = rag_chain.invoke({"question": "What is backpropagation?"})
51print(answer)
52
53# ── Streaming ─────────────────────────────────────────────────────
54for chunk in rag_chain.stream({"question": "Explain self-attention"}):
55    print(chunk, end="", flush=True)
← PREV6. Cross-Encoder RerankingNEXT →Full Pipeline