RAG & LLMs / 1. DOCUMENT LOADING

Stage 1: Document Loading

LangChain DocumentLoaders — every source, one interface

EXPLANATION

DocumentLoaders convert raw files into LangChain Document objects. Each Document has:
• page_content → the actual text
• metadata → source, page number, author, etc.

LangChain has 100+ loaders: PDF, CSV, HTML, Notion, Google Drive, YouTube, SQL, and more. They all return the same Document format so the rest of your pipeline doesn't change regardless of source.

DATA FLOW

PDF       ──→  PyPDFLoader     ──→  [Document(page_content, metadata)]
  .txt      ──→  TextLoader      ──→  [Document(page_content, metadata)]
  website   ──→  WebBaseLoader   ──→  [Document(page_content, metadata)]
  .csv      ──→  CSVLoader       ──→  [Document(page_content, metadata)]
  folder    ──→  DirectoryLoader ──→  [Document(page_content, metadata)]
                                               ↓
                                    Same interface for everything!

CODE

PYTHON
1from langchain_community.document_loaders import (
2    PyPDFLoader,
3    TextLoader,
4    WebBaseLoader,
5    DirectoryLoader,
6)
7
8# ── Single PDF ─────────────────────────────────────────────────────
9loader = PyPDFLoader("research_paper.pdf")
10docs = loader.load()
11
12print(f"Pages loaded : {len(docs)}")
13print(f"Preview      : {docs[0].page_content[:200]}")
14print(f"Metadata     : {docs[0].metadata}")
15# {'source': 'research_paper.pdf', 'page': 0}
16
17# ── Entire folder of PDFs ──────────────────────────────────────────
18dir_loader = DirectoryLoader(
19    path="./documents/",
20    glob="**/*.pdf",          # only PDF files
21    loader_cls=PyPDFLoader,
22    show_progress=True,
23)
24all_docs = dir_loader.load()
25print(f"Total pages from all PDFs: {len(all_docs)}")
26
27# ── Website ────────────────────────────────────────────────────────
28web_loader = WebBaseLoader(
29    web_paths=["https://arxiv.org/abs/1706.03762"],
30)
31web_docs = web_loader.load()
32
33# ── Inspect a document ────────────────────────────────────────────
34for doc in docs[:2]:
35    print("─" * 60)
36    print(f"Source : {doc.metadata['source']}")
37    print(f"Page   : {doc.metadata.get('page', 'N/A')}")
38    print(f"Length : {len(doc.page_content)} chars")
39    print(f"Content: {doc.page_content[:100]}...")
← PREVPipeline OverviewNEXT →2. Text Splitting