Full Compiler Pipeline — Source to Execution

Let's tie everything together by building a complete mini-compiler that takes a simple language all the way from source text to execution. This demonstrates every phase working as an integrated system.

The mini language supports:
- Integer arithmetic: +, -, *, /
- Variable assignment: x = expr
- If/else statements
- While loops
- Print statements
- Return values

The complete pipeline:
① Source text → Lexer → Token stream
② Token stream → Parser → AST
③ AST → Semantic Analyzer → Typed AST + Symbol Table
④ Typed AST → IR Generator → Three-Address Code
⑤ TAC → Optimizer → Optimized TAC
⑥ Optimized TAC → Code Generator → Target code
⑦ Target code → Execution

Real-world compiler architectures:

GCC (GNU Compiler Collection):
- Front-ends: C, C++, Fortran, Ada, Go
- IR: GIMPLE (high-level) → RTL (Register Transfer Language)
- Back-ends: x86, ARM, RISC-V, MIPS, PowerPC
- Optimization: 3 levels (-O1, -O2, -O3)

LLVM/Clang:
- Front-ends: Clang (C/C++), Rust, Swift, Kotlin/Native, Julia
- IR: LLVM IR (SSA form, typed, explicit memory model)
- Back-ends: x86, ARM, RISC-V, WebAssembly, GPU
- Middle-end: 100+ optimization passes

CPython:
- Front-end: Python tokenizer + parser → AST
- No separate IR: AST → bytecode directly
- Back-end: CPython VM interprets bytecode
- No machine code generation (unless PyPy, Cython, Numba)

PyPy:
- JIT compiles hot Python bytecode to native x86
- Uses tracing JIT: traces execution paths, compiles traces
- Can be 10-100× faster than CPython for CPU-bound code

The key insight connecting all your modules:
TOC → defines what's computable and what languages can be parsed (DFA for lexer, CFG for parser, TM for the full computation)
Compiler Design → implements those theoretical machines as practical tools
COA → defines the target (ISA, registers, instruction set the code generator targets)
OS → runs the compiled code (loads executable, manages process, handles syscalls)
FastAPI/Python → you write source code that all of this processes invisibly

1# Complete mini-compiler: source → execution
2
3class Interpreter:
4    """
5    Instead of generating real machine code,
6    we interpret the AST directly (like Python does).
7    This shows the full pipeline working end-to-end.
8    """
9    def __init__(self):
10        self.env    = {}
11        self.output = []
12
13    def eval_expr(self, node):
14        if isinstance(node, Num):
15            return node.value
16        if isinstance(node, Var):
17            if node.name not in self.env:
18                raise NameError(f"Undefined variable: {node.name}")
19            return self.env[node.name]
20        if isinstance(node, BinOp):
21            l = self.eval_expr(node.left)
22            r = self.eval_expr(node.right)
23            ops = {"+": lambda a,b: a+b, "-": lambda a,b: a-b,
24                   "*": lambda a,b: a*b, "/": lambda a,b: a//b,
25                   ">": lambda a,b: int(a>b), "<": lambda a,b: int(a<b),
26                   ">=":lambda a,b: int(a>=b),"<=":lambda a,b: int(a<=b),
27                   "==":lambda a,b: int(a==b),"!=":lambda a,b: int(a!=b)}
28            return ops[node.op](l, r)
29        if isinstance(node, UnaryOp):
30            v = self.eval_expr(node.operand)
31            return -v if node.op == "-" else not v
32        raise ValueError(f"Unknown expr: {node}")
33
34    def exec_stmt(self, node):
35        if isinstance(node, Assign):
36            self.env[node.name] = self.eval_expr(node.value)
37        elif isinstance(node, If):
38            cond = self.eval_expr(node.condition)
39            body = node.then_body if cond else node.else_body
40            for stmt in body:
41                result = self.exec_stmt(stmt)
42                if result is not None:
43                    return result
44        elif isinstance(node, Return):
45            return self.eval_expr(node.value)
46        return None
47
48    def run(self, program):
49        for stmt in program:
50            result = self.exec_stmt(stmt)
51            if result is not None:
52                return result
53
54# ── Full pipeline demo ─────────────────────────────────
55def compile_and_run(source: str, verbose=True):
56    print(f"
57{'='*50}")
58    print(f"SOURCE:
59{source}")
60
61    # Phase 1: Lex
62    lexer  = Lexer(source)
63    tokens = lexer.tokenize()
64    if verbose:
65        print(f"
66TOKENS: {[f'{t.type.name}({t.value})' for t in tokens if t.type != TT.EOF]}")
67
68    # Phase 2: Parse
69    parser   = Parser(tokens)
70    ast_tree = parser.parse_program()
71    if verbose:
72        print(f"
73AST: {ast_tree}")
74
75    # Phase 3+4: IR Generation
76    ir_gen = TACGenerator()
77    tac    = ir_gen.generate(ast_tree)
78    if verbose:
79        print("
80TAC:")
81        for i in tac: print(i)
82
83    # Phase 5: Optimize
84    optimized = constant_fold(tac)
85    optimized = algebraic_simplification(optimized)
86    optimized = constant_propagation(optimized)
87    optimized = dead_code_elimination(optimized)
88    if verbose and optimized != tac:
89        print("
90OPTIMIZED TAC:")
91        for i in optimized: print(i)
92
93    # Phase 6+7: Execute (via interpreter)
94    interp = Interpreter()
95    result = interp.run(ast_tree)
96    print(f"
97RESULT: {result}")
98    print(f"ENV:    {interp.env}")
99    return result
100
101# ── Test programs ──────────────────────────────────────
102compile_and_run("x = 2 + 3 * 4;
103result = x - 6;
104return result;
105")
106
107compile_and_run("""
108x = 10;
109y = 0;
110if (x) {
111    y = x + 5;
112} else {
113    y = 0;
114}
115return y;
116""", verbose=False)
117
118# ── What Python's compiler does to YOUR code ──────────
119import dis, ast as pyast
120
121source = """
122def fibonacci(n):
123    if n <= 1:
124        return n
125    return fibonacci(n-1) + fibonacci(n-2)
126"""
127
128print("
129" + "="*50)
130print("YOUR PYTHON CODE through the compiler:")
131print(source)
132
133tree = pyast.parse(source)
134print("AST (Phase 2):")
135print(pyast.dump(tree, indent=2)[:600])
136
137code_obj = compile(source, "<demo>", "exec")
138func_code = code_obj.co_consts[0]   # the fibonacci function code object
139print("
140Bytecode (Phase 4 IR — Python's TAC equivalent):")
141dis.dis(func_code)
142
143print("
144Key stats:")
145print(f"  Instructions: {len(list(dis.get_instructions(func_code)))}")
146print(f"  Local vars:   {func_code.co_varnames}")
147print(f"  Constants:    {func_code.co_consts}")