DEEP LEARNING / 1. PERCEPTRON & ACTIVATIONS

Perceptron & Activation Functions

The building block — one neuron, then many

EXPLANATION

A single neuron computes a weighted sum of inputs, adds a bias, then passes it through an activation function.

Without activation functions, stacking layers is useless — n linear layers = 1 linear layer. Activations introduce non-linearity, which is what gives neural networks their power to approximate any function.

Key activation functions:
• Sigmoid  → squashes to (0,1), used in output for binary classification. Problem: vanishing gradients
• Tanh     → squashes to (-1,1), zero-centered. Still has vanishing gradient problem
• ReLU     → max(0,x). Simple, fast, no vanishing gradient. Problem: dying ReLU
• Leaky ReLU → fixes dying ReLU with small slope for x<0
• GELU     → used in transformers (BERT, GPT). Smooth approximation of ReLU

DATA FLOW

Single Neuron:

  x1 ──(w1)──┐
  x2 ──(w2)──┼──→  z = w·x + b  ──→  a = activation(z)  ──→ output
  x3 ──(w3)──┘

  Activation functions:
  Sigmoid:   σ(z) = 1 / (1 + e^-z)          → (0, 1)
  Tanh:      tanh(z) = (e^z - e^-z)/(e^z + e^-z) → (-1, 1)
  ReLU:      max(0, z)                        → [0, ∞)
  GELU:      z · Φ(z)  (Φ = normal CDF)      → smooth

CODE

PYTHON
1import torch
2import torch.nn as nn
3import matplotlib.pyplot as plt
4
5# ── Single neuron manually ────────────────────────────────────────
6x = torch.tensor([1.0, 2.0, 3.0])
7w = torch.tensor([0.5, -0.3, 0.8])
8b = torch.tensor(0.1)
9
10z = torch.dot(w, x) + b       # linear: z = w·x + b
11a = torch.relu(z)              # activation
12print(f"z = {z.item():.4f}, a (relu) = {a.item():.4f}")
13
14# ── Activation functions side by side ────────────────────────────
15x = torch.linspace(-5, 5, 200)
16
17activations = {
18    "Sigmoid":    torch.sigmoid(x),
19    "Tanh":       torch.tanh(x),
20    "ReLU":       torch.relu(x),
21    "Leaky ReLU": nn.LeakyReLU(0.1)(x),
22    "GELU":       nn.GELU()(x),
23}
24
25fig, axes = plt.subplots(1, 5, figsize=(18, 3))
26for ax, (name, y) in zip(axes, activations.items()):
27    ax.plot(x.numpy(), y.detach().numpy(), color="#7B61FF", linewidth=2)
28    ax.set_title(name, fontsize=10)
29    ax.axhline(0, color="#333", linewidth=0.5)
30    ax.axvline(0, color="#333", linewidth=0.5)
31    ax.grid(alpha=0.2)
32plt.tight_layout()
33plt.savefig("activations.png", dpi=150, bbox_inches="tight")
34
35# ── MLP with different activations using nn.Module ────────────────
36class MLP(nn.Module):
37    def __init__(self, activation=nn.ReLU()):
38        super().__init__()
39        self.net = nn.Sequential(
40            nn.Linear(784, 256),
41            activation,
42            nn.Linear(256, 128),
43            activation,
44            nn.Linear(128, 10),
45        )
46
47    def forward(self, x):
48        return self.net(x)
49
50model_relu = MLP(nn.ReLU())
51model_gelu = MLP(nn.GELU())
52model_tanh = MLP(nn.Tanh())
53
54x = torch.randn(32, 784)   # batch of 32 flattened images
55out = model_relu(x)
56print(f"Output shape: {out.shape}")  # (32, 10)
← PREVOverviewNEXT →2. Backpropagation