{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Building Networks with nn.Module\n", "\n", "A crash course in Python classes, the anatomy of nn.Module, and an end-to-end MLP classifier — from raw data to accuracy.\n", "\n", "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/neural-networks-pytorch).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "`nn.Sequential` is great for straight-line networks, but real architectures\n", "branch, reuse blocks, and take configuration arguments. For that, PyTorch has\n", "one universal pattern: **subclass `nn.Module`**. Every model you'll ever see —\n", "from a two-layer MLP to GPT — is written this way. Since it's built on Python\n", "classes, we'll start with a five-minute object-oriented programming (OOP)\n", "refresher, then build a complete classifier end to end." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Just enough OOP\n", "\n", "A **class** is a blueprint; an **object** (instance) is one thing built from\n", "it. `__init__` runs at construction time and stores data on `self`; other\n", "methods define behavior. This runs right in your browser:" ] }, { "cell_type": "code", "id": "cell-0003", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "class Person:\n", " def __init__(self, height_cm, weight_kg, hair_color):\n", " self.height = height_cm # attributes live on self\n", " self.weight = weight_kg\n", " self.hair_color = hair_color\n", "\n", " def bmi(self): # a method: behavior using the data\n", " return self.weight / (self.height / 100) ** 2\n", "\n", " def dye_hair(self, color): # methods can also change state\n", " self.hair_color = color\n", "\n", "alice = Person(168, 62, \"black\") # calls __init__\n", "bob = Person(181, 90, \"brown\") # a second, independent object\n", "\n", "print(\"alice BMI:\", round(alice.bmi(), 1))\n", "print(\"bob BMI:\", round(bob.bmi(), 1))\n", "\n", "alice.dye_hair(\"green\")\n", "print(\"alice hair is now\", alice.hair_color)" ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "The second idea you need is **inheritance**: a class can extend another,\n", "getting all its attributes and methods for free. `super().__init__()` runs the\n", "parent's constructor first, so the parent's setup happens before yours:" ] }, { "cell_type": "code", "id": "cell-0005", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "class Person:\n", " def __init__(self, height_cm, weight_kg):\n", " self.height = height_cm\n", " self.weight = weight_kg\n", "\n", " def bmi(self):\n", " return self.weight / (self.height / 100) ** 2\n", "\n", "class Student(Person): # Student inherits from Person\n", " def __init__(self, height_cm, weight_kg, university):\n", " super().__init__(height_cm, weight_kg) # run Person's __init__\n", " self.university = university # then add our own attribute\n", "\n", "s = Student(168, 62, \"UGM\")\n", "print(s.university)\n", "print(\"inherited method still works:\", round(s.bmi(), 1))" ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "That's genuinely all the OOP you need: `class`, `__init__`, `self`, methods,\n", "and `super().__init__()`. Now look at how PyTorch uses exactly this pattern." ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "## Anatomy of an nn.Module\n", "\n", "A PyTorch model is a class that inherits from `nn.Module` and defines two\n", "things:\n", "\n", "- **`__init__`** — *what parts exist*: create the layers and store them on `self`.\n", "- **`forward`** — *how data flows*: take the input, pass it through the parts,\n", " return the output." ] }, { "cell_type": "code", "id": "cell-0008", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "\n", "class MLP(nn.Module):\n", " def __init__(self, input_size, hidden_size, output_size):\n", " super().__init__() # nn.Module's own setup — never skip\n", " self.fc1 = nn.Linear(input_size, hidden_size)\n", " self.fc2 = nn.Linear(hidden_size, hidden_size // 2)\n", " self.fc3 = nn.Linear(hidden_size // 2, output_size)\n", " self.relu = nn.ReLU()\n", "\n", " def forward(self, x):\n", " x = self.relu(self.fc1(x))\n", " x = self.relu(self.fc2(x))\n", " return self.fc3(x) # raw logits — no softmax\n", "\n", "model = MLP(input_size=8, hidden_size=16, output_size=2)\n", "print(model)" ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "Three details worth pausing on:\n", "\n", "- `super().__init__()` must run **before** you assign any layers — it's what\n", " lets `nn.Module` detect and register the parameters you attach to `self`\n", " (that's how `model.parameters()` later finds them for the optimizer).\n", "- Because the class takes arguments, the same blueprint builds networks of any\n", " size — try `MLP(20, 64, 5)`.\n", "- You call the model like a function — `model(x)` — never `model.forward(x)`\n", " directly. The call syntax runs important hooks around your `forward`." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "> **nn.Sequential is a shortcut, not a rival**\n", "> \n", "> For a plain layer-after-layer stack, `nn.Sequential(nn.Linear(8, 16),\n", "> nn.ReLU(), nn.Linear(16, 2))` says the same thing in one expression — and you\n", "> can use `nn.Sequential` blocks *inside* an `nn.Module` to group repeated\n", "> patterns. Reach for a full subclass when you need arguments, branching, skip\n", "> connections, or any logic in the forward pass." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "## End to end: classifying the moons dataset\n", "\n", "Time to put MCO and `nn.Module` together on a real (toy) classification\n", "problem — two interleaved crescents that no straight line can separate. Run\n", "this in Colab:" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import torch\n", "from torch import nn, optim\n", "from sklearn.datasets import make_moons\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "# 1. Data: numpy -> scaled -> tensors\n", "X, y = make_moons(n_samples=1000, noise=0.25, random_state=42)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.2, stratify=y, random_state=42\n", ")\n", "\n", "scaler = StandardScaler()\n", "X_train = scaler.fit_transform(X_train)\n", "X_test = scaler.transform(X_test)\n", "\n", "X_train = torch.tensor(X_train, dtype=torch.float32).to(device)\n", "X_test = torch.tensor(X_test, dtype=torch.float32).to(device)\n", "y_train = torch.tensor(y_train, dtype=torch.long).to(device) # class labels: long!\n", "y_test = torch.tensor(y_test, dtype=torch.long).to(device)\n", "\n", "# 2. MCO\n", "class MLP(nn.Module):\n", " def __init__(self, input_size, hidden_size, output_size):\n", " super().__init__()\n", " self.net = nn.Sequential(\n", " nn.Linear(input_size, hidden_size),\n", " nn.ReLU(),\n", " nn.Linear(hidden_size, hidden_size // 2),\n", " nn.ReLU(),\n", " nn.Linear(hidden_size // 2, output_size),\n", " )\n", "\n", " def forward(self, x):\n", " return self.net(x)\n", "\n", "model = MLP(2, 32, 2).to(device)\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.Adam(model.parameters(), lr=0.01)\n", "\n", "# 3. Train\n", "for epoch in range(300):\n", " model.train()\n", " output = model(X_train)\n", " loss = criterion(output, y_train)\n", "\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", "\n", " if (epoch + 1) % 100 == 0:\n", " print(f\"epoch {epoch+1} | train loss {loss.item():.4f}\")\n", "\n", "# 4. Evaluate\n", "model.eval()\n", "with torch.inference_mode():\n", " preds = model(X_test).argmax(1)\n", "acc = (preds == y_test).float().mean().item()\n", "print(f\"test accuracy: {acc:.3f}\") # ~0.96-0.98" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "Note the rhythm: features become `float32`, class labels become `long`, model\n", "and data both move `.to(device)`, logits come out, `argmax(1)` turns them into\n", "predicted classes. That skeleton carries you through the entire course." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "## How big should the hidden layers be?\n", "\n", "There's no formula, but these rules of thumb serve well:\n", "\n", "- **Start small** (one hidden layer, 16–64 units) and grow only if the model\n", " underfits — training loss stuck high.\n", "- **Funnel shapes work well**: sizes shrinking toward the output (e.g.\n", " 64 → 32 → 16), compressing information stage by stage.\n", "- **Powers of two** (16, 32, 64, 128...) are convention, not magic — they're\n", " just easy to reason about and hardware-friendly.\n", "- More width/depth means more capacity — and more overfitting risk on small\n", " data. Which brings us to dropout." ] }, { "cell_type": "markdown", "id": "cell-0015", "metadata": {}, "source": [ "## Dropout: organized forgetting\n", "\n", "**Dropout** randomly zeroes a fraction of a layer's activations during\n", "training (e.g. `p=0.2` drops 20%). Each step, a different random subset of\n", "units vanishes, so no unit can rely on a specific neighbor — the network is\n", "forced to learn redundant, robust features instead of brittle co-adaptations.\n", "It's one of the cheapest and most effective ways to fight overfitting.\n", "\n", "Placement: **after the activation, on hidden layers only** — never on the\n", "output:" ] }, { "cell_type": "code", "id": "cell-0016", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "self.net = nn.Sequential(\n", " nn.Linear(2, 32),\n", " nn.ReLU(),\n", " nn.Dropout(0.2), # after the activation\n", " nn.Linear(32, 16),\n", " nn.ReLU(),\n", " nn.Dropout(0.2),\n", " nn.Linear(16, 2), # output layer: no dropout\n", ")" ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "Dropout behaves differently in training and evaluation — it drops units while\n", "learning but must use *all* of them when predicting. That's exactly what\n", "`model.train()` and `model.eval()` toggle. Forgetting `model.eval()` before\n", "evaluation is a classic bug: your test predictions become randomly noisy and\n", "accuracy mysteriously fluctuates." ] }, { "cell_type": "markdown", "id": "cell-0018", "metadata": {}, "source": [ "### 🏋️ Exercise — A configurable MLP with dropout\n", "\n", "Rewrite the moons classifier so the model class takes `hidden_size` and\n", "`dropout` as constructor arguments (dropout after each hidden activation).\n", "Train three versions — `hidden_size` of 8, 32, and 128 — and for each print\n", "the parameter count (`sum(p.numel() for p in model.parameters())`) and test\n", "accuracy. Does the biggest model win, or does the problem saturate early?" ] }, { "cell_type": "code", "id": "cell-0019", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0020", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import torch\n", "from torch import nn, optim\n", "from sklearn.datasets import make_moons\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "X, y = make_moons(n_samples=1000, noise=0.25, random_state=42)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.2, stratify=y, random_state=42)\n", "scaler = StandardScaler()\n", "X_train = torch.tensor(scaler.fit_transform(X_train), dtype=torch.float32).to(device)\n", "X_test = torch.tensor(scaler.transform(X_test), dtype=torch.float32).to(device)\n", "y_train = torch.tensor(y_train, dtype=torch.long).to(device)\n", "y_test = torch.tensor(y_test, dtype=torch.long).to(device)\n", "\n", "class MLP(nn.Module):\n", " def __init__(self, input_size, output_size, hidden_size=32, dropout=0.2):\n", " super().__init__()\n", " self.net = nn.Sequential(\n", " nn.Linear(input_size, hidden_size),\n", " nn.ReLU(),\n", " nn.Dropout(dropout),\n", " nn.Linear(hidden_size, hidden_size // 2),\n", " nn.ReLU(),\n", " nn.Dropout(dropout),\n", " nn.Linear(hidden_size // 2, output_size),\n", " )\n", "\n", " def forward(self, x):\n", " return self.net(x)\n", "\n", "for hidden in [8, 32, 128]:\n", " torch.manual_seed(0)\n", " model = MLP(2, 2, hidden_size=hidden).to(device)\n", " criterion = nn.CrossEntropyLoss()\n", " optimizer = optim.Adam(model.parameters(), lr=0.01)\n", "\n", " for _ in range(300):\n", " model.train()\n", " loss = criterion(model(X_train), y_train)\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", "\n", " model.eval()\n", " with torch.inference_mode():\n", " acc = (model(X_test).argmax(1) == y_test).float().mean().item()\n", " n_params = sum(p.numel() for p in model.parameters())\n", " print(f\"hidden={hidden:4d} | params={n_params:6d} | test acc={acc:.3f}\")\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0021", "metadata": {}, "source": [ "Next up: right now we feed the *entire* dataset through the model every epoch —\n", "fine for 1,000 points, impossible for 1,000,000 images. Minibatches, `Dataset`,\n", "and `DataLoader` fix that." ] } ] }