{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Transfer Learning in Practice\n", "\n", "Freeze a pretrained ResNet, retrain its head on your own images, then fine-tune with a tiny learning rate — and see why this crushes training from scratch.\n", "\n", "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/transfer-learning-pytorch).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "The last lesson ended with a promise: every architecture in `torchvision` ships\n", "with weights already trained on ImageNet. In this lesson you'll cash that in.\n", "Instead of training a CNN from scratch, you'll take a pretrained ResNet-18,\n", "swap its final layer for one that fits *your* classes, and get a strong\n", "classifier from a few hundred images in minutes. All the PyTorch code here\n", "belongs in **Colab with a GPU runtime** — transfer learning is cheap, but not\n", "browser-cheap." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Why features transfer\n", "\n", "Recall what a CNN learns layer by layer: the earliest convolutions detect\n", "edges, color blobs, and simple textures; middle layers combine those into\n", "motifs like fur, mesh, or eyes; only the last layers become specific to the\n", "1,000 ImageNet categories. Here's the key observation — **the early layers are\n", "generic**. An edge detector trained on dogs and teapots works just as well on\n", "X-rays and satellite photos. Those generic features took 1.28 million labeled\n", "images and serious GPU time to learn. Your dataset of 2,000 photos can't\n", "reproduce them — but it doesn't have to, because you can download them.\n", "\n", "Transfer learning reuses a pretrained network as a **feature extractor** and\n", "only re-learns the part that's actually specific to your problem: the\n", "classifier head." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Two strategies\n", "\n", "There are two standard moves, usually applied in sequence:\n", "\n", "- **Feature extraction (adaptation).** Load the pretrained model, **freeze**\n", " every parameter in the backbone, replace the final classification layer with\n", " a fresh one sized for your classes, and train *only that new head*. Fast,\n", " data-efficient, and hard to mess up — the pretrained weights can't be\n", " damaged because they never change.\n", "- **Fine-tuning.** After the head has converged, **unfreeze** some or all of\n", " the backbone and keep training with a much lower learning rate — around 10×\n", " to 100× lower (say `1e-5` instead of `1e-3`). The small steps gently adapt\n", " the pretrained features to your domain without destroying them. If it helps,\n", " repeat with an even lower rate.\n", "\n", "Which strategy, and how much to unfreeze, depends on two questions: how much\n", "data do you have, and how similar is your domain to ImageNet?\n", "\n", "| Your dataset | Similar to ImageNet (photos of objects) | Different domain (medical, satellite, sketches) |\n", "|---|---|---|\n", "| **Small** | Feature extraction only | Feature extraction, heavy augmentation, expect a fight |\n", "| **Large** | Fine-tune the top layers | Fine-tune many or all layers |\n", "\n", "More data or a more different domain both push you toward unfreezing more." ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "## The pipeline: data\n", "\n", "`torchvision`'s `ImageFolder` turns a directory tree into a dataset — one\n", "subfolder per class, folder names become labels:" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "```text\n", "data/\n", " train/\n", " cats/ cat001.jpg cat002.jpg ...\n", " dogs/ dog001.jpg ...\n", " test/\n", " cats/ ...\n", " dogs/ ...\n", "```" ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "One rule is non-negotiable: your inputs must look like what the network saw\n", "during pretraining. That means **224×224 crops** and normalization with the\n", "**ImageNet channel statistics** — mean `[0.485, 0.456, 0.406]` and std\n", "`[0.229, 0.224, 0.225]`:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import torch\n", "from torch import nn, optim\n", "from torchvision import datasets, transforms\n", "from torch.utils.data import DataLoader\n", "\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "train_transform = transforms.Compose([\n", " transforms.RandomRotation(10),\n", " transforms.RandomResizedCrop(224),\n", " transforms.RandomHorizontalFlip(),\n", " transforms.ToTensor(),\n", " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", "])\n", "\n", "test_transform = transforms.Compose([\n", " transforms.Resize(256),\n", " transforms.CenterCrop(224),\n", " transforms.ToTensor(),\n", " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", "])\n", "\n", "train_set = datasets.ImageFolder(\"data/train\", transform=train_transform)\n", "test_set = datasets.ImageFolder(\"data/test\", transform=test_transform)\n", "trainloader = DataLoader(train_set, batch_size=64, shuffle=True, num_workers=2)\n", "testloader = DataLoader(test_set, batch_size=64)\n", "\n", "num_classes = len(train_set.classes)\n", "print(train_set.classes)" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "Augmentation (rotation, random crop, flip) goes on the *training* transform\n", "only — evaluation uses a deterministic resize and center crop.\n", "\n", "What does that `Normalize` actually do? A quick browser check:" ] }, { "cell_type": "code", "id": "cell-0009", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(42)\n", "img = rng.uniform(0, 1, (3, 4, 4)) # fake image, channels first, values in [0, 1]\n", "\n", "mean = np.array([0.485, 0.456, 0.406]).reshape(3, 1, 1)\n", "std = np.array([0.229, 0.224, 0.225]).reshape(3, 1, 1)\n", "normed = (img - mean) / std\n", "\n", "for c, name in enumerate([\"R\", \"G\", \"B\"]):\n", " print(f\"{name}: raw range [{img[c].min():.2f}, {img[c].max():.2f}]\"\n", " f\" -> normalized [{normed[c].min():.2f}, {normed[c].max():.2f}]\")" ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "Each channel is shifted and scaled exactly the way ImageNet images were during\n", "pretraining. Skip this (or use different stats) and the frozen backbone\n", "receives inputs from a distribution it has never seen — accuracy quietly tanks\n", "and nothing errors out." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "## The pipeline: model\n", "\n", "Load ResNet-18 with pretrained weights, freeze everything, then replace the\n", "head. In ResNet the head is a single layer called `fc`:" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from torchvision.models import resnet18, ResNet18_Weights\n", "\n", "model = resnet18(weights=ResNet18_Weights.DEFAULT)\n", "\n", "for param in model.parameters():\n", " param.requires_grad = False # freeze the backbone\n", "\n", "model.fc = nn.Linear(model.fc.in_features, num_classes) # new head, trainable\n", "model = model.to(device)\n", "\n", "trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "total = sum(p.numel() for p in model.parameters())\n", "print(f\"trainable: {trainable:,} / {total:,}\")" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "New layers are created with `requires_grad=True` by default, so only the\n", "fresh `fc` will learn. Out of 11 million parameters, you're training a few\n", "thousand." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "## Phase 1: train the head" ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.AdamW(model.fc.parameters(), lr=1e-3)\n", "\n", "def run_epoch(loader, train=True):\n", " model.train() if train else model.eval()\n", " total_loss = correct = n = 0\n", " ctx = torch.enable_grad() if train else torch.inference_mode()\n", " with ctx:\n", " for images, labels in loader:\n", " images, labels = images.to(device), labels.to(device)\n", " output = model(images)\n", " loss = criterion(output, labels)\n", " if train:\n", " loss.backward()\n", " optimizer.step()\n", " optimizer.zero_grad()\n", " total_loss += loss.item() * len(images)\n", " correct += (output.argmax(1) == labels).sum().item()\n", " n += len(images)\n", " return total_loss / n, correct / n\n", "\n", "for epoch in range(5):\n", " train_loss, train_acc = run_epoch(trainloader, train=True)\n", " test_loss, test_acc = run_epoch(testloader, train=False)\n", " print(f\"epoch {epoch}: train_acc={train_acc:.3f} test_acc={test_acc:.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "Because only the head learns, this converges in a handful of epochs." ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "## Phase 2: fine-tune\n", "\n", "Now unfreeze and continue with a *much* smaller learning rate. Give it more\n", "patience — improvements come slowly and gently:" ] }, { "cell_type": "code", "id": "cell-0018", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "for param in model.parameters():\n", " param.requires_grad = True # unfreeze everything\n", "\n", "optimizer = optim.AdamW(model.parameters(), lr=1e-5) # ~100x lower\n", "\n", "best_acc, patience, bad_epochs = 0.0, 3, 0\n", "for epoch in range(20):\n", " run_epoch(trainloader, train=True)\n", " _, test_acc = run_epoch(testloader, train=False)\n", " if test_acc > best_acc:\n", " best_acc, bad_epochs = test_acc, 0\n", " torch.save(model.state_dict(), \"resnet_best.pth\")\n", " else:\n", " bad_epochs += 1\n", " if bad_epochs >= patience:\n", " break\n", " print(f\"epoch {epoch}: test_acc={test_acc:.3f} (best {best_acc:.3f})\")" ] }, { "cell_type": "markdown", "id": "cell-0019", "metadata": {}, "source": [ "Why the tiny learning rate? The pretrained weights are already excellent —\n", "big steps would scramble them, and you'd be back to (badly) training from\n", "scratch. Fine-tuning is a polish, not a rebuild." ] }, { "cell_type": "markdown", "id": "cell-0020", "metadata": {}, "source": [ "## The benchmark mindset\n", "\n", "Never trust a technique without a baseline. On a typical small custom dataset\n", "(a few thousand images, a handful of classes), the comparison looks like this:\n", "\n", "| Approach | Trainable params | Epochs to converge | Test accuracy (typical) |\n", "|---|---|---|---|\n", "| Small CNN from scratch | ~1M | 25+ | ~70–75% |\n", "| ResNet-18, feature extraction | ~2.5K | ~5 | ~88–91% |\n", "| ResNet-18, fine-tuned | ~11M | +5–10 more | ~91–94% |\n", "\n", "The exact numbers depend on your data; the *pattern* is remarkably stable.\n", "Feature extraction alone usually captures most of the gain; fine-tuning adds a\n", "few extra points on top." ] }, { "cell_type": "markdown", "id": "cell-0021", "metadata": {}, "source": [ "> **A benchmark bonus: finding bad labels**\n", "> \n", "> Once your model is strong, inspect its most *confident wrong* predictions.\n", "> Surprisingly often, the model is right and the label is wrong — transfer\n", "> learning is good enough to expose mislabeled data in your dataset. Fix those\n", "> labels and everything improves." ] }, { "cell_type": "markdown", "id": "cell-0022", "metadata": {}, "source": [ "## Practical tips\n", "\n", "- **Match the preprocessing.** 224×224 inputs, ImageNet mean/std. Each\n", " weights object documents its own recipe: `ResNet18_Weights.DEFAULT.transforms()`\n", " returns the exact transform used at pretraining time.\n", "- **Head first, backbone second.** Training the head with a frozen backbone\n", " first prevents large random-head gradients from wrecking pretrained weights.\n", "- **Unfreeze in proportion to your data.** More images, or a domain further\n", " from ImageNet, justify unfreezing more layers. With tiny datasets, keep the\n", " backbone frozen.\n", "- **Lower the learning rate when you unfreeze.** Rule of thumb: divide by 10\n", " to 100. If fine-tuning makes things worse, your rate is too high." ] }, { "cell_type": "markdown", "id": "cell-0023", "metadata": {}, "source": [ "### 🏋️ Exercise — Feature extraction on CIFAR-10\n", "\n", "In Colab (GPU runtime), build a CIFAR-10 classifier by feature extraction:\n", "load `resnet18` with `ResNet18_Weights.DEFAULT`, freeze the backbone, replace\n", "`model.fc` with a 10-class head, and train the head for 1–2 epochs. Remember\n", "that CIFAR images are 32×32 — resize them to 224 and normalize with ImageNet\n", "statistics. How does your test accuracy after two epochs compare with a small\n", "CNN trained from scratch for much longer?" ] }, { "cell_type": "code", "id": "cell-0024", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0025", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import torch\n", "from torch import nn, optim\n", "from torchvision import datasets, transforms\n", "from torchvision.models import resnet18, ResNet18_Weights\n", "from torch.utils.data import DataLoader\n", "\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "\n", "tf = transforms.Compose([\n", " transforms.Resize(224),\n", " transforms.ToTensor(),\n", " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", "])\n", "\n", "train_set = datasets.CIFAR10(\"data\", train=True, download=True, transform=tf)\n", "test_set = datasets.CIFAR10(\"data\", train=False, download=True, transform=tf)\n", "trainloader = DataLoader(train_set, batch_size=128, shuffle=True, num_workers=2)\n", "testloader = DataLoader(test_set, batch_size=256)\n", "\n", "model = resnet18(weights=ResNet18_Weights.DEFAULT)\n", "for p in model.parameters():\n", " p.requires_grad = False\n", "model.fc = nn.Linear(model.fc.in_features, 10)\n", "model = model.to(device)\n", "\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.AdamW(model.fc.parameters(), lr=1e-3)\n", "\n", "for epoch in range(2):\n", " model.train()\n", " for images, labels in trainloader:\n", " images, labels = images.to(device), labels.to(device)\n", " loss = criterion(model(images), labels)\n", " loss.backward()\n", " optimizer.step()\n", " optimizer.zero_grad()\n", "\n", " model.eval()\n", " correct = 0\n", " with torch.inference_mode():\n", " for images, labels in testloader:\n", " images, labels = images.to(device), labels.to(device)\n", " correct += (model(images).argmax(1) == labels).sum().item()\n", " print(f\"epoch {epoch}: test accuracy = {correct / len(test_set):.3f}\")\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0026", "metadata": {}, "source": [ "Next module: images are done — we turn to data with *order*. Recurrent neural\n", "networks give a model memory, one time step at a time." ] } ] }