{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Famous CNN Architectures\n",
    "\n",
    "The ImageNet competition and the ideas it produced — AlexNet's GPUs, VGG's stacked 3×3 kernels, Inception's parallel branches, and ResNet's skip connections.\n",
    "\n",
    "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/cnn-architectures).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Every CNN you'll fine-tune in the next lesson descends from a handful of\n",
    "landmark architectures, most of them born in one competition. Understanding\n",
    "*why* each one won — not just its name — tells you what actually matters when\n",
    "designing or choosing a network. In this lesson you'll follow the evolution\n",
    "from LeNet to ResNet, do the arithmetic that justifies VGG's tiny kernels, and\n",
    "see why ResNet's skip connection is arguably the single most important trick\n",
    "in deep learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## The ImageNet moment\n",
    "\n",
    "Through the 2000s, computer vision was dominated by hand-crafted features.\n",
    "Then came **ILSVRC** — the ImageNet Large Scale Visual Recognition Challenge —\n",
    "a yearly competition on a dataset of over a million labeled images across\n",
    "1,000 categories. Classifiers were scored on **top-5 error**: the true label\n",
    "had to appear among the model's five best guesses.\n",
    "\n",
    "In 2012 a deep CNN called **AlexNet** cut the previous year's error nearly in\n",
    "half, and the field never looked back. Even better for the rest of us: the\n",
    "winning teams published their architectures *and* released the trained\n",
    "weights. Those pretrained weights are the raw material of transfer learning —\n",
    "which is exactly where this module is headed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## LeNet and AlexNet: the proof of concept\n",
    "\n",
    "**LeNet-5** (1998) established the template — convolutions, pooling, then\n",
    "fully connected layers — reading handwritten digits on checks with about 60K\n",
    "parameters. It worked, but compute and data limits kept CNNs niche for over a\n",
    "decade.\n",
    "\n",
    "**AlexNet** (2012) was essentially a scaled-up LeNet with three crucial\n",
    "upgrades:\n",
    "\n",
    "- **ReLU activations** instead of sigmoid/tanh — gradients stopped vanishing\n",
    "  in deeper stacks, so training became dramatically faster.\n",
    "- **Dropout** in the fully connected layers — 60 million parameters would\n",
    "  have memorized the dataset without it.\n",
    "- **GPU training** — two consumer graphics cards made a week-long training\n",
    "  run feasible at all.\n",
    "\n",
    "None of these ideas was brand new; combining them at ImageNet scale was. The\n",
    "result: 16.4% top-5 error versus roughly 26% for the best non-deep entry."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "## VGG: simplicity via small kernels\n",
    "\n",
    "**VGGNet** (2014, runner-up) asked: what if we use *only* 3×3 convolutions\n",
    "and just stack more of them? The insight is that two stacked 3×3 layers see\n",
    "the same input region as one 5×5 layer — the second layer's 3×3 window looks\n",
    "at outputs that each already summarize a 3×3 patch, so its effective\n",
    "**receptive field** is 5×5. Three stacked 3×3 layers see 7×7.\n",
    "\n",
    "The stacked version wins twice: fewer parameters, and an extra ReLU between\n",
    "the layers (more nonlinearity, more expressive power). Check the arithmetic —\n",
    "this runs in your browser:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0005",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "C = 64  # channels in and out\n",
    "\n",
    "one_5x5   = 5 * 5 * C * C\n",
    "two_3x3   = 2 * (3 * 3 * C * C)\n",
    "one_7x7   = 7 * 7 * C * C\n",
    "three_3x3 = 3 * (3 * 3 * C * C)\n",
    "\n",
    "print(f\"one 5x5 layer   : {one_5x5:>10,} weights\")\n",
    "print(f\"two 3x3 layers  : {two_3x3:>10,} weights  ({1 - two_3x3/one_5x5:.0%} fewer)\")\n",
    "print()\n",
    "print(f\"one 7x7 layer   : {one_7x7:>10,} weights\")\n",
    "print(f\"three 3x3 layers: {three_3x3:>10,} weights  ({1 - three_3x3/one_7x7:.0%} fewer)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "Same receptive field, 28–45% fewer weights, deeper nonlinearity. VGG-16\n",
    "reached 7.3% top-5 error with an architecture so regular you can recite it\n",
    "from memory — which is why it's still a favorite teaching example. Its\n",
    "weakness: those giant fully connected layers at the end push it to 138\n",
    "million parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## GoogLeNet / Inception: go wider, not just deeper\n",
    "\n",
    "**GoogLeNet** (the 2014 winner, named after LeNet) took a different route.\n",
    "Why choose between a 1×1, 3×3, or 5×5 kernel at each stage? The **Inception\n",
    "module** runs several kernel sizes *in parallel* on the same input and\n",
    "concatenates the resulting feature maps — the network learns multi-scale\n",
    "features at every layer.\n",
    "\n",
    "The trick that makes this affordable is the **1×1 convolution**. A 1×1 kernel\n",
    "doesn't look at spatial neighbors at all; it's a learned linear mix *across\n",
    "channels* at each pixel. Placed before an expensive 3×3 or 5×5 branch, it\n",
    "compresses, say, 256 channels down to 64 — slashing the multiplication count\n",
    "— and adds another nonlinearity for free. Thanks to this bottlenecking,\n",
    "22-layer GoogLeNet used only about 6.8 million parameters — 20× fewer than\n",
    "VGG-16 — while winning the competition at 6.7% top-5 error. (Its successor\n",
    "Inception-V3 refined the module further and was runner-up in 2015.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "## ResNet: the skip connection\n",
    "\n",
    "By 2015 everyone knew deeper was better — so why not 50, 100, 150 layers?\n",
    "Because in practice, very deep plain networks got *worse*. Not just on test\n",
    "data (that would be overfitting) but on **training data too**. A 56-layer\n",
    "plain CNN had *higher training error* than a 20-layer one. This is the\n",
    "**degradation problem**: in principle the deep network could copy the shallow\n",
    "one and set its extra layers to the identity function, but gradient-based\n",
    "optimization couldn't find that solution. Gradients had to survive\n",
    "multiplication through dozens of layers, and learning \"do nothing\" turns out\n",
    "to be hard for a stack of convolutions.\n",
    "\n",
    "**ResNet**'s fix is disarmingly simple. Instead of asking a block of layers\n",
    "to learn the full mapping `H(x)`, add the block's input back to its output,\n",
    "so the layers only learn the **residual** `F(x) = H(x) − x`:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0009",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# a residual block, in pseudocode\n",
    "def block(x):\n",
    "    out = conv_bn_relu(x)\n",
    "    out = conv_bn(out)\n",
    "    return relu(out + x)   # the skip connection: add the input back"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "Two consequences, and they're the key ideas of this lesson:\n",
    "\n",
    "1. **Identity is now the default.** If a block has nothing useful to add, it\n",
    "   just drives `F(x)` toward zero — trivially easy — and the input passes\n",
    "   through untouched. Extra depth can no longer hurt training.\n",
    "2. **Gradients get a highway.** In backprop, the derivative of `out + x` with\n",
    "   respect to `x` includes an identity term — a portion of the gradient flows\n",
    "   *around* every block, unattenuated, all the way to the first layer. No\n",
    "   more vanishing through 150 layers of multiplication.\n",
    "\n",
    "ResNet-152 won ILSVRC 2015 with 3.6% top-5 error — beating the ~5% score of\n",
    "a trained human — and later work showed skip connections also make the loss\n",
    "landscape dramatically smoother and easier to optimize. Today virtually every\n",
    "serious architecture, including transformers, is built from residual blocks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "## Scorecard\n",
    "\n",
    "| Model | Year | Depth (weight layers) | Params | ImageNet top-5 error |\n",
    "|---|---|---|---|---|\n",
    "| LeNet-5 | 1998 | 5 | ~60K | — (built for digits) |\n",
    "| AlexNet | 2012 | 8 | 60M | 16.4% |\n",
    "| VGG-16 | 2014 | 16 | 138M | 7.3% |\n",
    "| GoogLeNet | 2014 | 22 | 6.8M | 6.7% |\n",
    "| ResNet-152 | 2015 | 152 | 60M | 3.6% |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "## What came next\n",
    "\n",
    "Two threads are worth knowing. **EfficientNet** (2019) observed that depth,\n",
    "width, and input resolution should be scaled *together* by a fixed compound\n",
    "rule, giving a family of models that trace the best accuracy-per-FLOP curve.\n",
    "And in 2020 the **Vision Transformer (ViT)** dropped convolutions entirely:\n",
    "split the image into 16×16 patches, treat them as tokens, and apply the same\n",
    "transformer architecture that powers language models. With enough pretraining\n",
    "data, ViTs match or beat CNNs — but CNNs (and hybrids like ConvNeXt) remain\n",
    "excellent, especially on smaller datasets, and everything in the next lesson\n",
    "applies to both families.\n",
    "\n",
    "All of these live in `torchvision`, pretrained weights included. In Colab:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0013",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import torchvision.models as models\n",
    "\n",
    "# every architecture torchvision ships (ResNets, VGGs, EfficientNets, ViTs, ...)\n",
    "all_models = models.list_models()\n",
    "print(len(all_models), \"models available\")\n",
    "print([m for m in all_models if \"resnet\" in m])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Weigh the giants\n",
    "\n",
    "10}: {count_params(model)/1e6:6.1f}M parameters\")\n",
    "\n",
    "# Where do VGG's parameters live? Mostly the fully connected head:\n",
    "vgg = models.vgg16(weights=None)\n",
    "head = sum(p.numel() for p in vgg.classifier.parameters())\n",
    "print(f\"vgg16 classifier head alone: {head/1e6:.1f}M of {count_params(vgg)/1e6:.1f}M\")\n",
    "`}\n",
    ">\n",
    "In Colab, instantiate `resnet18`, `resnet50`, `vgg16`, and `googlenet` from\n",
    "`torchvision.models` (architecture only — no pretrained weights needed) and\n",
    "print each model's total parameter count in millions. Then investigate: what\n",
    "fraction of VGG-16's parameters sit in its fully connected classifier head?\n",
    "Does the result explain the table above?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0015",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import torchvision.models as models\n",
    "\n",
    "def count_params(model):\n",
    "    return sum(p.numel() for p in model.parameters())\n",
    "\n",
    "for name, fn in [(\"resnet18\", models.resnet18),\n",
    "                 (\"resnet50\", models.resnet50),\n",
    "                 (\"vgg16\", models.vgg16),\n",
    "                 (\"googlenet\", models.googlenet)]:\n",
    "    model = fn(weights=None)\n",
    "    print(f\"{name:>10}: {count_params(model)/1e6:6.1f}M parameters\")\n",
    "\n",
    "# Where do VGG's parameters live? Mostly the fully connected head:\n",
    "vgg = models.vgg16(weights=None)\n",
    "head = sum(p.numel() for p in vgg.classifier.parameters())\n",
    "print(f\"vgg16 classifier head alone: {head/1e6:.1f}M of {count_params(vgg)/1e6:.1f}M\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "Next: instead of admiring these architectures, we'll steal their pretrained\n",
    "weights and retrain them on our own data — transfer learning in practice."
   ]
  }
 ]
}