{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Autoencoders\n",
    "\n",
    "Train a network to compress and reconstruct its own input — and use the bottleneck for dimensionality reduction, denoising, and anomaly detection.\n",
    "\n",
    "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/autoencoders).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Every model so far learned to map inputs to labels. Autoencoders drop the\n",
    "labels entirely: the network's target is **its own input**. That sounds\n",
    "useless — copy x to x — until you add one constraint that changes everything:\n",
    "the copy must pass through a narrow bottleneck. In this lesson you'll see why\n",
    "that constraint forces the network to discover structure, build a full MNIST\n",
    "autoencoder, and use the same trick for denoising and anomaly detection."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Learning to compress\n",
    "\n",
    "An autoencoder has three parts:\n",
    "\n",
    "- **Encoder** — squeezes the input down, e.g. a 784-pixel MNIST image through\n",
    "  layers of 256 → 64 → 32 numbers.\n",
    "- **Bottleneck (latent space)** — the 32-number summary, also called the\n",
    "  latent code `z`.\n",
    "- **Decoder** — a mirror image that inflates 32 numbers back to 784 pixels.\n",
    "\n",
    "Training minimizes **reconstruction loss** — how far the decoder's output is\n",
    "from the original input, typically mean squared error over pixels. No labels\n",
    "anywhere: the data supervises itself, which is why this is called\n",
    "*self-supervised* (or classically, unsupervised) learning.\n",
    "\n",
    "The bottleneck is the whole point. If the latent space were as wide as the\n",
    "input, the network could learn the identity function and reconstruct\n",
    "perfectly while learning nothing. Forced through 32 numbers, it *cannot*\n",
    "memorize pixels — it has to find the 32 most useful facts about a digit\n",
    "image (roughly: which digit, how slanted, how thick the stroke, ...) and\n",
    "learn to redraw from them. Compression pressure is what turns copying into\n",
    "understanding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## The linear ancestor: PCA\n",
    "\n",
    "You've met this idea before. **PCA is exactly a linear autoencoder**: project\n",
    "onto the top k principal components (encode), then project back\n",
    "(decode) — and among all linear maps, PCA's reconstruction error is optimal.\n",
    "An autoencoder with no activation functions and MSE loss learns the same\n",
    "subspace as PCA; the deep, nonlinear version is what earns its keep on\n",
    "complex data.\n",
    "\n",
    "Watch reconstruction quality degrade as we shrink PCA's \"bottleneck\" on the\n",
    "digits dataset — this runs in your browser:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X = X / 16.0   # 8x8 images, 64 pixels, scaled to 0-1\n",
    "\n",
    "ks = [32, 8, 2]\n",
    "fig, axes = plt.subplots(len(ks) + 1, 6, figsize=(7, 5))\n",
    "for col in range(6):\n",
    "    axes[0, col].imshow(X[col].reshape(8, 8), cmap=\"gray\")\n",
    "    axes[0, col].axis(\"off\")\n",
    "axes[0, 0].set_title(\"original (64 pixels)\", loc=\"left\", fontsize=9)\n",
    "\n",
    "for row, k in enumerate(ks, start=1):\n",
    "    pca = PCA(n_components=k).fit(X)\n",
    "    X_rec = pca.inverse_transform(pca.transform(X[:6]))\n",
    "    var = pca.explained_variance_ratio_.sum()\n",
    "    for col in range(6):\n",
    "        axes[row, col].imshow(X_rec[col].reshape(8, 8), cmap=\"gray\")\n",
    "        axes[row, col].axis(\"off\")\n",
    "    axes[row, 0].set_title(f\"k={k} ({var:.0%} of variance)\", loc=\"left\", fontsize=9)\n",
    "    print(f\"bottleneck k={k:2d}: keeps {var:.1%} of the variance\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "With 32 components the digits are nearly perfect; at 8 they're blurry but\n",
    "recognizable; at 2 almost everything is lost. A deep autoencoder plays the\n",
    "same game, but its nonlinear encoder/decoder can pack far more structure into\n",
    "the same number of latent dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "## A full MNIST autoencoder in PyTorch\n",
    "\n",
    "Now the real thing. PyTorch doesn't run in the browser, so these cells belong\n",
    "in the downloadable notebook — Colab with a GPU trains this in a couple of\n",
    "minutes. Encoder and decoder are plain `nn.Sequential` stacks, mirror images\n",
    "of each other; the final `Sigmoid` keeps outputs in the 0–1 pixel range:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0007",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "train_set = datasets.MNIST(\"data\", train=True, download=True,\n",
    "                           transform=transforms.ToTensor())\n",
    "test_set = datasets.MNIST(\"data\", train=False, download=True,\n",
    "                          transform=transforms.ToTensor())\n",
    "train_loader = DataLoader(train_set, batch_size=128, shuffle=True)\n",
    "test_loader = DataLoader(test_set, batch_size=128)\n",
    "\n",
    "class Autoencoder(nn.Module):\n",
    "    def __init__(self, latent_dim=32):\n",
    "        super().__init__()\n",
    "        self.encoder = nn.Sequential(\n",
    "            nn.Flatten(),                      # (batch, 1, 28, 28) -> (batch, 784)\n",
    "            nn.Linear(784, 256), nn.ReLU(),\n",
    "            nn.Linear(256, 64), nn.ReLU(),\n",
    "            nn.Linear(64, latent_dim),         # the bottleneck: (batch, 32)\n",
    "        )\n",
    "        self.decoder = nn.Sequential(\n",
    "            nn.Linear(latent_dim, 64), nn.ReLU(),\n",
    "            nn.Linear(64, 256), nn.ReLU(),\n",
    "            nn.Linear(256, 784), nn.Sigmoid(), # back to 0-1 pixels\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        z = self.encoder(x)\n",
    "        recon = self.decoder(z).view(-1, 1, 28, 28)\n",
    "        return recon, z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "The training loop is the standard one with a twist: the loss compares the\n",
    "reconstruction to the *input*, and the labels are thrown away:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0009",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "model = Autoencoder().to(device)\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n",
    "\n",
    "for epoch in range(10):\n",
    "    model.train()\n",
    "    total = 0.0\n",
    "    for images, _ in train_loader:        # labels ignored!\n",
    "        images = images.to(device)\n",
    "        recon, z = model(images)\n",
    "        loss = criterion(recon, images)   # reconstruction vs original\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total += loss.item() * len(images)\n",
    "    print(f\"epoch {epoch + 1}: reconstruction MSE {total / len(train_loader.dataset):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "And the payoff — originals on top, reconstructions from 32 numbers below:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "model.eval()\n",
    "images, _ = next(iter(test_loader))\n",
    "with torch.no_grad():\n",
    "    recon, z = model(images.to(device))\n",
    "\n",
    "fig, ax = plt.subplots(2, 8, figsize=(14, 4))\n",
    "for i in range(8):\n",
    "    ax[0, i].imshow(images[i].squeeze(), cmap=\"gray\");        ax[0, i].axis(\"off\")\n",
    "    ax[1, i].imshow(recon[i].squeeze().cpu(), cmap=\"gray\");   ax[1, i].axis(\"off\")\n",
    "ax[0, 0].set_title(\"original\", loc=\"left\")\n",
    "ax[1, 0].set_title(\"reconstructed from 32 numbers\", loc=\"left\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "A 784-pixel image squeezed through 32 numbers and redrawn — a 24× compression\n",
    "learned purely from the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Application 1: denoising\n",
    "\n",
    "Here's a beautiful variation: corrupt the input, but keep the **clean** image\n",
    "as the target. The network can no longer succeed by copying — it must learn\n",
    "what digits *should* look like in order to repair them. This is the\n",
    "**denoising autoencoder**, and it takes a two-line change:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0014",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def add_noise(images, noise_factor=0.5):\n",
    "    noisy = images + noise_factor * torch.rand_like(images)\n",
    "    return noisy.clamp(0.0, 1.0)\n",
    "\n",
    "# inside the training loop, replace the forward pass with:\n",
    "#     recon, z = model(add_noise(images))\n",
    "#     loss = criterion(recon, images)     # target is still the CLEAN image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "Retrain, then feed it noisy test images: the reconstructions come out clean.\n",
    "The same recipe — corrupted input, clean target — powers document cleanup\n",
    "(removing coffee stains and shadows from scanned pages), audio denoising, and\n",
    "image inpainting where whole patches are masked out and repainted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "## Application 2: anomaly detection\n",
    "\n",
    "An autoencoder only learns to reconstruct **the kind of data it was trained\n",
    "on**. Show it something from a different distribution and the reconstruction\n",
    "fails — the error jumps. That gives a simple anomaly detector: score every\n",
    "sample by reconstruction error and flag the outliers.\n",
    "\n",
    "Since a linear autoencoder is PCA, we can demonstrate the whole idea in the\n",
    "browser: fit PCA on real digits, inject one fake \"image\" of pure random\n",
    "noise, and watch its reconstruction error stand out:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0017",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X = X / 16.0\n",
    "\n",
    "rng = np.random.default_rng(0)\n",
    "outlier = rng.random(64)              # random noise pretending to be a digit\n",
    "X_all = np.vstack([X, outlier])       # 1797 digits + 1 impostor\n",
    "\n",
    "pca = PCA(n_components=16).fit(X)     # 'autoencoder' trained on digits only\n",
    "recon = pca.inverse_transform(pca.transform(X_all))\n",
    "errors = np.mean((X_all - recon) ** 2, axis=1)\n",
    "\n",
    "print(f\"digits:  mean error {errors[:-1].mean():.4f}, max {errors[:-1].max():.4f}\")\n",
    "print(f\"impostor error:     {errors[-1]:.4f}\")\n",
    "rank = np.argsort(errors)[::-1].tolist().index(len(X_all) - 1) + 1\n",
    "print(f\"impostor ranks #{rank} of {len(X_all)} by reconstruction error\")\n",
    "\n",
    "plt.hist(errors[:-1], bins=50, label=\"real digits\")\n",
    "plt.axvline(errors[-1], color=\"red\", ls=\"--\", label=\"injected outlier\")\n",
    "plt.xlabel(\"reconstruction error\"); plt.ylabel(\"count\")\n",
    "plt.legend(); plt.title(\"Anomalies do not reconstruct well\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "The impostor's error is several times larger than any real digit's — it ranks\n",
    "first out of 1,798. In production this pattern detects fraudulent\n",
    "transactions, failing machines, and network intrusions: train on normal data\n",
    "only, then alert on whatever the model can't redraw."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0019",
   "metadata": {},
   "source": [
    "## The latent space, and a teaser\n",
    "\n",
    "The bottleneck isn't just small — it's *organized*. Encode all of MNIST and\n",
    "look at the 32-dimensional codes: images of the same digit cluster together,\n",
    "similar handwriting styles sit near each other, and walking in a straight\n",
    "line between the code for a 3 and the code for an 8 decodes into images that\n",
    "morph smoothly from one into the other. The encoder has arranged concepts\n",
    "geometrically, without ever seeing a label."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "> **Toward true generation: the VAE**\n",
    "> \n",
    "> Can you *sample* a random latent vector and decode a brand-new digit? With a\n",
    "> plain autoencoder, usually not — the latent space has gaps, and random points\n",
    "> often decode to mush. The **variational autoencoder (VAE)** fixes this by\n",
    "> forcing the latent codes toward a known distribution (a standard Gaussian),\n",
    "> so that every point you might sample decodes to something sensible. That one\n",
    "> change turns a compressor into a true generative model — and sets the stage\n",
    "> for the generation arms race in the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0021",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — A one-class digit detector\n",
    "\n",
    "Build a one-class detector in the anomaly PyRunner above: fit PCA with 16\n",
    "components on **only the images of digit 0**, then compute reconstruction\n",
    "errors for *all* digits. Compare the average error on zeros versus non-zeros\n",
    "and plot both histograms. Could you pick a threshold that flags most non-zero\n",
    "digits as anomalies?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0022",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0023",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X = X / 16.0\n",
    "\n",
    "pca = PCA(n_components=16).fit(X[y == 0])   # train on zeros ONLY\n",
    "\n",
    "recon = pca.inverse_transform(pca.transform(X))\n",
    "errors = np.mean((X - recon) ** 2, axis=1)\n",
    "\n",
    "print(f\"error on zeros:      {errors[y == 0].mean():.4f}\")\n",
    "print(f\"error on non-zeros:  {errors[y != 0].mean():.4f}\")\n",
    "\n",
    "plt.hist(errors[y == 0], bins=40, alpha=0.7, label=\"zeros (normal)\")\n",
    "plt.hist(errors[y != 0], bins=40, alpha=0.7, label=\"other digits (anomalies)\")\n",
    "plt.xlabel(\"reconstruction error\"); plt.ylabel(\"count\"); plt.legend()\n",
    "plt.show()\n",
    "\n",
    "# The zero-trained PCA redraws zeros with low error, but other digits\n",
    "# do not fit its compression scheme - their errors are clearly higher,\n",
    "# so a simple threshold separates 'normal' from 'anomaly'.\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0024",
   "metadata": {},
   "source": [
    "Next up: the final lesson — two networks locked in a forgery contest, better\n",
    "known as generative adversarial networks."
   ]
  }
 ]
}