{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# What Is Deep Learning?\n",
    "\n",
    "Where deep learning sits inside AI and ML, what a neuron actually computes, and why nonlinear activations give networks their power.\n",
    "\n",
    "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/what-is-deep-learning).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Deep learning powers image recognition, speech assistants, and language models —\n",
    "but under the hood it's built from a shockingly simple unit: a weighted sum, a\n",
    "bias, and a squashing function. In this lesson you'll place deep learning inside\n",
    "the AI landscape, build a single neuron in NumPy, and train a real (tiny) neural\n",
    "network live in your browser to see why depth and nonlinearity matter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## AI → ML → Deep Learning\n",
    "\n",
    "The three terms nest inside each other, and each level exists because the one\n",
    "above it hit a wall:\n",
    "\n",
    "- **Artificial intelligence** — making machines behave intelligently. The classic\n",
    "  approach was hand-written rules, but many problems (is this photo a cat?) are\n",
    "  far too complex to program explicitly.\n",
    "- **Machine learning** — instead of writing the rules, let the machine find\n",
    "  patterns from examples. This works, but classic ML lives or dies on **feature\n",
    "  engineering**: a human must decide which input combinations matter (ratios,\n",
    "  interactions, edge detectors...), and the space of possible combinations is huge.\n",
    "- **Deep learning** — let the machine learn the feature engineering too. A deep\n",
    "  network's hidden layers automatically build useful intermediate features from\n",
    "  raw inputs, stacking simple combinations into increasingly abstract ones.\n",
    "\n",
    "That last point is the whole sales pitch: **deep learning trades hand-crafted\n",
    "features for learned ones**. You pay for it with more data and more compute."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## A neuron: weighted sum + bias + activation\n",
    "\n",
    "A single artificial neuron does three things: multiply each input by a weight,\n",
    "add a bias, then pass the result through an **activation function**:\n",
    "\n",
    "**a = f(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)**\n",
    "\n",
    "Look familiar? With the identity function as `f`, this is exactly **linear\n",
    "regression**. With a sigmoid as `f`, it's **logistic regression**. A neural\n",
    "network is just many of these units wired together — you already know the atoms.\n",
    "\n",
    "Let's build one in NumPy, right here:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def neuron(x, w, b, activation):\n",
    "    z = np.dot(w, x) + b        # weighted sum + bias\n",
    "    return activation(z)        # squash\n",
    "\n",
    "x = np.array([2.0, -1.0, 0.5])   # three input features\n",
    "w = np.array([0.4, 0.8, -1.2])   # one weight per input\n",
    "b = 0.1\n",
    "\n",
    "print(\"pre-activation z :\", np.dot(w, x) + b)\n",
    "print(\"identity  (regression-style):\", neuron(x, w, b, lambda z: z))\n",
    "print(\"sigmoid   (probability-style):\", round(neuron(x, w, b, sigmoid), 4))\n",
    "print(\"ReLU      (hidden-layer favourite):\", neuron(x, w, b, lambda z: max(0, z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "The activation function is the neuron's personality. Explore the common ones\n",
    "and their derivatives — the derivative matters because training flows gradients\n",
    "backwards through these functions:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/deep-learning/what-is-deep-learning)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "Notice that sigmoid and tanh flatten out at the extremes (their derivative goes\n",
    "to ~0 — a problem called *vanishing gradients*), while ReLU keeps a constant\n",
    "slope of 1 for positive inputs. That's a big part of why **ReLU is the default\n",
    "hidden-layer activation** in modern networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "## Why activations must be nonlinear\n",
    "\n",
    "Here's the key theoretical fact of this lesson. Suppose you stack two linear\n",
    "layers with no activation in between:\n",
    "\n",
    "**h = W₁x + b₁**, then **y = W₂h + b₂ = W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂)**\n",
    "\n",
    "The result is... still a linear function of x. **Stacking linear layers\n",
    "collapses into a single linear layer** — a hundred of them have exactly the\n",
    "same expressive power as one. Depth buys you nothing without nonlinearity.\n",
    "\n",
    "A nonlinear activation between layers breaks the collapse: each hidden unit can\n",
    "now carve out its own bent piece of the input space, and the next layer\n",
    "combines those pieces into shapes no straight line could make."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "## See it live: train a network in your browser\n",
    "\n",
    "Time to test the theory. The playground below trains a small MLP on 2-D toy\n",
    "data in real time. Try this experiment on the **circles** dataset:\n",
    "\n",
    "1. Set hidden units to **0** (no hidden layer — a linear model). Watch it fail:\n",
    "   the best a line can do on a ring-inside-a-ring is roughly 50% territory.\n",
    "2. Now give it a hidden layer with **4–6 units** and ReLU. Each unit learns one\n",
    "   \"tilted line\" feature; combined, they bend into a closed boundary around the\n",
    "   inner cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/deep-learning/what-is-deep-learning)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "Try **moons** too — two or three hidden units are usually enough, because the\n",
    "boundary only needs one gentle bend. The harder the shape, the more units (width)\n",
    "or layers (depth) you need. This is the \"power of combination\" in action:\n",
    "straight lines → triangles → curves → arbitrary blobs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "## Vocabulary you'll use constantly\n",
    "\n",
    "- **Input layer** — one node per feature. Not counted as a \"real\" layer (it does no computation).\n",
    "- **Hidden layers** — the layers between input and output. Their units are the learned features.\n",
    "- **Output layer** — one node per regression target, or per class in classification.\n",
    "- **Width** — how many units in a layer. **Depth** — how many layers. \"Deep\" learning literally means \"more than one hidden layer\".\n",
    "- **Weights and biases** — the learnable parameters. A linear regression has a handful; modern networks have millions to billions, which is why they need so much more data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "> **Universal approximation, in one paragraph**\n",
    "> \n",
    "> A classic theorem says a network with a single, sufficiently wide hidden layer\n",
    "> and a nonlinear activation can approximate any continuous function to arbitrary\n",
    "> precision. So why go deep instead of just wide? Because depth is exponentially\n",
    "> more efficient: features composed of features reuse structure, so a deep network\n",
    "> often needs far fewer total units than an equally capable shallow one — and it\n",
    "> learns hierarchies (edges → shapes → objects) that match how real data is built."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "## When deep learning wins — and when it doesn't\n",
    "\n",
    "Deep learning dominates on **unstructured data**: images, audio, video, and\n",
    "text. These have raw, high-dimensional inputs where hand-crafting features is\n",
    "hopeless, and learned feature hierarchies shine.\n",
    "\n",
    "But on **small tabular datasets** — a few thousand rows of customer records —\n",
    "gradient-boosted trees and even logistic regression routinely beat neural\n",
    "networks, train in seconds, and are easier to interpret. More parameters means\n",
    "more data hunger: a network that can learn anything will happily memorize noise\n",
    "when examples are scarce.\n",
    "\n",
    "Rule of thumb: reach for deep learning when the data is unstructured or truly\n",
    "huge; reach for classic ML first when it's a modest spreadsheet."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — A two-layer network by hand\n",
    "\n",
    "3 hidden units\n",
    "b1 = rng.normal(0, 1, 3)\n",
    "W2 = rng.normal(0, 1, (1, 3))   # 3 hidden -> 1 output\n",
    "b2 = rng.normal(0, 1, 1)\n",
    "\n",
    "# With ReLU\n",
    "a1 = np.maximum(0, W1 @ x + b1)\n",
    "out = sigmoid(W2 @ a1 + b2)\n",
    "print(\"with ReLU   :\", out)\n",
    "\n",
    "# Without activation -> collapses to one linear layer\n",
    "z = W2 @ (W1 @ x + b1) + b2\n",
    "collapsed = (W2 @ W1) @ x + (W2 @ b1 + b2)\n",
    "print(\"no activation, stacked :\", sigmoid(z))\n",
    "print(\"single collapsed layer :\", sigmoid(collapsed))   # identical!\n",
    "`}\n",
    ">\n",
    "Extend the single-neuron code into a **two-layer network** in NumPy: 2 inputs →\n",
    "3 hidden units with ReLU → 1 output with sigmoid. Use `np.random.default_rng(42)`\n",
    "to create the weight matrices (shapes `3×2` and `1×3`) and biases. Then prove\n",
    "the linear-collapse claim: remove the ReLU and show that the stacked computation\n",
    "gives exactly the same output as a single pre-multiplied linear layer."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "x = np.array([1.5, -0.5])\n",
    "\n",
    "W1 = rng.normal(0, 1, (3, 2))   # 2 inputs -> 3 hidden units\n",
    "b1 = rng.normal(0, 1, 3)\n",
    "W2 = rng.normal(0, 1, (1, 3))   # 3 hidden -> 1 output\n",
    "b2 = rng.normal(0, 1, 1)\n",
    "\n",
    "# With ReLU\n",
    "a1 = np.maximum(0, W1 @ x + b1)\n",
    "out = sigmoid(W2 @ a1 + b2)\n",
    "print(\"with ReLU   :\", out)\n",
    "\n",
    "# Without activation -> collapses to one linear layer\n",
    "z = W2 @ (W1 @ x + b1) + b2\n",
    "collapsed = (W2 @ W1) @ x + (W2 @ b1 + b2)\n",
    "print(\"no activation, stacked :\", sigmoid(z))\n",
    "print(\"single collapsed layer :\", sigmoid(collapsed))   # identical!\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "Next up: the tool we'll use to build real networks — PyTorch, its tensors, and\n",
    "the autograd engine that computes gradients for free."
   ]
  }
 ]
}