{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Linear Regression & Gradient Descent\n",
    "\n",
    "Fit your first model by hand, understand loss functions, and watch gradient descent find the best weights automatically.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/linear-regression-gradient-descent).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Linear regression is the \"hello world\" of machine learning — but almost every\n",
    "idea you'll use later (weights, bias, loss, gradient descent) shows up here\n",
    "first, in its simplest form. In this lesson you'll fit a line by hand, define\n",
    "what \"best fit\" means mathematically, and then let gradient descent do the\n",
    "work for you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## The model: a line with two knobs\n",
    "\n",
    "A linear model predicts a number as a weighted sum of the input plus an offset:\n",
    "\n",
    "**ŷ = w·x + b**\n",
    "\n",
    "- **w (weight / slope)** — how much the prediction changes when x increases by 1\n",
    "- **b (bias / intercept)** — the prediction when x = 0\n",
    "\n",
    "Training a model just means finding good values for these two knobs. Try it\n",
    "yourself — move the sliders until the line goes through the middle of the\n",
    "cloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/machine-learning/linear-regression-gradient-descent)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "You probably ended up somewhere near **w ≈ 1.6, b ≈ 1.2** — and you likely\n",
    "used the dashed *residual* lines to guide you. That intuition — \"make the\n",
    "vertical gaps small\" — is exactly what the loss function formalizes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "## The loss: measuring \"how wrong\"\n",
    "\n",
    "For each point, the **residual** is the gap between truth and prediction,\n",
    "`y − ŷ`. The **mean squared error (MSE)** averages the squared residuals:\n",
    "\n",
    "**MSE = (1/n) Σ (yᵢ − ŷᵢ)²**\n",
    "\n",
    "Squaring does two useful things: errors can't cancel each other out, and big\n",
    "misses are punished much more than small ones. Let's compute it ourselves —\n",
    "run this right here in your browser:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0006",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "x = rng.uniform(0.5, 9.5, 28)\n",
    "y = 1.6 * x + 1.2 + rng.normal(0, 1.4, 28)\n",
    "\n",
    "def mse(w, b):\n",
    "    y_pred = w * x + b\n",
    "    return np.mean((y - y_pred) ** 2)\n",
    "\n",
    "print(f\"w=0.4, b=3.5  ->  MSE = {mse(0.4, 3.5):.3f}   (the playground's start)\")\n",
    "print(f\"w=1.6, b=1.2  ->  MSE = {mse(1.6, 1.2):.3f}   (near the truth)\")\n",
    "print(f\"w=2.5, b=-3   ->  MSE = {mse(2.5, -3.0):.3f}   (way off)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "Lower loss = better fit. Training is now an **optimization problem**: find the\n",
    "`(w, b)` pair that minimizes MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "## Gradient descent: rolling downhill\n",
    "\n",
    "Imagine the loss as a landscape where every location is a `(w, b)` choice and\n",
    "the altitude is the MSE. Gradient descent starts anywhere and repeatedly takes\n",
    "a small step **downhill**:\n",
    "\n",
    "**w ← w − η · ∂L/∂w**  and  **b ← b − η · ∂L/∂b**\n",
    "\n",
    "where **η (the learning rate)** controls the step size. Watch how the choice\n",
    "of η changes everything — try a tiny value, then crank it up past 0.7:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/machine-learning/linear-regression-gradient-descent)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "Three regimes to remember:\n",
    "\n",
    "- **η too small** — converges, but painfully slowly\n",
    "- **η just right** — smooth, fast descent to the minimum\n",
    "- **η too large** — overshoots the valley and bounces or diverges"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "> **Where do the gradients come from?**\n",
    "> \n",
    "> For MSE they have closed forms: ∂L/∂w = (2/n) Σ (ŷᵢ − yᵢ)·xᵢ and\n",
    "> ∂L/∂b = (2/n) Σ (ŷᵢ − yᵢ). Deep learning frameworks like PyTorch compute\n",
    "> gradients automatically for any model — that's what \"autograd\" means."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "## Implementing gradient descent from scratch\n",
    "\n",
    "Fifteen lines of NumPy are enough. This is the same algorithm the playground's\n",
    "\"Auto-fit\" button runs:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0013",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "x = rng.uniform(0.5, 9.5, 28)\n",
    "y = 1.6 * x + 1.2 + rng.normal(0, 1.4, 28)\n",
    "\n",
    "w, b, lr = 0.0, 0.0, 0.01\n",
    "\n",
    "for step in range(1, 401):\n",
    "    y_pred = w * x + b\n",
    "    grad_w = 2 * np.mean((y_pred - y) * x)\n",
    "    grad_b = 2 * np.mean(y_pred - y)\n",
    "    w -= lr * grad_w\n",
    "    b -= lr * grad_b\n",
    "    if step % 100 == 0:\n",
    "        print(f\"step {step:3d}:  w={w:.3f}  b={b:.3f}  MSE={np.mean((y - y_pred)**2):.3f}\")\n",
    "\n",
    "print(f\"\\\\nLearned:  y = {w:.2f}x + {b:.2f}   (true: y = 1.60x + 1.20)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "Notice `b` converges more slowly than `w` — the bias gradient doesn't get the\n",
    "\"leverage\" of being multiplied by x. Feature scaling (a later lesson) fixes\n",
    "exactly this kind of imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "## The scikit-learn way\n",
    "\n",
    "In practice you won't hand-roll gradient descent for linear regression —\n",
    "scikit-learn solves it directly:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "x = rng.uniform(0.5, 9.5, 28).reshape(-1, 1)   # sklearn expects 2-D features\n",
    "y = 1.6 * x.ravel() + 1.2 + rng.normal(0, 1.4, 28)\n",
    "\n",
    "model = LinearRegression().fit(x, y)\n",
    "y_pred = model.predict(x)\n",
    "\n",
    "print(f\"w = {model.coef_[0]:.3f}, b = {model.intercept_:.3f}\")\n",
    "print(f\"MSE = {mean_squared_error(y, y_pred):.3f}\")\n",
    "print(f\"R²  = {r2_score(y, y_pred):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "Same answer, one line of fitting. `LinearRegression` uses a closed-form\n",
    "solution (ordinary least squares); gradient descent matters when models get\n",
    "too big for closed forms — which is every neural network you'll ever train."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Gradient descent on a different dataset\n",
    "\n",
    "Generate a dataset with a **negative** slope — `y = -2x + 8` plus Gaussian\n",
    "noise (σ = 0.8) for 40 points with x between 0 and 5. Fit it two ways: with\n",
    "your own gradient-descent loop, and with scikit-learn's `LinearRegression`.\n",
    "Do the two answers agree to two decimal places?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0019",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "rng = np.random.default_rng(0)\n",
    "x = rng.uniform(0, 5, 40)\n",
    "y = -2.0 * x + 8.0 + rng.normal(0, 0.8, 40)\n",
    "\n",
    "w, b, lr = 0.0, 0.0, 0.02\n",
    "for _ in range(2000):\n",
    "    y_pred = w * x + b\n",
    "    w -= lr * 2 * np.mean((y_pred - y) * x)\n",
    "    b -= lr * 2 * np.mean(y_pred - y)\n",
    "\n",
    "print(f\"scratch : w={w:.3f}, b={b:.3f}\")\n",
    "\n",
    "model = LinearRegression().fit(x.reshape(-1, 1), y)\n",
    "print(f\"sklearn : w={model.coef_[0]:.3f}, b={model.intercept_:.3f}\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0021",
   "metadata": {},
   "source": [
    "Next up: what happens when a straight line isn't enough — polynomial\n",
    "regression, and the overfitting trap that comes with it."
   ]
  }
 ]
}