{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# The Bias–Variance Tradeoff\n",
    "\n",
    "Why models fail in two opposite ways — and how to diagnose which one is happening to you, with validation curves, learning curves, and a cheat sheet.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/bias-variance-tradeoff).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "In the last lesson you watched test error trace a U as model complexity grew.\n",
    "The **bias–variance tradeoff** is the theory behind that U — and more\n",
    "usefully, it's a *diagnostic framework*: models fail in two opposite ways,\n",
    "each with its own symptoms and its own cures. Prescribing the wrong cure\n",
    "(adding data to a high-bias model, adding complexity to a high-variance one)\n",
    "wastes weeks. This lesson teaches you to tell the two apart."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Three ingredients of prediction error\n",
    "\n",
    "The expected error of a model on new data decomposes, conceptually, into\n",
    "three parts:\n",
    "\n",
    "**expected error = bias² + variance + irreducible noise**\n",
    "\n",
    "- **Bias** — error from wrong *assumptions*. A straight line fitted to a\n",
    "  curve is biased: no matter how much data you give it, it systematically\n",
    "  misses the shape. High bias = **underfitting**.\n",
    "- **Variance** — error from *sensitivity to the training sample*. A flexible\n",
    "  model fitted to 40 noisy points would look completely different if you drew\n",
    "  40 different points. High variance = **overfitting**.\n",
    "- **Irreducible noise** — randomness in the data itself (measurement error,\n",
    "  unmodeled factors). No model can remove it; it's the floor under your error.\n",
    "\n",
    "The tradeoff: making a model more flexible reduces bias but raises variance,\n",
    "and vice versa. The best model balances the two — the bottom of the U."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## Seeing variance with your own eyes\n",
    "\n",
    "\"Variance\" sounds abstract until you watch it. Below we draw several\n",
    "**bootstrap samples** (resamples of the same dataset) and fit the same model\n",
    "to each. A degree-2 fit barely notices which sample it got; a degree-9 fit\n",
    "changes shape completely every time:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "rng = np.random.default_rng(3)\n",
    "x = np.sort(rng.uniform(-3, 3, 40))\n",
    "y = np.sin(1.3 * x) + rng.normal(0, 0.35, 40)\n",
    "grid = np.linspace(-3, 3, 200).reshape(-1, 1)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(9, 3.8), sharey=True)\n",
    "for ax, degree in [(axes[0], 2), (axes[1], 9)]:\n",
    "    for _ in range(6):                        # 6 bootstrap resamples\n",
    "        idx = rng.integers(0, len(x), len(x))\n",
    "        model = Pipeline([\n",
    "            (\"poly\", PolynomialFeatures(degree, include_bias=False)),\n",
    "            (\"lr\", LinearRegression()),\n",
    "        ]).fit(x[idx].reshape(-1, 1), y[idx])\n",
    "        ax.plot(grid, model.predict(grid), alpha=0.6, lw=1.5)\n",
    "    ax.scatter(x, y, s=15, color=\"black\", zorder=5)\n",
    "    ax.set_ylim(-2.5, 2.5)\n",
    "    ax.set_title(f\"degree {degree}: 6 fits on 6 resamples\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "Left: six nearly identical curves — **low variance** (but they all miss the\n",
    "sine wiggle the same way: that consistent miss is **bias**). Right: six wildly\n",
    "different curves — each one chased the noise of its particular sample. That\n",
    "disagreement *is* variance, and it's why the degree-9 model's test error is\n",
    "so bad: on average, a randomly-drawn wiggly curve is far from the truth."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "## Symptoms and cures\n",
    "\n",
    "**High bias (underfitting)** looks like:\n",
    "\n",
    "- Training error is high — the model can't even fit the data it has seen.\n",
    "- Test error is about equally high; the train/test gap is small.\n",
    "- More data doesn't help — the curves just confirm the same wrong shape.\n",
    "\n",
    "Fixes: **add complexity** — more/better features (polynomial terms,\n",
    "interactions, domain-driven features like the taxi `distance` you saw), a\n",
    "more flexible model family, or *less* regularization.\n",
    "\n",
    "**High variance (overfitting)** looks like:\n",
    "\n",
    "- Training error is very low, sometimes near zero.\n",
    "- Test error is much higher — a big train/test gap.\n",
    "- Results change a lot between random seeds or resamples.\n",
    "\n",
    "Fixes: **constrain or stabilize** — more training data, a simpler model\n",
    "(lower degree), regularization (next lessons), or averaging many models\n",
    "(ensembles, later in the course)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "> **This logic drives hyperparameter tuning**\n",
    "> \n",
    "> Almost every hyperparameter is a complexity dial you can reason about. KNN:\n",
    "> increasing `n_neighbors` averages over more points → less complexity → fights\n",
    "> overfitting. Random forests: increasing `max_depth` adds decisions → more\n",
    "> complexity → risks overfitting, while increasing `n_estimators` averages more\n",
    "> trees → less variance. When a model overfits, ask: which dial turns\n",
    "> complexity *down*?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "## Validation curves: error vs complexity\n",
    "\n",
    "scikit-learn automates the \"U-curve\" experiment with `validation_curve`: it\n",
    "sweeps one hyperparameter and cross-validates at each value. Here we sweep\n",
    "KNN's `n_neighbors` (note: for KNN, *small* `n_neighbors` = high complexity,\n",
    "so the x-axis runs complex → simple):"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0009",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import validation_curve\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "rng = np.random.default_rng(0)\n",
    "X = rng.uniform(-3, 3, (120, 1))\n",
    "y = np.sin(1.3 * X.ravel()) + rng.normal(0, 0.3, 120)\n",
    "\n",
    "ks = np.array([1, 2, 3, 5, 8, 12, 20, 40, 80])\n",
    "train_sc, val_sc = validation_curve(\n",
    "    KNeighborsRegressor(), X, y,\n",
    "    param_name=\"n_neighbors\", param_range=ks, cv=4)\n",
    "\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(ks, train_sc.mean(axis=1), \"o-\", label=\"train R2\")\n",
    "plt.plot(ks, val_sc.mean(axis=1), \"s-\", label=\"validation R2\")\n",
    "plt.xscale(\"log\")\n",
    "plt.xlabel(\"n_neighbors  (left = complex, right = simple)\")\n",
    "plt.ylabel(\"R2\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "Read it like a doctor: on the left (k = 1), train R² is perfect while\n",
    "validation lags — **variance zone**. On the far right (k = 80, nearly\n",
    "averaging everything), both scores collapse together — **bias zone**. The\n",
    "sweet spot is where validation peaks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "## Learning curves: will more data help?\n",
    "\n",
    "The second diagnostic sweeps *training set size* instead. Its shape answers\n",
    "the most expensive question in ML — \"should we collect more data?\":"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0012",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import learning_curve\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "rng = np.random.default_rng(1)\n",
    "X = rng.uniform(-3, 3, (150, 1))\n",
    "y = np.sin(1.3 * X.ravel()) + rng.normal(0, 0.3, 150)\n",
    "\n",
    "model = Pipeline([\n",
    "    (\"poly\", PolynomialFeatures(6, include_bias=False)),\n",
    "    (\"lr\", LinearRegression()),\n",
    "])\n",
    "sizes, train_sc, val_sc = learning_curve(\n",
    "    model, X, y, train_sizes=np.linspace(0.15, 1.0, 6), cv=4)\n",
    "\n",
    "plt.figure(figsize=(7, 4))\n",
    "plt.plot(sizes, train_sc.mean(axis=1), \"o-\", label=\"train R2\")\n",
    "plt.plot(sizes, val_sc.mean(axis=1), \"s-\", label=\"validation R2\")\n",
    "plt.xlabel(\"training set size\")\n",
    "plt.ylabel(\"R2\")\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "With little data the flexible model aces training but flops on validation\n",
    "(variance). As data grows, the two curves **converge** — more data is\n",
    "shrinking the variance. Two endgames to recognize:\n",
    "\n",
    "- Curves still converging with a gap → **more data will help**.\n",
    "- Curves already converged at a mediocre score → the model has hit its bias\n",
    "  floor; **more data won't help — you need a better model or features**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "## Diagnosis cheat sheet\n",
    "\n",
    "| Observation | Diagnosis | Try |\n",
    "|---|---|---|\n",
    "| Train error high, test error similarly high | High bias (underfit) | More features, higher degree, more flexible model, less regularization |\n",
    "| Train error tiny, test error much worse | High variance (overfit) | More data, simpler model, regularization, ensembling |\n",
    "| Learning curves converged, both mediocre | Bias floor reached | Better features / model family — more data is wasted money |\n",
    "| Learning curves still converging with a gap | Variance, curable | Collect more data |\n",
    "| Fits change wildly across random seeds | High variance | Same as overfitting fixes |\n",
    "| Test error can't go below some level no matter what | Irreducible noise | Accept it, or measure better data |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Diagnose a tree with a validation curve\n",
    "\n",
    "bias zone.\n",
    "# Deep trees: train R2 -> 1 while validation falls -> variance zone.\n",
    "# Increasing max_depth increases complexity, so it can cause overfitting.\n",
    "`}\n",
    ">\n",
    "Decision trees (coming later in the course) have a complexity dial called\n",
    "`max_depth`. Using the same synthetic sine data as above, run\n",
    "`validation_curve` on a `DecisionTreeRegressor` with `max_depth` from 1 to 12\n",
    "and plot train vs validation R². Identify the bias zone, the variance zone,\n",
    "and the best depth. Then answer in a comment: does *increasing* `max_depth`\n",
    "fight overfitting or cause it?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import validation_curve\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "rng = np.random.default_rng(5)\n",
    "X = rng.uniform(-3, 3, (150, 1))\n",
    "y = np.sin(1.3 * X.ravel()) + rng.normal(0, 0.3, 150)\n",
    "\n",
    "depths = np.arange(1, 13)\n",
    "train_sc, val_sc = validation_curve(\n",
    "    DecisionTreeRegressor(random_state=0), X, y,\n",
    "    param_name=\"max_depth\", param_range=depths, cv=4)\n",
    "\n",
    "plt.plot(depths, train_sc.mean(axis=1), \"o-\", label=\"train R2\")\n",
    "plt.plot(depths, val_sc.mean(axis=1), \"s-\", label=\"validation R2\")\n",
    "plt.xlabel(\"max_depth (complexity)\")\n",
    "plt.ylabel(\"R2\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "best = depths[np.argmax(val_sc.mean(axis=1))]\n",
    "print(f\"Best max_depth by validation: {best}\")\n",
    "# Depth 1-2: both scores low -> bias zone.\n",
    "# Deep trees: train R2 -> 1 while validation falls -> variance zone.\n",
    "# Increasing max_depth increases complexity, so it can cause overfitting.\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "Next: a quiet culprit behind unstable models — features on wildly different\n",
    "scales, and the transforms that fix them."
   ]
  }
 ]
}