{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Regression Metrics\n",
    "\n",
    "Learn how to measure how good a numeric prediction really is — MAE, MSE, RMSE, R², MAPE, and the residual plots that reveal what the numbers hide.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/regression-metrics).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Before you fit a single regression model, you need an answer to a deceptively\n",
    "simple question: **what does \"good\" mean when you're predicting a number?** A\n",
    "classifier is right or wrong; a regression model is *off by some amount*, and\n",
    "how you summarize those amounts changes which model looks best. In this lesson\n",
    "you'll meet the standard regression metrics, learn where each one shines or\n",
    "lies, and use residual plots to see what a single score can't tell you."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## It all starts with residuals\n",
    "\n",
    "Suppose a model predicts house prices. For each house we have the true price\n",
    "`y` and the prediction `ŷ`. The **residual** is the gap between them:\n",
    "\n",
    "**residual = y − ŷ**\n",
    "\n",
    "A positive residual means the model predicted too low; negative means too\n",
    "high. Every regression metric is just a different way of squashing a list of\n",
    "residuals into a single number — and each way of squashing makes a different\n",
    "trade-off."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## MAE, MSE, and RMSE\n",
    "\n",
    "The three workhorses:\n",
    "\n",
    "- **Mean Absolute Error (MAE)** — average of the absolute residuals. \"On\n",
    "  average, we're off by this much.\" Same units as the target.\n",
    "- **Mean Squared Error (MSE)** — average of the *squared* residuals. Punishes\n",
    "  big misses much harder, but the units are squared (dollars² — awkward).\n",
    "- **Root Mean Squared Error (RMSE)** — the square root of MSE. Back in the\n",
    "  target's units, but still outlier-sensitive because the squaring happened\n",
    "  first.\n",
    "\n",
    "The key behavioral difference is **outlier sensitivity**. Watch what a single\n",
    "terrible prediction does to each metric:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
    "\n",
    "y_true = np.array([10.0, 12.0, 15.0, 18.0, 20.0, 22.0, 25.0, 28.0])\n",
    "y_good = np.array([11.0, 11.0, 16.0, 17.0, 21.0, 21.0, 26.0, 27.0])  # all small misses\n",
    "y_out  = np.array([11.0, 11.0, 16.0, 17.0, 21.0, 21.0, 26.0, 48.0])  # one huge miss\n",
    "\n",
    "for name, y_pred in [(\"small misses only\", y_good), (\"with one outlier\", y_out)]:\n",
    "    mae = mean_absolute_error(y_true, y_pred)\n",
    "    mse = mean_squared_error(y_true, y_pred)\n",
    "    rmse = np.sqrt(mse)\n",
    "    print(f\"{name:19s} MAE={mae:5.2f}   MSE={mse:6.2f}   RMSE={rmse:5.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "One bad prediction out of eight barely moved the MAE (each point contributes\n",
    "linearly), but MSE exploded and RMSE tripled — the squared 20-unit miss\n",
    "dominates everything. Neither behavior is \"correct\":\n",
    "\n",
    "- If big misses are disproportionately costly in your problem (a 20% error in\n",
    "  a drug dose is not 2× as bad as a 10% error), **RMSE's sensitivity is a\n",
    "  feature**.\n",
    "- If your data contains a few noisy, unrepresentative extremes you don't want\n",
    "  dominating model selection, **MAE is the more robust summary**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "## R²: better than guessing the mean?\n",
    "\n",
    "MAE and RMSE are in the target's units, which is great for stakeholders but\n",
    "hard to compare across problems. **R² (the coefficient of determination)**\n",
    "fixes that by comparing your model against the dumbest possible baseline:\n",
    "always predicting the mean of `y`.\n",
    "\n",
    "- **R² = 1** — perfect predictions, zero error.\n",
    "- **R² = 0** — your model is exactly as good as predicting the mean.\n",
    "- **R² negative** — your model is *worse* than predicting the mean. Yes, this\n",
    "  happens, and it's a loud alarm bell."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0007",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "y_true = rng.uniform(10, 50, 30)\n",
    "\n",
    "y_close = y_true + rng.normal(0, 2, 30)          # good model\n",
    "y_mean  = np.full(30, y_true.mean())             # baseline: always the mean\n",
    "y_bad   = 60 - y_true + rng.normal(0, 2, 30)     # confidently wrong\n",
    "\n",
    "print(f\"good model      R2 = {r2_score(y_true, y_close):7.3f}\")\n",
    "print(f\"predict mean    R2 = {r2_score(y_true, y_mean):7.3f}\")\n",
    "print(f\"terrible model  R2 = {r2_score(y_true, y_bad):7.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "That last score is far below zero: the model's errors are *bigger* than the\n",
    "spread of the data itself. When you see negative R² on a test set, the model\n",
    "learned something that actively misleads it — often a sign of leakage,\n",
    "a bug, or severe overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "> **R² is the default score in scikit-learn**\n",
    "> \n",
    "> Calling `.score(X, y)` on any scikit-learn regressor returns R². When you read\n",
    "> \"score = 0.87\" in regression code, it almost always means R²."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## MAPE: intuitive, with a fatal flaw\n",
    "\n",
    "**Mean Absolute Percentage Error** reports the average relative error — \"off\n",
    "by 12% on average\" — which non-technical audiences love. But it divides by\n",
    "the true value, so it breaks when `y` is zero or near zero:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import mean_absolute_percentage_error\n",
    "\n",
    "y_true = np.array([100.0, 200.0, 300.0])\n",
    "y_pred = np.array([110.0, 190.0, 315.0])\n",
    "print(f\"normal targets:    MAPE = {mean_absolute_percentage_error(y_true, y_pred):.3f}\")\n",
    "\n",
    "y_true2 = np.array([100.0, 200.0, 0.001])   # one near-zero target\n",
    "y_pred2 = np.array([110.0, 190.0, 1.0])     # tiny absolute miss...\n",
    "print(f\"near-zero target:  MAPE = {mean_absolute_percentage_error(y_true2, y_pred2):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "A prediction that missed by *one unit* blew the MAPE up to astronomical\n",
    "levels, because the miss was divided by 0.001. Avoid MAPE when the target can\n",
    "be zero or close to it (demand forecasting with zero-sale days is the classic\n",
    "trap)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Which metric should you report?\n",
    "\n",
    "| Situation | Reach for |\n",
    "|---|---|\n",
    "| You want an error in the target's units, robust to outliers | MAE |\n",
    "| Big misses are disproportionately costly | RMSE |\n",
    "| You're optimizing / comparing models mathematically | MSE (smooth, differentiable) |\n",
    "| You need a unit-free \"how much better than baseline\" score | R² |\n",
    "| A relative \"% off\" story for stakeholders (targets far from 0) | MAPE |\n",
    "\n",
    "In practice, report **two**: one absolute metric (MAE or RMSE) plus R². They\n",
    "answer different questions — \"how far off are we?\" and \"how much of the\n",
    "variation do we explain?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "## Residual plots: seeing what the score hides\n",
    "\n",
    "A single number can't tell you *where* the model fails. A **residual plot** —\n",
    "predictions on the x-axis, residuals on the y-axis — can. For a healthy model\n",
    "the residuals look like a structureless, symmetric band around zero. Patterns\n",
    "mean trouble:\n",
    "\n",
    "- **A curve or U-shape** — the model is missing a nonlinear relationship.\n",
    "- **A funnel (spread grows with prediction)** — heteroscedasticity; errors\n",
    "  grow with the target, often fixed by transforming `y` (a later lesson).\n",
    "- **Train residuals tiny, test residuals huge** — overfitting."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0015",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y = load_diabetes(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = LinearRegression().fit(X_train, y_train)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(9, 3.5), sharey=True)\n",
    "for ax, Xs, ys, title in [(axes[0], X_train, y_train, \"train\"), (axes[1], X_test, y_test, \"test\")]:\n",
    "    pred = model.predict(Xs)\n",
    "    ax.scatter(pred, ys - pred, alpha=0.6, s=18)\n",
    "    ax.axhline(0, color=\"crimson\", lw=1.5)\n",
    "    ax.set_xlabel(\"predicted\")\n",
    "    ax.set_title(f\"{title} residuals\")\n",
    "axes[0].set_ylabel(\"residual (y - pred)\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "Both panels show a roughly symmetric cloud around zero with similar spread —\n",
    "no obvious curvature, no funnel, and train/test look alike. That's the visual\n",
    "signature of a model that is honest, if not spectacular. Get in the habit of\n",
    "looking at this plot *every time* you fit a regressor; it catches problems\n",
    "that MAE and R² silently average away."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Judge two models by more than one number\n",
    "\n",
    "Load `load_diabetes`, split into train/test, and fit a `LinearRegression`.\n",
    "Call its test predictions **Model A**. Create **Model B** by copying Model A's\n",
    "predictions and adding 300 to a single one (simulating one terrible miss).\n",
    "Compute MAE, RMSE, and R² for both. Which metrics barely notice the corruption,\n",
    "and which ones panic? Does the ranking of \"how bad is Model B\" depend on the\n",
    "metric you choose?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0018",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0019",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
    "\n",
    "X, y = load_diabetes(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "\n",
    "model = LinearRegression().fit(X_train, y_train)\n",
    "pred_a = model.predict(X_test)\n",
    "\n",
    "# Model B: identical, but one prediction corrupted by +300\n",
    "pred_b = pred_a.copy()\n",
    "pred_b[0] += 300\n",
    "\n",
    "for name, pred in [(\"A (clean)\", pred_a), (\"B (one bad miss)\", pred_b)]:\n",
    "    mae = mean_absolute_error(y_test, pred)\n",
    "    rmse = np.sqrt(mean_squared_error(y_test, pred))\n",
    "    r2 = r2_score(y_test, pred)\n",
    "    print(f\"{name:18s} MAE={mae:6.2f}  RMSE={rmse:6.2f}  R2={r2:.3f}\")\n",
    "\n",
    "# MAE barely moves (+300/n), RMSE jumps, R2 drops sharply.\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "Now that you can measure what \"good\" means, it's time to earn a score of your\n",
    "own — next: fit your first model with linear regression and gradient descent."
   ]
  }
 ]
}