{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Regularization: Ridge, Lasso & ElasticNet\n",
    "\n",
    "Tame exploding coefficients with L1 and L2 penalties — shrink weights with Ridge, select features with Lasso, and blend both with ElasticNet.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/regularization).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "You've seen that flexible models overfit, and that one cure is \"regularize.\"\n",
    "This lesson delivers on that promise. **Regularization** adds a penalty for\n",
    "large weights to the loss function, so the optimizer must balance fitting the\n",
    "data against keeping the model tame. It's the practical answer to a real\n",
    "dilemma: you rarely know the *right* polynomial degree or feature set in\n",
    "advance — so use a generous model and let the penalty rein it in."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## The symptom: exploding coefficients\n",
    "\n",
    "When features are strongly **correlated** (or plentiful and noisy), ordinary\n",
    "least squares becomes unstable. If two columns carry nearly the same\n",
    "information, the model can put a huge positive weight on one and a huge\n",
    "negative weight on the other — the two nearly cancel, training error looks\n",
    "fine, and the coefficients are meaningless and fragile:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0003",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "n = 60\n",
    "x1 = rng.normal(0, 1, n)\n",
    "x2 = x1 + rng.normal(0, 0.01, n)   # nearly a copy of x1\n",
    "x3 = x1 + rng.normal(0, 0.01, n)   # another near-copy\n",
    "X = np.column_stack([x1, x2, x3])\n",
    "y = 3 * x1 + rng.normal(0, 0.5, n) # truth: only x1 matters, weight 3\n",
    "\n",
    "ols = LinearRegression().fit(X, y)\n",
    "ridge = Ridge(alpha=1.0).fit(X, y)\n",
    "\n",
    "print(\"true weights:    [ 3.00  0.00  0.00 ]\")\n",
    "print(\"OLS coef:       \", np.round(ols.coef_, 2))\n",
    "print(\"Ridge coef:     \", np.round(ridge.coef_, 2))\n",
    "print(f\"\\\\nOLS R2   = {ols.score(X, y):.3f}\")\n",
    "print(f\"Ridge R2 = {ridge.score(X, y):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "Both models fit equally well — but OLS invented enormous, opposite-signed\n",
    "weights (rerun mentally with a different noise seed and they'd be completely\n",
    "different), while Ridge quietly spread a sensible total of ~3 across the three\n",
    "near-identical columns. That stability is what the penalty buys."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "## How the penalty works\n",
    "\n",
    "Regularized regression minimizes **loss = MSE + α · penalty(w)**, and the two\n",
    "classic penalties differ in one exponent:\n",
    "\n",
    "- **L2 (Ridge):** penalty = Σ wᵢ² — the *squared* sizes. Big weights are\n",
    "  punished quadratically, so Ridge **shrinks** all weights smoothly toward\n",
    "  zero but almost never exactly *to* zero.\n",
    "- **L1 (Lasso):** penalty = Σ |wᵢ| — the *absolute* sizes. The pressure on a\n",
    "  weight doesn't fade as it approaches zero, so Lasso pushes unhelpful\n",
    "  weights **exactly to zero** — it *sparsifies*.\n",
    "\n",
    "The mnemonic from the source course: **L2 for simplicity** (smooth,\n",
    "stable, keeps everything a little), **L1 for feature selection** (keeps a\n",
    "few, kills the rest). The bias term `b` is not penalized — only the weights."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "> **Scale before you regularize**\n",
    "> \n",
    "> The penalty compares raw coefficient sizes. An unscaled feature measured in\n",
    "> tiny units needs a huge coefficient just to participate — and gets crushed by\n",
    "> the penalty for reasons that have nothing to do with usefulness. **Always put\n",
    "> a scaler before Ridge/Lasso/ElasticNet in your pipeline.** (This is why the\n",
    "> previous lesson came first.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## Alpha: the strength dial\n",
    "\n",
    "`alpha` sets how loudly the penalty speaks. At α = 0 you recover plain OLS; as\n",
    "α grows toward infinity every weight is forced to zero (predicting only the\n",
    "mean). Tracing each coefficient as α grows produces the **coefficient path** —\n",
    "and it makes the L1/L2 difference visible:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0008",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import Ridge, Lasso\n",
    "\n",
    "X, y = load_diabetes(return_X_y=True)\n",
    "X = StandardScaler().fit_transform(X)\n",
    "\n",
    "alphas = np.logspace(-2, 3, 30)\n",
    "ridge_paths, lasso_paths = [], []\n",
    "for a in alphas:\n",
    "    ridge_paths.append(Ridge(alpha=a).fit(X, y).coef_)\n",
    "    lasso_paths.append(Lasso(alpha=a, max_iter=5000).fit(X, y).coef_)\n",
    "\n",
    "fig, axes = plt.subplots(1, 2, figsize=(9, 3.8), sharey=True)\n",
    "for ax, paths, title in [(axes[0], ridge_paths, \"Ridge (L2): shrink\"),\n",
    "                         (axes[1], lasso_paths, \"Lasso (L1): sparsify\")]:\n",
    "    ax.plot(alphas, np.array(paths))\n",
    "    ax.set_xscale(\"log\")\n",
    "    ax.axhline(0, color=\"black\", lw=0.8)\n",
    "    ax.set_xlabel(\"alpha\")\n",
    "    ax.set_title(title)\n",
    "axes[0].set_ylabel(\"coefficient value\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "Read the difference: Ridge coefficients glide smoothly toward zero together\n",
    "but stay alive until enormous α. Lasso coefficients hit **exactly zero** one\n",
    "by one — by moderate α only a handful of features survive. Reading which\n",
    "features survive longest is a legitimate (and popular) form of feature\n",
    "selection: fit `Lasso`, keep the features with non-zero coefficients.\n",
    "\n",
    "Choose α the same way you chose polynomial degree: **cross-validation**.\n",
    "Small α = complex model (variance risk); large α = rigid model (bias risk) —\n",
    "it's the same U-curve, just with the dial reversed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## ElasticNet: why not both?\n",
    "\n",
    "Lasso has quirks: with a group of correlated features it tends to pick *one*\n",
    "arbitrarily and zero the rest, and it can behave erratically when features\n",
    "outnumber samples. **ElasticNet** blends both penalties:\n",
    "\n",
    "**loss = MSE + α · ( l1_ratio · Σ|wᵢ| + (1 − l1_ratio) · Σwᵢ² / 2 )**\n",
    "\n",
    "`l1_ratio` slides from 0 (pure Ridge) to 1 (pure Lasso). Values in between\n",
    "give you sparsity *and* the stabilizing, group-friendly behavior of L2 — a\n",
    "sensible default when you suspect correlated features but still want\n",
    "selection."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge, Lasso, ElasticNet\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, PolynomialFeatures\n",
    "\n",
    "# The pattern for all three — generous features, then a penalty:\n",
    "model = Pipeline([\n",
    "    (\"scaler\", StandardScaler()),\n",
    "    (\"poly\", PolynomialFeatures(10, include_bias=False)),\n",
    "    (\"reg\", ElasticNet(alpha=0.01, l1_ratio=0.5)),   # or Ridge(0.01) / Lasso(0.01)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "## Spotting multicollinearity before it bites\n",
    "\n",
    "The exploding-coefficient problem has a name — **multicollinearity** — and a\n",
    "cheap early-warning system: the **correlation matrix**. Compute pairwise\n",
    "correlations between features (and the target) and look for near-±1 blocks:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0013",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_diabetes\n",
    "\n",
    "data = load_diabetes()\n",
    "df = pd.DataFrame(data.data, columns=data.feature_names)\n",
    "df[\"target\"] = data.target\n",
    "\n",
    "corr = df.corr()\n",
    "\n",
    "plt.figure(figsize=(6.5, 5.5))\n",
    "plt.imshow(corr, cmap=\"coolwarm\", vmin=-1, vmax=1)\n",
    "plt.colorbar(label=\"Pearson correlation\")\n",
    "plt.xticks(range(len(corr)), corr.columns, rotation=45, ha=\"right\")\n",
    "plt.yticks(range(len(corr)), corr.columns)\n",
    "for i in range(len(corr)):\n",
    "    for j in range(len(corr)):\n",
    "        plt.text(j, i, f\"{corr.iloc[i, j]:.1f}\", ha=\"center\", va=\"center\", fontsize=7)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "The bright cell between `s1` and `s2` (two blood-serum measurements,\n",
    "correlation ≈ 0.9) is exactly the situation from our first demo: OLS\n",
    "coefficients for those two are untrustworthy, and regularization is the\n",
    "standard remedy. (Pearson measures *linear* relationships; Spearman and\n",
    "Kendall rank-based variants catch monotonic ones — `df.corr(method=\"spearman\")`.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "## Rules of thumb\n",
    "\n",
    "- **Default:** Ridge with cross-validated α. Stable, smooth, rarely a bad idea.\n",
    "- **Many features, suspect most are useless:** Lasso — get a sparse,\n",
    "  interpretable model for free.\n",
    "- **Correlated feature groups + want sparsity:** ElasticNet\n",
    "  (tune `l1_ratio` ∈ 0.1–0.9).\n",
    "- Good feature engineering still beats brute-force regularization — a\n",
    "  well-chosen degree-3 model can outscore a regularized degree-10 one. But\n",
    "  when you *don't know* the right features, regularization is the practical\n",
    "  path.\n",
    "- And always: **scaler first, penalty second, α by cross-validation.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Tune an ElasticNet end to end\n",
    "\n",
    "Recreate this module's original capstone exercise on an offline dataset: load\n",
    "`load_diabetes`, split 75/25, and build a `Pipeline` of `StandardScaler` →\n",
    "`PolynomialFeatures(2)` → `ElasticNet`. Use `GridSearchCV` (cv=5) to tune\n",
    "`alpha` over 0.01–10 and `l1_ratio` over 0.1, 0.5, 0.9. Report the best\n",
    "parameters, cross-validated R², and test R² — and count how many of the\n",
    "polynomial features ElasticNet zeroed out."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0017",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, PolynomialFeatures\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "X, y = load_diabetes(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    (\"scaler\", StandardScaler()),\n",
    "    (\"poly\", PolynomialFeatures(2, include_bias=False)),\n",
    "    (\"enet\", ElasticNet(max_iter=5000)),\n",
    "])\n",
    "\n",
    "params = {\n",
    "    \"enet__alpha\": [0.01, 0.1, 1.0, 10.0],\n",
    "    \"enet__l1_ratio\": [0.1, 0.5, 0.9],\n",
    "}\n",
    "search = GridSearchCV(pipeline, params, cv=5)\n",
    "search.fit(X_train, y_train)\n",
    "\n",
    "print(\"best params:\", search.best_params_)\n",
    "print(f\"CV R2:   {search.best_score_:.3f}\")\n",
    "print(f\"test R2: {search.score(X_test, y_test):.3f}\")\n",
    "\n",
    "coefs = search.best_estimator_.named_steps[\"enet\"].coef_\n",
    "print(f\"features: {len(coefs)} total, {(coefs == 0).sum()} zeroed out by the L1 part\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0019",
   "metadata": {},
   "source": [
    "That wraps up regression — next module, we switch from predicting numbers to\n",
    "predicting categories: classification, starting with logistic regression."
   ]
  }
 ]
}