{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Gradient Boosting & XGBoost\n", "\n", "Boosting as gradient descent in function space — fit trees to residuals, control the learning-rate/tree-count tradeoff, and use HistGradientBoosting and XGBoost like a practitioner.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/gradient-boosting-xgboost).*" ] }, { "cell_type": "code", "id": "cell-0001", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q xgboost" ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "AdaBoost fixes mistakes by re-weighting samples. Gradient boosting fixes them\n", "more directly: each new tree is trained to predict the **residuals** — the\n", "part of the target the ensemble hasn't explained yet. That one reframing\n", "turns boosting into gradient descent, works for any differentiable loss, and\n", "leads straight to XGBoost and friends: the models that win most tabular ML\n", "competitions. This lesson builds the mechanism by hand, then hands you the\n", "production tools." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Fit the residuals: boosting as golf\n", "\n", "Think of a golfer: the first stroke covers most of the distance to the hole,\n", "the second corrects what's left, the third corrects what's left after *that*.\n", "Gradient boosting for regression is exactly this. Start with a first model,\n", "compute the errors it leaves behind, and train the next model **on those\n", "errors** — then predict with the *sum* of all strokes:\n", "\n", "**F(x) = tree₁(x) + tree₂(x) + tree₃(x) + …**\n", "\n", "Let's do three strokes manually on a 1-D problem and watch the sum improve:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.tree import DecisionTreeRegressor\n", "\n", "rng = np.random.default_rng(42)\n", "X = np.sort(rng.uniform(0, 6, 120)).reshape(-1, 1)\n", "y = np.sin(X.ravel()) * 2 + rng.normal(0, 0.3, 120)\n", "grid = np.linspace(0, 6, 300).reshape(-1, 1)\n", "\n", "# Stroke 1: fit the target itself\n", "t1 = DecisionTreeRegressor(max_depth=2).fit(X, y)\n", "r1 = y - t1.predict(X) # what's left unexplained\n", "\n", "# Stroke 2: fit the residuals of stroke 1\n", "t2 = DecisionTreeRegressor(max_depth=2).fit(X, r1)\n", "r2 = r1 - t2.predict(X)\n", "\n", "# Stroke 3: fit the residuals of stroke 2\n", "t3 = DecisionTreeRegressor(max_depth=2).fit(X, r2)\n", "\n", "stages = [\n", " (\"1 tree\", t1.predict(grid)),\n", " (\"1 + 2\", t1.predict(grid) + t2.predict(grid)),\n", " (\"1 + 2 + 3\", t1.predict(grid) + t2.predict(grid) + t3.predict(grid)),\n", "]\n", "\n", "fig, axes = plt.subplots(1, 3, figsize=(11, 3), sharey=True)\n", "for ax, (label, pred) in zip(axes, stages):\n", " ax.scatter(X, y, s=8, alpha=0.4)\n", " ax.plot(grid, pred, color=\"crimson\", lw=2)\n", " ax.set_title(label)\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "for (label, _), r in zip(stages, [r1, r2, r2 - t3.predict(X)]):\n", " print(f\"{label:9s}: residual MSE = {np.mean(r**2):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "Each shallow tree is a lousy model of the sine wave on its own — but each one\n", "only has to model *what its predecessors missed*, and the sum sharpens with\n", "every stage. The residual MSE printout is the \"distance to the hole\" shrinking\n", "stroke by stroke." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Why \"gradient\" boosting?\n", "\n", "For squared-error loss, the residual `y − F(x)` is exactly the **negative\n", "gradient** of the loss with respect to the current prediction. So \"fit a tree\n", "to the residuals, add it to the ensemble\" is literally a gradient-descent\n", "step — not in parameter space like lesson 2's gradient descent, but in\n", "**function space**: each iteration nudges the whole prediction function\n", "downhill on the loss.\n", "\n", "That's the generalization AdaBoost was missing. Swap in a different loss —\n", "absolute error for robustness, log-loss for classification, quantile loss for\n", "prediction intervals — compute its negative gradient instead of plain\n", "residuals, and the same machinery works. Classification with gradient\n", "boosting is just this recipe applied to log-loss." ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "## The knobs: learning rate, tree count, depth, subsample\n", "\n", "In practice each tree's contribution is shrunk by a **learning rate** ν:\n", "\n", "**Fₘ(x) = Fₘ₋₁(x) + ν · treeₘ(x)**\n", "\n", "- **`learning_rate` vs `n_estimators`** — the fundamental tradeoff. A lower\n", " rate takes smaller, more cautious steps and needs more trees to get there,\n", " but almost always generalizes better. The recipe: set `learning_rate` low\n", " (0.05–0.1), make `n_estimators` large, and stop when validation stops\n", " improving.\n", "- **`max_depth`** — keep the trees shallow (2–5). Just as with AdaBoost, the\n", " *depth* of the base learner is the dangerous knob, not the number of rounds:\n", " deep trees fit each round's residuals (noise included) too eagerly. With\n", " many features, allow a bit more depth so trees can combine features.\n", "- **`subsample`** — train each tree on a random fraction of rows (e.g. 0.8).\n", " This \"stochastic gradient boosting\" adds bagging-style diversity and often\n", " improves generalization for free.\n", "\n", "scikit-learn ships this as `GradientBoostingClassifier` and\n", "`GradientBoostingRegressor` — faithful, but slow on large data because every\n", "split search scans every unique feature value." ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "## The modern default: HistGradientBoosting\n", "\n", "`HistGradientBoostingClassifier` (and its regressor twin) is scikit-learn's\n", "LightGBM-inspired rewrite: it bins each feature into at most 255 buckets and\n", "searches splits over bins instead of raw values. It's orders of magnitude\n", "faster on big data, handles missing values natively, and supports early\n", "stopping out of the box. If you're gradient boosting in scikit-learn today,\n", "start here:" ] }, { "cell_type": "code", "id": "cell-0009", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.ensemble import HistGradientBoostingClassifier\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, stratify=y, random_state=42)\n", "\n", "hgb = HistGradientBoostingClassifier(\n", " learning_rate=0.1,\n", " max_iter=200, # boosting rounds\n", " early_stopping=True, # hold out part of train, stop when it stalls\n", " validation_fraction=0.15,\n", " random_state=42,\n", ")\n", "hgb.fit(X_train, y_train)\n", "\n", "print(f\"rounds actually used: {hgb.n_iter_} of 200\")\n", "print(f\"test accuracy : {hgb.score(X_test, y_test):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "Early stopping neatly solves the \"how many trees?\" question: ask for plenty\n", "and let the validation curve decide when to quit." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "## XGBoost\n", "\n", "**XGBoost** (eXtreme Gradient Boosting) took gradient boosting from a good\n", "idea to a phenomenon: regularized objectives (L1/L2 penalties on the leaves),\n", "clever handling of sparse and missing data, column subsampling, and ruthless\n", "systems engineering. The API mirrors scikit-learn. It doesn't run in the\n", "browser, so run these cells in the downloaded notebook or Colab:" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from xgboost import XGBClassifier\n", "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)\n", "X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42)\n", "\n", "model = XGBClassifier(\n", " n_estimators=1000, # an upper bound - early stopping picks the real number\n", " learning_rate=0.05,\n", " max_depth=4,\n", " subsample=0.8, # row subsampling per tree\n", " colsample_bytree=0.8, # feature subsampling per tree\n", " eval_metric=\"logloss\",\n", " early_stopping_rounds=25, # stop if val logloss hasn't improved in 25 rounds\n", " random_state=42,\n", ")\n", "model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)\n", "\n", "print(f\"best iteration: {model.best_iteration}\")\n", "print(f\"test accuracy : {model.score(X_test, y_test):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "Note the pattern: a separate **validation set** in `eval_set` drives early\n", "stopping, and the untouched test set gives the final honest number. XGBoost\n", "also reports feature importances — with the same caveats as random forests\n", "(prefer `importance_type=\"gain\"` over the default split counts, and prefer\n", "permutation importance over both):" ] }, { "cell_type": "code", "id": "cell-0014", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import pandas as pd\n", "from sklearn.datasets import load_breast_cancer\n", "\n", "names = load_breast_cancer().feature_names\n", "booster = model.get_booster()\n", "gain = booster.get_score(importance_type=\"gain\")\n", "imp = (pd.Series({names[int(k[1:])]: v for k, v in gain.items()})\n", " .sort_values(ascending=False))\n", "print(imp.head(8).round(1))" ] }, { "cell_type": "markdown", "id": "cell-0015", "metadata": {}, "source": [ "## Choosing your booster — and when to skip boosting\n", "\n", "The three big gradient-boosting libraries are more alike than different:\n", "\n", "| Library | Notable for | Reach for it when |\n", "|---|---|---|\n", "| **XGBoost** | The battle-tested original; huge ecosystem | You want maximum community support and portability |\n", "| **LightGBM** | Histogram splits, leaf-wise growth — usually fastest | Large datasets, many features, speed matters |\n", "| **CatBoost** | Native categorical handling, strong defaults | Lots of categorical features, minimal tuning time |\n", "\n", "Honestly, on most tabular problems all three (and HistGradientBoosting) land\n", "within a whisker of each other once tuned. The bigger question is **boosting\n", "vs random forest**: boosting usually squeezes out a few extra points of\n", "accuracy because it reduces bias as well as variance — but it demands tuning\n", "(learning rate, rounds, depth) and is touchier about noisy labels. A random\n", "forest is nearly tuning-free, trains in parallel, and is hard to badly\n", "misconfigure. A sensible workflow: baseline with a forest, then bring in\n", "gradient boosting with early stopping when you need the last few points." ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "### 🏋️ Exercise — The learning-rate / tree-count tradeoff, measured\n", "\n", "Using `HistGradientBoostingClassifier` on the breast cancer dataset (70/30\n", "stratified split, `max_iter=200`, early stopping off), train models with\n", "learning rates 1.0, 0.3, 0.1, and 0.03. Print train and test accuracy for\n", "each. Which rate gives the best test score — and what do you notice about the\n", "train scores of the aggressive rates?" ] }, { "cell_type": "code", "id": "cell-0017", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0018", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.ensemble import HistGradientBoostingClassifier\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.3, stratify=y, random_state=42)\n", "\n", "for lr in [1.0, 0.3, 0.1, 0.03]:\n", " hgb = HistGradientBoostingClassifier(\n", " learning_rate=lr, max_iter=200, early_stopping=False, random_state=42)\n", " hgb.fit(X_train, y_train)\n", " print(f\"lr={lr:<5} train={hgb.score(X_train, y_train):.3f} \"\n", " f\"test={hgb.score(X_test, y_test):.3f}\")\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0019", "metadata": {}, "source": [ "That wraps up trees and ensembles — next module: unsupervised learning, where\n", "we find structure in data that has no labels at all, starting with K-Means." ] } ] }