Gradient Boosting & XGBoost

AdaBoost fixes mistakes by re-weighting samples. Gradient boosting fixes them more directly: each new tree is trained to predict the residuals — the part of the target the ensemble hasn't explained yet. That one reframing turns boosting into gradient descent, works for any differentiable loss, and leads straight to XGBoost and friends: the models that win most tabular ML competitions. This lesson builds the mechanism by hand, then hands you the production tools.

Fit the residuals: boosting as golf

Think of a golfer: the first stroke covers most of the distance to the hole, the second corrects what's left, the third corrects what's left after that. Gradient boosting for regression is exactly this. Start with a first model, compute the errors it leaves behind, and train the next model on those errors — then predict with the sum of all strokes:

F(x) = tree₁(x) + tree₂(x) + tree₃(x) + …

Let's do three strokes manually on a 1-D problem and watch the sum improve:

Each shallow tree is a lousy model of the sine wave on its own — but each one only has to model what its predecessors missed, and the sum sharpens with every stage. The residual MSE printout is the "distance to the hole" shrinking stroke by stroke.

Why "gradient" boosting?

For squared-error loss, the residual y − F(x) is exactly the negative gradient of the loss with respect to the current prediction. So "fit a tree to the residuals, add it to the ensemble" is literally a gradient-descent step — not in parameter space like lesson 2's gradient descent, but in function space: each iteration nudges the whole prediction function downhill on the loss.

That's the generalization AdaBoost was missing. Swap in a different loss — absolute error for robustness, log-loss for classification, quantile loss for prediction intervals — compute its negative gradient instead of plain residuals, and the same machinery works. Classification with gradient boosting is just this recipe applied to log-loss.

The knobs: learning rate, tree count, depth, subsample

In practice each tree's contribution is shrunk by a learning rate ν:

Fₘ(x) = Fₘ₋₁(x) + ν · treeₘ(x)

• learning_rate vs n_estimators — the fundamental tradeoff. A lower rate takes smaller, more cautious steps and needs more trees to get there, but almost always generalizes better. The recipe: set learning_rate low (0.05–0.1), make n_estimators large, and stop when validation stops improving.
• max_depth — keep the trees shallow (2–5). Just as with AdaBoost, the depth of the base learner is the dangerous knob, not the number of rounds: deep trees fit each round's residuals (noise included) too eagerly. With many features, allow a bit more depth so trees can combine features.
• subsample — train each tree on a random fraction of rows (e.g. 0.8). This "stochastic gradient boosting" adds bagging-style diversity and often improves generalization for free.

scikit-learn ships this as GradientBoostingClassifier and GradientBoostingRegressor — faithful, but slow on large data because every split search scans every unique feature value.

The modern default: HistGradientBoosting

HistGradientBoostingClassifier (and its regressor twin) is scikit-learn's LightGBM-inspired rewrite: it bins each feature into at most 255 buckets and searches splits over bins instead of raw values. It's orders of magnitude faster on big data, handles missing values natively, and supports early stopping out of the box. If you're gradient boosting in scikit-learn today, start here:

Early stopping neatly solves the "how many trees?" question: ask for plenty and let the validation curve decide when to quit.

XGBoost

XGBoost (eXtreme Gradient Boosting) took gradient boosting from a good idea to a phenomenon: regularized objectives (L1/L2 penalties on the leaves), clever handling of sparse and missing data, column subsampling, and ruthless systems engineering. The API mirrors scikit-learn. It doesn't run in the browser, so run these cells in the downloaded notebook or Colab:

from xgboost import XGBClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
 
X, y = load_breast_cancer(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42)
 
model = XGBClassifier(
    n_estimators=1000,          # an upper bound - early stopping picks the real number
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,              # row subsampling per tree
    colsample_bytree=0.8,       # feature subsampling per tree
    eval_metric="logloss",
    early_stopping_rounds=25,   # stop if val logloss hasn't improved in 25 rounds
    random_state=42,
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
 
print(f"best iteration: {model.best_iteration}")
print(f"test accuracy : {model.score(X_test, y_test):.3f}")

Note the pattern: a separate validation set in eval_set drives early stopping, and the untouched test set gives the final honest number. XGBoost also reports feature importances — with the same caveats as random forests (prefer importance_type="gain" over the default split counts, and prefer permutation importance over both):

import pandas as pd
from sklearn.datasets import load_breast_cancer
 
names = load_breast_cancer().feature_names
booster = model.get_booster()
gain = booster.get_score(importance_type="gain")
imp = (pd.Series({names[int(k[1:])]: v for k, v in gain.items()})
         .sort_values(ascending=False))
print(imp.head(8).round(1))

Choosing your booster — and when to skip boosting

The three big gradient-boosting libraries are more alike than different:

Honestly, on most tabular problems all three (and HistGradientBoosting) land within a whisker of each other once tuned. The bigger question is boosting vs random forest: boosting usually squeezes out a few extra points of accuracy because it reduces bias as well as variance — but it demands tuning (learning rate, rounds, depth) and is touchier about noisy labels. A random forest is nearly tuning-free, trains in parallel, and is hard to badly misconfigure. A sensible workflow: baseline with a forest, then bring in gradient boosting with early stopping when you need the last few points.

That wraps up trees and ensembles — next module: unsupervised learning, where we find structure in data that has no labels at all, starting with K-Means.