Regularization: Ridge, Lasso & ElasticNet

You've seen that flexible models overfit, and that one cure is "regularize." This lesson delivers on that promise. Regularization adds a penalty for large weights to the loss function, so the optimizer must balance fitting the data against keeping the model tame. It's the practical answer to a real dilemma: you rarely know the right polynomial degree or feature set in advance — so use a generous model and let the penalty rein it in.

The symptom: exploding coefficients

When features are strongly correlated (or plentiful and noisy), ordinary least squares becomes unstable. If two columns carry nearly the same information, the model can put a huge positive weight on one and a huge negative weight on the other — the two nearly cancel, training error looks fine, and the coefficients are meaningless and fragile:

Both models fit equally well — but OLS invented enormous, opposite-signed weights (rerun mentally with a different noise seed and they'd be completely different), while Ridge quietly spread a sensible total of ~3 across the three near-identical columns. That stability is what the penalty buys.

How the penalty works

Regularized regression minimizes loss = MSE + α · penalty(w), and the two classic penalties differ in one exponent:

• L2 (Ridge): penalty = Σ wᵢ² — the squared sizes. Big weights are punished quadratically, so Ridge shrinks all weights smoothly toward zero but almost never exactly to zero.
• L1 (Lasso): penalty = Σ |wᵢ| — the absolute sizes. The pressure on a weight doesn't fade as it approaches zero, so Lasso pushes unhelpful weights exactly to zero — it sparsifies.

The mnemonic from the source course: L2 for simplicity (smooth, stable, keeps everything a little), L1 for feature selection (keeps a few, kills the rest). The bias term b is not penalized — only the weights.

Alpha: the strength dial

alpha sets how loudly the penalty speaks. At α = 0 you recover plain OLS; as α grows toward infinity every weight is forced to zero (predicting only the mean). Tracing each coefficient as α grows produces the coefficient path — and it makes the L1/L2 difference visible:

Read the difference: Ridge coefficients glide smoothly toward zero together but stay alive until enormous α. Lasso coefficients hit exactly zero one by one — by moderate α only a handful of features survive. Reading which features survive longest is a legitimate (and popular) form of feature selection: fit Lasso, keep the features with non-zero coefficients.

Choose α the same way you chose polynomial degree: cross-validation. Small α = complex model (variance risk); large α = rigid model (bias risk) — it's the same U-curve, just with the dial reversed.

ElasticNet: why not both?

Lasso has quirks: with a group of correlated features it tends to pick one arbitrarily and zero the rest, and it can behave erratically when features outnumber samples. ElasticNet blends both penalties:

loss = MSE + α · ( l1_ratio · Σ|wᵢ| + (1 − l1_ratio) · Σwᵢ² / 2 )

l1_ratio slides from 0 (pure Ridge) to 1 (pure Lasso). Values in between give you sparsity and the stabilizing, group-friendly behavior of L2 — a sensible default when you suspect correlated features but still want selection.

from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
 
# The pattern for all three — generous features, then a penalty:
model = Pipeline([
    ("scaler", StandardScaler()),
    ("poly", PolynomialFeatures(10, include_bias=False)),
    ("reg", ElasticNet(alpha=0.01, l1_ratio=0.5)),   # or Ridge(0.01) / Lasso(0.01)
])

Spotting multicollinearity before it bites

The exploding-coefficient problem has a name — multicollinearity — and a cheap early-warning system: the correlation matrix. Compute pairwise correlations between features (and the target) and look for near-±1 blocks:

The bright cell between s1 and s2 (two blood-serum measurements, correlation ≈ 0.9) is exactly the situation from our first demo: OLS coefficients for those two are untrustworthy, and regularization is the standard remedy. (Pearson measures linear relationships; Spearman and Kendall rank-based variants catch monotonic ones — df.corr(method="spearman").)

Rules of thumb

• Default: Ridge with cross-validated α. Stable, smooth, rarely a bad idea.
• Many features, suspect most are useless: Lasso — get a sparse, interpretable model for free.
• Correlated feature groups + want sparsity: ElasticNet (tune l1_ratio ∈ 0.1–0.9).
• Good feature engineering still beats brute-force regularization — a well-chosen degree-3 model can outscore a regularized degree-10 one. But when you don't know the right features, regularization is the practical path.
• And always: scaler first, penalty second, α by cross-validation.

That wraps up regression — next module, we switch from predicting numbers to predicting categories: classification, starting with logistic regression.