The Bias–Variance Tradeoff

In the last lesson you watched test error trace a U as model complexity grew. The bias–variance tradeoff is the theory behind that U — and more usefully, it's a diagnostic framework: models fail in two opposite ways, each with its own symptoms and its own cures. Prescribing the wrong cure (adding data to a high-bias model, adding complexity to a high-variance one) wastes weeks. This lesson teaches you to tell the two apart.

Three ingredients of prediction error

The expected error of a model on new data decomposes, conceptually, into three parts:

expected error = bias² + variance + irreducible noise

• Bias — error from wrong assumptions. A straight line fitted to a curve is biased: no matter how much data you give it, it systematically misses the shape. High bias = underfitting.
• Variance — error from sensitivity to the training sample. A flexible model fitted to 40 noisy points would look completely different if you drew 40 different points. High variance = overfitting.
• Irreducible noise — randomness in the data itself (measurement error, unmodeled factors). No model can remove it; it's the floor under your error.

The tradeoff: making a model more flexible reduces bias but raises variance, and vice versa. The best model balances the two — the bottom of the U.

Seeing variance with your own eyes

"Variance" sounds abstract until you watch it. Below we draw several bootstrap samples (resamples of the same dataset) and fit the same model to each. A degree-2 fit barely notices which sample it got; a degree-9 fit changes shape completely every time:

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.preprocessing import PolynomialFeatures from sklearn.pipeline import Pipeline from sklearn.linear_model import LinearRegression rng = np.random.default_rng(3) x = np.sort(rng.uniform(-3, 3, 40)) y = np.sin(1.3 * x) + rng.normal(0, 0.35, 40) grid = np.linspace(-3, 3, 200).reshape(-1, 1) fig, axes = plt.subplots(1, 2, figsize=(9, 3.8), sharey=True) for ax, degree in [(axes[0], 2), (axes[1], 9)]: for _ in range(6): # 6 bootstrap resamples idx = rng.integers(0, len(x), len(x)) model = Pipeline([ ("poly", PolynomialFeatures(degree, include_bias=False)), ("lr", LinearRegression()), ]).fit(x[idx].reshape(-1, 1), y[idx]) ax.plot(grid, model.predict(grid), alpha=0.6, lw=1.5) ax.scatter(x, y, s=15, color="black", zorder=5) ax.set_ylim(-2.5, 2.5) ax.set_title(f"degree {degree}: 6 fits on 6 resamples") plt.tight_layout() plt.show()

Left: six nearly identical curves — low variance (but they all miss the sine wiggle the same way: that consistent miss is bias). Right: six wildly different curves — each one chased the noise of its particular sample. That disagreement is variance, and it's why the degree-9 model's test error is so bad: on average, a randomly-drawn wiggly curve is far from the truth.

Symptoms and cures

High bias (underfitting) looks like:

• Training error is high — the model can't even fit the data it has seen.
• Test error is about equally high; the train/test gap is small.
• More data doesn't help — the curves just confirm the same wrong shape.

Fixes: add complexity — more/better features (polynomial terms, interactions, domain-driven features like the taxi distance you saw), a more flexible model family, or less regularization.

High variance (overfitting) looks like:

• Training error is very low, sometimes near zero.
• Test error is much higher — a big train/test gap.
• Results change a lot between random seeds or resamples.

Fixes: constrain or stabilize — more training data, a simpler model (lower degree), regularization (next lessons), or averaging many models (ensembles, later in the course).

Validation curves: error vs complexity

scikit-learn automates the "U-curve" experiment with validation_curve: it sweeps one hyperparameter and cross-validates at each value. Here we sweep KNN's n_neighbors (note: for KNN, small n_neighbors = high complexity, so the x-axis runs complex → simple):

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.model_selection import validation_curve from sklearn.neighbors import KNeighborsRegressor rng = np.random.default_rng(0) X = rng.uniform(-3, 3, (120, 1)) y = np.sin(1.3 * X.ravel()) + rng.normal(0, 0.3, 120) ks = np.array([1, 2, 3, 5, 8, 12, 20, 40, 80]) train_sc, val_sc = validation_curve( KNeighborsRegressor(), X, y, param_name="n_neighbors", param_range=ks, cv=4) plt.figure(figsize=(7, 4)) plt.plot(ks, train_sc.mean(axis=1), "o-", label="train R2") plt.plot(ks, val_sc.mean(axis=1), "s-", label="validation R2") plt.xscale("log") plt.xlabel("n_neighbors (left = complex, right = simple)") plt.ylabel("R2") plt.legend() plt.tight_layout() plt.show()

Read it like a doctor: on the left (k = 1), train R² is perfect while validation lags — variance zone. On the far right (k = 80, nearly averaging everything), both scores collapse together — bias zone. The sweet spot is where validation peaks.

Learning curves: will more data help?

The second diagnostic sweeps training set size instead. Its shape answers the most expensive question in ML — "should we collect more data?":

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.model_selection import learning_curve from sklearn.preprocessing import PolynomialFeatures from sklearn.pipeline import Pipeline from sklearn.linear_model import LinearRegression rng = np.random.default_rng(1) X = rng.uniform(-3, 3, (150, 1)) y = np.sin(1.3 * X.ravel()) + rng.normal(0, 0.3, 150) model = Pipeline([ ("poly", PolynomialFeatures(6, include_bias=False)), ("lr", LinearRegression()), ]) sizes, train_sc, val_sc = learning_curve( model, X, y, train_sizes=np.linspace(0.15, 1.0, 6), cv=4) plt.figure(figsize=(7, 4)) plt.plot(sizes, train_sc.mean(axis=1), "o-", label="train R2") plt.plot(sizes, val_sc.mean(axis=1), "s-", label="validation R2") plt.xlabel("training set size") plt.ylabel("R2") plt.legend() plt.tight_layout() plt.show()

With little data the flexible model aces training but flops on validation (variance). As data grows, the two curves converge — more data is shrinking the variance. Two endgames to recognize:

• Curves still converging with a gap → more data will help.
• Curves already converged at a mediocre score → the model has hit its bias floor; more data won't help — you need a better model or features.

Diagnosis cheat sheet

Observation	Diagnosis	Try
Train error high, test error similarly high	High bias (underfit)	More features, higher degree, more flexible model, less regularization
Train error tiny, test error much worse	High variance (overfit)	More data, simpler model, regularization, ensembling
Learning curves converged, both mediocre	Bias floor reached	Better features / model family — more data is wasted money
Learning curves still converging with a gap	Variance, curable	Collect more data
Fits change wildly across random seeds	High variance	Same as overfitting fixes
Test error can't go below some level no matter what	Irreducible noise	Accept it, or measure better data

Check your understanding

5 questions · free

1. Q1.A model has high training error AND high test error, with almost no gap between them. What's the diagnosis?

   AHigh variance — collect more dataBHigh bias — the model is too simple to capture the patternCIrreducible noise — nothing can be doneDData leakage

2. Q2.In the bootstrap demo, the six degree-9 curves disagreed wildly with each other. That disagreement is a direct picture of…

   ABiasBVarianceCIrreducible noiseDMulticollinearity

3. Q3.Your learning curve shows train and validation scores that have already converged — to a disappointing R² of 0.45. What's the best next step?

   ACollect 10× more dataBLower the learning rateCImprove the model or features — you've hit the bias floorDUse a smaller test set

4. Q4.Your KNN regressor is overfitting. Based on complexity reasoning, which change fights it?

   ADecrease n_neighbors to 1BIncrease n_neighbors so each prediction averages more pointsCRemove the test setDDuplicate the training rows

5. Q5.Even a perfect model can't reach zero test error on real data. Why?

   AGradient descent never fully convergesBThe target contains irreducible noise from measurement error and unmodeled factorsCR² is capped below 1 by scikit-learnDTest sets are always mislabeled

Exercise: Diagnose a tree with a validation curve

Decision trees (coming later in the course) have a complexity dial called max_depth. Using the same synthetic sine data as above, run validation_curve on a DecisionTreeRegressor with max_depth from 1 to 12 and plot train vs validation R². Identify the bias zone, the variance zone, and the best depth. Then answer in a comment: does increasing max_depth fight overfitting or cause it?

Next: a quiet culprit behind unstable models — features on wildly different scales, and the transforms that fix them.