Polynomial Regression & Overfitting

Linear regression draws straight lines, but the world is full of curves: diminishing returns, saturation effects, U-shaped costs. Polynomial regression lets the same linear machinery fit curves — and in doing so introduces machine learning's most important dial: model complexity. Turn it too low and the model can't learn; turn it too high and it memorizes noise. This lesson is where you learn to feel that trade-off in your hands.

When a line isn't enough

Play with the degree slider below. Watch both curves: at degree 1 the line misses the shape entirely, around degree 3–4 the fit hugs the data's true pattern, and at high degrees the curve contorts itself through every training point while the test error climbs back up.

That U-shaped test-error curve is the single most important picture in this module. Keep it in mind as we rebuild it in code.

The trick: manufacture new features

Polynomial regression doesn't change the algorithm — it changes the inputs. Instead of feeding the model just x, we feed it x, x², x³, … and let plain linear regression find a weight for each:

ŷ = w₁·x + w₂·x² + w₃·x³ + b

scikit-learn's PolynomialFeatures manufactures those columns, and a Pipeline chains it with LinearRegression so the whole thing behaves like one model:

Degree is a complexity dial

Each extra degree gives the curve one more way to bend. A degree-1 model has two parameters; a degree-15 model has sixteen — enough to wiggle through nearly every training point. More flexibility always reduces training error, but past some point it stops modeling the signal and starts modeling the noise. That's overfitting, and you can only detect it by checking data the model never saw.

Train vs test: rebuilding the U-curve

Let's reproduce exactly what the playground showed you, in code. We fit every degree from 1 to 12 and track both errors:

The two curves tell the whole story:

• Training error only goes down. More degrees can never hurt the fit on data the model is allowed to see.
• Test error is a U. It falls while extra flexibility captures real structure, bottoms out, then rises as the model starts fitting noise.
• The gap between the curves is your overfitting meter — a small gap at the U's bottom, a chasm at high degrees.

Choosing the degree with validation

The rule is simple: pick the degree by performance on held-out data, never on training data — and among models with similar validation scores, prefer the simplest. In the source course's words: if degree 3 gives the best test score, use degree 3, not the flashier degree 10 that ties it.

One refinement: if you compare many degrees against the same test set, you slowly overfit to that test set too. The standard fix is cross-validation — split the training data into folds, average the score across them, and keep the test set untouched for a final honest estimate:

from sklearn.model_selection import cross_val_score
 
for d in range(1, 9):
    model = Pipeline([
        ("poly", PolynomialFeatures(d, include_bias=False)),
        ("lr", LinearRegression()),
    ])
    scores = cross_val_score(model, X_train, y_train, cv=5)
    print(f"degree {d}:  CV R2 = {scores.mean():.3f} +/- {scores.std():.3f}")

You'll formalize this workflow (and automate it with grid search) later in the course; for now the principle is what matters.

You've now seen overfitting with your own eyes — next we give it a proper theory: the bias–variance tradeoff.