Regression Metrics

Before you fit a single regression model, you need an answer to a deceptively simple question: what does "good" mean when you're predicting a number? A classifier is right or wrong; a regression model is off by some amount, and how you summarize those amounts changes which model looks best. In this lesson you'll meet the standard regression metrics, learn where each one shines or lies, and use residual plots to see what a single score can't tell you.

It all starts with residuals

Suppose a model predicts house prices. For each house we have the true price y and the prediction ŷ. The residual is the gap between them:

residual = y − ŷ

A positive residual means the model predicted too low; negative means too high. Every regression metric is just a different way of squashing a list of residuals into a single number — and each way of squashing makes a different trade-off.

MAE, MSE, and RMSE

The three workhorses:

• Mean Absolute Error (MAE) — average of the absolute residuals. "On average, we're off by this much." Same units as the target.
• Mean Squared Error (MSE) — average of the squared residuals. Punishes big misses much harder, but the units are squared (dollars² — awkward).
• Root Mean Squared Error (RMSE) — the square root of MSE. Back in the target's units, but still outlier-sensitive because the squaring happened first.

The key behavioral difference is outlier sensitivity. Watch what a single terrible prediction does to each metric:

Python — runs in your browser

Run

import numpy as np from sklearn.metrics import mean_absolute_error, mean_squared_error y_true = np.array([10.0, 12.0, 15.0, 18.0, 20.0, 22.0, 25.0, 28.0]) y_good = np.array([11.0, 11.0, 16.0, 17.0, 21.0, 21.0, 26.0, 27.0]) # all small misses y_out = np.array([11.0, 11.0, 16.0, 17.0, 21.0, 21.0, 26.0, 48.0]) # one huge miss for name, y_pred in [("small misses only", y_good), ("with one outlier", y_out)]: mae = mean_absolute_error(y_true, y_pred) mse = mean_squared_error(y_true, y_pred) rmse = np.sqrt(mse) print(f"{name:19s} MAE={mae:5.2f} MSE={mse:6.2f} RMSE={rmse:5.2f}")

One bad prediction out of eight barely moved the MAE (each point contributes linearly), but MSE exploded and RMSE tripled — the squared 20-unit miss dominates everything. Neither behavior is "correct":

• If big misses are disproportionately costly in your problem (a 20% error in a drug dose is not 2× as bad as a 10% error), RMSE's sensitivity is a feature.
• If your data contains a few noisy, unrepresentative extremes you don't want dominating model selection, MAE is the more robust summary.

R²: better than guessing the mean?

MAE and RMSE are in the target's units, which is great for stakeholders but hard to compare across problems. R² (the coefficient of determination) fixes that by comparing your model against the dumbest possible baseline: always predicting the mean of y.

• R² = 1 — perfect predictions, zero error.
• R² = 0 — your model is exactly as good as predicting the mean.
• R² negative — your model is worse than predicting the mean. Yes, this happens, and it's a loud alarm bell.

Python — runs in your browser

Run

import numpy as np from sklearn.metrics import r2_score rng = np.random.default_rng(42) y_true = rng.uniform(10, 50, 30) y_close = y_true + rng.normal(0, 2, 30) # good model y_mean = np.full(30, y_true.mean()) # baseline: always the mean y_bad = 60 - y_true + rng.normal(0, 2, 30) # confidently wrong print(f"good model R2 = {r2_score(y_true, y_close):7.3f}") print(f"predict mean R2 = {r2_score(y_true, y_mean):7.3f}") print(f"terrible model R2 = {r2_score(y_true, y_bad):7.3f}")

That last score is far below zero: the model's errors are bigger than the spread of the data itself. When you see negative R² on a test set, the model learned something that actively misleads it — often a sign of leakage, a bug, or severe overfitting.

MAPE: intuitive, with a fatal flaw

Mean Absolute Percentage Error reports the average relative error — "off by 12% on average" — which non-technical audiences love. But it divides by the true value, so it breaks when y is zero or near zero:

Python — runs in your browser

Run

import numpy as np from sklearn.metrics import mean_absolute_percentage_error y_true = np.array([100.0, 200.0, 300.0]) y_pred = np.array([110.0, 190.0, 315.0]) print(f"normal targets: MAPE = {mean_absolute_percentage_error(y_true, y_pred):.3f}") y_true2 = np.array([100.0, 200.0, 0.001]) # one near-zero target y_pred2 = np.array([110.0, 190.0, 1.0]) # tiny absolute miss... print(f"near-zero target: MAPE = {mean_absolute_percentage_error(y_true2, y_pred2):.3f}")

A prediction that missed by one unit blew the MAPE up to astronomical levels, because the miss was divided by 0.001. Avoid MAPE when the target can be zero or close to it (demand forecasting with zero-sale days is the classic trap).

Which metric should you report?

Situation	Reach for
You want an error in the target's units, robust to outliers	MAE
Big misses are disproportionately costly	RMSE
You're optimizing / comparing models mathematically	MSE (smooth, differentiable)
You need a unit-free "how much better than baseline" score	R²
A relative "% off" story for stakeholders (targets far from 0)	MAPE

In practice, report two: one absolute metric (MAE or RMSE) plus R². They answer different questions — "how far off are we?" and "how much of the variation do we explain?"

Residual plots: seeing what the score hides

A single number can't tell you where the model fails. A residual plot — predictions on the x-axis, residuals on the y-axis — can. For a healthy model the residuals look like a structureless, symmetric band around zero. Patterns mean trouble:

• A curve or U-shape — the model is missing a nonlinear relationship.
• A funnel (spread grows with prediction) — heteroscedasticity; errors grow with the target, often fixed by transforming y (a later lesson).
• Train residuals tiny, test residuals huge — overfitting.

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_diabetes from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split X, y = load_diabetes(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42) model = LinearRegression().fit(X_train, y_train) fig, axes = plt.subplots(1, 2, figsize=(9, 3.5), sharey=True) for ax, Xs, ys, title in [(axes[0], X_train, y_train, "train"), (axes[1], X_test, y_test, "test")]: pred = model.predict(Xs) ax.scatter(pred, ys - pred, alpha=0.6, s=18) ax.axhline(0, color="crimson", lw=1.5) ax.set_xlabel("predicted") ax.set_title(f"{title} residuals") axes[0].set_ylabel("residual (y - pred)") plt.tight_layout() plt.show()

Both panels show a roughly symmetric cloud around zero with similar spread — no obvious curvature, no funnel, and train/test look alike. That's the visual signature of a model that is honest, if not spectacular. Get in the habit of looking at this plot every time you fit a regressor; it catches problems that MAE and R² silently average away.

Check your understanding

5 questions · free

1. Q1.Your predictions are mostly accurate, but one sample is wildly off. Which metric changes the most?

   AMAEBRMSECR² moves more than bothDThey all change equally

2. Q2.A model scores R² = −0.4 on the test set. What does that mean?

   AThe model explains 40% of the varianceBR² can't be negative — it's a bug in the codeCThe model performs worse than always predicting the mean of yDThe correlation between y and ŷ is −0.4

3. Q3.Why is MAPE risky for a demand-forecasting problem where some days have zero sales?

   AMAPE ignores negative residualsBPercentage errors divide by the true value, which explodes near zeroCMAPE only works for classificationDMAPE requires normally distributed targets

4. Q4.A residual plot shows the spread of residuals growing steadily as predictions increase. What is this pattern called?

   AMulticollinearityBHeteroscedasticityCUnderfittingDAutocorrelation

5. Q5.You must report one error metric in the same units as house prices, and the dataset has a few extreme luxury sales you consider noise. Best choice?

   AMSEBR²CMAEDMAPE

Exercise: Judge two models by more than one number

Load load_diabetes, split into train/test, and fit a LinearRegression. Call its test predictions Model A. Create Model B by copying Model A's predictions and adding 300 to a single one (simulating one terrible miss). Compute MAE, RMSE, and R² for both. Which metrics barely notice the corruption, and which ones panic? Does the ranking of "how bad is Model B" depend on the metric you choose?

Now that you can measure what "good" means, it's time to earn a score of your own — next: fit your first model with linear regression and gradient descent.