Feature Scaling & Transforms

A dataset rarely arrives ready for modeling: one column measures age (20–70), another income (thousands to millions), a third is skewed so hard that 95% of values huddle near zero. Many algorithms quietly assume features live on comparable scales — and misbehave when they don't. This lesson covers why scaling matters, which scaler to use when, and how power transforms fix skewed distributions — plus the one mistake (fitting scalers on test data) that silently invalidates your results.

Why scale at all?

Three concrete reasons:

1. Gradient descent converges better. Remember in the linear-regression lesson how b learned slower than w? With features of wildly different magnitudes, the loss surface becomes a long, narrow valley: the gradient is steep in one direction and nearly flat in another, so a learning rate that's safe for one weight is glacial for another. Scaling rounds the valley into a bowl, and the same steps reach the bottom far faster.
2. Distance-based models need it. KNN, K-Means, and SVMs compare points by distance. If income ranges over millions and age over decades, distance is effectively just income — age becomes invisible.
3. Regularization must be fair. The next lesson penalizes large weights. But a feature measured in millimeters needs a coefficient 1000× larger than the same feature in meters — penalizing raw coefficient size across unscaled features punishes features for their units, not their usefulness.

(Tree-based models are the notable exception — they split on thresholds and don't care about scale.)

Three scalers, one outlier

scikit-learn's big three:

• StandardScaler — subtract the mean, divide by the standard deviation. Result: mean 0, std 1. The default choice, best when data is roughly bell-shaped.
• MinMaxScaler — shift the minimum to 0, stretch the maximum to 1. Intuitive bounded range, but one outlier squeezes everyone else into a tiny sliver.
• RobustScaler — subtract the median, divide by the interquartile range (IQR). Quantiles barely move when an outlier appears, so it's the scaler of choice for contaminated data.

Watch how a single outlier affects each:

Python — runs in your browser

Run

import numpy as np from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler rng = np.random.default_rng(42) clean = rng.normal(50, 10, 200) dirty = np.append(clean, 500.0) # one data-entry error def report(name, x): print(f"{name:28s} mean={x.mean():7.2f} std={x.std():6.2f} " f"min={x.min():6.2f} max={x.max():6.2f}") for label, data in [("clean", clean), ("with outlier", dirty)]: X = data.reshape(-1, 1) print(f"--- {label} ---") report("raw", data) report("StandardScaler", StandardScaler().fit_transform(X).ravel()) report("MinMaxScaler", MinMaxScaler().fit_transform(X).ravel()) report("RobustScaler", RobustScaler().fit_transform(X).ravel()) print()

Look at the MinMax row in the dirty block: the outlier grabbed max = 1.0 for itself and crushed all 200 real points below ~0.1 — most of the scale is wasted on one bad value. StandardScaler suffered too (the outlier inflated the std, shrinking everyone). RobustScaler's output is nearly identical in both blocks — the median and IQR barely noticed.

Skewed data: transform before (or instead of) scaling

Scaling shifts and stretches, but it can't change a distribution's shape. Income, prices, trip distances — many real features are right-skewed: a heavy pile near zero and a long tail. Linear models and standardization both work better when values are roughly symmetric. The fix is a nonlinear transform:

• log — the classic for positive right-skewed data (np.log1p handles zeros).
• Box-Cox — a family of power transforms that learns the best exponent; requires strictly positive data.
• Yeo-Johnson — Box-Cox's sibling that also accepts zero and negative values. scikit-learn's PowerTransformer default.

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.preprocessing import PowerTransformer rng = np.random.default_rng(0) skewed = rng.lognormal(mean=2.0, sigma=0.8, size=1000) # right-skewed, positive X = skewed.reshape(-1, 1) logged = np.log1p(skewed) boxcox = PowerTransformer(method="box-cox").fit_transform(X).ravel() yeojohnson = PowerTransformer(method="yeo-johnson").fit_transform(X).ravel() fig, axes = plt.subplots(1, 4, figsize=(11, 2.8)) for ax, data, title in [ (axes[0], skewed, "raw (skewed)"), (axes[1], logged, "log1p"), (axes[2], boxcox, "Box-Cox"), (axes[3], yeojohnson, "Yeo-Johnson"), ]: ax.hist(data, bins=40, color="steelblue") ax.set_title(title, fontsize=10) ax.set_yticks([]) plt.tight_layout() plt.show()

The raw histogram slumps against zero with a long tail; all three transforms produce a much more symmetric bell. PowerTransformer even standardizes the output for you (mean 0, std 1) by default, so it often replaces the scaler entirely for skewed columns. This works on targets too — if y is skewed (like the taxi fares in this module's exercises), modeling log(y) often fixes the funnel-shaped residual plots you learned to spot in lesson 1.

QuantileTransformer, briefly: it maps values to their quantiles, forcing any distribution into a uniform (or normal) shape. It's a blunt but effective instrument for very messy features — just know it's non-linear and rank-based, so it distorts distances within the tails.

Fit on train, transform everywhere

Scalers are learned from data — the mean, the min/max, the quantiles are statistics. If you compute them on the full dataset before splitting, information about the test set bleeds into training. That's data leakage, and it makes your evaluation optimistic.

The foolproof way to obey this rule is to put the scaler inside a Pipeline. Then fit only ever sees training data, cross-validation re-fits the scaler per fold automatically, and you can't leak even if you try:

Python — runs in your browser

Run

from sklearn.datasets import load_diabetes from sklearn.model_selection import train_test_split, cross_val_score from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.neighbors import KNeighborsRegressor X, y = load_diabetes(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42) model = Pipeline([ ("scaler", StandardScaler()), # fit on train folds only, automatically ("knn", KNeighborsRegressor(n_neighbors=12)), ]) cv = cross_val_score(model, X_train, y_train, cv=5) model.fit(X_train, y_train) print(f"CV R2: {cv.mean():.3f} +/- {cv.std():.3f}") print(f"test R2: {model.score(X_test, y_test):.3f}")

From now on in this course, preprocessing always lives inside the pipeline — it's not just tidier, it's the only leak-proof way to work.

Choosing quickly

Data looks like	Use
Roughly bell-shaped	StandardScaler
Need a bounded 0–1 range, no outliers (e.g., pixel values)	MinMaxScaler
Contains outliers you can't remove	RobustScaler
Right-skewed, strictly positive	log / Box-Cox
Skewed with zeros or negatives	Yeo-Johnson (PowerTransformer)
Bizarre multi-modal mess	QuantileTransformer

Check your understanding

4 questions · free

1. Q1.Why does feature scaling speed up gradient descent?

   AIt reduces the number of featuresBIt turns the loss surface's narrow valley into a rounder bowl, so one learning rate works for all weightsCIt removes the need for a bias termDIt lowers the irreducible noise

2. Q2.Your feature has a few extreme outliers you can't remove. Which scaler distorts the non-outlier points the least?

   AMinMaxScalerBStandardScalerCRobustScalerDThey're all identical

3. Q3.You want to power-transform a feature containing negative values. Which method works?

   ABox-CoxBYeo-JohnsonCNeither — power transforms need positive dataDlog

4. Q4.A colleague fits StandardScaler on the full dataset, then splits into train/test. What's wrong?

   ANothing — the scaler is deterministicBThe test set's statistics leaked into preprocessing, making evaluation optimisticCStandardScaler only works on training data by designDThe split must happen on scaled data

Exercise: Does scaling rescue KNN?

Load load_diabetes and deliberately sabotage it: multiply one feature column by 1000 so it dominates all distances. Split into train/test, then fit a KNeighborsRegressor twice — once on the raw sabotaged data, once inside a Pipeline with StandardScaler. Compare test R². Explain in a comment why the unscaled version suffers.

With features on a fair, common scale, we can finally penalize model weights fairly too — next: regularization with Ridge, Lasso, and ElasticNet.