{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Feature Scaling & Transforms\n", "\n", "Put features on a common scale with StandardScaler, MinMaxScaler, and RobustScaler, and tame skewed distributions with log and power transforms.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/scaling-and-transforms).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "A dataset rarely arrives ready for modeling: one column measures age (20–70),\n", "another income (thousands to millions), a third is skewed so hard that 95% of\n", "values huddle near zero. Many algorithms quietly assume features live on\n", "comparable scales — and misbehave when they don't. This lesson covers **why**\n", "scaling matters, **which** scaler to use when, and how **power transforms**\n", "fix skewed distributions — plus the one mistake (fitting scalers on test\n", "data) that silently invalidates your results." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Why scale at all?\n", "\n", "Three concrete reasons:\n", "\n", "1. **Gradient descent converges better.** Remember in the linear-regression\n", " lesson how `b` learned slower than `w`? With features of wildly different\n", " magnitudes, the loss surface becomes a long, narrow valley: the gradient\n", " is steep in one direction and nearly flat in another, so a learning rate\n", " that's safe for one weight is glacial for another. Scaling rounds the\n", " valley into a bowl, and the same steps reach the bottom far faster.\n", "2. **Distance-based models need it.** KNN, K-Means, and SVMs compare points\n", " by distance. If income ranges over millions and age over decades, distance\n", " is effectively *just income* — age becomes invisible.\n", "3. **Regularization must be fair.** The next lesson penalizes large weights.\n", " But a feature measured in millimeters needs a coefficient 1000× larger\n", " than the same feature in meters — penalizing raw coefficient size across\n", " unscaled features punishes features for their units, not their usefulness.\n", "\n", "(Tree-based models are the notable exception — they split on thresholds and\n", "don't care about scale.)" ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Three scalers, one outlier\n", "\n", "scikit-learn's big three:\n", "\n", "- **StandardScaler** — subtract the mean, divide by the standard deviation.\n", " Result: mean 0, std 1. The default choice, best when data is roughly\n", " bell-shaped.\n", "- **MinMaxScaler** — shift the minimum to 0, stretch the maximum to 1.\n", " Intuitive bounded range, but **one outlier squeezes everyone else** into a\n", " tiny sliver.\n", "- **RobustScaler** — subtract the *median*, divide by the *interquartile\n", " range (IQR)*. Quantiles barely move when an outlier appears, so it's the\n", " scaler of choice for contaminated data.\n", "\n", "Watch how a single outlier affects each:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler\n", "\n", "rng = np.random.default_rng(42)\n", "clean = rng.normal(50, 10, 200)\n", "dirty = np.append(clean, 500.0) # one data-entry error\n", "\n", "def report(name, x):\n", " print(f\"{name:28s} mean={x.mean():7.2f} std={x.std():6.2f} \"\n", " f\"min={x.min():6.2f} max={x.max():6.2f}\")\n", "\n", "for label, data in [(\"clean\", clean), (\"with outlier\", dirty)]:\n", " X = data.reshape(-1, 1)\n", " print(f\"--- {label} ---\")\n", " report(\"raw\", data)\n", " report(\"StandardScaler\", StandardScaler().fit_transform(X).ravel())\n", " report(\"MinMaxScaler\", MinMaxScaler().fit_transform(X).ravel())\n", " report(\"RobustScaler\", RobustScaler().fit_transform(X).ravel())\n", " print()" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "Look at the MinMax row in the dirty block: the outlier grabbed `max = 1.0`\n", "for itself and crushed all 200 real points below ~0.1 — most of the scale is\n", "wasted on one bad value. StandardScaler suffered too (the outlier inflated\n", "the std, shrinking everyone). RobustScaler's output is nearly identical in\n", "both blocks — the median and IQR barely noticed." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Skewed data: transform before (or instead of) scaling\n", "\n", "Scaling shifts and stretches, but it can't change a distribution's **shape**.\n", "Income, prices, trip distances — many real features are right-skewed: a heavy\n", "pile near zero and a long tail. Linear models and standardization both work\n", "better when values are roughly symmetric. The fix is a **nonlinear\n", "transform**:\n", "\n", "- **log** — the classic for positive right-skewed data (`np.log1p` handles\n", " zeros).\n", "- **Box-Cox** — a family of power transforms that *learns* the best exponent;\n", " **requires strictly positive** data.\n", "- **Yeo-Johnson** — Box-Cox's sibling that also accepts zero and negative\n", " values. scikit-learn's `PowerTransformer` default." ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.preprocessing import PowerTransformer\n", "\n", "rng = np.random.default_rng(0)\n", "skewed = rng.lognormal(mean=2.0, sigma=0.8, size=1000) # right-skewed, positive\n", "\n", "X = skewed.reshape(-1, 1)\n", "logged = np.log1p(skewed)\n", "boxcox = PowerTransformer(method=\"box-cox\").fit_transform(X).ravel()\n", "yeojohnson = PowerTransformer(method=\"yeo-johnson\").fit_transform(X).ravel()\n", "\n", "fig, axes = plt.subplots(1, 4, figsize=(11, 2.8))\n", "for ax, data, title in [\n", " (axes[0], skewed, \"raw (skewed)\"),\n", " (axes[1], logged, \"log1p\"),\n", " (axes[2], boxcox, \"Box-Cox\"),\n", " (axes[3], yeojohnson, \"Yeo-Johnson\"),\n", "]:\n", " ax.hist(data, bins=40, color=\"steelblue\")\n", " ax.set_title(title, fontsize=10)\n", " ax.set_yticks([])\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "The raw histogram slumps against zero with a long tail; all three transforms\n", "produce a much more symmetric bell. `PowerTransformer` even standardizes the\n", "output for you (mean 0, std 1) by default, so it often replaces the\n", "scaler entirely for skewed columns. This works on **targets** too — if `y` is\n", "skewed (like the taxi fares in this module's exercises), modeling `log(y)`\n", "often fixes the funnel-shaped residual plots you learned to spot in lesson 1.\n", "\n", "**QuantileTransformer**, briefly: it maps values to their quantiles, forcing\n", "*any* distribution into a uniform (or normal) shape. It's a blunt but\n", "effective instrument for very messy features — just know it's non-linear and\n", "rank-based, so it distorts distances within the tails." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "## Fit on train, transform everywhere\n", "\n", "Scalers are *learned* from data — the mean, the min/max, the quantiles are\n", "statistics. If you compute them on the full dataset before splitting,\n", "information about the test set bleeds into training. That's **data leakage**,\n", "and it makes your evaluation optimistic." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "> **Never fit a scaler on test data**\n", "> \n", "> Always: `scaler.fit_transform(X_train)` then `scaler.transform(X_test)` — fit\n", "> on train **only**, reuse those statistics for the test set. The test set must\n", "> be processed exactly as truly-new data would be: with statistics it had no\n", "> part in computing." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "The foolproof way to obey this rule is to put the scaler **inside a\n", "Pipeline**. Then `fit` only ever sees training data, cross-validation\n", "re-fits the scaler per fold automatically, and you can't leak even if you try:" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_diabetes\n", "from sklearn.model_selection import train_test_split, cross_val_score\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.neighbors import KNeighborsRegressor\n", "\n", "X, y = load_diabetes(return_X_y=True)\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n", "\n", "model = Pipeline([\n", " (\"scaler\", StandardScaler()), # fit on train folds only, automatically\n", " (\"knn\", KNeighborsRegressor(n_neighbors=12)),\n", "])\n", "\n", "cv = cross_val_score(model, X_train, y_train, cv=5)\n", "model.fit(X_train, y_train)\n", "print(f\"CV R2: {cv.mean():.3f} +/- {cv.std():.3f}\")\n", "print(f\"test R2: {model.score(X_test, y_test):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "From now on in this course, preprocessing always lives inside the pipeline —\n", "it's not just tidier, it's the only leak-proof way to work." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "## Choosing quickly\n", "\n", "| Data looks like | Use |\n", "|---|---|\n", "| Roughly bell-shaped | StandardScaler |\n", "| Need a bounded 0–1 range, no outliers (e.g., pixel values) | MinMaxScaler |\n", "| Contains outliers you can't remove | RobustScaler |\n", "| Right-skewed, strictly positive | log / Box-Cox |\n", "| Skewed with zeros or negatives | Yeo-Johnson (PowerTransformer) |\n", "| Bizarre multi-modal mess | QuantileTransformer |" ] }, { "cell_type": "markdown", "id": "cell-0015", "metadata": {}, "source": [ "### 🏋️ Exercise — Does scaling rescue KNN?\n", "\n", "Load `load_diabetes` and deliberately sabotage it: multiply one feature column\n", "by 1000 so it dominates all distances. Split into train/test, then fit a\n", "`KNeighborsRegressor` twice — once on the raw sabotaged data, once inside a\n", "`Pipeline` with `StandardScaler`. Compare test R². Explain in a comment *why*\n", "the unscaled version suffers." ] }, { "cell_type": "code", "id": "cell-0016", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.datasets import load_diabetes\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.neighbors import KNeighborsRegressor\n", "\n", "X, y = load_diabetes(return_X_y=True)\n", "X_bad = X.copy()\n", "X_bad[:, 2] *= 1000 # BMI column now dwarfs every other feature\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X_bad, y, test_size=0.25, random_state=42)\n", "\n", "raw = KNeighborsRegressor(n_neighbors=12).fit(X_train, y_train)\n", "print(f\"KNN, no scaling: R2 = {raw.score(X_test, y_test):.3f}\")\n", "\n", "scaled = Pipeline([\n", " (\"scaler\", StandardScaler()),\n", " (\"knn\", KNeighborsRegressor(n_neighbors=12)),\n", "]).fit(X_train, y_train)\n", "print(f\"KNN, with scaling: R2 = {scaled.score(X_test, y_test):.3f}\")\n", "\n", "# Without scaling, distances are computed almost entirely on the inflated\n", "# column, so the other nine features are ignored. Scaling restores them.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0018", "metadata": {}, "source": [ "With features on a fair, common scale, we can finally penalize model weights\n", "fairly too — next: regularization with Ridge, Lasso, and ElasticNet." ] } ] }