{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# EDA & Feature Engineering\n", "\n", "Interrogate a dataset before modeling — distributions, missing values, outliers, target balance — then craft features that make models smarter.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/eda-feature-engineering).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "Models are only as good as the data you feed them, and most real-world\n", "accuracy gains come from understanding and reshaping that data — not from\n", "fancier algorithms. This lesson covers the two crafts that dominate working\n", "data science: **exploratory data analysis (EDA)**, where you interrogate the\n", "data before modeling, and **feature engineering**, where you turn raw columns\n", "into signals a model can actually use." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## What EDA is looking for\n", "\n", "EDA isn't aimless plotting — it's a checklist of questions, each of which\n", "changes what you do next:\n", "\n", "- **What's the target, and is it balanced?** A 95/5 class split changes your\n", " metrics and your baseline (next lesson digs into this).\n", "- **What type is each feature?** Numeric, categorical, ordinal, datetime,\n", " free text — each needs different preprocessing.\n", "- **How much is missing, and where?** A column that's 80% empty is probably a\n", " drop; a column missing 2% is an impute.\n", "- **What do the distributions look like?** Skewed features (like incomes or\n", " fares) may benefit from a log transform or binning.\n", "- **Any outliers?** Are they data-entry errors or real (and important) rare\n", " events?\n", "- **Which features relate to the target?** Group means, correlations, and\n", " simple plots reveal which columns carry signal.\n", "\n", "Let's run a compact EDA on a synthetic passenger dataset — a small ship\n", "manifest with survival labels, in the spirit of the famous Titanic problem:" ] }, { "cell_type": "code", "id": "cell-0003", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "rng = np.random.default_rng(42)\n", "n = 500\n", "pclass = rng.choice([1, 2, 3], n, p=[0.25, 0.20, 0.55])\n", "sex = rng.choice([\"male\", \"female\"], n, p=[0.65, 0.35])\n", "age = rng.normal(30, 13, n).clip(1, 80).round(0)\n", "age[rng.random(n) < 0.20] = np.nan # 20% of ages unknown\n", "fare = (rng.lognormal(2.2, 0.8, n) * (4 - pclass)).round(2)\n", "family = rng.poisson(0.9, n)\n", "p_survive = 0.12 + 0.55 * (sex == \"female\") + 0.15 * (pclass == 1)\n", "survived = (rng.random(n) < p_survive).astype(int)\n", "\n", "df = pd.DataFrame({\"pclass\": pclass, \"sex\": sex, \"age\": age,\n", " \"fare\": fare, \"family\": family, \"survived\": survived})\n", "\n", "print(\"--- target balance ---\")\n", "print(df[\"survived\"].value_counts(normalize=True).round(3))\n", "\n", "print(\"\\\\n--- missing values (fraction per column) ---\")\n", "print(df.isna().mean().round(3))\n", "\n", "print(\"\\\\n--- numeric distributions ---\")\n", "print(df[[\"age\", \"fare\", \"family\"]].describe().round(1))\n", "\n", "print(\"\\\\n--- survival rate by group ---\")\n", "print(df.groupby(\"sex\")[\"survived\"].mean().round(3))\n", "print(df.groupby(\"pclass\")[\"survived\"].mean().round(3))" ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "Five minutes of EDA already wrote our modeling plan: the target is\n", "imbalanced (roughly one third survived), `age` needs imputation, `fare` is\n", "heavily right-skewed (compare its mean to its median), and `sex` and\n", "`pclass` clearly carry signal. On larger datasets you'd add histograms\n", "(`df[\"fare\"].hist()`), correlation matrices (`df.corr(numeric_only=True)`),\n", "and count plots per category — same questions, more pictures." ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "## Handling missing values\n", "\n", "Two families of fixes:\n", "\n", "- **Drop.** Remove rows (`df.dropna()`) when very few are affected, or drop a\n", " whole column when most of it is missing — there's little left to learn\n", " from. Always drop rows whose *target* is missing.\n", "- **Impute.** Fill numeric gaps with the median (robust to outliers) or mean;\n", " fill categorical gaps with the most frequent value. In scikit-learn that's\n", " `SimpleImputer`, which you met inside pipelines — and the pipeline is\n", " exactly where imputation belongs, so its statistics come from training\n", " folds only.\n", "\n", "A third, underrated option: add a boolean `age_missing` indicator column.\n", "Sometimes *the fact that a value is missing* is itself predictive." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Encoding categorical features\n", "\n", "Models compute with numbers, so categories must be encoded:\n", "\n", "- **One-hot encoding** (`OneHotEncoder`, or `pd.get_dummies` for quick\n", " exploration) creates one 0/1 column per category. Right choice for\n", " *nominal* categories with no order: port, city, color.\n", "- **Ordinal encoding** (`OrdinalEncoder`) maps categories to integers. Only\n", " right when the order is real — small, medium, large — because models will\n", " treat the numbers as ordered and evenly spaced.\n", "\n", "Beware one-hot exploding on high-cardinality columns (thousands of zip\n", "codes); grouping rare categories into an `\"other\"` bucket is a simple,\n", "effective fix." ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "## Creating new features\n", "\n", "This is where domain thinking beats algorithms. Classic moves:\n", "\n", "- **Boolean flags** — `is_alone` from a family count.\n", "- **Ratios and combinations** — fare per family member; total family size\n", " from siblings plus parents.\n", "- **Binning** — turn numeric `age` into categories like child / teen / adult,\n", " which can capture non-linear effects and tame outliers.\n", "- **Datetime parts** — from a timestamp, extract hour, weekday, month;\n", " \"purchases spike on weekends\" is invisible to a raw timestamp.\n", "- **Text extraction** — pull a title like \"Mr\" or \"Dr\" out of a name string." ] }, { "cell_type": "code", "id": "cell-0008", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "rng = np.random.default_rng(42)\n", "n = 500\n", "pclass = rng.choice([1, 2, 3], n, p=[0.25, 0.20, 0.55])\n", "sex = rng.choice([\"male\", \"female\"], n, p=[0.65, 0.35])\n", "age = rng.normal(30, 13, n).clip(1, 80).round(0)\n", "fare = (rng.lognormal(2.2, 0.8, n) * (4 - pclass)).round(2)\n", "family = rng.poisson(0.9, n)\n", "p_survive = 0.12 + 0.55 * (sex == \"female\") + 0.15 * (pclass == 1)\n", "survived = (rng.random(n) < p_survive).astype(int)\n", "df = pd.DataFrame({\"pclass\": pclass, \"sex\": sex, \"age\": age,\n", " \"fare\": fare, \"family\": family, \"survived\": survived})\n", "\n", "# 1. Boolean flag\n", "df[\"is_alone\"] = (df[\"family\"] == 0).astype(int)\n", "\n", "# 2. Ratio feature\n", "df[\"fare_per_person\"] = (df[\"fare\"] / (df[\"family\"] + 1)).round(2)\n", "\n", "# 3. Binning a numeric column\n", "df[\"age_group\"] = pd.cut(df[\"age\"], bins=[0, 12, 18, 40, 80],\n", " labels=[\"child\", \"teen\", \"adult\", \"senior\"])\n", "\n", "# 4. Datetime parts (synthetic boarding timestamps)\n", "boarding = pd.to_datetime(\"2026-04-01\") + pd.to_timedelta(\n", " rng.integers(0, 72, n), unit=\"h\")\n", "df[\"boarding_hour\"] = boarding.hour\n", "df[\"boarding_day\"] = boarding.day_name()\n", "\n", "print(df[[\"fare\", \"family\", \"is_alone\", \"fare_per_person\",\n", " \"age_group\", \"boarding_hour\", \"boarding_day\"]].head())\n", "\n", "print(\"\\\\nsurvival rate by age group:\")\n", "print(df.groupby(\"age_group\", observed=True)[\"survived\"].mean().round(3))" ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "After creating features, always sanity-check them against the target the way\n", "we did with `groupby` — a new feature that doesn't separate the target at all\n", "is probably not pulling its weight." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "## Scaling, one more time\n", "\n", "Recap from the KNN lesson: distance- and gradient-based models (KNN, SVMs,\n", "linear models with regularization, neural networks) need features on\n", "comparable scales — `StandardScaler` or `MinMaxScaler` inside your pipeline.\n", "Tree-based models split one feature at a time and don't care about scale.\n", "When in doubt, scale: it never hurts, and forgetting it can quietly cripple\n", "half your model zoo." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "> **Feature engineering can leak too**\n", "> \n", "> Any transformation whose parameters are computed from data — imputation\n", "> statistics, scaling means, bin edges chosen from quantiles, target-based\n", "> encodings — must be fit on the training split only. Even EDA can leak in a\n", "> subtle way: if you choose features by studying the full dataset, the test\n", "> set has quietly influenced your decisions. Do exploratory work on the\n", "> training split, and let pipelines handle the mechanics." ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "### 🏋️ Exercise — Engineer and evaluate two new features\n", "\n", "Regenerate the synthetic passenger dataset and engineer two new features:\n", "`is_child` (1 when age is below 12) and `family_bucket` (bin the family\n", "count into `alone`, `small` for 1–2, and `large` for 3+ using `pd.cut`).\n", "Print the survival rate for each group of both features, plus the group\n", "sizes. Do the new features separate the target — and are any groups too\n", "small to trust?" ] }, { "cell_type": "code", "id": "cell-0013", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import numpy as np\n", "import pandas as pd\n", "\n", "rng = np.random.default_rng(42)\n", "n = 500\n", "pclass = rng.choice([1, 2, 3], n, p=[0.25, 0.20, 0.55])\n", "sex = rng.choice([\"male\", \"female\"], n, p=[0.65, 0.35])\n", "age = rng.normal(30, 13, n).clip(1, 80).round(0)\n", "fare = (rng.lognormal(2.2, 0.8, n) * (4 - pclass)).round(2)\n", "family = rng.poisson(0.9, n)\n", "p_survive = 0.12 + 0.55 * (sex == \"female\") + 0.15 * (pclass == 1)\n", "survived = (rng.random(n) < p_survive).astype(int)\n", "df = pd.DataFrame({\"pclass\": pclass, \"sex\": sex, \"age\": age,\n", " \"fare\": fare, \"family\": family, \"survived\": survived})\n", "\n", "# Feature 1: is_child\n", "df[\"is_child\"] = (df[\"age\"] < 12).astype(int)\n", "\n", "# Feature 2: family size bucket\n", "df[\"family_bucket\"] = pd.cut(df[\"family\"], bins=[-1, 0, 2, 20],\n", " labels=[\"alone\", \"small\", \"large\"])\n", "\n", "print(\"survival rate by is_child:\")\n", "print(df.groupby(\"is_child\")[\"survived\"].mean().round(3))\n", "print(\"\\\\nsurvival rate by family bucket:\")\n", "print(df.groupby(\"family_bucket\", observed=True)[\"survived\"].mean().round(3))\n", "print(\"\\\\ngroup sizes:\")\n", "print(df[\"family_bucket\"].value_counts())\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0015", "metadata": {}, "source": [ "Next up: before celebrating any model's score, you need something to compare\n", "it against — baselines and benchmarks." ] } ] }