{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Ensemble Learning\n",
    "\n",
    "The general theory behind combining models — voting classifiers, bagging any estimator, stacking with a meta-learner, and the bagging-vs-boosting map.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/ensemble-learning).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Random forests aren't a one-off trick — they're one member of a whole family\n",
    "of techniques that combine multiple models into something better than any\n",
    "single one. That family is called **ensemble learning**, and it powers most\n",
    "winning solutions on tabular data. In this lesson you'll build three kinds of\n",
    "ensembles — voting, bagging, and stacking — and set up the bagging-vs-boosting\n",
    "distinction that drives the rest of this module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Why several models beat one\n",
    "\n",
    "Suppose you have three classifiers, each independently right 70% of the time,\n",
    "and you take a majority vote. The vote is correct when at least two of the\n",
    "three are right:\n",
    "\n",
    "**P(majority correct) = 3 · (0.7² · 0.3) + 0.7³ = 0.441 + 0.343 = 0.784**\n",
    "\n",
    "Three mediocre models, one 78.4% ensemble — and the effect compounds with more\n",
    "voters. But reread the assumption: *independently* right. If all three models\n",
    "make the **same** mistakes, the vote just repeats those mistakes with more\n",
    "confidence. Ensembles only work when the members are both\n",
    "\n",
    "1. **better than random**, and\n",
    "2. **diverse** — their errors are (at least partly) uncorrelated.\n",
    "\n",
    "Everything in this lesson is a different strategy for manufacturing that\n",
    "diversity: use different *algorithms* (voting), different *data samples*\n",
    "(bagging), or a *learned combination* of both (stacking)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## Voting: different algorithms, one ballot\n",
    "\n",
    "The most direct ensemble: train a few genuinely different models — say\n",
    "logistic regression (linear boundary), KNN (local boundary), and a decision\n",
    "tree (rectangular boundary) — and combine their predictions. Two flavors:\n",
    "\n",
    "- **Hard voting** — each model casts one vote for a class; majority wins.\n",
    "- **Soft voting** — average the models' predicted *probabilities* and pick\n",
    "  the highest. A model that's 99% sure counts for more than one that's 51%\n",
    "  sure, so soft voting usually edges out hard voting (when the members\n",
    "  produce calibrated probabilities)."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "X, y = make_moons(n_samples=500, noise=0.30, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, stratify=y, random_state=42)\n",
    "\n",
    "members = [\n",
    "    (\"logreg\", LogisticRegression()),\n",
    "    (\"knn\", KNeighborsClassifier(n_neighbors=7)),\n",
    "    (\"tree\", DecisionTreeClassifier(max_depth=5, random_state=42)),\n",
    "]\n",
    "\n",
    "for name, clf in members:\n",
    "    clf.fit(X_train, y_train)\n",
    "    print(f\"{name:6s}: {clf.score(X_test, y_test):.3f}\")\n",
    "\n",
    "hard = VotingClassifier(members, voting=\"hard\").fit(X_train, y_train)\n",
    "soft = VotingClassifier(members, voting=\"soft\").fit(X_train, y_train)\n",
    "print(f\"\\\\nhard voting: {hard.score(X_test, y_test):.3f}\")\n",
    "print(f\"soft voting: {soft.score(X_test, y_test):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "The ensemble matches or beats its best member — even though one member\n",
    "(logistic regression) is clearly too simple for moon-shaped data. Its votes\n",
    "still help on the samples where the boundary happens to be locally linear,\n",
    "and the other two cover the curves."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "## Bagging: same algorithm, different data\n",
    "\n",
    "**Bagging** (bootstrap aggregating) manufactures diversity from data instead\n",
    "of from algorithms: train N copies of the *same* estimator, each on a\n",
    "different bootstrap sample, and aggregate. You already know its most famous\n",
    "incarnation — a random forest is bagged decision trees *plus* random feature\n",
    "subsets. But scikit-learn's `BaggingClassifier` will bag anything:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0007",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "X, y = make_moons(n_samples=500, noise=0.30, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, stratify=y, random_state=42)\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)\n",
    "bag = BaggingClassifier(DecisionTreeClassifier(random_state=42),\n",
    "                        n_estimators=50, random_state=42)\n",
    "bag.fit(X_train, y_train)\n",
    "\n",
    "print(f\"one deep tree  : {tree.score(X_test, y_test):.3f}\")\n",
    "print(f\"50 bagged trees: {bag.score(X_test, y_test):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "Bagging shines with **high-variance** base models — deep trees, KNN with tiny\n",
    "k — because averaging is a variance-reduction machine. Bagging a very stable\n",
    "model (like logistic regression) barely helps: fifty nearly identical models\n",
    "vote nearly identically. Set `bootstrap=False` and you get *pasting* (sampling\n",
    "without replacement); `max_features` gives you random feature subsets for any\n",
    "estimator, forest-style."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "## Stacking: let a model learn how to combine\n",
    "\n",
    "Voting weighs every member equally (or with weights you hand-pick). Stacking\n",
    "asks: *why not learn the combination?* Train the base models, collect their\n",
    "predictions, and feed those predictions as features into a **meta-learner**\n",
    "(often logistic regression) that learns which member to trust in which\n",
    "situation. To avoid leakage, `StackingClassifier` generates the base models'\n",
    "training predictions with internal cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0010",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import StackingClassifier\n",
    "\n",
    "X, y = make_moons(n_samples=500, noise=0.30, random_state=42)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, stratify=y, random_state=42)\n",
    "\n",
    "stack = StackingClassifier(\n",
    "    estimators=[\n",
    "        (\"knn\", KNeighborsClassifier(n_neighbors=7)),\n",
    "        (\"tree\", DecisionTreeClassifier(max_depth=5, random_state=42)),\n",
    "        (\"logreg\", LogisticRegression()),\n",
    "    ],\n",
    "    final_estimator=LogisticRegression(),\n",
    "    cv=3,\n",
    ")\n",
    "stack.fit(X_train, y_train)\n",
    "print(f\"stacking: {stack.score(X_test, y_test):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "Stacking is the heavyweight of the family — more training cost, more moving\n",
    "parts, and a real risk of overfitting on small datasets — but on large, messy\n",
    "problems a well-built stack is hard to beat, which is why it dominates Kaggle\n",
    "leaderboards."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "> **Diversity is the budget you spend**\n",
    "> \n",
    "> All three techniques answer the same question differently: where does\n",
    "> disagreement come from? Voting buys it with different algorithms, bagging\n",
    "> with different data samples, stacking with both plus a learned referee. An\n",
    "> ensemble of clones is just one model with extra compute."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Bagging vs boosting: the fork in the road\n",
    "\n",
    "There's one strategy we haven't touched: instead of training members\n",
    "**in parallel and independently**, train them **in sequence**, where each new\n",
    "model deliberately focuses on the mistakes of the ones before it. That's\n",
    "**boosting**, and it behaves very differently:\n",
    "\n",
    "| | Bagging (e.g. random forest) | Boosting (e.g. AdaBoost, XGBoost) |\n",
    "|---|---|---|\n",
    "| Training | Parallel, independent members | Sequential — each member fixes the last one's errors |\n",
    "| Base models | Strong, high-variance (deep trees) | Weak, high-bias (shallow trees, stumps) |\n",
    "| Mainly reduces | Variance | Bias |\n",
    "| More members | Never overfits, just plateaus | Can overfit — member count needs tuning |\n",
    "| Sensitivity to noisy labels | Low | Higher (errors get chased) |\n",
    "| Combination rule | Equal vote / average | Weighted sum built during training |\n",
    "\n",
    "A useful slogan: **bagging turns strong-but-unstable learners into a good\n",
    "learner; boosting turns weak learners into a good learner.** The next two\n",
    "lessons walk down the boosting branch — starting with the algorithm that\n",
    "invented it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Assemble a voting classifier for breast cancer\n",
    "\n",
    "Build a `VotingClassifier` on the breast cancer dataset from three members:\n",
    "a scaled `LogisticRegression`, a scaled `KNeighborsClassifier`, and a\n",
    "`DecisionTreeClassifier` with `max_depth=4`. Print each member's individual\n",
    "test accuracy, then the hard-voting and soft-voting ensemble accuracies. Does\n",
    "the ensemble beat the best individual — and which voting mode wins?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0015",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, stratify=y, random_state=42)\n",
    "\n",
    "members = [\n",
    "    (\"logreg\", make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))),\n",
    "    (\"knn\", make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=7))),\n",
    "    (\"tree\", DecisionTreeClassifier(max_depth=4, random_state=42)),\n",
    "]\n",
    "\n",
    "for name, clf in members:\n",
    "    clf.fit(X_train, y_train)\n",
    "    print(f\"{name:6s}: {clf.score(X_test, y_test):.3f}\")\n",
    "\n",
    "for mode in [\"hard\", \"soft\"]:\n",
    "    vote = VotingClassifier(members, voting=mode).fit(X_train, y_train)\n",
    "    print(f\"{mode} voting: {vote.score(X_test, y_test):.3f}\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "Next up: AdaBoost — the original boosting algorithm, where each new model is\n",
    "trained to obsess over exactly the samples the previous ones got wrong."
   ]
  }
 ]
}