Ensemble Learning

Random forests aren't a one-off trick — they're one member of a whole family of techniques that combine multiple models into something better than any single one. That family is called ensemble learning, and it powers most winning solutions on tabular data. In this lesson you'll build three kinds of ensembles — voting, bagging, and stacking — and set up the bagging-vs-boosting distinction that drives the rest of this module.

Why several models beat one

Suppose you have three classifiers, each independently right 70% of the time, and you take a majority vote. The vote is correct when at least two of the three are right:

P(majority correct) = 3 · (0.7² · 0.3) + 0.7³ = 0.441 + 0.343 = 0.784

Three mediocre models, one 78.4% ensemble — and the effect compounds with more voters. But reread the assumption: independently right. If all three models make the same mistakes, the vote just repeats those mistakes with more confidence. Ensembles only work when the members are both

1. better than random, and
2. diverse — their errors are (at least partly) uncorrelated.

Everything in this lesson is a different strategy for manufacturing that diversity: use different algorithms (voting), different data samples (bagging), or a learned combination of both (stacking).

Voting: different algorithms, one ballot

The most direct ensemble: train a few genuinely different models — say logistic regression (linear boundary), KNN (local boundary), and a decision tree (rectangular boundary) — and combine their predictions. Two flavors:

• Hard voting — each model casts one vote for a class; majority wins.
• Soft voting — average the models' predicted probabilities and pick the highest. A model that's 99% sure counts for more than one that's 51% sure, so soft voting usually edges out hard voting (when the members produce calibrated probabilities).

Python — runs in your browser

Run

from sklearn.datasets import make_moons from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import VotingClassifier X, y = make_moons(n_samples=500, noise=0.30, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) members = [ ("logreg", LogisticRegression()), ("knn", KNeighborsClassifier(n_neighbors=7)), ("tree", DecisionTreeClassifier(max_depth=5, random_state=42)), ] for name, clf in members: clf.fit(X_train, y_train) print(f"{name:6s}: {clf.score(X_test, y_test):.3f}") hard = VotingClassifier(members, voting="hard").fit(X_train, y_train) soft = VotingClassifier(members, voting="soft").fit(X_train, y_train) print(f"\nhard voting: {hard.score(X_test, y_test):.3f}") print(f"soft voting: {soft.score(X_test, y_test):.3f}")

The ensemble matches or beats its best member — even though one member (logistic regression) is clearly too simple for moon-shaped data. Its votes still help on the samples where the boundary happens to be locally linear, and the other two cover the curves.

Bagging: same algorithm, different data

Bagging (bootstrap aggregating) manufactures diversity from data instead of from algorithms: train N copies of the same estimator, each on a different bootstrap sample, and aggregate. You already know its most famous incarnation — a random forest is bagged decision trees plus random feature subsets. But scikit-learn's BaggingClassifier will bag anything:

Python — runs in your browser

Run

from sklearn.datasets import make_moons from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import BaggingClassifier X, y = make_moons(n_samples=500, noise=0.30, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train) bag = BaggingClassifier(DecisionTreeClassifier(random_state=42), n_estimators=50, random_state=42) bag.fit(X_train, y_train) print(f"one deep tree : {tree.score(X_test, y_test):.3f}") print(f"50 bagged trees: {bag.score(X_test, y_test):.3f}")

Bagging shines with high-variance base models — deep trees, KNN with tiny k — because averaging is a variance-reduction machine. Bagging a very stable model (like logistic regression) barely helps: fifty nearly identical models vote nearly identically. Set bootstrap=False and you get pasting (sampling without replacement); max_features gives you random feature subsets for any estimator, forest-style.

Stacking: let a model learn how to combine

Voting weighs every member equally (or with weights you hand-pick). Stacking asks: why not learn the combination? Train the base models, collect their predictions, and feed those predictions as features into a meta-learner (often logistic regression) that learns which member to trust in which situation. To avoid leakage, StackingClassifier generates the base models' training predictions with internal cross-validation:

Python — runs in your browser

Run

from sklearn.datasets import make_moons from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.neighbors import KNeighborsClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import StackingClassifier X, y = make_moons(n_samples=500, noise=0.30, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) stack = StackingClassifier( estimators=[ ("knn", KNeighborsClassifier(n_neighbors=7)), ("tree", DecisionTreeClassifier(max_depth=5, random_state=42)), ("logreg", LogisticRegression()), ], final_estimator=LogisticRegression(), cv=3, ) stack.fit(X_train, y_train) print(f"stacking: {stack.score(X_test, y_test):.3f}")

Stacking is the heavyweight of the family — more training cost, more moving parts, and a real risk of overfitting on small datasets — but on large, messy problems a well-built stack is hard to beat, which is why it dominates Kaggle leaderboards.

Bagging vs boosting: the fork in the road

There's one strategy we haven't touched: instead of training members in parallel and independently, train them in sequence, where each new model deliberately focuses on the mistakes of the ones before it. That's boosting, and it behaves very differently:

	Bagging (e.g. random forest)	Boosting (e.g. AdaBoost, XGBoost)
Training	Parallel, independent members	Sequential — each member fixes the last one's errors
Base models	Strong, high-variance (deep trees)	Weak, high-bias (shallow trees, stumps)
Mainly reduces	Variance	Bias
More members	Never overfits, just plateaus	Can overfit — member count needs tuning
Sensitivity to noisy labels	Low	Higher (errors get chased)
Combination rule	Equal vote / average	Weighted sum built during training

A useful slogan: bagging turns strong-but-unstable learners into a good learner; boosting turns weak learners into a good learner. The next two lessons walk down the boosting branch — starting with the algorithm that invented it.

Check your understanding

5 questions · free

1. Q1.Three independent classifiers are each correct 70% of the time. Roughly what accuracy does their majority vote achieve?

   A70%B78%C90%D100%

2. Q2.What is the difference between hard and soft voting?

   AHard voting uses more modelsBSoft voting averages predicted probabilities; hard voting counts one class vote per modelCHard voting only works for regressionDSoft voting requires identical base models

3. Q3.Why does bagging logistic regression barely improve on a single logistic regression?

   ALogistic regression cannot be baggedBBootstrap samples are too smallCLogistic regression is a low-variance model, so the bagged copies are nearly identical and there is little disagreement to average awayDBagging only works with exactly 100 estimators

4. Q4.In stacking, why does StackingClassifier use cross-validation to produce the base models' training predictions?

   ATo make training fasterBSo the meta-learner trains on out-of-fold predictions rather than predictions the base models made on data they memorizedCBecause base models cannot predict on their own training dataDTo reduce the number of base models needed

5. Q5.Which statement correctly contrasts bagging and boosting?

   ABagging trains members sequentially; boosting trains them in parallelBBagging reduces variance with independent strong learners; boosting reduces bias by sequentially combining weak learnersCBoth require deep trees as base modelsDBoosting can never overfit no matter how many members are added

Exercise: Assemble a voting classifier for breast cancer

Build a VotingClassifier on the breast cancer dataset from three members: a scaled LogisticRegression, a scaled KNeighborsClassifier, and a DecisionTreeClassifier with max_depth=4. Print each member's individual test accuracy, then the hard-voting and soft-voting ensemble accuracies. Does the ensemble beat the best individual — and which voting mode wins?

Next up: AdaBoost — the original boosting algorithm, where each new model is trained to obsess over exactly the samples the previous ones got wrong.