Random Forests

A single decision tree is interpretable but twitchy — nudge the training data and the whole tree can restructure itself. Random forests fix this with a wonderfully simple idea: train many trees, each on a slightly different view of the data, and let them vote. In this lesson you'll see why averaging works, where the randomness comes from, and how to read a forest's feature importances without being fooled by them.

Wisdom of crowds

Ask one person to guess the number of beans in a jar and they'll probably be far off. Average a hundred independent guesses and the result is often startlingly good — individual errors point in different directions and cancel out. The same logic applies to models: if each tree overfits in its own random way, the average of their predictions keeps the signal (which all trees agree on) and washes out the noise (which they don't).

The crucial word is independent. A hundred copies of the same tree vote unanimously and average to... the same tree. Diversity is not a nice-to-have; it's the entire mechanism. So a random forest goes out of its way to make its trees disagree.

Two sources of randomness

A random forest decorrelates its trees in two ways:

1. Bootstrap sampling (bagging). Each tree trains on a bootstrap sample: n rows drawn from the n training rows with replacement. Some rows appear twice or three times, and on average about 37% of rows are left out of any given tree's sample entirely. Every tree therefore sees a slightly different dataset.

2. Random feature subsets per split. At every node, the tree is only allowed to consider a random subset of features (max_features, by default the square root of the feature count for classification). Without this, one dominant feature would win the root split in every tree and the trees would all look alike. Restricting the menu forces different trees to discover different structure.

Bagging alone helps; the feature restriction is what makes it a random forest rather than just bagged trees. To classify a new sample, every tree votes and the majority wins (for regression, predictions are averaged).

Forest vs a single tree

Talk is cheap — let's race them on the same split of the breast cancer dataset:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train) forest = RandomForestClassifier(n_estimators=100, random_state=42) forest.fit(X_train, y_train) print(f"single tree : train={tree.score(X_train, y_train):.3f} test={tree.score(X_test, y_test):.3f}") print(f"forest (100): train={forest.score(X_train, y_train):.3f} test={forest.score(X_test, y_test):.3f}")

Both models hit a perfect train score — every individual tree still overfits its bootstrap sample. But the forest's test accuracy is clearly higher: the overfitting of a hundred different trees averages out. That's variance reduction in action, and it's why forests are such a strong default for tabular data.

The knobs that matter

• n_estimators — number of trees. More is monotonically better (the average just gets more stable) until it plateaus; the only cost is compute. 100–500 is typical. You cannot overfit by adding trees.
• max_features — the size of the random feature menu per split. Smaller values mean more diverse (less correlated) trees but each tree is weaker. The default ("sqrt" for classification) is a solid starting point.
• max_depth / min_samples_leaf — same meaning as for a single tree. Forests tolerate deep trees much better than a lone tree does, so these often stay at their defaults; tune them if the forest is slow or still overfits.

Free validation: the OOB score

Remember that each tree never sees about 37% of the training rows. Those rows are out-of-bag (OOB) for that tree — so we can use them as a private little test set. For every training row, collect votes only from the trees that didn't train on it, and score the result. You get an honest performance estimate without sacrificing any data to a validation split:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) forest = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42) forest.fit(X_train, y_train) print(f"OOB score : {forest.oob_score_:.3f}") print(f"test score: {forest.score(X_test, y_test):.3f}")

The OOB score lands very close to the held-out test score — it's a built-in cross-validation you get almost for free.

Feature importances

Forests come with a bonus: a ranking of which features mattered. Each split reduces impurity by some amount; add up the reductions credited to each feature across all trees and you get impurity-based feature importance (feature_importances_):

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_wine from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.inspection import permutation_importance wine = load_wine() X_train, X_test, y_train, y_test = train_test_split( wine.data, wine.target, test_size=0.3, stratify=wine.target, random_state=42) forest = RandomForestClassifier(n_estimators=100, random_state=42) forest.fit(X_train, y_train) order = np.argsort(forest.feature_importances_) plt.barh(np.array(wine.feature_names)[order], forest.feature_importances_[order]) plt.xlabel("impurity-based importance") plt.tight_layout() plt.show() # The more trustworthy alternative: shuffle a feature, watch the score drop perm = permutation_importance(forest, X_test, y_test, n_repeats=5, random_state=42) top = np.argsort(perm.importances_mean)[::-1][:5] print("top 5 by permutation importance:") for i in top: print(f" {wine.feature_names[i]:<30s} {perm.importances_mean[i]:.3f}")

One more caution, no matter which importance you use: important ≠ causal. A feature can rank highly because it's correlated with the true driver. Importance tells you what the model leaned on, not what makes the outcome happen in the real world.

Regression forests

Everything transfers directly to regression: RandomForestRegressor averages each tree's numeric prediction instead of taking a vote, and splits minimize MSE instead of gini. Same knobs, same OOB trick, same importances:

Python — runs in your browser

Run

from sklearn.datasets import load_diabetes from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestRegressor X, y = load_diabetes(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) forest = RandomForestRegressor(n_estimators=100, random_state=42).fit(X_train, y_train) print(f"train R2: {forest.score(X_train, y_train):.3f}") print(f"test R2 : {forest.score(X_test, y_test):.3f}")

And like all tree models, forests need no feature scaling — you can drop the StandardScaler from your pipeline entirely.

Check your understanding

5 questions · free

1. Q1.Why does averaging many trees improve on a single tree?

   AEach tree is individually more accurate inside a forestBThe trees' independent errors cancel out, reducing varianceCAveraging increases the depth of each treeDIt reduces the number of features needed

2. Q2.What are the two sources of randomness in a random forest?

   ARandom learning rates and random depthsBRandom labels and random thresholdsCBootstrap sampling of rows and random feature subsets at each splitDRandom train/test splits and random seeds

3. Q3.What is the out-of-bag (OOB) score?

   AThe forest's accuracy on the test setBEach row scored using only the trees whose bootstrap sample excluded it — a free validation estimateCThe fraction of trees that agree with the majorityDThe training accuracy of the worst tree

4. Q4.You keep increasing n_estimators from 100 to 10,000. What happens?

   AThe forest overfits badlyBPerformance improves then plateaus — more trees only cost compute, they don't overfitCThe forest underfitsDOOB score becomes unavailable

5. Q5.Impurity-based feature importance ranks a random-noise ID column as the top feature. What is the best explanation?

   AThe ID column truly drives the targetBThe forest is broken and must be retrainedCImpurity importance is biased toward high-cardinality features; permutation importance on held-out data would expose itDn_estimators is too low

Exercise: How many trees are enough?

On the breast cancer dataset (same 70/30 stratified split as above), train random forests with n_estimators set to 1, 5, 10, 25, 50, 100, and 200. Plot test accuracy against the number of trees. Where does the curve flatten out — and is a 200-tree forest meaningfully better than a 50-tree one here?

Next up: forests are just one member of a bigger family — voting, bagging, stacking, and boosting all combine models, and ensemble learning is the map that ties them together.