Decision Trees

Every model you've met so far draws smooth curves through the data. A decision tree does something completely different: it plays twenty questions. "Is petal length below 2.45 cm? Yes → it's a setosa. No → ask another question." The result is a model you can literally read out loud — and the building block for random forests and gradient boosting, the workhorses of tabular machine learning.

A model made of if/else rules

A trained tree is just a flowchart of yes/no questions about the features. Each internal node tests one feature against one threshold, each branch is an answer, and each leaf holds a prediction. Prediction is cheap: drop a sample in at the root and follow the answers down to a leaf.

Training is where the magic happens — the algorithm learns which questions to ask, in which order, from the data. Each question splits the feature space with an axis-aligned cut, so the decision boundary is built from rectangles. Watch what happens to the boundary as you let the tree ask more questions:

max depth3leaves5train acc91.3%

Max depth3

At depth 1 the tree makes a single cut — one question. Each extra level lets it subdivide every region again, so the boundary gets more intricate. Push the depth high enough and the tree starts fencing off individual points: it has memorized the training set, noise included. Keep that picture in mind — depth is the tree's main complexity knob.

How a split is chosen: gini impurity

At every node the algorithm tries many candidate splits — every feature, many thresholds — and keeps the one that makes the resulting child nodes as pure as possible. The default purity measure in scikit-learn is gini impurity:

G = 1 − Σ pₖ²

where pₖ is the fraction of samples in the node belonging to class k. A pure node (all one class) has G = 0; a 50/50 node has G = 0.5. A candidate split is scored by the weighted average of its children's impurities — lower is better. Let's work a tiny example by hand:

Python — runs in your browser

Run

import numpy as np def gini(counts): p = np.array(counts) / sum(counts) return 1 - np.sum(p ** 2) # Parent node: 6 apples, 4 oranges print(f"parent gini: {gini([6, 4]):.3f}") # Split A -> left: 4 apples, 0 oranges | right: 2 apples, 4 oranges gA = (4/10) * gini([4, 0]) + (6/10) * gini([2, 4]) print(f"split A weighted gini: {gA:.3f}") # Split B -> left: 3 apples, 2 oranges | right: 3 apples, 2 oranges gB = (5/10) * gini([3, 2]) + (5/10) * gini([3, 2]) print(f"split B weighted gini: {gB:.3f}") print("\nSplit A wins: it creates a perfectly pure left child.")

The parent starts at 0.48. Split A drops the weighted impurity to 0.267 — mostly because its left child is completely pure — while split B barely helps at all. The tree greedily picks the biggest impurity drop, then repeats the whole search inside each child. Entropy (criterion="entropy") is an alternative measure with the same spirit; in practice the two produce very similar trees, and gini is slightly cheaper to compute.

Depth and leaf size: the overfitting controls

Left alone, a tree keeps splitting until every leaf is pure — which usually means memorizing the training data. Two hyperparameters rein it in:

• max_depth — hard cap on the number of questions along any path. Smaller = simpler boundary (exactly what you saw in the playground).
• min_samples_leaf — a split is only allowed if each child keeps at least this many samples. Larger values stop the tree from carving out tiny regions around individual noisy points, which indirectly limits depth too.

There's also min_samples_split (don't split nodes smaller than this) and max_leaf_nodes. You rarely need all of them — tuning max_depth plus min_samples_leaf covers most situations.

A real tree on iris

Let's train one and — this is the fun part — print its rules as plain text:

Python — runs in your browser

Run

from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier, export_text X, y = load_iris(return_X_y=True) iris = load_iris() X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42) tree = DecisionTreeClassifier(max_depth=3, random_state=42) tree.fit(X_train, y_train) print(f"train accuracy: {tree.score(X_train, y_train):.3f}") print(f"test accuracy : {tree.score(X_test, y_test):.3f}\n") print(export_text(tree, feature_names=iris.feature_names))

Read the printout top to bottom: the very first question — petal length vs 2.45 cm — separates all the setosas in one cut, and the rest of the tree works on telling versicolor from virginica. No coefficients, no probabilities to decode: the model is the explanation.

Notice what we didn't do: no feature scaling. A tree only asks "is this feature above this threshold?", so stretching or squashing a feature's scale changes the threshold but not the tree. Standardization, so critical for KNN and SVMs, is simply irrelevant here.

Drawing the tree

For reports and sanity checks, plot_tree renders the same structure graphically — each node shows its split rule, gini, sample count, and class mix:

Python — runs in your browser

Run

import matplotlib.pyplot as plt from sklearn.datasets import load_iris from sklearn.tree import DecisionTreeClassifier, plot_tree iris = load_iris() tree = DecisionTreeClassifier(max_depth=2, random_state=42) tree.fit(iris.data, iris.target) plt.figure(figsize=(9, 5)) plot_tree(tree, feature_names=iris.feature_names, class_names=iris.target_names, filled=True, rounded=True) plt.show()

Darker node colors mean purer nodes. Follow any root-to-leaf path and you can state the exact rule that produces that prediction — try explaining a neural network's prediction that easily.

Strengths — and the flaw that motivates forests

Decision trees have a lot going for them:

• Interpretable — the model is a readable set of rules.
• No scaling needed — thresholds don't care about units.
• Mixed feature types — numeric and (encoded) categorical features coexist happily, and monotone transformations of features change nothing.
• Nonlinear out of the box — no kernels or polynomial features required.

But they have one serious weakness: instability. Because each split is a greedy, winner-takes-all choice, removing a handful of training samples can flip which question wins at the root — and everything below the root then changes too. Two nearly identical datasets can produce wildly different trees. In statistics terms, a deep tree is a high-variance model.

Here's the beautiful trick: if one tree is unstable, train hundreds of slightly different trees and average them. The individual wobbles cancel out. That's a random forest — the subject of the next lesson.

Check your understanding

5 questions · free

1. Q1.A node contains 8 samples of class A and 0 of class B. What is its gini impurity?

   A1.0B0.5C0.0D0.8

2. Q2.How does a decision tree pick which split to make at a node?

   AIt picks a random feature and thresholdBIt tries candidate splits and keeps the one with the lowest weighted child impurityCIt always splits on the feature with the largest valuesDIt minimizes the number of leaves

3. Q3.Why is feature scaling unnecessary for decision trees?

   ATrees scale the data internallyBSplits are threshold comparisons on one feature at a time, so units don't affect the tree's structureCTrees only work with categorical featuresDScaling would break gini impurity

4. Q4.Your tree gets 100% train accuracy but 78% test accuracy. Which change is most likely to help?

   AIncrease max_depthBDecrease min_samples_leaf to 1CReduce max_depth or increase min_samples_leafDStandardize the features

5. Q5.What does it mean that decision trees are unstable?

   AThey give different predictions each time you call predictBSmall changes in the training data can produce a completely different treeCThey crash on large datasetsDTheir accuracy decreases over time

Exercise: Find the sweet-spot depth on the wine dataset

Load load_wine from scikit-learn, split it 70/30 (stratified), and train DecisionTreeClassifier models with max_depth from 1 to 10. Print train and test accuracy for each depth and plot both curves. At what depth does test accuracy peak, and where does the train–test gap start to widen?

Next up: random forests — how averaging hundreds of deliberately randomized trees turns one unstable learner into one of the most reliable models in machine learning.