Multiclass & Multilabel Classification

So far every classifier we've built answered a yes/no question: malignant or benign, fraud or legit. Real problems are rarely that tidy — an iris flower is one of three species, a handwritten digit is one of ten, and a movie can be an action comedy and a romance all at once. This lesson extends everything you know about binary classification to many classes, and then to many labels per sample.

One question, how many answers?

Three different problem shapes hide behind the word "classification":

• Binary — exactly one of 2 classes. Is this tumor malignant?
• Multiclass — exactly one of K classes. Which iris species is this: setosa, versicolor, or virginica? The classes are mutually exclusive — a flower can't be two species.
• Multilabel — any subset of K labels. Which genres describe this movie? ["action", "comedy"] is a perfectly valid answer, and so is an empty set or all of them.

The distinction matters because it changes the model's output, the loss, and the metrics. Confuse multiclass with multilabel and you'll force a movie to have exactly one genre — or let a flower be two species at once.

Turning binary classifiers into multiclass ones

Logistic regression natively separates two classes. Two classic strategies recycle it for K classes:

One-vs-Rest (OvR) trains K binary classifiers, each answering "is it class k, or anything else?" — setosa vs rest, versicolor vs rest, virginica vs rest. At prediction time, all K models score the sample and the most confident one wins. K models total, each trained on the full dataset.

One-vs-One (OvO) trains one binary classifier per pair of classes — K·(K−1)/2 models — and lets them vote. That's more models, but each trains on only two classes' worth of data, which is a win for algorithms whose training cost grows steeply with dataset size (scikit-learn's SVC uses OvO internally for exactly this reason).

Softmax: the multiclass sigmoid

There's a more elegant third option. Instead of gluing binary models together, generalize the sigmoid itself. Softmax takes K raw scores and turns them into K probabilities that sum to 1: exponentiate every score, then divide each by the total.

Python — runs in your browser

Run

import numpy as np classes = ["setosa", "versicolor", "virginica"] scores = np.array([2.0, 1.0, 0.1]) # raw linear scores, one per class exp_scores = np.exp(scores) softmax = exp_scores / exp_scores.sum() for c, s, p in zip(classes, scores, softmax): print(f"{c:11s} score={s:4.1f} -> P={p:.3f}") print(f"\nprobabilities sum to {softmax.sum():.1f}")

A logistic regression trained this way — one weight vector per class, softmax on top, cross-entropy loss — is called multinomial (softmax) logistic regression. It's one coherent model rather than K independent ones, so the probabilities are directly comparable. Modern scikit-learn does this automatically: hand LogisticRegression a target with three classes and it fits the multinomial model, no extra arguments needed.

Multiclass in practice: iris

Everything else about the workflow is unchanged — same pipeline, same fit, same predict_proba (now with three columns). The confusion matrix just grows to K×K:

Python — runs in your browser

Run

from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import ConfusionMatrixDisplay import matplotlib.pyplot as plt iris = load_iris() X_tr, X_te, y_tr, y_te = train_test_split(iris.data, iris.target, stratify=iris.target, random_state=42) model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr) print(f"test accuracy: {model.score(X_te, y_te):.3f}") ConfusionMatrixDisplay.from_estimator(model, X_te, y_te, display_labels=iris.target_names) plt.title("Iris test set") plt.show()

The matrix tells a story accuracy can't: setosa is never confused with anything (its petals are unmistakably small), while the mistakes live entirely in the versicolor/virginica corner — those two species genuinely overlap.

Macro, micro, weighted: averaging metrics across classes

Precision, recall, and F1 are defined per class — with K classes you get K of each. To report a single number you must average them, and how you average changes the answer:

• macro — average the K per-class scores equally. Every class counts the same, so a tiny minority class can drag the score down. Use it when small classes matter.
• micro — pool all predictions first, then compute the metric once. Big classes dominate. (For single-label multiclass, micro-F1 equals plain accuracy.)
• weighted — macro, but each class weighted by its number of samples. A compromise that still mostly reflects the majority.

On balanced iris these barely differ. On imbalanced data they diverge wildly:

Python — runs in your browser

Run

from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import f1_score, recall_score, accuracy_score X, y = make_classification(n_samples=1000, n_classes=3, n_informative=6, weights=[0.7, 0.2, 0.1], random_state=42) X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42) model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr) pred = model.predict(X_te) print("per-class recall:", recall_score(y_te, pred, average=None).round(3)) print() for avg in ["macro", "micro", "weighted"]: print(f"F1 ({avg:8s}) = {f1_score(y_te, pred, average=avg):.3f}") print(f"accuracy = {accuracy_score(y_te, pred):.3f}")

The model nails the 70% majority class and largely misses the 10% minority. Micro-F1 (= accuracy here) looks respectable; macro-F1 exposes the failure. When someone reports "the F1", always ask which averaging.

Multilabel: many tags per sample

Now drop the "exactly one" constraint. A multilabel target is a binary matrix: one column per label, one row per sample, with 1s wherever a label applies. MultiLabelBinarizer converts lists of tags into that matrix, and OneVsRestClassifier fits one independent binary classifier per column — each label gets its own yes/no decision, so a sample can light up several:

Python — runs in your browser

Run

import numpy as np from sklearn.preprocessing import MultiLabelBinarizer from sklearn.multiclass import OneVsRestClassifier from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.metrics import hamming_loss, f1_score # synthetic "movies": 3 features drive which genre tags apply rng = np.random.default_rng(42) X = rng.normal(0, 1, (300, 3)) tags = [] for a, r, c in X: t = [] if a > 0.2: t.append("action") if r > 0.4: t.append("romance") if c > 0.1 or a > 1.2: t.append("comedy") if not t: t.append("drama") tags.append(t) mlb = MultiLabelBinarizer() Y = mlb.fit_transform(tags) print("labels:", list(mlb.classes_)) print("first 3 movies:", tags[:3]) print("as a binary matrix:\n", Y[:3]) X_tr, X_te, Y_tr, Y_te = train_test_split(X, Y, random_state=42) clf = OneVsRestClassifier(LogisticRegression()).fit(X_tr, Y_tr) P = clf.predict(X_te) print(f"\nhamming loss : {hamming_loss(Y_te, P):.3f}") print(f"micro-F1 : {f1_score(Y_te, P, average='micro'):.3f}") print(f"exact-match rate: {(P == Y_te).all(axis=1).mean():.3f}")

Two multilabel-specific metrics appear here. Hamming loss is the fraction of individual label decisions that were wrong — every cell of the matrix counts, so 0.09 means 9% of all yes/no calls missed. The exact-match rate (subset accuracy) is far stricter: a sample only counts if every label is right. It's normal for hamming loss to look great while exact match looks mediocre — one wrong tag out of four ruins the exact match but barely moves the hamming loss.

Check your understanding

5 questions · free

1. Q1.Tagging news articles with topics (politics, sports, tech — possibly several per article) is which kind of problem?

   ABinary classificationBMulticlass classificationCMultilabel classificationDRegression

2. Q2.For a 10-class problem, how many binary models do One-vs-Rest and One-vs-One train, respectively?

   A10 and 10B10 and 45C45 and 10D100 and 10

3. Q3.What does the softmax function guarantee about its outputs?

   AEach output is either 0 or 1BThe largest output is always above 0.5CAll outputs are positive and sum to 1, so they can be read as class probabilitiesDThe outputs equal the raw scores, just sorted

4. Q4.A 3-class dataset is 90% class A. Your model predicts class A perfectly but fails on B and C. Which F1 average will look worst?

   AMicro — it pools all predictionsBWeighted — it weights by class sizeCMacro — it gives B and C equal say despite their sizeDThey are always identical

5. Q5.A multilabel model has hamming loss 0.05 but exact-match rate 0.60. What does this mean?

   AThe metrics contradict each other — one must be a bugBOnly 5% of individual label decisions are wrong, but 40% of samples have at least one wrong labelCThe model predicts the empty label set most of the timeD60% of the labels are unused

Exercise: Find the hardest genre

Using the synthetic movie-tag dataset from this lesson, compute the per-label F1 score (one number per genre) and each label's frequency in the data. Which genre is hardest to predict — and looking at how the tags were generated, can you explain why that label is fundamentally harder than the others?

Next module: a classifier with a completely different philosophy — support vector machines, which ignore most of the data and let a handful of boundary points decide everything.