Classification Metrics Beyond Accuracy

Accuracy is the first metric everyone learns — and the first one that betrays them. On real-world classification problems (fraud, disease, churn) the classes are rarely balanced, and a model can score 95%+ accuracy while being completely useless. This lesson gives you the full evaluation toolkit: the confusion matrix, precision and recall, F1, ROC/AUC, and the precision-recall curve.

The accuracy trap

Suppose only 5% of transactions are fraudulent. Here's a "model" that never predicts fraud at all — it just answers "legit" every time:

95% accuracy, zero frauds caught. Any metric that rewards this behavior can't be trusted on imbalanced data. To do better we need to look at which kinds of mistakes the model makes — not just how many.

The confusion matrix: four kinds of outcome

For a binary problem, every prediction lands in one of four cells:

• True Positive (TP) — predicted positive, actually positive
• False Positive (FP) — predicted positive, actually negative (a false alarm)
• False Negative (FN) — predicted negative, actually positive (a miss)
• True Negative (TN) — predicted negative, actually negative

Accuracy is (TP + TN) / total — it lumps both error types together. The confusion matrix keeps them separate:

Rows are the truth, columns are the predictions. The diagonal is what the model got right; the off-diagonal cells are the two flavors of mistake.

Precision vs recall: which mistake hurts more?

Two metrics zoom in on the positive class, each punishing a different error:

Precision = TP / (TP + FP) — of everything I flagged as positive, how much really was? High precision = few false alarms.

Recall = TP / (TP + FN) — of everything that is positive, how much did I catch? High recall = few misses. (Also called sensitivity.)

Which one matters depends entirely on the cost of each mistake:

• Spam filtering — a false positive means a real email (maybe a job offer) lands in spam. That's the expensive mistake, so optimize precision.
• Cancer screening — a false negative means telling a sick patient they're healthy. That's catastrophic, so optimize recall; a few false alarms that trigger a follow-up test are an acceptable price.

F1 and the threshold tradeoff

The F1-score is the harmonic mean of precision and recall:

F1 = 2 · (precision · recall) / (precision + recall)

The harmonic mean is deliberately harsh: if either precision or recall is near zero, F1 is near zero too. Use it when you want a single number that balances both — the "do nothing" fraud model above has an F1 of exactly 0.

Most classifiers actually output a probability, and the default 0.5 cutoff is just a choice. Moving the threshold trades precision for recall:

Lower the threshold and recall climbs (you flag more) while precision drops. For a per-class summary of everything at once, classification_report prints precision, recall, F1, and support for each class:

from sklearn.metrics import classification_report
print(classification_report(y_te, model.predict(X_te),
                            target_names=["malignant", "benign"]))

ROC curve and AUC

Instead of picking one threshold, the ROC curve sweeps through all of them, plotting the true-positive rate (recall) against the false-positive rate at every cutoff. The AUC (area under the curve) summarizes it: 1.0 is a perfect ranker, 0.5 is coin-flipping.

A nice property of AUC: it doesn't depend on any particular threshold, so it measures how well the model ranks positives above negatives.

The PR curve: when imbalance is heavy

ROC has a blind spot. The false-positive rate divides by the number of negatives — and when negatives are 95%+ of the data, even thousands of false alarms barely move the curve. ROC-AUC then looks flattering while precision is actually awful.

The precision-recall curve plots precision against recall across thresholds, and both quantities focus on the positive class — so it stays honest under heavy imbalance. Rules of thumb from practice:

• Roughly balanced data (20–80% positive) — ROC-AUC and PR-AUC both work.
• Rare positives (under ~5%) — prefer the PR curve and average precision; ROC-AUC tends to overestimate how good the model is.
• Very little data — every metric becomes unstable; treat all scores with suspicion.

from sklearn.metrics import PrecisionRecallDisplay
PrecisionRecallDisplay.from_estimator(model, X_te, y_te)

Next: the workhorse classifier behind those probability scores — logistic regression, the sigmoid, and why it needs a different loss than MSE.