Baselines & Benchmarks

"My model gets 92% accuracy." Is that good? You genuinely cannot know without something to compare against. If guessing the majority class already scores 91%, that model has learned almost nothing. This lesson gives you the habit that separates rigorous practitioners from hopeful ones: establish a dumb baseline first, benchmark everything against it, and only trust improvements you can measure fairly.

Why every project starts with a dumb model

A baseline is the score of the simplest strategy imaginable — random guessing, always predicting the most common class, or one hand-written rule ("predict survived for every female passenger"). A benchmark is a stronger reference to beat: a simple standard model, a previous production system, or a public leaderboard.

Starting simple pays off repeatedly:

• It calibrates every number that follows. 92% only means something relative to the baseline's 91% — or 55%.
• It catches bugs and leaks. If your first fancy model scores below a dummy, something is broken. If it scores suspiciously near 100%, suspect leakage.
• It delivers value early. A working end-to-end pipeline with a simple model beats a half-finished sophisticated one, and effort is often not proportional to payoff — a huge grid search frequently buys a fraction of a percent.

DummyClassifier: the honest zero point

scikit-learn ships baseline models that deliberately ignore the features: DummyClassifier and DummyRegressor. Let's pit one against a real model on the breast cancer dataset:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.dummy import DummyClassifier from sklearn.model_selection import train_test_split from sklearn.neighbors import KNeighborsClassifier from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42 ) dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train) print(f"dummy (majority class): {dummy.score(X_test, y_test):.3f}") real = Pipeline([ ("scaler", StandardScaler()), ("knn", KNeighborsClassifier(n_neighbors=7)), ]).fit(X_train, y_train) print(f"scaled KNN : {real.score(X_test, y_test):.3f}")

The dummy scores about 63% without ever looking at a single feature — because 63% of patients in this dataset have benign tumors. So the honest reading of the KNN result is not "97% accurate" but "34 points above chance." (DummyRegressor plays the same role for regression, predicting the training mean or median; its R² is essentially zero by construction.)

The accuracy trap: imbalanced classes

Now the trap this protects you from. When one class dominates, accuracy becomes nearly meaningless:

Python — runs in your browser

Run

from sklearn.datasets import make_classification from sklearn.dummy import DummyClassifier from sklearn.linear_model import LogisticRegression from sklearn.metrics import recall_score from sklearn.model_selection import train_test_split # Fraud-style data: only ~3% positives X, y = make_classification(n_samples=2000, n_features=8, n_informative=4, weights=[0.97, 0.03], flip_y=0, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42 ) dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train) model = LogisticRegression(max_iter=1000).fit(X_train, y_train) for name, clf in [("dummy", dummy), ("logistic", model)]: acc = clf.score(X_test, y_test) rec = recall_score(y_test, clf.predict(X_test)) print(f"{name:8s} accuracy={acc:.3f} recall={rec:.3f}")

The dummy hits about 97% accuracy while catching zero fraud cases — its recall (the fraction of true positives it finds) is 0. Anyone who reports "97% accurate" on this problem is reporting the class ratio, not model skill. On imbalanced problems, lead with metrics like recall, precision, F1, or balanced accuracy — the classification module covers them in depth — and always publish the dummy's score next to yours.

Benchmarking models fairly

Once the baseline is planted, compare candidate models under identical conditions: same data, same preprocessing, same cross-validation splits. A loop over a dictionary of pipelines does it cleanly:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.dummy import DummyClassifier from sklearn.linear_model import LogisticRegression from sklearn.model_selection import cross_val_score from sklearn.neighbors import KNeighborsClassifier from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier X, y = load_breast_cancer(return_X_y=True) def scaled(model): return Pipeline([("scaler", StandardScaler()), ("model", model)]) candidates = { "dummy": DummyClassifier(strategy="most_frequent"), "knn": scaled(KNeighborsClassifier(n_neighbors=7)), "logistic": scaled(LogisticRegression(max_iter=1000)), "tree (d=4)": DecisionTreeClassifier(max_depth=4, random_state=42), } for name, model in candidates.items(): scores = cross_val_score(model, X, y, cv=5) print(f"{name:12s} {scores.mean():.3f} ± {scores.std():.3f}")

Reading the table properly:

• Cross-validation, not a single split, so no model wins by luck of the draw. Report the mean and the standard deviation.
• Overlapping error bars mean "roughly tied." If two models are within a standard deviation of each other, prefer the simpler, faster one.
• Each model gets the preprocessing it needs (scaling for KNN and logistic regression; trees don't care), bundled in a pipeline so nothing leaks.

Keep an experiments log

Within a day of iterating you will forget which combination produced which score. Professionals keep a log — a spreadsheet or a plain text file is enough. For every run record:

• the data version and features used (e.g., "added is_alone, binned age"),
• the model and hyperparameters (or the grid searched),
• the validation scheme (5-fold CV, seed 42) — it must stay constant, or scores aren't comparable,
• the score with its spread, and one line of notes ("helped, keep" / "no change, revert").

Two related habits multiply your speed: build yourself reusable templates for the boilerplate (imports, split, preprocessor, grid search) so each new experiment costs minutes, and resist tinkering past the point of diminishing returns — when three experiments in a row move the score by less than its standard deviation, the remaining gains probably live in the data, not the model.

Check your understanding

5 questions · free

1. Q1.Your classifier reports 94% accuracy on a dataset where 94% of samples belong to one class. What have you learned about the model?

   AIt's excellent — 94% is a high scoreBAlmost nothing — always predicting the majority class scores the sameCIt's overfittingDIt needs more training data

2. Q2.What does DummyClassifier(strategy="most_frequent") do?

   ATrains a fast approximate model on the most frequent featuresBAlways predicts the most common class from the training labels, ignoring all featuresCRemoves infrequent classes before trainingDPredicts randomly with equal probability per class

3. Q3.Why benchmark models with cross-validation instead of one train/test split?

   AIt's required by scikit-learnBA single split can favor one model by chance; averaging over k folds gives a more robust, comparable estimateCCross-validation makes models train fasterDIt eliminates the need for a test set entirely

4. Q4.Model A scores 0.951 ± 0.020 and model B scores 0.958 ± 0.025 in the same 5-fold CV. B is much slower. What's the reasonable call?

   AShip B — higher mean always winsBShip A — the difference is well within the noise, so prefer the simpler, faster modelCNeither — both models failedDAverage the two models' predictions

5. Q5.Which item is LEAST important to record in an experiments log?

   AThe features and data version usedBThe validation scheme and random seedCThe score and its standard deviationDThe time of day the experiment ran

Exercise: Benchmark on an imbalanced problem with the right metric

Rebuild the fraud-style imbalanced dataset from this lesson and benchmark three candidates — a majority-class dummy, scaled KNN, and scaled logistic regression — with 5-fold cross-validation, reporting both accuracy and F1 (scoring="f1") for each. Which metric actually distinguishes the real models from the dummy, and which model would you pick?

That wraps the foundations — next module: regression, where you'll fit your first line with gradient descent and meet the loss functions behind almost every model.