Logistic Regression

Despite the name, logistic regression is a classifier — and probably the most widely deployed one in industry. The trick is beautifully simple: keep the linear model you already know, then squash its output into a probability. In this lesson you'll see why a plain line fails on 0/1 labels, why the loss function has to change too, and how to use LogisticRegression properly.

Why not just fit a line?

Labels in binary classification are 0 or 1. What if we fit ordinary linear regression to them and predict "1" whenever the line is above 0.5? Play with the threshold and the data below and watch what the sigmoid does that a line can't:

A straight line has two fatal problems here:

1. Unbounded outputs. The line happily predicts 3.7 or −1.2 — meaningless as a probability of a class.
2. Outliers drag the boundary. Add one very obvious positive far to the right and MSE forces the line to tilt toward it, moving the decision boundary and misclassifying points near the middle. The most confident examples shouldn't be the ones wrecking the fit.

The sigmoid: scores in, probabilities out

The fix: pass the linear score z = w·x + b through the sigmoid function:

σ(z) = 1 / (1 + e^(−z))

• Large positive z → σ(z) → 1 (confident positive)
• Large negative z → σ(z) → 0 (confident negative)
• z = 0 → σ(z) = 0.5 (right on the boundary)

The output is always between 0 and 1, so we can read it as P(y = 1 given x). The model is still linear at heart — the sigmoid just translates its score into probability language.

Why the loss must change too

You might be tempted to keep MSE and just wrap the prediction in a sigmoid. Don't — this is where the weaknesses of gradient descent bite. Gradient descent only works well when the loss surface cooperates; it can get stuck in local minima, stall on plateaus (flat regions where gradients vanish), and crawl through saddle points. MSE-through-a-sigmoid creates exactly these pathologies: the loss surface becomes non-convex, with plateaus wherever the sigmoid saturates. A confidently-wrong prediction (σ ≈ 0 when the truth is 1) sits on flat ground — the gradient is nearly zero and learning stalls precisely when the model most needs correcting.

The right loss is log-loss (binary cross-entropy):

L = −(1/n) Σ [ yᵢ·log(pᵢ) + (1 − yᵢ)·log(1 − pᵢ) ]

Being wrong with high confidence costs almost infinitely much — log(p) blows up as p → 0 — so the gradient stays large exactly where MSE goes flat. Bonus: with log-loss the optimization problem is convex, one bowl-shaped valley with a single minimum. Gradient descent can't get trapped.

Decision boundary and threshold

The model's probability is continuous; the decision needs a cutoff. By default predict uses 0.5, which corresponds to the line (or hyperplane) where w·x + b = 0. But as you saw in the metrics lesson, the threshold is a business decision, not a math constant — lower it to catch more positives, raise it to reduce false alarms. Keep predict_proba around and apply the threshold yourself when the costs are asymmetric.

Logistic regression on real data

Let's train on the breast-cancer dataset. Scaling matters: logistic regression is trained by an iterative optimizer, and wildly different feature scales make the loss valley long and narrow — slow, unstable convergence. A pipeline keeps it honest:

predict_proba returns one column per class, ordered by model.classes_ (here 0 = malignant, 1 = benign); each row sums to 1. Notice how some predictions are near-certain (0.99+) while others hover near 0.5 — that uncertainty information is free, and thresholding is how you use it.

Coefficients are log-odds

Because the model is linear inside the sigmoid, its coefficients are interpretable: increasing feature j by one (scaled) unit adds w_j to the log-odds log(p / (1 − p)) of the positive class. You don't need to think in log-odds day to day — the sign and size are what matter:

• positive coefficient → larger feature value pushes toward class 1
• negative coefficient → pushes toward class 0
• larger magnitude → stronger influence (comparable across features only because we standardized them)

import pandas as pd
coefs = pd.Series(model[-1].coef_[0],
                  index=load_breast_cancer().feature_names)
print(coefs.sort_values())     # most malignant-leaning features first

Regularization: the C parameter

LogisticRegression applies L2 regularization by default, controlled by C — the inverse regularization strength (the same convention SVMs use):

• small C (e.g. 0.01) → strong regularization → smaller coefficients, simpler model, may underfit
• large C (e.g. 100) → weak regularization → coefficients roam free, may overfit

Watch the coefficients grow with C while cross-validation accuracy peaks somewhere in the middle — tune C with cross-validation rather than guessing.

Next: what happens when there are more than two classes — and when a single sample can carry several labels at once.