Linear Regression & Gradient Descent

Linear regression is the "hello world" of machine learning — but almost every idea you'll use later (weights, bias, loss, gradient descent) shows up here first, in its simplest form. In this lesson you'll fit a line by hand, define what "best fit" means mathematically, and then let gradient descent do the work for you.

The model: a line with two knobs

A linear model predicts a number as a weighted sum of the input plus an offset:

ŷ = w·x + b

• w (weight / slope) — how much the prediction changes when x increases by 1
• b (bias / intercept) — the prediction when x = 0

Training a model just means finding good values for these two knobs. Try it yourself — move the sliders until the line goes through the middle of the cloud:

ŷ =0.40·x + 3.50MSE23.762

Weight w0.40Bias b3.5

▶ Auto-fit with gradient descentHide residuals

You probably ended up somewhere near w ≈ 1.6, b ≈ 1.2 — and you likely used the dashed residual lines to guide you. That intuition — "make the vertical gaps small" — is exactly what the loss function formalizes.

The loss: measuring "how wrong"

For each point, the residual is the gap between truth and prediction, y − ŷ. The mean squared error (MSE) averages the squared residuals:

MSE = (1/n) Σ (yᵢ − ŷᵢ)²

Squaring does two useful things: errors can't cancel each other out, and big misses are punished much more than small ones. Let's compute it ourselves — run this right here in your browser:

Python — runs in your browser

Run

import numpy as np rng = np.random.default_rng(42) x = rng.uniform(0.5, 9.5, 28) y = 1.6 * x + 1.2 + rng.normal(0, 1.4, 28) def mse(w, b): y_pred = w * x + b return np.mean((y - y_pred) ** 2) print(f"w=0.4, b=3.5 -> MSE = {mse(0.4, 3.5):.3f} (the playground's start)") print(f"w=1.6, b=1.2 -> MSE = {mse(1.6, 1.2):.3f} (near the truth)") print(f"w=2.5, b=-3 -> MSE = {mse(2.5, -3.0):.3f} (way off)")

Lower loss = better fit. Training is now an optimization problem: find the (w, b) pair that minimizes MSE.

Gradient descent: rolling downhill

Imagine the loss as a landscape where every location is a (w, b) choice and the altitude is the MSE. Gradient descent starts anywhere and repeatedly takes a small step downhill:

w ← w − η · ∂L/∂w and b ← b − η · ∂L/∂b

where η (the learning rate) controls the step size. Watch how the choice of η changes everything — try a tiny value, then crank it up past 0.7:

x-2.800L(x)2.917∇L(x)-4.125

Learning rate η0.12

▶ PlayStep onceReset (left)Reset (right)

Three regimes to remember:

• η too small — converges, but painfully slowly
• η just right — smooth, fast descent to the minimum
• η too large — overshoots the valley and bounces or diverges

Implementing gradient descent from scratch

Fifteen lines of NumPy are enough. This is the same algorithm the playground's "Auto-fit" button runs:

Python — runs in your browser

Run

import numpy as np rng = np.random.default_rng(42) x = rng.uniform(0.5, 9.5, 28) y = 1.6 * x + 1.2 + rng.normal(0, 1.4, 28) w, b, lr = 0.0, 0.0, 0.01 for step in range(1, 401): y_pred = w * x + b grad_w = 2 * np.mean((y_pred - y) * x) grad_b = 2 * np.mean(y_pred - y) w -= lr * grad_w b -= lr * grad_b if step % 100 == 0: print(f"step {step:3d}: w={w:.3f} b={b:.3f} MSE={np.mean((y - y_pred)**2):.3f}") print(f"\nLearned: y = {w:.2f}x + {b:.2f} (true: y = 1.60x + 1.20)")

Notice b converges more slowly than w — the bias gradient doesn't get the "leverage" of being multiplied by x. Feature scaling (a later lesson) fixes exactly this kind of imbalance.

The scikit-learn way

In practice you won't hand-roll gradient descent for linear regression — scikit-learn solves it directly:

Python — runs in your browser

Run

import numpy as np from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error, r2_score rng = np.random.default_rng(42) x = rng.uniform(0.5, 9.5, 28).reshape(-1, 1) # sklearn expects 2-D features y = 1.6 * x.ravel() + 1.2 + rng.normal(0, 1.4, 28) model = LinearRegression().fit(x, y) y_pred = model.predict(x) print(f"w = {model.coef_[0]:.3f}, b = {model.intercept_:.3f}") print(f"MSE = {mean_squared_error(y, y_pred):.3f}") print(f"R² = {r2_score(y, y_pred):.3f}")

Same answer, one line of fitting. LinearRegression uses a closed-form solution (ordinary least squares); gradient descent matters when models get too big for closed forms — which is every neural network you'll ever train.

Check your understanding

4 questions · free

1. Q1.In ŷ = w·x + b, what happens to predictions if you increase b by 2?

   AThey all increase by 2·xBThey all increase by exactly 2COnly predictions at x = 0 changeDThe slope of the line doubles

2. Q2.Why does MSE square the residuals instead of just averaging them?

   ASquaring is faster to computeBSo positive and negative errors don't cancel, and large errors are penalized moreCBecause predictions are always positiveDTo keep the loss between 0 and 1

3. Q3.You set the learning rate very high and the loss starts growing every step. What's happening?

   AThe model is underfittingBThe data needs more featuresCEach step overshoots the minimum, so the ball bounces higher and higherDGradient descent found a saddle point

4. Q4.Gradient descent updates w with w ← w − η·∂L/∂w. Why subtract the gradient?

   AThe gradient points uphill — subtracting it moves downhill toward lower lossBSubtraction keeps w positiveCIt's a convention with no effectDTo normalize the weights

Exercise: Gradient descent on a different dataset

Generate a dataset with a negative slope — y = -2x + 8 plus Gaussian noise (σ = 0.8) for 40 points with x between 0 and 5. Fit it two ways: with your own gradient-descent loop, and with scikit-learn's LinearRegression. Do the two answers agree to two decimal places?

Next up: what happens when a straight line isn't enough — polynomial regression, and the overfitting trap that comes with it.