What Is Machine Learning?

Before touching any algorithm, you need three things: a clear picture of what "learning from data" actually means, the vocabulary to talk about it, and one non-negotiable habit — always keeping some data hidden from your model so you can measure how well it really performs. This lesson builds all three.

From writing rules to learning them

Suppose you're asked to build a spam filter. The classical approach is to write rules by hand: if the subject contains "FREE!!!", flag it; if the sender is unknown and there are more than three links, flag it… This works until spammers change tactics, and every fix adds another brittle rule.

Machine learning flips the recipe. Instead of writing the rules, you collect examples — thousands of emails already labeled spam or not spam — and let an algorithm find the patterns itself. The program's behavior is learned from data rather than hard-coded.

That's the relationship between the two famous buzzwords: artificial intelligence is the broad goal of making machines behave intelligently (rule-based expert systems count too), while machine learning is the subfield where that behavior is learned from examples. Almost everything called "AI" today is machine learning underneath.

Three flavors of learning

• Supervised learning — every example comes with the correct answer (a label). The model learns to map inputs to answers: spam detection, house price prediction, medical diagnosis. This is most of the course.
• Unsupervised learning — no labels at all. The model looks for structure on its own: grouping similar customers, compressing features. We'll get there in the clustering and PCA module.
• Reinforcement learning — an agent learns by acting and receiving rewards, like a game-playing bot. Fascinating, but outside our scope here.

Within supervised learning there are two main task types: classification (predict a category: spam / not spam) and regression (predict a number: tomorrow's temperature).

Samples, features, labels

Data almost always arrives as a table, and ML has names for its parts:

Term	In the table	Convention
Sample (observation, instance)	one row	n_samples of them
Feature (attribute, predictor)	one input column	matrix X, shape (n_samples, n_features)
Label (target)	the column to predict	vector y

So "train a model" means: given X and y, find a function that maps a new row of features to a good prediction of its label.

The golden rule: never grade a model on its homework

Here's a trap every beginner falls into. You fit a model, evaluate it on the same data it trained on, see 100% accuracy, and celebrate. But a model that memorizes its training data perfectly can still be useless on new data — and new data is the only thing we care about.

Think of it like studying for an exam: if the exam questions are the exact homework problems you practiced, a perfect score proves you memorized the homework, not that you understand the subject. To measure understanding, the exam must contain questions you've never seen.

In ML the fix is the train/test split: set aside a portion of the data (typically 20–30%), train only on the rest, and evaluate on the held-out part. Watch how dramatic the difference can be — this decision tree is allowed to grow as deep as it likes on noisy data:

Python — runs in your browser

Run

from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier # Noisy data: only 3 of 10 features matter, and 25% of labels are flipped X, y = make_classification(n_samples=400, n_features=10, n_informative=3, flip_y=0.25, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42 ) tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train) print(f"train accuracy: {tree.score(X_train, y_train):.3f}") print(f"test accuracy : {tree.score(X_test, y_test):.3f}")

Perfect on training data, mediocre on test data. If we had evaluated on the training set, we would have believed we had a flawless model.

A few details in that split call matter:

• test_size=0.25 — hold out 25% of the rows for testing.
• stratify=y — keep the class proportions the same in both splits, so neither set is accidentally easier.
• random_state=42 — make the shuffle reproducible.

Overfitting vs underfitting

The gap you just saw has a name. A model overfits when it learns the training data too well — noise, quirks, and all — so its training score is high but its test score is much lower. It memorized the homework.

The opposite failure is underfitting: the model is too simple to capture the real pattern, so both scores are low. Between the two lies the sweet spot, and controlling model complexity is how you find it:

Python — runs in your browser

Run

from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier X, y = make_classification(n_samples=400, n_features=10, n_informative=3, flip_y=0.25, random_state=42) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42 ) for depth in [1, 3, 5, None]: tree = DecisionTreeClassifier(max_depth=depth, random_state=42) tree.fit(X_train, y_train) name = f"depth={depth}" print(f"{name:12s} train={tree.score(X_train, y_train):.3f} " f"test={tree.score(X_test, y_test):.3f}")

Depth 1 underfits (both scores low), unlimited depth overfits (huge gap), and a moderate depth does best on the test set. You'll see this pattern in every model family for the rest of the course.

The basic workflow

Every supervised project in this course follows the same four beats:

1. Split — train_test_split before anything else touches the data.
2. Fit — model.fit(X_train, y_train) learns patterns from training data only.
3. Predict — model.predict(X_test) produces answers for unseen rows.
4. Evaluate — compare predictions against y_test with a metric such as accuracy.

Everything else — preprocessing, feature engineering, hyperparameter tuning — is elaboration on this skeleton. Memorize the beats; the next lessons add the instruments.

Check your understanding

5 questions · free

1. Q1.What is the key difference between a rule-based system and a machine learning system?

   AML systems are always more accurateBRule-based systems need data, ML systems don'tCIn ML, the decision logic is learned from labeled examples instead of being hand-writtenDRule-based systems can't do classification

2. Q2.In a table of house sales, each row is one house and one column is the sale price you want to predict. What is the sale price column called?

   AA sampleBA featureCThe label (target)DAn estimator

3. Q3.A model scores 99% accuracy on the training set and 68% on the test set. What's the most likely diagnosis?

   AUnderfitting — the model is too simpleBOverfitting — the model memorized the training dataCThe test set is too largeDNothing is wrong; training accuracy is what matters

4. Q4.Why pass stratify=y to train_test_split for classification?

   ATo shuffle the rows more thoroughlyBTo keep class proportions the same in train and test setsCTo make training fasterDTo remove duplicate rows

5. Q5.Which sequence describes the basic supervised ML workflow?

   Afit → split → evaluate → predictBsplit → fit → predict → evaluateCpredict → fit → split → evaluateDevaluate → predict → fit → split

Exercise: Watch overfitting grow with noise

Using the unlimited-depth decision tree from this lesson, generate three datasets with make_classification at noise levels flip_y = 0.0, 0.1, and 0.3 (keep everything else the same). For each, print the train accuracy, test accuracy, and the gap between them. How does label noise affect how badly the tree overfits?

Next up: your first real algorithm — K-Nearest Neighbors, a classifier so intuitive you already use it in daily life.