Recall from the previous chapter that any (supervised) ML method can be described by:
X: the input variable, always observed.y: the output variable, not observed except on the training data.This chapter zooms in on two of these elements: the response and the error metric. Together, they determine which tools apply to a given problem.
As a reminder, here’s the data matrix X and target y for our two running examples:
import pandas as pd
penguins = pd.read_csv("assets/data/penguins.csv", sep=" ")
X_penguins = penguins[["bill_length_mm", "bill_depth_mm", "flipper_length_mm", "body_mass_g"]]
y_penguins = penguins["species"]
X_penguins.head()| bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | |
|---|---|---|---|---|
| 0 | 39.1 | 18.7 | 181 | 3750 |
| 1 | 39.5 | 17.4 | 186 | 3800 |
| 2 | 40.3 | 18.0 | 195 | 3250 |
| 3 | 36.7 | 19.3 | 193 | 3450 |
| 4 | 39.3 | 20.6 | 190 | 3650 |
X_penguins has shape n_samples x n_features, and y_penguins (the species) has length n_samples, one label per row of X_penguins.
Depending on y, supervised learning methods fall into two families:
y is discrete, made of a finite number of classes, e.g. Adelie, Chinstrap, Gentoo.
y is continuous, a numerical quantity, e.g. income.Unsupervised learning has no y at all: there’s only X, and the goal is to find structure within it (e.g. grouping similar observations). That’s for later in the course.
Again, the distinction between regression and classification depends on the type of output. Inputs can be quantitative or qualitative, independent of the output.
In practice, categorical outputs are encoded using numbers:
Class 3 would be encoded by [0, 0, 1, 0, 0], a vector of 5 elements, zero everywhere except in the third position. The other approach is to arbitrarily order the classes from \(1\) to \(K\) and assign an integer to each class.Recall, there is a third kind of variable: ordinal, made of categories with a natural order but no meaningful notion of distance, e.g. small/medium/large. The gap between “small” and “medium” need not equal the gap between “medium” and “large”. The tools for handling this type are slightly different from those we’re going to discuss here.
Notation conventions. We’ll try to stick to the following throughout the course:
X (uppercase) denotes an input data matrix. One observation is \(X_j\), or X[j] in Python. \(y\) (lowercase) denotes a generic output. In our code, it’ll often be y_true.y_pred.Once we’ve picked a model, we need a way to say how good its predictions are. An error metric compares a model’s predictions y_pred to the actual values y_true and returns a single number summarising the fit.
The right metric to use depends on the type of response:
| Task | Metric | What it measures | Direction |
|---|---|---|---|
| Classification | Accuracy | Fraction of predictions that are exactly correct | Higher is better |
| Classification | F1 score | Balance of precision and recall | Higher is better |
| Regression | Mean squared error (MSE) | Average squared difference between predicted and true values | Lower is better |
Some metrics are scores: higher values mean the model is doing better (accuracy, F1). Others are losses or errors: smaller values mean the model is doing better (MSE).
Always check which kind of metric you’re looking at before comparing models. Picking the model with the highest MSE, or the lowest accuracy, is a very common mistake.
Accuracy. The proportion of predictions that match the true label:
\[ \text{accuracy} = \frac{\text{number of correct predictions}}{\text{total number of predictions}} \]
F1 score. The harmonic mean of precision and recall, for classification tasks (particularly useful when classes are imbalanced, e.g. detecting a rare disease). For this metric, classes are not treated equally: one is often the discovery, the other the absence of discovery.
Mean squared error (MSE). The average of the squared differences between predicted and true values, for regression tasks:
\[ \text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 \]
where \(y_i\) is the true value and \(\hat{y}_i\) is the predicted value for observation \(i\).
scikit-learn is the standard Python library for classical machine learning: models, preprocessing, and metrics. Its sklearn.metrics module implements the error metrics above.
Every metric function in sklearn.metrics follows the same syntax:
metric_function(y_true, y_pred)y_true comes first: the ground-truth values.y_pred comes second: the model’s predictions.This order (y_true, then y_pred) is a scikit-learn-wide convention. Getting it backwards doesn’t raise an error for metrics like accuracy (the formula is symmetric), but it silently gives the wrong answer for others. Get in the habit of writing y_true first.
Classification example. Let’s fit a model to predict penguin species from measurements, then score its predictions.
We haven’t covered how models like LogisticRegression work yet. For now, treat .fit(X, y) and .predict(X) as a black box that learns a rule from data and applies it.
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
model = LogisticRegression(max_iter=1000)
model.fit(X_penguins, y_penguins)
y_true = y_penguins
y_pred = model.predict(X_penguins)
accuracy_score(y_true, y_pred)/usr/local/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:599: ConvergenceWarning: lbfgs failed to converge after 1000 iteration(s) (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT
Increase the number of iterations to improve the convergence (max_iter=1000).
You might also want to scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(0.9970760233918129f1_score(y_true, y_pred, average="macro")0.9964307541865297average="macro" computes the F1 score for each class separately and averages the results, unweighted. It’s the standard choice for multiclass problems; scikit-learn will raise an error if you omit it with more than two classes.
Notice that we scored the model on X_penguins/y_penguins, the very data it was trained on. This is a mistake we won’t make later: a high score here doesn’t tell us how the model performs on new penguins. We’ll fix this with a train/test split.
Regression example. Let’s predict body_mass_g from the other measurements:
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
X_mass = penguins[["bill_length_mm", "bill_depth_mm", "flipper_length_mm"]]
y_mass = penguins["body_mass_g"]
reg = LinearRegression()
reg.fit(X_mass, y_mass)
y_true = y_mass
y_pred = reg.predict(X_mass)
mean_squared_error(y_true, y_pred)152957.1897089392The same y_true/y_pred syntax, the same import location (sklearn.metrics), regardless of the task. Only the direction of “better” changes.