6 Preprocessing: scaling and encoding

Before feeding data to a model, it’s very useful to apply preprocessing to scale variables, remove trends and outliers, and convert categorical features to numerical values. The objective is to massage data to improve the final score. We’re going to look at the most common preprocessing step for numerical variables (scaling) and for categorical variables (encoding), and at the Pipeline object scikit-learn uses to chain a transformer¹ with a predictor so that the two are fit and used together as a single unit.

6.1 Loading the data

Let’s predict a person’s income from features such as age, marital status, education, hours worked per week, or workclass.

import pandas as pd

adult_census = pd.read_csv("assets/data/income-data-labeled.csv")

target_name = "class"
target = adult_census[target_name]
data = adult_census.drop(columns=target_name)
data.head()

	age	workclass	fnlwgt	education	educational-num	marital-status	occupation	relationship	race	gender	hours-per-week	native-country
0	39	Self-emp-not-inc	327120	HS-grad	9	Married-civ-spouse	Craft-repair	Husband	White	Male	40	Portugal
1	32	Private	123253	Assoc-acdm	12	Married-civ-spouse	Craft-repair	Husband	White	Male	42	United-States
2	47	Private	232628	HS-grad	9	Married-civ-spouse	Craft-repair	Husband	Black	Male	40	United-States
3	19	Private	374262	12th	8	Never-married	Handlers-cleaners	Own-child	White	Male	20	United-States
4	46	Self-emp-not-inc	311231	HS-grad	9	Married-civ-spouse	Farming-fishing	Husband	White	Male	40	United-States

target.value_counts()

class
<=50K    682
>50K     217
Name: count, dtype: int64

target is discrete (<=50K or >50K), so this is a classification problem. data mixes two kinds of columns, which we’ll deal with in turn: numerical columns like age or hours-per-week, and categorical columns like workclass or native-country.

6.2 Numerical preprocessing

6.2.1 Why scale numerical features

Let’s restrict data to the four numerical columns.

numerical_columns = ["age", "capital-gain", "capital-loss", "hours-per-week"]
data_numeric = data[numerical_columns]

from sklearn.model_selection import train_test_split

data_train, data_test, target_train, target_test = train_test_split(
    data_numeric, target, random_state=42
)
data_train.describe()

	age	capital-gain	capital-loss	hours-per-week
count	674.000000	674.000000	674.000000	674.000000
mean	38.608309	632.001484	102.704748	40.860534
std	13.350381	2790.230587	430.380726	12.501821
min	17.000000	0.000000	0.000000	2.000000
25%	28.000000	0.000000	0.000000	40.000000
50%	36.500000	0.000000	0.000000	40.000000
75%	48.000000	0.000000	0.000000	45.000000
max	90.000000	27828.000000	2415.000000	99.000000

The four columns span very different ranges: age is in the tens, hours-per-week is in the tens too, but on a different scale, and capital-gain is in the tens of thousands. Some models are sensitive to this:

Models that rely on distances between samples (e.g. k-nearest neighbours) should be trained on normalised features, so that no single feature dominates the distance just because its raw values happen to be larger.
Models fit by gradient descent (e.g. LogisticRegression’s solver) converge faster, in fewer iterations, when the features are on comparable scales.

Whether scaling matters depends on the model family: linear models generally benefit from it, while decision trees don’t care.

6.2.2 The `StandardScaler` API

StandardScaler shifts and scales each column independently so that it has 0 mean and unit standard deviation. Like any transformer, it has a .fit() step, which learns the per-column mean and standard deviation from the data, and a .transform() step, which applies that learned transformation.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(data_train)

StandardScaler()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

StandardScaler

?Documentation for StandardScaleriFitted

Parameters

	copy copy: bool, default=True If False, try to avoid a copy and do inplace scaling instead. This is not guaranteed to always work inplace; e.g. if the data is not a NumPy array or scipy.sparse CSR matrix, a copy may still be returned.	True
	with_mean with_mean: bool, default=True If True, center the data before scaling. This does not work (and will raise an exception) when attempted on sparse matrices, because centering them entails building a dense matrix which in common use cases is likely to be too large to fit in memory.	True
	with_std with_std: bool, default=True If True, scale the data to unit variance (or equivalently, unit standard deviation).	True

Fitted attributes

Name	Type	Value
feature_names_in_ feature_names_in_: ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0	ndarray[object](4,)	['age','capital-gain','capital-loss','hours-per-week']
mean_ mean_: ndarray of shape (n_features,) or None The mean value for each feature in the training set. Equal to ``None`` when ``with_mean=False`` and ``with_std=False``.	ndarray[float64](4,)	[ 38.61,632. ,102.7 , 40.86]
n_features_in_ n_features_in_: int Number of features seen during :term:`fit`. .. versionadded:: 0.24	int	4
n_samples_seen_ n_samples_seen_: int or ndarray of shape (n_features,) The number of samples processed by the estimator for each feature. If there are no missing samples, the ``n_samples_seen`` will be an integer, otherwise it will be an array of dtype int. If `sample_weights` are used it will be a float (if no missing data) or an array of dtype float that sums the weights seen so far. Will be reset on new calls to fit, but increments across ``partial_fit`` calls.	float64	674
scale_ scale_: ndarray of shape (n_features,) or None Per feature relative scaling of the data to achieve zero mean and unit variance. Generally this is calculated using `np.sqrt(var_)`. If a variance is zero, we can't achieve unit variance, and the data is left as-is, giving a scaling factor of 1. `scale_` is equal to `None` when `with_std=False`. .. versionadded:: 0.17 scale_	ndarray[float64](4,)	[ 13.34,2788.16, 430.06, 12.49]
var_ var_: ndarray of shape (n_features,) or None The variance for each feature in the training set. Used to compute `scale_`. Equal to ``None`` when ``with_mean=False`` and ``with_std=False``.	ndarray[float64](4,)	[ 177.97,7773835.71, 184952.75, 156.06]

4 features

age
capital-gain
capital-loss
hours-per-week

Note

scikit-learn convention: an attribute learned from the data has a name that ends with _. Here, the scaler’s model state is mean_ and scale_, one value per column.

scaler.mean_

array([ 38.60830861, 632.00148368, 102.70474777,  40.86053412])

scaler.scale_

array([  13.34047374, 2788.1599143 ,  430.06133412,   12.49254319])

.transform() then applies the learned shift-and-scale to any data (the training data, or later, new data):

data_train_scaled = scaler.transform(data_train)
data_train_scaled[:5]

array([[-0.64527758, -0.22667333, -0.23881419, -0.06888382],
       [-0.49535787, -0.22667333, -0.23881419,  0.57149819],
       [-1.16999658, -0.22667333, -0.23881419, -0.06888382],
       [ 1.60351812, -0.22667333, -0.23881419,  0.41140269],
       [-0.79519729, -0.22667333, -0.23881419, -0.06888382]])

.fit_transform() is a shorthand for calling .fit() then .transform() on the same data.

scaler = StandardScaler().set_output(transform="pandas")
data_train_scaled = scaler.fit_transform(data_train)
data_train_scaled.describe()

	age	capital-gain	capital-loss	hours-per-week
count	6.740000e+02	6.740000e+02	6.740000e+02	6.740000e+02
mean	2.457645e-16	-7.906633e-18	-3.162653e-17	-2.424701e-16
std	1.000743e+00	1.000743e+00	1.000743e+00	1.000743e+00
min	-1.619756e+00	-2.266733e-01	-2.388142e-01	-3.110698e+00
25%	-7.951973e-01	-2.266733e-01	-2.388142e-01	-6.888382e-02
50%	-1.580385e-01	-2.266733e-01	-2.388142e-01	-6.888382e-02
75%	7.039998e-01	-2.266733e-01	-2.388142e-01	3.313549e-01
max	3.852314e+00	9.754103e+00	5.376664e+00	4.653934e+00

Notice that every column of data_train_scaled now has a mean of 0 and a standard deviation of 1. For convenience, set_output(transform="pandas") makes the transformer return a dataframe (with the original column names) instead of a plain NumPy array.

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].scatter(data_train["age"], data_train["hours-per-week"], alpha=0.4)
axes[0].set_xlabel("age")
axes[0].set_ylabel("hours-per-week")
axes[0].set_title("Before StandardScaler")

axes[1].scatter(data_train_scaled["age"], data_train_scaled["hours-per-week"], alpha=0.4)
axes[1].set_xlabel("age (scaled)")
axes[1].set_ylabel("hours-per-week (scaled)")
axes[1].set_title("After StandardScaler")
plt.tight_layout()
plt.show()

Figure 6.1: Age vs. hours-per-week, before and after StandardScaler. The scaler shifts and rescales the axes; it does not change the shape of the point cloud.

As shown in Figure 6.1, scaling only shifts and stretches the axes; it does not change the relative position of the points.

6.3 The `Pipeline` syntax

Fitting a scaler and a predictor as two separate steps, and remembering to call .transform() before every .predict(), is exactly the kind of bookkeeping scikit-learn’s Pipeline automates. make_pipeline builds one from a sequence of transformers followed by a predictor:

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(StandardScaler(), LogisticRegression())
model

Pipeline(steps=[('standardscaler', StandardScaler()),
                ('logisticregression', LogisticRegression())])

make_pipeline names each step automatically from its class name ("standardscaler", "logisticregression"); model.named_steps exposes them by name if you need to inspect one directly.

model.named_steps

{'standardscaler': StandardScaler(),
 'logisticregression': LogisticRegression()}

The resulting pipeline exposes the same .fit()/.predict()/.score() methods as a bare predictor, but under the hood, it does more:

flowchart LR
    subgraph fit["model.fit(X, y)"]
        direction LR
        A1(("X")) --> B1["scaler.fit_transform"] --> C1["predictor.fit"]
        A2(("y")) --> C1
    end

flowchart LR
    subgraph predict["model.predict(X)"]
        direction LR
        A(("X")) --> B["scaler.transform"] --> C["predictor.predict"] --> D(("ŷ"))
    end

Calling model.fit(X, y) calls .fit_transform() on the scaler to learn and apply the scaling, then hands the scaled data to the predictor’s .fit(). Calling model.predict(X) only calls .transform() on the scaler (it must not re-fit the scaler on new data), then the predictor’s .predict().

model.fit(data_train, target_train)
model.score(data_test, target_test)

0.8177777777777778

Does the scaling actually help here? Let’s compare against a LogisticRegression fit directly on the unscaled data.

model.fit(data_train, target_train)
n_iter_scaled = model[-1].n_iter_[0]

unscaled_model = LogisticRegression()
unscaled_model.fit(data_train, target_train)
n_iter_unscaled = unscaled_model.n_iter_[0]

print(f"Iterations with scaling:    {n_iter_scaled}")
print(f"Iterations without scaling: {n_iter_unscaled}")

Iterations with scaling:    8
Iterations without scaling: 59

The number of iterations is larger without scaling, meaning that the optimisation has to work harder to converge.

Warning

An unscaled model that requires more iterations than max_iter will stop before converging, silently returning a suboptimal fit. Before raising max_iter to silence a convergence warning, check whether scaling the data instead fixes the problem.

We can also get a cross-validated estimate of accuracy for the scaled pipeline: cross_validate doesn’t care whether model is a bare predictor or a Pipeline, only that it has .fit()/.predict().

from sklearn.model_selection import cross_validate

cv_results = cross_validate(model, data_numeric, target)
scores = cv_results["test_score"]
print(f"Accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

Accuracy: 0.794 ± 0.019

6.4 Categorical preprocessing

Unlike a numerical variable, a categorical variable takes discrete values from a finite set of possibilities, typically represented as strings. native-country is one example:

data["native-country"].value_counts().sort_index()

native-country
Cambodia                1
Canada                  4
China                   2
Columbia                3
Cuba                    2
Dominican-Republic      1
Ecuador                 2
El-Salvador             4
England                 2
France                  2
Germany                 3
Greece                  1
Guatemala               1
Haiti                   1
India                   2
Italy                   3
Jamaica                 3
Laos                    1
Mexico                 13
Nicaragua               3
Philippines             3
Poland                  8
Portugal                3
Puerto-Rico             5
South                   2
Taiwan                  3
Thailand                1
Trinadad&Tobago         1
United-States         819
Name: count, dtype: int64

Let’s restrict data to the categorical columns.

categorical_columns = [
    "workclass", "education", "marital-status", "occupation",
    "relationship", "race", "gender", "native-country",
]
data_categorical = data[categorical_columns]
print(f"The dataset is composed of {data_categorical.shape[1]} categorical features")

The dataset is composed of 8 categorical features

6.4.1 Encoding ordinal categories: `OrdinalEncoder`

The most direct strategy is to replace each category with a distinct integer. Let’s try it on a single column first:

from sklearn.preprocessing import OrdinalEncoder

education_column = data_categorical[["education"]]

encoder = OrdinalEncoder().set_output(transform="pandas")
encoder.fit_transform(education_column)

	education
0	11.0
1	7.0
2	11.0
3	2.0
4	11.0
...	...
894	12.0
895	1.0
896	15.0
897	15.0
898	12.0

899 rows × 1 columns

The mapping between categories and integers is stored in the fitted categories_ attribute:

encoder.categories_

[array(['10th', '11th', '12th', '1st-4th', '5th-6th', '7th-8th', '9th',
        'Assoc-acdm', 'Assoc-voc', 'Bachelors', 'Doctorate', 'HS-grad',
        'Masters', 'Preschool', 'Prof-school', 'Some-college'],
       dtype=object)]

The same encoder works on the full categorical dataframe, encoding every column independently:

data_encoded = encoder.fit_transform(data_categorical)
print(f"The dataset encoded contains {data_encoded.shape[1]} features")
data_encoded[:5]

The dataset encoded contains 8 features

	workclass	education	marital-status	occupation	relationship	race	gender	native-country
0	4.0	11.0	2.0	2.0	0.0	4.0	1.0	22.0
1	2.0	7.0	2.0	2.0	0.0	4.0	1.0	28.0
2	2.0	11.0	2.0	2.0	0.0	2.0	1.0	28.0
3	2.0	2.0	4.0	5.0	3.0	4.0	1.0	28.0
4	4.0	11.0	2.0	4.0	0.0	4.0	1.0	28.0

Warning

OrdinalEncoder makes downstream models assume the categories are ordered (0 < 1 < 2 < ...). By default, it orders them lexicographically, which is often meaningless: with categories "S", "M", "L", "XL", the alphabetical order would map them to 2, 1, 0, 3, not the size order we’d want. Use the categories argument to pass an explicit ordering when one exists; when it doesn’t, prefer one-hot encoding instead.

6.4.2 Encoding nominal categories: `OneHotEncoder`

OneHotEncoder does not imply any order: for a given feature, it creates one new column per category, set to 1 for the category a sample belongs to and 0 for all other categories.

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse_output=False).set_output(transform="pandas")
encoder.fit_transform(education_column)

	education_10th	education_11th	education_12th	education_1st-4th	education_5th-6th	education_7th-8th	education_9th	education_Assoc-acdm	education_Assoc-voc	education_Bachelors	education_Doctorate	education_HS-grad	education_Masters	education_Preschool	education_Prof-school	education_Some-college
0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
3	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
894	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0
895	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
896	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0
897	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0
898	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0

899 rows × 16 columns

Note

sparse_output=False is used here only so the result prints as an ordinary dense dataframe. With many categorical columns, one-hot encoding produces mostly-zero matrices, for which scikit-learn’s default sparse output is far more memory-efficient.

Applied to the whole categorical dataframe:

data_encoded = encoder.fit_transform(data_categorical)
data_encoded[:5]

	workclass_Private	workclass_Self-emp-not-inc	education_12th	...	native-country_Portugal	native-country_United-States
0	0.0	1.0	0.0	...	1.0	0.0
1	1.0	0.0	0.0	...	0.0	1.0
2	1.0	0.0	0.0	...	0.0	1.0
3	1.0	0.0	1.0	...	0.0	1.0
4	0.0	1.0	0.0	...	0.0	1.0

5 rows × 85 columns

print(
    f"The encoded dataset contains {data_encoded.shape[1]} features, "
    f"up from {data_categorical.shape[1]} original categorical columns"
)

The encoded dataset contains 85 features, up from 8 original categorical columns

The column count jumps well above the original number of categorical columns, because columns like occupation or native-country each have many possible values, and every one of them gets its own 0/1 column.

Choosing between the two encoders

OneHotEncoder is the usual choice when the downstream model is linear, since it avoids imposing a false ordering. OrdinalEncoder is often fine for tree-based models, which don’t rely on distances between encoded values, and it avoids the column explosion one-hot encoding causes with high-cardinality columns.

6.4.3 A rare-category problem

data["native-country"].value_counts().tail()

native-country
Greece                1
Laos                  1
Guatemala             1
Haiti                 1
Dominican-Republic    1
Name: count, dtype: int64

Some countries appear only once or twice in this data. If such a row appears in the test set during cross-validation, the encoder (trained only on the training set) has never seen that category and would not know how to encode it at prediction time. Passing handle_unknown="ignore" to OneHotEncoder handles this: an unseen category is encoded as all-zeros for that feature’s columns, instead of raising an error.

6.4.4 Fitting a pipeline on the categorical features

model = make_pipeline(
    OneHotEncoder(handle_unknown="ignore"), LogisticRegression(max_iter=500)
)
cv_results = cross_validate(model, data_categorical, target)
scores = cv_results["test_score"]
print(f"Accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

Accuracy: 0.809 ± 0.028

Note

max_iter is raised here because, unlike the scaled numerical features above, one-hot encoded columns are already all on the same 0/1 scale (they wouldn’t benefit from StandardScaler), so increasing max_iter is the right way to help the solver converge.

This categorical-only pipeline scores comparably to, or better than, the numerical-only pipeline from earlier, mostly because it has many more (predictive) columns to work with, one per category value.

6.5 Combining numerical and categorical preprocessing

So far, we’ve scaled the numerical columns and encoded the categorical columns in two separate pipelines, each ignoring the other kind of column. ColumnTransformer applies a different transformer to a different subset of columns, then concatenates the results into a single feature matrix, so one pipeline can use both numerical_columns and categorical_columns, defined earlier, at once.

ColumnTransformer takes a list of (name, transformer, columns) triples: each transformer is fit and applied only to its own subset of columns.

from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer([
    ("standard-scaler", StandardScaler(), numerical_columns),
    ("one-hot-encoder", OneHotEncoder(handle_unknown="ignore"), categorical_columns),
])

preprocessor is itself a transformer, with a .fit_transform(), so it can be the first step of a Pipeline, exactly like StandardScaler or OneHotEncoder were on their own:

Note

data has a couple of columns (fnlwgt, educational-num) that appear in neither numerical_columns nor categorical_columns. By default, ColumnTransformer silently drops any column not listed in one of its transformers, so passing the whole data below is fine; only the two listed groups reach the model.

model = make_pipeline(preprocessor, LogisticRegression(max_iter=500))
model

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(transformers=[('standard-scaler',
                                                  StandardScaler(),
                                                  ['age', 'capital-gain',
                                                   'capital-loss',
                                                   'hours-per-week']),
                                                 ('one-hot-encoder',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  ['workclass', 'education',
                                                   'marital-status',
                                                   'occupation', 'relationship',
                                                   'race', 'gender',
                                                   'native-country'])])),
                ('logisticregression', LogisticRegression(max_iter=500))])

cv_results = cross_validate(model, data, target)
scores = cv_results["test_score"]
print(f"Accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

Accuracy: 0.834 ± 0.018

This combined pipeline uses both groups of columns at once, numerical and categorical, and its cross-validated accuracy improves on both single-kind pipelines from earlier.

6.6 Main takeaways

StandardScaler shifts and rescales numerical columns to comparable ranges. It helps distance-based models and speeds up gradient-based solvers, without changing the relative position of the data points.
OrdinalEncoder maps each category to an integer, which implies an order on the categories, so use it only when that order is meaningful (or a tree-based model is downstream).
OneHotEncoder maps each category to its own 0/1 column, avoiding a false order at the cost of many more columns; handle_unknown="ignore" keeps it robust to categories only seen at prediction time.
A Pipeline (built with make_pipeline) chains transformers and a predictor into one object: .fit() fits and applies every transformer in sequence before fitting the predictor. In contrast, .predict() only applies (never refits) the transformers before calling the predictor’s .predict().
ColumnTransformer applies different transformers to different groups of columns and concatenates the results, so numerical and categorical preprocessing can happen side by side in one step.
Tools like cross_validate treat a Pipeline exactly like any other predictor, since both only need .fit()/.predict().

Recall that a transformer is any estimator with a .transform() method.↩︎

	steps steps: list of tuples List of (name of step, estimator) tuples that are to be chained in sequential order. To be compatible with the scikit-learn API, all steps must define `fit`. All non-last steps must also define `transform`. See :ref:`Combining Estimators <combining_estimators>` for more details.	[('standardscaler', ...), ('logisticregression', ...)]
	transform_input transform_input: list of str, default=None The names of the :term:`metadata` parameters that should be transformed by the pipeline before passing it to the step consuming it. This enables transforming some input arguments to ``fit`` (other than ``X``) to be transformed by the steps of the pipeline up to the step which requires them. Requirement is defined via :ref:`metadata routing <metadata_routing>`. For instance, this can be used to pass a validation set through the pipeline. You can only set this if metadata routing is enabled, which you can enable using ``sklearn.set_config(enable_metadata_routing=True)``. .. versionadded:: 1.6	None
	memory memory: str or object with the joblib.Memory interface, default=None Used to cache the fitted transformers of the pipeline. The last step will never be cached, even if it is a transformer. By default, no caching is performed. If a string is given, it is the path to the caching directory. Enabling caching triggers a clone of the transformers before fitting. Therefore, the transformer instance given to the pipeline cannot be inspected directly. Use the attribute ``named_steps`` or ``steps`` to inspect estimators within the pipeline. Caching the transformers is advantageous when fitting is time consuming. See :ref:`sphx_glr_auto_examples_neighbors_plot_caching_nearest_neighbors.py` for an example on how to enable caching.	None
	verbose verbose: bool, default=False If True, the time elapsed while fitting each step will be printed as it is completed.	False

	penalty penalty: {'l1', 'l2', 'elasticnet', None}, default='l2' Specify the norm of the penalty: - `None`: no penalty is added; - `'l2'`: add an L2 penalty term and it is the default choice; - `'l1'`: add an L1 penalty term; - `'elasticnet'`: both L1 and L2 penalty terms are added. .. warning:: Some penalties may not work with some solvers. See the parameter `solver` below, to know the compatibility between the penalty and solver. .. versionadded:: 0.19 l1 penalty with SAGA solver (allowing 'multinomial' + L1) .. deprecated:: 1.8 `penalty` was deprecated in version 1.8 and will be removed in 1.10. Use `l1_ratio` and `C` instead. `l1_ratio=0` for `penalty='l2'`, `l1_ratio=1` for `penalty='l1'`, `l1_ratio` set to any float between 0 and 1 for `penalty='elasticnet'`, and `C=np.inf` for `penalty=None`.	'deprecated'
	C C: float, default=1.0 Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization. `C=np.inf` results in unpenalized logistic regression. For a visual example on the effect of tuning the `C` parameter with an L1 penalty, see: :ref:`sphx_glr_auto_examples_linear_model_plot_logistic_path.py`.	1.0
	l1_ratio l1_ratio: float, default=0.0 The Elastic-Net mixing parameter, with `0 <= l1_ratio <= 1`. Setting `l1_ratio=1` gives a pure L1-penalty, setting `l1_ratio=0` a pure L2-penalty. Any value between 0 and 1 gives an Elastic-Net penalty of the form `l1_ratio * L1 + (1 - l1_ratio) * L2`. .. warning:: Certain values of `l1_ratio`, i.e. some penalties, may not work with some solvers. See the parameter `solver` below, to know the compatibility between the penalty and solver. .. versionchanged:: 1.8 Default value changed from None to 0.0. .. deprecated:: 1.8 `None` is deprecated and will be removed in version 1.10. Always use `l1_ratio` to specify the penalty type.	0.0
	dual dual: bool, default=False Dual (constrained) or primal (regularized, see also :ref:`this equation <regularized-logistic-loss>`) formulation. Dual formulation is only implemented for l2 penalty with liblinear solver. Prefer `dual=False` when n_samples > n_features.	False
	tol tol: float, default=1e-4 Tolerance for stopping criteria.	0.0001
	fit_intercept fit_intercept: bool, default=True Specifies if a constant (a.k.a. bias or intercept) should be added to the decision function.	True
	intercept_scaling intercept_scaling: float, default=1 Useful only when the solver `liblinear` is used and `self.fit_intercept` is set to `True`. In this case, `x` becomes `[x, self.intercept_scaling]`, i.e. a "synthetic" feature with constant value equal to `intercept_scaling` is appended to the instance vector. The intercept becomes ``intercept_scaling * synthetic_feature_weight``. .. note:: The synthetic feature weight is subject to L1 or L2 regularization as all other features. To lessen the effect of regularization on synthetic feature weight (and therefore on the intercept) `intercept_scaling` has to be increased.	1
	class_weight class_weight: dict or 'balanced', default=None Weights associated with classes in the form ``{class_label: weight}``. If not given, all classes are supposed to have weight one. The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as ``n_samples / (n_classes * np.bincount(y))``. Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified. .. versionadded:: 0.17 class_weight='balanced'	None
	random_state random_state: int, RandomState instance, default=None Only used for `solver` == 'sag', 'saga' or 'liblinear' to shuffle the data. It has no effect on the other solvers. See :term:`Glossary <random_state>` for details.	None
	solver solver: {'lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga'}, default='lbfgs' Algorithm to use in the optimization problem. Default is 'lbfgs'. To choose a solver, you might want to consider the following aspects: - 'lbfgs' is a good default solver because it works reasonably well for a wide class of problems. - For :term:`multiclass` problems (`n_classes >= 3`), all solvers except 'liblinear' minimize the full multinomial loss, 'liblinear' will raise an error. - 'newton-cholesky' is a good choice for `n_samples` >> `n_features * n_classes`, especially with one-hot encoded categorical features with rare categories. Be aware that the memory usage of this solver has a quadratic dependency on `n_features * n_classes` because it explicitly computes the full Hessian matrix. - For small datasets, 'liblinear' is a good choice, whereas 'sag' and 'saga' are faster for large ones; - 'liblinear' can only handle binary classification by default. To apply a one-versus-rest scheme for the multiclass setting one can wrap it with the :class:`~sklearn.multiclass.OneVsRestClassifier`. .. warning:: The choice of the algorithm depends on the penalty chosen (`l1_ratio=0` for L2-penalty, `l1_ratio=1` for L1-penalty and `0 < l1_ratio < 1` for Elastic-Net) and on (multinomial) multiclass support: ================= ======================== ====================== solver l1_ratio multinomial multiclass ================= ======================== ====================== 'lbfgs' l1_ratio=0 yes 'liblinear' l1_ratio=1 or l1_ratio=0 no 'newton-cg' l1_ratio=0 yes 'newton-cholesky' l1_ratio=0 yes 'sag' l1_ratio=0 yes 'saga' 0<=l1_ratio<=1 yes ================= ======================== ====================== .. note:: 'sag' and 'saga' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from :mod:`sklearn.preprocessing`. .. seealso:: Refer to the :ref:`User Guide <Logistic_regression>` for more information regarding :class:`LogisticRegression` and more specifically the :ref:`Table <logistic_regression_solvers>` summarizing solver/penalty supports. .. versionadded:: 0.17 Stochastic Average Gradient (SAG) descent solver. Multinomial support in version 0.18. .. versionadded:: 0.19 SAGA solver. .. versionchanged:: 0.22 The default solver changed from 'liblinear' to 'lbfgs' in 0.22. .. versionadded:: 1.2 newton-cholesky solver. Multinomial support in version 1.6.	'lbfgs'
	max_iter max_iter: int, default=100 Maximum number of iterations taken for the solvers to converge.	100
	verbose verbose: int, default=0 For the liblinear and lbfgs solvers set verbose to any positive number for verbosity.	0
	warm_start warm_start: bool, default=False When set to True, reuse the solution of the previous call to fit as initialization, otherwise, just erase the previous solution. Useless for liblinear solver. See :term:`the Glossary <warm_start>`. .. versionadded:: 0.17 warm_start to support lbfgs, newton-cg, sag, saga solvers.	False
	n_jobs n_jobs: int, default=None Does not have any effect. .. deprecated:: 1.8 `n_jobs` is deprecated in version 1.8 and will be removed in 1.10.	None

	steps steps: list of tuples List of (name of step, estimator) tuples that are to be chained in sequential order. To be compatible with the scikit-learn API, all steps must define `fit`. All non-last steps must also define `transform`. See :ref:`Combining Estimators <combining_estimators>` for more details.	[('columntransformer', ...), ('logisticregression', ...)]
	transform_input transform_input: list of str, default=None The names of the :term:`metadata` parameters that should be transformed by the pipeline before passing it to the step consuming it. This enables transforming some input arguments to ``fit`` (other than ``X``) to be transformed by the steps of the pipeline up to the step which requires them. Requirement is defined via :ref:`metadata routing <metadata_routing>`. For instance, this can be used to pass a validation set through the pipeline. You can only set this if metadata routing is enabled, which you can enable using ``sklearn.set_config(enable_metadata_routing=True)``. .. versionadded:: 1.6	None
	memory memory: str or object with the joblib.Memory interface, default=None Used to cache the fitted transformers of the pipeline. The last step will never be cached, even if it is a transformer. By default, no caching is performed. If a string is given, it is the path to the caching directory. Enabling caching triggers a clone of the transformers before fitting. Therefore, the transformer instance given to the pipeline cannot be inspected directly. Use the attribute ``named_steps`` or ``steps`` to inspect estimators within the pipeline. Caching the transformers is advantageous when fitting is time consuming. See :ref:`sphx_glr_auto_examples_neighbors_plot_caching_nearest_neighbors.py` for an example on how to enable caching.	None
	verbose verbose: bool, default=False If True, the time elapsed while fitting each step will be printed as it is completed.	False

	transformers transformers: list of tuples List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data. name : str Like in Pipeline and FeatureUnion, this allows the transformer and its parameters to be set using ``set_params`` and searched in grid search. transformer : {'drop', 'passthrough'} or estimator Estimator must support :term:`fit` and :term:`transform`. Special-cased strings 'drop' and 'passthrough' are accepted as well, to indicate to drop the columns or to pass them through untransformed, respectively. columns : str, array-like of str, int, array-like of int, array-like of bool, slice or callable Indexes the data on its second axis. Integers are interpreted as positional columns, while strings can reference DataFrame columns by name. A scalar string or int should be used where ``transformer`` expects X to be a 1d array-like (vector), otherwise a 2d array will be passed to the transformer. A callable is passed the input data `X` and can return any of the above. To select multiple columns by name or dtype, you can use :obj:`make_column_selector`.	[('standard-scaler', ...), ('one-hot-encoder', ...)]
	remainder remainder: {'drop', 'passthrough'} or estimator, default='drop' By default, only the specified columns in `transformers` are transformed and combined in the output, and the non-specified columns are dropped. (default of ``'drop'``). By specifying ``remainder='passthrough'``, all remaining columns that were not specified in `transformers`, but present in the data passed to `fit` will be automatically passed through. This subset of columns is concatenated with the output of the transformers. For dataframes, extra columns not seen during `fit` will be excluded from the output of `transform`. By setting ``remainder`` to be an estimator, the remaining non-specified columns will use the ``remainder`` estimator. The estimator must support :term:`fit` and :term:`transform`. Note that using this feature requires that the DataFrame columns input at :term:`fit` and :term:`transform` have identical order.	'drop'
	sparse_threshold sparse_threshold: float, default=0.3 If the output of the different transformers contains sparse matrices, these will be stacked as a sparse matrix if the overall density is lower than this value. Use ``sparse_threshold=0`` to always return dense. When the transformed output consists of all dense data, the stacked result will be dense, and this keyword will be ignored.	0.3
	n_jobs n_jobs: int, default=None Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary <n_jobs>` for more details.	None
	transformer_weights transformer_weights: dict, default=None Multiplicative weights for features per transformer. The output of the transformer is multiplied by these weights. Keys are transformer names, values the weights.	None
	verbose verbose: bool, default=False If True, the time elapsed while fitting each transformer will be printed as it is completed.	False
	verbose_feature_names_out verbose_feature_names_out: bool, str or Callable[[str, str], str], default=True - If True, :meth:`ColumnTransformer.get_feature_names_out` will prefix all feature names with the name of the transformer that generated that feature. It is equivalent to setting `verbose_feature_names_out="{transformer_name}__{feature_name}"`. - If False, :meth:`ColumnTransformer.get_feature_names_out` will not prefix any feature names and will error if feature names are not unique. - If ``Callable[[str, str], str]``, :meth:`ColumnTransformer.get_feature_names_out` will rename all the features using the name of the transformer. The first argument of the callable is the transformer name and the second argument is the feature name. The returned string will be the new feature name. - If ``str``, it must be a string ready for formatting. The given string will be formatted using two field names: ``transformer_name`` and ``feature_name``. e.g. ``"{feature_name}__{transformer_name}"``. See :meth:`str.format` method from the standard library for more info. .. versionadded:: 1.0 .. versionchanged:: 1.6 `verbose_feature_names_out` can be a callable or a string to be formatted.	True

	handle_unknown handle_unknown: {'error', 'ignore', 'infrequent_if_exist', 'warn'}, default='error' Specifies the way unknown categories are handled during :meth:`transform`. - 'error' : Raise an error if an unknown category is present during transform. - 'ignore' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None. - 'infrequent_if_exist' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will map to the infrequent category if it exists. The infrequent category will be mapped to the last position in the encoding. During inverse transform, an unknown category will be mapped to the category denoted `'infrequent'` if it exists. If the `'infrequent'` category does not exist, then :meth:`transform` and :meth:`inverse_transform` will handle an unknown category as with `handle_unknown='ignore'`. Infrequent categories exist based on `min_frequency` and `max_categories`. Read more in the :ref:`User Guide <encoder_infrequent_categories>`. - 'warn' : When an unknown category is encountered during transform a warning is issued, and the encoding then proceeds as described for `handle_unknown="infrequent_if_exist"`. .. versionchanged:: 1.1 `'infrequent_if_exist'` was added to automatically handle unknown categories and infrequent categories. .. versionadded:: 1.6 The option `"warn"` was added in 1.6.	'ignore'
	categories categories: 'auto' or a list of array-like, default='auto' Categories (unique values) per feature: - 'auto' : Determine categories automatically from the training data. - list : ``categories[i]`` holds the categories expected in the ith column. The passed categories should not mix strings and numeric values within a single feature, and should be sorted in case of numeric values. The used categories can be found in the ``categories_`` attribute. .. versionadded:: 0.20	'auto'
	drop drop: {'first', 'if_binary'} or an array-like of shape (n_features,), default=None Specifies a methodology to use to drop one of the categories per feature. This is useful in situations where perfectly collinear features cause problems, such as when feeding the resulting data into an unregularized linear regression model. However, dropping one category breaks the symmetry of the original representation and can therefore induce a bias in downstream models, for instance for penalized linear classification or regression models. - None : retain all features (the default). - 'first' : drop the first category in each feature. If only one category is present, the feature will be dropped entirely. - 'if_binary' : drop the first category in each feature with two categories. Features with 1 or more than 2 categories are left intact. - array : ``drop[i]`` is the category in feature ``X[:, i]`` that should be dropped. When `max_categories` or `min_frequency` is configured to group infrequent categories, the dropping behavior is handled after the grouping. .. versionadded:: 0.21 The parameter `drop` was added in 0.21. .. versionchanged:: 0.23 The option `drop='if_binary'` was added in 0.23. .. versionchanged:: 1.1 Support for dropping infrequent categories.	None
	sparse_output sparse_output: bool, default=True When ``True``, it returns a SciPy sparse matrix/array in "Compressed Sparse Row" (CSR) format. .. versionadded:: 1.2 `sparse` was renamed to `sparse_output`	True
	dtype dtype: number type, default=np.float64 Desired dtype of output.	<class 'numpy.float64'>
	min_frequency min_frequency: int or float, default=None Specifies the minimum frequency below which a category will be considered infrequent. - If `int`, categories with a smaller cardinality will be considered infrequent. - If `float`, categories with a smaller cardinality than `min_frequency * n_samples` will be considered infrequent. .. versionadded:: 1.1 Read more in the :ref:`User Guide <encoder_infrequent_categories>`.	None
	max_categories max_categories: int, default=None Specifies an upper limit to the number of output features for each input feature when considering infrequent categories. If there are infrequent categories, `max_categories` includes the category representing the infrequent categories along with the frequent categories. If `None`, there is no limit to the number of output features. .. versionadded:: 1.1 Read more in the :ref:`User Guide <encoder_infrequent_categories>`.	None
	feature_name_combiner feature_name_combiner: "concat" or callable, default="concat" Callable with signature `def callable(input_feature, category)` that returns a string. This is used to create feature names to be returned by :meth:`get_feature_names_out`. `"concat"` concatenates encoded feature name and category with `feature + "_" + str(category)`.E.g. feature X with values 1, 6, 7 create feature names `X_1, X_6, X_7`. .. versionadded:: 1.3	'concat'

6.1 Loading the data

6.2 Numerical preprocessing

6.2.1 Why scale numerical features

6.2.2 The StandardScaler API

6.3 The Pipeline syntax

6.4 Categorical preprocessing

6.4.1 Encoding ordinal categories: OrdinalEncoder

6.4.2 Encoding nominal categories: OneHotEncoder

6.4.3 A rare-category problem

6.4.4 Fitting a pipeline on the categorical features

6.5 Combining numerical and categorical preprocessing

6.6 Main takeaways

6.2.2 The `StandardScaler` API

6.3 The `Pipeline` syntax

6.4.1 Encoding ordinal categories: `OrdinalEncoder`

6.4.2 Encoding nominal categories: `OneHotEncoder`