Pipeline, Cross-Validation, and GridSearch: the Real Workflow

The rest of lecture 4, the "real work" hiding behind every accuracy number I've shown so far. Without this, every K I picked in the previous two posts was, without exaggeration, an educated guess.

Pipeline: what I already knew, now with classification

Pipeline already showed up in the other playlist: it chains normalization and the model into a single object, so the scaler never sees data that should stay out of training. Here the professor confirms the same conclusion, just on a classification problem with 3 classes:

model = KNeighborsClassifier()
model.fit(X_train, y_train)

Output (no normalization): accuracy 0.67.

Then he normalizes four different ways (min-max by hand, MinMaxScaler, z-score by hand, StandardScaler), and they all match:

Output (any normalization): accuracy 0.92.

And Pipeline lands on the same 0.92, chaining both steps automatically:

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', KNeighborsClassifier())
])
pipeline.fit(X_train, y_train)

Nothing new mechanically (I'd already seen Pipeline from the outside), but the jump from 0.67 to 0.92 is the biggest I've seen so far just from normalizing, reinforcing the reason I already explained: 13 variables at very different scales, and KNN decides everything by distance.

Separating validation from testing

aula04c starts slow: it splits off a slice of training just for validation (X_tr/X_val), tries several K values, and picks the one that wins on validation:

X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2)

for k in range(1, 21, 2):
    model = KNeighborsClassifier(n_neighbors=k)
    model.fit(X_tr, y_tr)
    acc = accuracy_score(y_val, model.predict(X_val))

Output: the best K found was k=9, with 0.759 validation accuracy.

That's already much better than "I tested on the test set and saw which K won" (which would be committing exactly the mistake any stats course warns against: using the test set to pick a hyperparameter is a form of leakage, it biases the final estimate because it stops being about genuinely unseen data). But there's a visible problem here: neither train_test_split call, holdout or validation, fixes random_state, so every time I rerun this cell, X_tr/X_val change, and the winning K can change with it. A single validation split is just one sample, and it can get lucky or unlucky.

A real bug, hiding inside a loop

The professor then moves on to cross-validation, implemented by hand:

def cross_validation(model, X, y, k=3):
    n = int(len(y)/k)
    idx = np.random.permutation(len(y))
    X = X[idx]
    y = y[idx]
    for i in range(k):
        X_tr = np.concatenate([X[:i*n], X[(i+1)*n:]])
        y_tr = np.concatenate([y[:i*n], y[(i+1)*n:]])
        X_val = X[i*n:(i+1)*n]
        y_val = y[i*n:(i+1)*n]
        model.fit(X_tr, y_tr)
        y_pred = model.predict(X_val)
        acc = accuracy_score(y_val, y_pred)
        return acc

This function is worth reading closely, because it teaches a lesson that has nothing to do with machine learning: the return sits inside the for. The loop runs i from 0 to k-1 to build k different folds (the right idea behind cross-validation: every fold becomes validation once, the rest becomes training), but the function exits and returns as soon as the first iteration (i=0) finishes. The other k-1 folds never run. cross_validation(), despite the name, computes just one train/validation split, the exact same limitation as the section before, just hiding behind a name that promises more than it delivers.

Output: cross_validation(model, X_train, y_train) returns 0.723, a single number from a single fold, shuffled differently on every call (because np.random.permutation runs again each time).

That explains something odd that shows up right after: repeated_cross_validation, which calls cross_validation() ten times and averages, kind of works by accident. It isn't doing "10 repeats of real cross-validation" (which would be a full fold pass, repeated 10 times), it's doing repeated holdout: 10 different random splits, each evaluated once. The average of those 10 (0.717, std 0.057, computed right after in the notebook) is still a more stable estimate than a single split, because it cuts down the "I got lucky or unlucky on one split" variance. It just isn't cross-validation in the technical sense of the term: no training point is guaranteed to become validation across those 10 rounds, since that only happens by chance, not by the full-coverage guarantee real k-fold gives you.

Bishop describes exactly this real k-fold (he calls it $S$-fold): split the data into $S$ blocks, use $S-1$ to train and 1 to validate, repeat $S$ times swapping which block is held out, and average the $S$ scores. The important guarantee the professor's cross_validation() loses to the bug: in real k-fold, every point becomes validation exactly once, covering the whole dataset with no overlap. With repeated holdout (even repeated many times), some points might never land in validation, and others might land there repeatedly, purely by the luck of random sampling.

Doing it the right way

Scikit-learn already implements Bishop's $S$-fold correctly:

from sklearn.model_selection import KFold

kf = KFold(n_splits=3, shuffle=True)
accs = []
for train_index, val_index in kf.split(X_train):
    model = KNeighborsClassifier(n_neighbors=5)
    model.fit(X_train[train_index], y_train[train_index])
    accs.append(accuracy_score(y_train[val_index], model.predict(X_train[val_index])))

print(np.mean(accs))

Output: 0.726 with KFold(n_splits=3). With RepeatedKFold(n_splits=3, n_repeats=10) (real k-fold, repeated 10 times with different shuffles, to shrink the estimate's variance even further): 0.676. And the one-line shortcut, cross_val_score, matches both: 0.704 and 0.701 respectively.

Those numbers aren't wildly different from the "buggy" repeated holdout (0.717), and that's expected: even with the broken implementation, the general idea (test on chunks that weren't used for training) already captured most of the signal. The gain from real k-fold is robustness, not necessarily a dramatically different number on this specific dataset. But you don't know that without comparing, which is exactly why it's worth implementing (or using) the correct thing instead of trusting that "it produced a plausible-looking number" means "the code is right."

GridSearchCV: automating the search

Instead of writing a for k in range(1, 21, 2) every time, GridSearchCV runs the search (and the cross-validation behind it) automatically:

params = {'n_neighbors': range(1, 21, 2)}
grid = GridSearchCV(KNeighborsClassifier(), params, scoring='accuracy')
grid.fit(X_train, y_train)
print(grid.best_params_, grid.best_score_)

Output: {'n_neighbors': 1}, with a cross-validation score of 0.761.

And expanding the search to three hyperparameters at once (n_neighbors, weights, metric), with KFold(n_splits=5):

Output: {'metric': 'manhattan', 'n_neighbors': 11, 'weights': 'distance'}, score 0.803.

Bishop already warns about exactly this situation in chapter 1.3: once you have more than one hyperparameter to tune, testing every combination by hand turns into a combinatorial explosion fast. GridSearchCV is organized brute force: it tries every combination in the grid, cross-validates each one, and returns the best.

Pipeline + GridSearch: this is where it takes off

The turning point of the whole post: putting GridSearchCV inside a Pipeline, together with the scaler, and tuning even the KNeighborsClassifier's own hyperparameters (using the model__ prefix to point at which pipeline step each parameter belongs to):

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', KNeighborsClassifier())
])
params = {
    'model__n_neighbors': range(1, 21, 2),
    'model__weights': ['uniform', 'distance'],
    'model__metric': ['euclidean', 'manhattan', 'minkowski'],
}
grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True))
scores = cross_val_score(grid, X_train, y_train, cv=KFold(n_splits=5, shuffle=True))
print(np.mean(scores))

Notice the structure: there's a cross_val_score wrapped around an entire GridSearchCV. That's nested cross-validation: the outer loop measures how well the whole process (normalize, search for the best hyperparameters, train) generalizes, and the inner loop (inside GridSearchCV) only picks the hyperparameters. Without that nesting, GridSearchCV's own cross-validation score (best_score_) runs slightly optimistic, because the same data that chose the hyperparameters also evaluated the final result.

Output: average of 0.957 (versus 0.81 without normalizing inside the pipeline, and versus 0.68 for raw KNN with no grid at all). Adding scaler__with_mean and scaler__with_std to the search grid (letting even the normalization be part of what's optimized): 0.979.

From raw KNN (0.67-0.68) to the full pipeline with nested hyperparameter search (0.979): the entire distance between "I ran the default model" and "I did this properly."

Faster than a full grid: random search

RandomizedSearchCV only tries a random sample of combinations (here, 20) instead of all of them:

grid = RandomizedSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True), n_iter=20)

Output: 0.957, basically tied with the full grid, while testing far fewer combinations.

That makes sense once the search grid gets too big to fully test (here it's already $10 \times 2 \times 3 = 60$ combinations, each with 5 folds, 300 model fits. Add more hyperparameters and this explodes fast). The notebook takes a quick peek at Optuna, a Bayesian search library that picks the next combination to try based on what's worked so far instead of sampling or testing blindly, but that's just a passing mention, not the lecture's focus.

Wrapping up

Practical application

I reproduce the full pipeline (normalization + hyperparameter search + nested cross-validation) on the same wine dataset, with a fixed seed for a reproducible result, something none of the searches in the original notebook have.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

pipeline = Pipeline([('scaler', StandardScaler()), ('model', KNeighborsClassifier())])
params = {
    'model__n_neighbors': range(1, 21, 2),
    'model__weights': ['uniform', 'distance'],
    'model__metric': ['euclidean', 'manhattan', 'minkowski'],
}
grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True, random_state=42))
grid.fit(X_train, y_train)

The nested cross-validation score (0.9791) and the test accuracy (0.9722) land close to each other, and that's exactly what I want to see: it means the nested cross-validation wasn't running too optimistic, it genuinely predicted how the pipeline would do on data it had never seen. Against raw KNN (0.7222), the entire gap (25 percentage points) came just from normalizing and picking hyperparameters properly, without touching the algorithm itself.