Just a Little Extra: Scikit-Learn Pitfalls

This post doesn't come from any required section of the lab, it's what was left over after I split off the essentials of scikit-learn and stochastic gradient descent into the main post. These are the traps that only show up once the code leaves the educational notebook and heads into a real project.

fit_transform and transform aren't the same thing

This is where most people slip. fit_transform learns and applies. transform only applies what was already learned. On training data, fit_transform. On anything else (validation, test, production), transform.

I split the 100-house dataset into 75 train / 25 test and compared size's mean computed both ways:

mu_correct = size[train].mean()          # correct: train only
mu_wrong = size[test].mean()             # wrong: if I recomputed on test

Output: train mean = 1449.93, test mean = 1305.04, a difference of 144.89

If I called fit_transform on the test set instead of transform, the scaler would start using 1305.04 instead of 1449.93, a difference large enough to distort any prediction made afterward. And then the validation metric ends up looking better than it actually is, without me noticing, because the model "saw" statistics from data that should have been unknown.

SGD without normalizing simply explodes

Unlike LinearRegression, normalizing here isn't optional. With features on very different scales, the default step size becomes a giant leap in the direction of the largest-scale feature. I ran stochastic gradient descent directly on the raw data (no normalization), on house size in sqft:

The same scale phenomenon I already saw with batch gradient descent in earlier posts, just now with SGD. The difference is that here normalizing isn't just "recommended," it's practically mandatory, and the reason is that SGD already starts from a shakier place before scale even enters the picture: every step uses the gradient of a single example, not the smoothed-out average over the whole dataset like batch does, so the step is already jumping around noisily from example to example. Multiply that noisy gradient by a feature on a giant scale (house size in square feet, in the thousands) and the step size starts swinging wildly from one iteration to the next, with none of the batch's averaging left to absorb the jolt. It's the same wrong-direction problem as always, just without the shock absorber batch got for free.

Pipeline and cross-validation: the professional way

Every metric I showed in the main post was measured on training data, it tells me how much the model memorized, not how well it generalizes. And there's a second problem: if I normalize once and then run cross-validation, the scaler has already seen the data from every fold. Leakage again, just more subtle.

Pipeline fixes both: it chains normalization and the model into a single estimator, and on every fit (including inside each cross-validation fold) the scaler gets refit using only that fold's training portion.

I ran a real 5-fold cross-validation, comparing LinearRegression and Ridge(alpha=1):

Notice the validation average (0.945) is lower than the training R² I cited in the main post (0.9594), exactly as expected: validation is always more honest than training. And the two models end up nearly tied, with 100 examples and 4 features there isn't excess capacity Ridge needs to hold back.

The other pitfalls, in a list

• intercept_ is an array in SGDRegressor and a scalar in LinearRegression. Code assuming one of the two shapes breaks on the other.
• max_iter is a ceiling, not a target (already covered in the main post). Always check n_iter_.
• penalty='l2' is on by default in SGDRegressor (main post). For pure least squares, pass penalty=None.
• random_state isn't optional if you want a reproducible result.
• ConvergenceWarning means the model hit max_iter without satisfying tol. Don't ignore it: raise max_iter, adjust the learning rate, or check whether you normalized.
• Normalizing the target y isn't done automatically by any of this. If y has an extreme magnitude, consider normalizing it, and remember to denormalize predictions before reporting any metric.
• fit resets the model from scratch. For incremental training, with data arriving gradually (instead of all at once), there's partial_fit: the model updates itself without ever needing to see the whole dataset at once. That's the real use case behind "data that doesn't fit in memory" I mentioned in the main post, and it's literally what the main post's stochastic simulator already shows: every step only sees one house, never the whole dataset at once, and the model still walks toward the same place batch does.

Exercises

Try them before opening the answer.

Exercise 1: reproduce the scaler

Implement your own z-score normalizer with fit, transform, and fit_transform methods, following the trailing-underscore attribute convention. Check it matches the result from the normalization post.

class MyScaler:
    def fit(self, X):
        self.mean_ = X.mean(axis=0)
        self.scale_ = X.std(axis=0)
        self.scale_[self.scale_ == 0] = 1.0
        return self
    def transform(self, X):
        return (X - self.mean_) / self.scale_
    def fit_transform(self, X):
        return self.fit(X).transform(X)

Exercise 2: the effect of SGD's learning rate

Run stochastic gradient descent with a constant rate at several values. What happens at both extremes?

Exercise 3: how much is the hidden Ridge costing?

Compare penalty=None against several $\alpha$ values using cross-validation. At what point does regularization start hurting? What if the dataset only had 20 examples instead of 100?

Exercise 4: cross-validation with and without Pipeline

Compare cross-validation R² in two scenarios: normalizing once beforehand and feeding the result into cross-validation, versus using a Pipeline. Do the numbers match?

Exercise 5: does the model generalize to houses outside the range?

Set aside the 20 most expensive houses as test and train on the remaining 80. What happens to the test R²?