Linear Regression with Scikit-Learn

After four posts implementing gradient descent, normalization, and feature engineering entirely by hand, it's finally time to use the real tool. And the first shock was finding out the "default" model the course uses here isn't quite what it looks like.

The API convention, more important than the model itself

scikit-learn ships ready-made, tested implementations of most of what I've already built by hand. But what matters most to learn here isn't the specific model, it's the API convention, because it repeats identically across hundreds of models and transformers:

Learned attributes end with a trailing underscore: coef_, intercept_, mean_, scale_, n_iter_. That trailing underscore is the convention distinguishing "learned from the data" from "configured by me."

The golden rule that comes with it: fit only ever gets to see training data. On test data, only transform and predict. Breaking that is data leakage, and it's the most common mistake for anyone starting out (I already hit this note in the previous two bonus posts).

StandardScaler: the z-score I already built by hand

StandardScaler does exactly the math I implemented in the normalization post: $x \leftarrow \frac{x-\mu}{\sigma}$, column by column. I checked and it matches my manual z-score digit for digit, including the same population-standard-deviation convention (divide by $m$, not $m-1$) I was already using.

SGDRegressor and what the "S" means

The course uses SGDRegressor without ever explaining the acronym. Worth pausing on, because it's the real difference between this model and the gradient descent I implemented in the previous posts.

Batch gradient descent (what I've done so far): every step uses all $m$ examples to compute the gradient.

$\mathbf{w} \leftarrow \mathbf{w} - \alpha\,\frac{1}{m}\sum_{i=0}^{m-1}\left(f(\mathbf{x}^{(i)}) - y^{(i)}\right)\mathbf{x}^{(i)}$

Stochastic gradient descent (SGD): every step uses a single example, picked at random.

$\mathbf{w} \leftarrow \mathbf{w} - \alpha\left(f(\mathbf{x}^{(i)}) - y^{(i)}\right)\mathbf{x}^{(i)}$

Dirt-cheap step, noisy direction, a zigzag path, but $m$ updates for the price of one batch step. I built an SGD engine from scratch (the same GradientDescentSimulator as always, just swapping the gear underneath) so you can see the difference live, on the same 8-house dataset from the normalization post:

Batch:

Stochastic:

Click "Rodar 100" on both with the same default alpha. After 50 steps, batch already has cost near 922 (very close to the real minimum, 919), while stochastic, with the same number of steps but each one only seeing one house at a time, is still bouncing around 7102. The stochastic path on the chart is visibly "messier" too, back and forth instead of a smooth descent.

One full pass through the examples is called an epoch. In scikit-learn, max_iter counts epochs, not individual updates.

With a constant learning rate, SGD never fully stops trembling around the minimum. That's why scikit-learn defaults to a rate that decreases over training.

Two things max_iter hides

I ask for max_iter=1000 and SGDRegressor usually stops well before that. Not a bug: there's early stopping (tol and n_iter_no_change) that detects when the loss has genuinely stopped improving and halts on its own. max_iter is a ceiling, not a target.

And without fixing random_state, every fit samples a different order for the examples, and the result changes run to run. The variation is usually small, but real, especially on smaller or harder datasets. Fixing the seed is what makes the result reproducible.

The surprise: by default, this is Ridge, not plain least squares

This is the one that caught me most off guard. SGDRegressor's defaults are penalty='l2' and alpha=0.0001. In other words, by default it doesn't minimize

$J(\mathbf{w},b) = \frac{1}{2m}\sum_i \left(f(\mathbf{x}^{(i)}) - y^{(i)}\right)^2$

it minimizes

$J(\mathbf{w},b) = \frac{1}{2m}\sum_i \left(f(\mathbf{x}^{(i)}) - y^{(i)}\right)^2 + \alpha\|\mathbf{w}\|^2$

That extra term is exactly the L2 regularization I built from scratch in the previous bonus post. With $\alpha=0.0001$ (the default) the effect is too small to notice on this dataset, but the mechanism is real. I fit with $\alpha=1$ (much stronger, just to make it visible) on our 100-house dataset:

The weight shrinks, the training error gets a little worse, exactly the trade-off I already saw in the Ridge post. Know the defaults of the library you're using: an SGDRegressor() called with no arguments isn't "raw linear regression," it's Ridge with a small, discrete $\alpha$.

Predicting, and a fragile way to compare

Two mathematically equivalent calculations can differ in the last bit of precision because of floating-point operation order. Comparing predictions with == is fragile (0.1 + 0.2 == 0.3 is false in any language using 64-bit floats, for the exact same reason). The right way is to check the difference falls within a small tolerance, not demand exact equality.

Metrics: how good is this model, really

Up to now I'd used the cost $J$ to train, but never a metric meant to be read by a person. I fit the model with all 4 features on the full 100-house dataset and computed three standard metrics:

For reference: a "dumb" model that always guesses the mean has an RMSE of 104.07. Ours misses 4.96 times less. That's the real payoff of having a model, not just the R² number alone.

SGDRegressor or LinearRegression? The real choice

The course presents SGDRegressor as "scikit-learn's linear regression," but in practice it's the least common choice. LinearRegression solves the normal equation in closed form, $\boldsymbol{\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}$, no iteration, no learning rate, no random seed. The cost grows with the cube of the number of features, $O(n^3)$. SGDRegressor iterates, costs $O(mn)$ per epoch, and never needs everything loaded into memory at once.

Rule of thumb: I default to LinearRegression (or Ridge). I only reach for SGDRegressor when the data doesn't fit in memory, arrives as a stream, or the feature count is huge.

Wrapping up

Three takeaways:

1. The fit/predict/transform/score convention matters more than memorizing one specific model, it repeats across the whole library.
2. SGD trades precision for speed per step: same general direction, a cheaper, noisier path.
3. Know the defaults of what you're using: penalty='l2' turned on by default in SGDRegressor is exactly the kind of detail that changes what your code is actually doing.

Practical application

Same real housing dataset from the previous posts. I already know batch gradient descent, with square_feet/100 and price/1000, alpha=0.01, needs 4000 full iterations to reach $(w,b) = (116.5, 398.3)$, final cost 5189.72. Each one of those 4000 iterations looks at all 50 houses, so that's 200 thousand example evaluations total.

I ran the stochastic version with the same alpha, but counting individual steps instead of full iterations:

w, b, hist = sgd_gradient_descent(
    square_feet_norm, price,
    w_in=0, b_in=0,
    alpha=0.01, num_steps=4000)  # 4000 examples seen, not 4000 full passes

print(f"(w, b) found: ({w:.1f}, {b:.1f})")

Output: (w, b) found: (112.1, 396.9), final cost 5233.79

Practically the same result as batch (cost 5233.79 versus 5189.72), using 50 times fewer example evaluations (4000 versus 200 thousand). Compare both live:

Loading real data...

Loading real data...

Click "Rodar 2000" on each and notice: the stochastic one reaches a good neighborhood much faster in terms of total work, even with the messier path on the chart.