Feature Engineering and Polynomial Regression

This is Week 2's fourth lab, and what I expected to be "just another math trick" ended up being the post that made me replan the most. The core idea fits in one sentence, but its consequences (including one the original lab never mentions at all) took over the whole post.

The core idea: linear in the parameters, not the feature

Linear regression fits models of the form

$f_{\mathbf{w},b}(\mathbf{x}) = w_0x_0 + w_1x_1 + \ldots + w_{n-1}x_{n-1} + b$

What makes this "linear" is the relationship between the output and the parameters $\mathbf{w}, b$, not the relationship between the output and the original variable. Nothing stops $x_1$ from being $x^2$ itself, or $\log x$, or the product of two other columns. No matter how I tune $w$ and $b$, the equation never becomes a curve if the features are straight lines. But if I engineer a curved feature and plug it in, the exact same machinery fits the curve. That's feature engineering.

Polynomial features in practice

I tested this with a simple target: $y = 1 + x^2$, for $x$ from 0 to 19. First, the naive attempt: feeding the model raw $x$.

x = list(range(20))
y = [1 + xi**2 for xi in x]

Drag the degree to 1 in the simulator below and notice: no value of $w$ and $b$ fixes this, because the model is a line and the target is a parabola. Now drag it to 2:

At degree 2 the fit becomes exact, zero RMSE. What I swapped wasn't the algorithm, it was the feature: instead of $x$, I fed in $x^2$ (plus the rest of the polynomial) and let the same old linear regression find the right weight.

Choosing features, without knowing the answer beforehand

Up there I already knew the right term was $x^2$. In practice, you don't. One strategy is to throw several candidates at it and let the fit decide: I tested $y = w_0x + w_1x^2 + w_2x^3 + b$ against a pure $x^2$ target, and the solver zeroed out $x$'s and $x^3$'s weights on its own, leaving basically only $x^2$'s standing. (The original notebook runs this same test with gradient descent and only manages to shrink the wrong weights, not zero them, because convergence never fully finishes. With an exact solution, the result comes out clean.)

Another way to think about it: after creating the features, I'm still doing linear regression. So the best feature is the one with a linear relationship to the target. That turns into an easy-to-compute Pearson correlation:

$x^2$ has perfect correlation because it is the target, up to scale. I already mentioned this same heuristic (plot feature against target, look for the line) in the normalization post, it's the same idea, just now applied to choose the right shape of the feature, not just its scale.

Normalizing, again (the extreme version)

Polynomial features are the most extreme case of different scales I've seen so far: $x$ goes up to 19, $x^2$ up to 361, $x^3$ up to 6859, a ratio of 361 times between $x$ and $x^2$ alone. The same trick from the previous post fixes it: z-score each column, computed from training data only. The component I built for this post already normalizes under the hood before solving, exactly like I did back there.

A genuinely complicated function

With feature engineering I can model much wilder things than a parabola. I tested $y = \cos(x/2)$ with a polynomial up to degree 13:

Drag it up to degree 13: RMSE drops to $1.1\times10^{-5}$, a nearly perfect fit on the 20 training points. And here I deliberately swapped tools: instead of gradient descent with an $\alpha$ hand-tuned for every degree (what the original notebook does, and it's a lot of work), this component solves via the normal equation, the same exact solution I already used to check results in the previous post. That makes exploring many degrees instant, no alpha-hunting needed, leaving all the attention for what actually matters here: what happens once the model gets too flexible.

The elephant in the room: overfitting

I just fit 14 parameters (13 weights + the bias) to 20 data points. The fit looked gorgeous. That should raise an alarm, not a celebration, and that's exactly what the original lab never does.

First surprise: without noise, flexibility isn't a sin

I split the 20 points into 10 train (even indices) and 10 test (odd indices), with zero noise in the data:

I noticed something: at degree 9 (10 parameters for 10 training points, the exact boundary for a unique solution), the ratio looks catastrophic (24529 times!), but the absolute test error stays small (0.009). That contradicts the slogan "lots of parameters always cause overfitting." The correct statement is more subtle: overfitting is the model fitting noise, not simply having many parameters. With no noise to fit, a flexible model interpolates well, even at the extreme edge.

With noise, the phenomenon shows up for real

Real data always has noise. I added modest noise (standard deviation 0.15) and redid the test, now comparing several degrees at once:

Train error only ever drops (more parameters always fit what's already been seen at least as well, this is nearly a theorem, not a coincidence). Test error drops, hits a minimum around degree 6, then climbs sharply, reaching 1.72 at degree 9, worse than a degree-1 fit. That minimum is what matters, not whichever degree zeroes the training error.

Try it yourself, using only the 10 noisy training points (drag the degree and watch the live test RMSE, computed on the 10 points the fit never saw):

Extrapolation: where it gets genuinely dangerous

Everything so far was inside the training range ($x$ from 0 to 19). Outside it, the behavior surprised me, and not in the way I expected. Drag the degree to 13 in the simulator below (the green band marks where the training data actually was) and notice: right past the edge, the degree-13 fit stays better than a low degree for a few points, because it learned the curve's shape with a lot of precision right at the boundary. That's exactly the trap: false confidence. Keep dragging the axis forward, and the same degree 13 that looked safe rockets off to infinity much faster than any low degree.

I computed the numbers to confirm what the eye sees: at $x=22$, three units past the boundary, degree 3 already misses by $-5.42$ (the true value is $0.004$) while degree 13 lands at $0.03$, nearly perfect. But by $x=30$, degree 3 has only grown to $-31$ (bad, but growing slowly), while degree 13 is already at $86$, an absurd value for a function that never leaves $[-1,1]$. The rule of thumb I'm keeping: never trust a polynomial prediction outside the training range, especially at higher degree, because the disaster isn't immediate, it's treacherous.

Wrapping up

Three takeaways:

1. Feature engineering doesn't change the algorithm, it changes the data going into it. The same linear regression learns any curve if you hand it the right feature.
2. Overfitting is about noise, not parameter count: a flexible model with no noise to fit generalizes well, even at the extreme edge. The danger shows up once there's noise to memorize.
3. Extrapolating with a high-degree polynomial is treacherous: it can look better than a low degree right past the edge of the data, and still blow up far worse a bit further out.

One question remains: can I use all the flexibility of a high degree without paying overfitting's price? There's an answer with a continuous dial instead of the discrete choice of degree, called regularization, and it's the subject of the next post.

Practical application

Same real 50-house dataset from the previous posts. In the normalization bonus post I already left a promise: "feature engineering is the topic of the next lab." Following through: I created the feature size_per_bedroom = square_feet / num_bedrooms and compared the fit with and without it.

size_per_bedroom = [s / b for s, b in zip(square_feet, num_bedrooms)]

rmse_before = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center], price)
rmse_after = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center, size_per_bedroom], price)

Output: RMSE without the new feature: 67.25 / RMSE with the new feature: 65.13 (thousand dollars)

A real, if modest, improvement (about 3%). Curious detail: the direct correlation between size_per_bedroom and price is nearly zero (0.0069, very close to none), so if I'd only looked at the isolated correlation I'd have discarded this feature. It only helps once it joins the model together with the others, the same kind of hidden effect I already saw with the bedrooms coefficient in the previous post.