Just a Little Extra: Feature Scaling Pitfalls

This post doesn't come from any required section of the lab, it's what was left over after I split off the essentials of normalization into the main post. These are the traps I'd only run into on a real project, outside the comfort of an educational notebook.

Data leakage: normalizing before splitting train and test

The most common mistake of all. If I compute $\mu$ and $\sigma$ using the whole dataset (train and test together) before splitting, information from the test set leaks into training, and my validation metric ends up looking better than it actually is, without me noticing.

I split the 100-house dataset into 80 train / 20 test and compared size's mean computed both ways:

mu_all = X_train.mean(axis=0)          # WRONG: uses train + test
mu_train_only = X_train[i_tr].mean(axis=0)  # RIGHT: train only

Output: mean with everything combined = 1413.7, train-only mean = 1385.3, a difference of 28.4.

That doesn't look like much on its own, so I planted a worse scenario: a giant 9000 sqft house showing up only in the test set. If I computed $\mu$ using the whole dataset (including that "invisible" house that should still be locked away in test), the mean would jump to 1488.8, a shift of over 100 units caused by one single row I shouldn't have even looked at yet.

The rule I carry with me: fit only on training data, transform on everyone. This applies to anything that learns statistics from the data before using it (normalization, dimensionality reduction, feature selection), not just linear regression. scikit-learn's Pipeline (shown further down) exists basically to make this mistake impossible to make by accident.

Constant feature: division by zero

If a column has $\sigma_j = 0$ (everyone shares the same value), the z-score formula divides by zero and produces inf or nan.

mu, sigma = 1413.71, 0.0   # sigma is zero, constant column
(1500 - mu) / sigma

Output without protection: inf

def safe_zscore(X, eps=1e-12):
    mu = X.mean(axis=0)
    sigma = X.std(axis=0)
    constant = sigma < eps
    sigma_safe = np.where(constant, 1.0, sigma)  # forces sigma=1 on constant columns
    return (X - mu) / sigma_safe, mu, sigma_safe

Output with protection: 86.29 (the constant column becomes zero instead of inf, without breaking the rest of the calculation)

A constant column carries zero information for the model either way, so locking $\sigma$ at 1 is just a safe way to "turn off" that feature without crashing the whole program.

When I do NOT normalize

• Decision trees, Random Forest, Gradient Boosting: these models split by thresholds on each isolated feature. Scale doesn't matter to them at all, normalizing just wastes processing time.
• Normal equation / LinearRegression: solved in closed form, no iterative step involved. It doesn't need normalization (though very extreme scales can still cause numerical precision issues).
• When the unit matters for interpretation: if I need to say "each extra sqft is worth $X", I have to denormalize the coefficients back (I did that in the main post's denormalization section).

I always normalize for: gradient descent, SVM, k-NN, k-means, PCA, neural networks, and any model with L1/L2 regularization (otherwise the penalty unfairly falls on small-scale features).

What about the target $y$? Not required for linear regression with gradient descent, but it helps if $y$ has a huge magnitude (avoids inf in the cost). If I normalize the target, I need to remember to denormalize the predictions before reporting any metric to anyone.

Checking against an exact solution

Is my gradient descent implementation correct? I don't have to trust it blindly. Linear regression has a closed-form solution (the normal equation), so I can compare my iterative result against the exact answer.

theta_exact = np.linalg.lstsq(X_aug, y_train, rcond=None)[0]
w_exact, b_exact = theta_exact[:4], theta_exact[4]

In the main post I already showed that, with a single feature, denormalizing my gradient descent's result ($w = 0.267$, $b = 7.16$) matches digit for digit what the closed-form formula gives directly in raw space. I ran the same check with all 4 real features: the theoretical minimum cost from the normal equation and my gradient descent's cost after 1000 iterations end up a negligible distance apart. Both calculations agree, which gives me a lot more confidence that I didn't write a silent bug into the implementation.

I don't have scikit-learn installed in this environment to run a third comparison method (LinearRegression/SGDRegressor), but the principle is the same as in the original notebook: running the same calculation through independent methods and checking they all agree is the cheapest way to catch an implementation bug before trusting a number.

The idiomatic way to do all of this in production is a Pipeline:

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import SGDRegressor
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(
    StandardScaler(),                                   # equivalent to my zscore_normalize_features
    SGDRegressor(max_iter=2000, tol=1e-6, eta0=0.1),
)
pipe.fit(X_train, y_train)

The Pipeline guarantees the StandardScaler only gets fit on training data in each validation fold, eliminating data leakage automatically. It's the correct way to do this in practice, instead of normalizing by hand like I did in this post to understand the mechanics underneath.

Why learn gradient descent, if LinearRegression solves it exactly? Because the normal equation only exists for linear models, and costs $O(n^3)$ to invert the matrix, infeasible with millions of features. Gradient descent is the algorithm that actually trains neural networks, logistic regression, SVMs, and pretty much everything else. Linear regression is just the playground where you can watch the mechanics work with an exact answer to check against.

Exercises

Try them before opening the answer.

Exercise 1: sensitivity to units

Convert size from sqft to square meters (1 sqft = 0.092903 m²) and run gradient descent on the raw data with $\alpha = 9\times10^{-7}$. What happens? Then normalize and run with $\alpha = 0.1$.

Exercise 2: min-max vs z-score

Normalize with min-max instead of z-score and run gradient descent with $\alpha = 0.1$, 1000 iterations. Compare the final cost and $\kappa$. Then insert an outlier (a 20000 sqft house) and repeat.

Exercise 3: implement a stopping criterion

My gradient_descent always runs the full num_iters iterations. Add early stopping when $\|\nabla J\|_\infty < \texttt{tol}$.

grad_norm = max(np.max(np.abs(dj_dw)), abs(dj_db))
if grad_norm < tol:
    print(f"Converged at iteration {i}: |grad|_inf = {grad_norm:.2e}")
    break

Exercise 4: predict the critical $\alpha$

Without running gradient descent, compute the critical $\alpha$ for min-max normalized data. Then confirm it empirically by running at $0.9\times$ and $1.1\times$ that value.

X_mm = minmax_scaling(X_train)
Ha = (np.column_stack([X_mm, np.ones(m)]).T @ np.column_stack([X_mm, np.ones(m)])) / m
a_crit = 2 / np.linalg.eigvalsh(Ha)[-1]

Exercise 5: a new feature

Add the feature size_per_bedroom = size / bedrooms, normalize, and train. Does the error improve?