Feature Scaling and Learning Rate

I picked up the optional lab from Course 1, Week 2 (C1_W2_Lab03) expecting just another follow-up post, and ended up finding the topic that annoyed me the most in the whole specialization so far: the exact gradient descent I built in the last post, same algorithm, zero code changes, simply stops working depending on the unit I use to measure a feature. I swapped meters for centimeters and the algorithm diverged. That's not a bug, it's math, and after this post you'll never pick an $\alpha$ in the dark again without understanding why.

The problem: 4 features, wildly different scales

The original notebook uses a 100-house dataset (derived from the Ames Housing dataset, the same one used in the course) with 4 features: size in sqft, number of bedrooms, number of floors, and age. The goal is to predict the price of a 1200 sqft, 3-bedroom, 1-floor, 40-year-old house.

I noticed this number the moment I ran the cell: the feature with the biggest range (size) is 2406 times bigger than the one with the smallest range (floors). I kept that number in mind, because it's the root of everything that follows.

Before jumping into modeling, I looked at each feature against price, one at a time:

Size carries real signal (the cloud of points clearly climbs from left to right). Bedrooms and floors are discrete and noisy, you can see 2-bedroom houses costing more than 3-bedroom ones. Age pulls price down, but with a lot of scatter. That doesn't mean bedrooms and floors are useless, just that alone they explain little, their value shows up once they join the model together with the others (the negative bedrooms coefficient I compute further down is exactly that kind of hidden effect).

I also computed the correlation between every pair of features (and between each feature and price):

Size is by far the feature most correlated with price (0.86), which matches the chart above. And notice size and bedrooms aren't independent (0.56), bigger houses tend to have more bedrooms, which is kind of obvious once you stop to think about it, but it's always good to confirm with a number instead of a guess.

Quick recap, with a single feature

To keep the foundation I already built, I'll simplify to 1 feature (size) for this first part, exactly like the previous posts: $f_{w,b}(x) = wx + b$, cost $J(w,b) = \frac{1}{2m}\sum(f_{w,b}(x^{(i)}) - y^{(i)})^2$. The gradient descent I built two posts ago doesn't change a single line:

$w = w - \alpha \frac{\partial J}{\partial w} \qquad b = b - \alpha \frac{\partial J}{\partial b}$

The real notebook runs this with all 4 features at once (one $w$ per feature), but the core lesson shows up completely already with a single feature, and it's much easier to visualize on a 2D chart. Keep this simplification in mind, it comes back later when I show the result with the real 4 features.

Gradient descent diverges on its own

I picked 8 real houses from the notebook's dataset (size in sqft, price in thousands of dollars) to test this by hand:

x_train = [952, 1244, 1947, 1725, 1959, 1314, 864, 1836]
y_train = [271.5, 300, 509.8, 394, 540, 415, 230, 560]

I ran the same gradient_descent from the previous post with three different $\alpha$ values, just 10 iterations, starting at $w=0, b=0$:

Output with $\alpha = 9\times10^{-7}$: cost climbs every iteration, from $8.8\times10^4$ to $9.4\times10^5$. w flips sign constantly.

Output with $\alpha = 8\times10^{-7}$: cost drops from $8.8\times10^4$ to $9.4\times10^3$, but w still oscillates (0 → 0.51 → 0.06 → 0.46 → 0.10...).

Output with $\alpha = 1\times10^{-7}$: cost drops from $8.8\times10^4$ to $1.3\times10^3$, w climbs straight up without oscillating (0 → 0.06 → 0.11 → 0.15...).

Try it yourself in the simulator below. I started with a fairly small alpha on purpose, drag the slider up slowly and notice exactly where the cost flips from "dropping" to "climbing":

The pattern I saw matches exactly what the notebook describes for the full dataset (with all 100 houses and all 4 features, the threshold sits between $9\times10^{-7}$ and $9.9\times10^{-7}$, very close to what I found here with just size and a smaller subset). That's not a coincidence, and the next section explains exactly why.

Why exactly there: the $2/L$ bound and the condition number

This isn't trial and error, it can be calculated. For a quadratic cost function like ours, gradient descent converges if and only if

$0 < \alpha < \frac{2}{L}, \qquad L = \lambda_{\max}(\mathbf{H})$

where $\mathbf{H}$ is the Hessian of the cost and $\lambda_{\max}$ is its largest eigenvalue. $L$ is the bowl's maximum curvature: the sharper the steepest direction, the smaller the step allowed before it overshoots to the other side of the valley.

I computed the Hessian for our 8-house example (a $2\times2$ calculation, since it's just $w$ and $b$) and got $L \approx 2.36\times10^6$, giving a critical $\alpha \approx 8.46\times10^{-7}$. That matches what I saw in the simulator: $9\times10^{-7}$ crosses that threshold and diverges, $8\times10^{-7}$ stays under it and converges.

The original notebook runs this same calculation with the real 4 features (the Hessian becomes $5\times5$, counting $b$) and lands on a critical $\alpha \approx 9.22\times10^{-7}$, practically the same number I found by simplifying to 1 feature. That makes sense: size is the feature with the most absurd scale, so it alone already dominates most of the bowl's curvature.

There's a second number that comes out of this same calculation, the condition number $\kappa = L/\mu$ (where $\mu$ is the smallest curvature). It measures how "stretched" the bowl is. I found $\kappa \approx 5.8\times10^7$ for the full 100-house dataset with all 4 features. I also computed how many iterations would be needed, in the best case, just to reduce the error by 10x: around 67 million. That's when I fully understood why "lower alpha a bit and wait longer" isn't a real solution here.

A practical recipe for picking $\alpha$

I won't always want to compute an eigenvalue. The recipe Andrew Ng teaches, and that I've already been using since the gradient descent post:

1. Start small, like $\alpha = 0.001$.
2. Multiply by roughly 3 each attempt: 0.001, 0.003, 0.01, 0.03, 0.1...
3. Run a few iterations and look at the cost-vs-iteration chart.
4. Pick the largest $\alpha$ that still gives a smooth, decreasing curve.

I ran this sweep on the full dataset and found the usable range of $\alpha$ (with no scaling at all) sits between $10^{-8}$ and $10^{-6}$. An extremely narrow range, and one that depends entirely on the unit I used to measure size. If I'd measured in square meters instead of sqft, every one of these numbers would shift. That's fragile, and it's not how I want to train any model.

The scaling techniques

The fix isn't hunting for a better $\alpha$, it's fixing the scale of the features before running anything. There are a few ways to do that:

I used z-score from here on, it's the most robust option (it doesn't depend on just two extreme points in the dataset, unlike min-max does). $\mu_j$ and $\sigma_j$ are the mean and standard deviation of feature $j$, computed only from the training data. That matters: I store those two numbers and reuse them for any new data that shows up later, I never recompute them.

def zscore_normalize_features(X):
    mu = X.mean(axis=0)      # mean of each column
    sigma = X.std(axis=0)    # std of each column
    X_norm = (X - mu) / sigma
    return X_norm, mu, sigma

For our 8-house example, size's mean is 1480.1 sqft and its std is 414.7. I normalized it and the range, which used to span hundreds to thousands, turned into something between roughly -1 and 2.

One detail I almost slid past: z-score does not make the data normally distributed, it only rescales. The shape of the distribution stays exactly the same, only the center and scale change.

Gradient descent with normalized features

I ran the same algorithm, on the same 8-house dataset, but with size now z-score normalized, using $\alpha = 0.1$, six orders of magnitude bigger than the $10^{-7}$ values from before:

After about 50 iterations the cost already flattened near the minimum (I got $J \approx 919$, almost identical to the real minimum of $919.37$ I computed directly through a closed-form solution). In the raw example, with $\alpha = 10^{-7}$, cost had barely moved off $1.3\times10^3$ after 10 whole iterations. Normalizing didn't change where the minimum is, it completely changed how fast I get there.

I also noticed something small but satisfying to understand: when $x$ is centered at zero (mean zero, thanks to z-score), the optimal $b$ turns out to be exactly the mean of the training prices. That makes sense geometrically: if every x is zero on average, the best line passes through the average y at that point.

Predicting a new house and denormalizing

I fit the model with all 4 real features (not just size) on the full 100-house dataset, using the normal equation (the exact closed-form solution, no gradient descent):

w = [0.268, -32.90, -67.29, -1.465]   # size, bedrooms, floors, age
b = 221.50

I predicted the target house's price (1200 sqft, 3 bedrooms, 1 floor, 40 years old) and got $318.9 thousand.

What caught my attention most was the bedrooms coefficient: negative. On its own, more bedrooms sounds like a purely good thing, but controlling for the house's size (which is already in the model), more bedrooms in a same-sized house usually means smaller bedrooms, which pulls the price down a bit. That's exactly the kind of thing that only shows up once you look at the coefficients of a model with several features at the same time, you can't see it in a bedrooms-vs-price chart alone.

If I'd fit with normalized features, the weights would come out on a different scale (each $w_j$ would represent "how much the price changes per 1 standard deviation of that feature", not per 1 real unit). To get back to real-unit coefficients, the denormalization formula is:

$w_j^{original} = \frac{w_j^{normalized}}{\sigma_j} \qquad b^{original} = b^{normalized} - \sum_j \frac{w_j^{normalized} \cdot \mu_j}{\sigma_j}$

I checked this on our 1-feature example: denormalizing the $w$ and $b$ that gradient descent found in normalized space, I got $w = 0.267$ and $b = 7.16$, the exact same numbers the closed-form solution gives directly in raw space. Both paths land in the same place, one of them just converged in 50 iterations and the other would need millions.

The contours: the geometric picture

Everything I've described in numbers, I can see at once in a single image. From here on I switched datasets: I'm using the 50-house one that already showed up in the previous posts in this playlist (square_feet, num_bedrooms, location score, distance to center), not the 100-house one from the notebook I used earlier. It's a different set of houses, so the correlation numbers here differ from the ones I computed in "The problem" section.

Before going abstract (the shape of the cost), it's worth seeing the far more direct, concrete effect first: the shape of the data points themselves. I took size and number of bedrooms from these 50 houses and plotted one against the other, raw and then normalized:

Points, without normalizing:

Points, after normalizing (z-score):

Notice the cloud of points doesn't change shape, it just shifts and rescales so both axes end up in the same unit (standard deviations instead of sqft/bedrooms). And it's exactly that change in relative scale between the two axes that makes all the difference further down.

Now the cost map. I fit a model with all 4 features (size, bedrooms, location score, distance to center), fixed two of them at their fitted values, and drew a heatmap varying just size's weight and bedrooms' weight, the same trick the original notebook uses. The red dot marks exactly where my fit landed, and the line right below the chart shows the exact values.

Without normalizing:

After normalizing (z-score):

Notice the difference in shape. In the raw contour, the size axis is far more sensitive than the bedrooms axis (a tiny 0.5 change in size's weight already sends the cost skyrocketing, while bedrooms' weight needs to move dozens of units to do the same damage). It's a heavily elongated valley, almost a corridor, and the red dot (the real fit) sits squeezed at the bottom of it. In the normalized contour, both directions end up with similar sensitivity, the shape turns much rounder, and the dot lands near the center of a circle.

It's not a perfect circle (size and number of bedrooms, in this 50-house dataset, still have a small residual correlation, I calculated it and it came out around 0.02, very close to zero but not exactly zero), but the difference in shape is dramatic. And that's not a coincidence or a visual approximation: for a single z-score normalized feature, it's provable that the Hessian becomes exactly the identity matrix, $\kappa = 1$, a perfect circle, an exact result and not an approximation. With two real features, the result comes out nearly circular, and the "nearly" is literally the size of the correlation between them.

Wrapping up

Three takeaways:

1. The relative scale of the features decides the shape of the cost bowl, not the optimization algorithm itself.
2. The condition number $\kappa$ is the metric that quantifies this: close to 1 is a round, fast-to-descend bowl, much bigger than 1 is a canyon that any fixed $\alpha$ takes forever to cross.
3. Normalizing doesn't change the answer, it changes the path to it: same minimum, orders of magnitude fewer iterations to get near it.

Practical application

Same real housing dataset from the previous posts (Housing Prices Regression, Kaggle). We already saw in the gradient descent post that the raw algorithm, with square_feet/100 and price/1000, converges slowly, needing $\alpha=0.01$ and 4000 iterations to reach $(w,b) = (116.5, 398.3)$. Now I z-score normalized size before running it:

size_norm, mu, sigma = zscore_normalize_features(square_feet)

w, b, J_hist = gradient_descent(
    size_norm, price,
    w_in=0, b_in=0,
    alpha=0.3, num_iters=200,
    cost_function=compute_cost, gradient_function=compute_gradient)

print(f"(w, b) found: ({w:.1f}, {b:.1f})")

Output: (w, b) found: (88.1, 593.0)

The final cost matches the raw example almost exactly ($J \approx 5189.6$ in both cases, it's the same minimum), except here I got there in 200 iterations at $\alpha = 0.3$, versus 4000 iterations at $\alpha = 0.01$ before. Compare both live:

Loading real data...

Loading real data...

Click "Rodar 100" a few times on both and time it in your head: one of them already flattened out, the other is still climbing the slope. Same house, same final price, and the only difference between the two simulators is one extra line of code at the very start.