Optional Lab: Gradient Descent

Quick recap of the last two posts: you built the model $f_{w,b}(x) = wx + b$, then built a way to measure how wrong it is, $J(w,b)$. But to find the best $(w,b)$ you were still doing the most primitive thing possible: dragging a slider and watching the number drop. That works for 2 points. For a real dataset, with thousands of examples and dozens of parameters, it's impossible.

This post closes the loop with the algorithm that does that search on its own: gradient descent.

$w = w - \alpha \frac{\partial J(w,b)}{\partial w} \qquad b = b - \alpha \frac{\partial J(w,b)}{\partial b}$

The idea in one sentence

Remember the soup bowl from the last post? Gradient descent is literally that: you start at some point on the bowl and take steps downhill, always in the direction that descends fastest, until you land near the bottom.

The "feeling which direction descends fastest" part is the derivative's job, that $\frac{\partial J}{\partial w}$ in the formula. It's the cost surface's slope at that specific point. And $\alpha$ (alpha) is how big a step you take each time.

Why subtract the derivative

The derivative points toward where the cost increases. Since you want a smaller cost, you walk in the opposite direction. Hence the minus sign in the formula.

Notice the last row: the algorithm doesn't need an "if you've reached the minimum, stop" check. It simply stops moving on its own, because the derivative hits zero. And since the derivative shrinks as you get closer to the bottom, the steps also get smaller on their own, even with a fixed $\alpha$. That's a free property, not something you code separately.

The two partial derivatives

For single-variable linear regression, the math works out to these two formulas (I didn't need to memorize the derivation, just get the pattern):

$\frac{\partial J(w,b)}{\partial w} = \frac{1}{m} \sum_{i=0}^{m-1} \left(f_{w,b}(x^{(i)}) - y^{(i)}\right) x^{(i)}$

$\frac{\partial J(w,b)}{\partial b} = \frac{1}{m} \sum_{i=0}^{m-1} \left(f_{w,b}(x^{(i)}) - y^{(i)}\right)$

Notice the two are nearly twins: the one for $w$ has an $x^{(i)}$ multiplying the error, the one for $b$ doesn't. That makes sense geometrically: changing $w$ affects examples with large $x$ more strongly (the slope carries more weight farther from the origin), while $b$ shifts the entire line the same amount for everyone.

And remember that "2" we put in $2m$ in the last post, where I said it was just there to simplify the math later? Here's where it pays off: differentiating the squared term leaves behind a factor of 2 that cancels exactly with the 2 in the denominator. Without that 2 back there, these formulas here would carry a leftover 2.

Simultaneous updates matter. You compute both derivatives first, using the current values of $w$ and $b$, and only then swap both parameters at once. Using the new $w$ to compute $b$'s derivative is a classic mistake that changes the algorithm's behavior.

Putting this into code

def compute_gradient(x, y, w, b):
    """
    Computes the gradient of the cost function for linear regression.

    Args:
      x (ndarray (m,)) : input data, m examples
      y (ndarray (m,)) : target values
      w, b (scalar)    : model parameters

    Returns:
      dj_dw (scalar): partial derivative of the cost with respect to w
      dj_db (scalar): partial derivative of the cost with respect to b
    """
    m = x.shape[0]

    dj_dw = 0
    dj_db = 0

    for i in range(m):
        f_wb = w * x[i] + b
        dj_dw_i = (f_wb - y[i]) * x[i]   # example i's contribution to dj_dw
        dj_db_i = f_wb - y[i]            # example i's contribution to dj_db
        dj_db += dj_db_i
        dj_dw += dj_dw_i

    dj_dw = dj_dw / m
    dj_db = dj_db / m

    return dj_dw, dj_db

And the main loop, which repeats the update until the iterations run out:

def gradient_descent(x, y, w_in, b_in, alpha, num_iters, cost_function, gradient_function):
    """
    Runs gradient descent to fit w and b.

    Args:
      x, y                : training data
      w_in, b_in (scalar) : INITIAL parameter values
      alpha (float)       : learning rate
      num_iters (int)     : how many iterations to run
      cost_function       : function to compute the cost
      gradient_function   : function to compute the gradient

    Returns:
      w, b (scalar)    : parameters after training
      J_history (list) : cost at every iteration
    """
    J_history = []
    w = w_in
    b = b_in

    for i in range(num_iters):
        dj_dw, dj_db = gradient_function(x, y, w, b)

        b = b - alpha * dj_db   # simultaneous update: both derivatives were
        w = w - alpha * dj_dw   # already computed from the old values

        J_history.append(cost_function(x, y, w, b))

    return w, b, J_history

Now it's your turn, but with the algorithm doing the work

No more dragging a slider until you land on the right value by hand. Below is the real algorithm running, with full control: pick a learning rate, take one step at a time or run a batch at once, and watch the little red dot walk downhill on its own, on the heatmap, on the 3D surface, or on the parabola.

Start at $w = 0$, $b = 0$ (far from the answer) and click "Rodar 2000" a few times with the default alpha of 0.01. Notice how the cost drops fast at first and then slows down on its own, with no input from you.

When the learning rate is too big

Now click the alpha 0.8 preset (way bigger than the 0.01 that worked) and run a few steps. You'll watch $w$ and $b$ become increasingly absurd, and instead of dropping, the cost climbs.

That's divergence, and the reason is easy to picture: the step size is proportional to the derivative. If $\alpha$ is large, the step overshoots the bottom of the bowl and lands on the other side, higher up than it started. From that new spot, the derivative is even bigger (in magnitude) and flipped in sign, so the next step is even bigger in the opposite direction. It turns into a self-feeding loop that explodes, like pushing a swing harder and harder until it flips upside down.

In practice, if you're training a real model and see the cost climbing or bouncing back and forth without settling, the first thing I try is lowering $\alpha$ (divide by 3 or by 10, for instance). The course suggests trying a sequence like 0.001, 0.003, 0.01, 0.03, 0.1 and comparing the cost curves until you find one that drops smoothly.

Try these values yourself in the simulator above and compare against the table.

Bonus: not every bowl is this well-behaved

Every cost surface we've drawn so far looks like the same soup bowl, because it comes from squared error, and that guarantees convexity (last post). But gradient descent doesn't only live on nicely-behaved bowls. Here are two classics that anyone studying optimization runs into sooner or later, just so you see that the trouble we hit with a big alpha is only the tip of the iceberg.

Rosenbrock's banana valley

This one's practically the standard stress test of the field, it even has its own name: the Rosenbrock function.

$f(w,b) = (1-w)^2 + 100(b - w^2)^2$

The global minimum sits at $(1,1)$, cost zero, but look at the shape:

It's not a round bowl, it's a curved valley, shaped like a banana. That's a real problem for gradient descent: the direction that descends fastest almost never points toward the bottom of the valley, it points toward the nearest wall. The algorithm ends up bouncing from one side of the banana to the other, barely making progress with each zig-zag, even near the bottom.

Try alpha 0.01 here (the same value that worked smoothly in our housing example) and watch what happens:

Alpha 0.01 diverges almost instantly here, the same value that was the "good balance" above. There's no universal alpha, it depends entirely on the shape of the surface you're descending. Drop it to 0.002 and click "Rodar 2000" a few times: now the red dot snakes slowly through the valley until it lands near $(1,1)$.

The saddle point

$f(w,b) = w^2 - b^2$

Rotate this one slowly. It's a minimum if you only look along the $w$ axis, and a maximum if you only look along the $b$ axis, at the same time, at the same point. That's called a saddle point, the red dot in the center marks exactly that.

Remember the rule "when the derivative hits zero, the algorithm stops on its own"? Well, right at this point it hits zero just the same. If gradient descent landed exactly there, it would think it was done, without being done at all, just balanced on top of a mountain pass. Any tiny nudge away from that exact center, and it slides downhill in whichever direction actually descends (the $b$ axis), away from any real minimum.

This never happens on our own regression cost (it's always convex, no hidden saddle anywhere), but this exact kind of surface shows up constantly in more complex models, like neural networks. Keep that name in your back pocket, it comes back.

Wrapping up the trilogy

Three takeaways:

1. The gradient points toward where the cost increases, so the algorithm always walks the opposite way.
2. The update is simultaneous, compute both derivatives first, swap the parameters after.
3. The learning rate is the single most sensitive knob you'll touch: too small is slow, too big diverges.

Coming up next: everything you've seen so far used one feature (the house's size). In Week 2 of the course, the model gains several features at once, and computing with a for loop stops cutting it. Before touching a model with multiple features, the next post covers the tool that makes it viable: NumPy and vectorization.

Practical application

Same real housing dataset from the last two posts (Housing Prices Regression, Kaggle), same scale (size in "hundreds of sqft", price in "thousands of dollars"). Let's watch the algorithm find, on its own, in real data, the fit you went hunting for by hand in posts 1 and 2.

w, b, J_hist = gradient_descent(
    x_sqft, y_price,          # the 50 real houses
    w_in=0, b_in=0,           # starting from zero, same as the toy example
    alpha=0.01, num_iters=4000,
    cost_function=compute_cost, gradient_function=compute_gradient)

print(f"(w, b) found: ({w:.1f}, {b:.1f})")

Output: (w, b) found: (116.5, 398.3)

Matches almost exactly the fit cited in the last two posts. Click "Rodar 2000" twice on the simulator below and watch it live:

Loading real data...

Notice convergence here is quite a bit slower than in the 2-point toy example, even already using the same small scale as before. That happens because $w$ and $b$ still live on pretty different ranges from each other (one goes up to 300, the other up to 600), which stretches the cost valley. This is exactly the kind of situation feature scaling, later in the course, fixes for good.