Optional Lab: the Cost Function

In the last post you found $w = 200$ and $b = 100$ by dragging a slider until the error hit zero. Cool, it worked. But wait. How did you know you'd found the "best" fit? You watched the total error number drop to zero and trusted it.

This post is about giving that number a name, a last name, and a formula. It's called the cost function, written $J(w,b)$.

$J(w,b) = \frac{1}{2m}\sum_{i=0}^{m-1}\left(f_{w,b}(x^{(i)}) - y^{(i)}\right)^2$

I know, it looks scary. Let's take that formula apart piece by piece, like disassembling a Lego set.

Watch out for the word "cost"

Before anything else: here cost has nothing to do with a house's price. It's the measure of how wrong the model is. For the house's value we keep using the word price. This vocabulary mix-up trips a lot of people up early on, because "cost" in everyday speech always means money.

Taking the formula apart

Think of a target-shooting contest. Each house in your training set is an attempt to hit the bullseye (the real price, $y^{(i)}$). $f_{w,b}(x^{(i)})$ is where your arrow actually landed (the model's prediction). The distance between where it landed and the bullseye is that shot's error.

Two things fall right out of this formula, and I didn't have to memorize any of it:

1. $J(w,b)$ is never negative. It's a sum of squared terms, so the smallest possible value is zero.
2. $J(w,b) = 0$ means a perfect fit. The line passes exactly through every point.

And why square the error instead of, say, taking its absolute value? Because squaring disproportionately punishes big misses. Missing by twice as much costs four times as much in $J$, not twice as much. That makes the model hate really bad predictions in a way absolute value wouldn't, and it's exactly this behavior that gives the cost its U shape, which we'll see in a moment. Back to the target-shooting contest: it's as if the judge scored exponentially worse the farther the arrow lands from the bullseye. Missing by a little barely stings, missing badly stings way more than twice as much.

Putting this into code

def compute_cost(x, y, w, b):
    """
    Computes the cost function for linear regression.

    Args:
      x (ndarray (m,)) : input data, m examples
      y (ndarray (m,)) : target values
      w, b (scalar)    : model parameters

    Returns:
      total_cost (float): the cost of using w and b as parameters to fit
                          the points (x, y)
    """
    m = x.shape[0]

    cost_sum = 0

    for i in range(m):
        f_wb = w * x[i] + b           # model's prediction for example i
        cost = (f_wb - y[i]) ** 2     # example i's error, squared
        cost_sum = cost_sum + cost    # add it to the total

    total_cost = (1 / (2 * m)) * cost_sum

    return total_cost

Notice it's the same for loop from the last post's compute_model_output, except instead of storing the predictions, we accumulate each one's squared error. Same structure, different purpose.

Freezing one parameter to see the other

With two parameters to move ($w$ and $b$), I had a hard time visualizing the cost directly. Classic trick: freeze one and look only at the other. Fix $b = 100$ and move only $w$.

Drag the slider below and watch two things at once: how the line (left panel) gets closer to or farther from the points, and where the red dot sits on the $J(w)$ curve (right panel). As the fit improves, the dot moves down the curve.

Notice three things:

• The cost is at its lowest exactly at $w = 200$, the same value you found by eye in the last post. With $b = 100$, the cost there hits zero, because the line passes through both points.
• The cost shoots up fast when $w$ gets too big or too small. That's the square in the formula doing its job.
• That's when it clicked: minimizing the cost is the same thing as finding the best fit. It's not a coincidence, it's the definition. Choosing $w$ and $b$ that minimize $J$ is literally what we mean by training a model.

Try dragging the slider to $w = 0$: the line goes flat (predicts the same price for any house size) and the cost spikes.

The real world doesn't hand you 2 perfect points

So far, with 2 points and 2 parameters, I could get the cost to hit zero. That's rare. Let's swap the training set for a more realistic one, 6 houses, where the prices don't fall exactly on any single line (real noise, like similar houses selling for slightly different prices):

The open question: is there still some $(w, b)$ that zeroes out the cost here?

Move both sliders below (now $w$ and $b$ at once) and try to reach the center of the heatmap, where the cost is lowest. Switch to the 3D view and rotate the surface with your finger or mouse to feel the shape of the "valley".

You probably noticed: this time the cost doesn't hit zero. The best you can get lands around $w \approx 209$ and $b \approx 2.4$, with a cost around 1736. Why? Because these 6 houses aren't aligned. No single line passes exactly through all of them. The best fit is the one that leaves the smallest possible total squared error, and that's exactly what the cost function defines as "the best".

Hold on to this: a nonzero minimum cost is the normal case, not a bug. Zero cost tends to happen when you have too little data (like our 2-point case) or when the model memorized the data instead of learning its pattern (that has a name, overfitting, but that's a topic for further down the course).

Why the surface is always a bowl

Rotating the 3D surface above, you probably noticed it always looks like a soup bowl, a single valley, no fake peaks along the way. That's not a coincidence of our specific example, it's a direct consequence of squaring the error in the formula. Any time you square something, the result is a convex surface.

To see this more cleanly, without the scale distortion the real data brings (notice $w$ ranges from 0 to 400 and $b$ from -200 to 200 in the chart above, which stretches the valley), here's the idealized version of the same shape, just $w^2 + b^2$, with both axes on the same scale:

Why does this shape matter so much? Because a convex surface guarantees there's only one minimum, the global minimum. There's no hidden valley elsewhere for a search algorithm to fall into and get stuck, mistakenly thinking it already reached the bottom when it hasn't. That guarantee is what makes the next step of the course reliable: an automatic way to walk down to the bottom of that bowl, without you having to drag a slider for the rest of your life.

Wrapping up

Coming up next: you've now felt firsthand what it's like to hunt for the bottom of the bowl by dragging sliders. It doesn't scale. The next post introduces gradient descent, an algorithm that uses the slope of the cost surface to walk down to the bottom on its own, with no guessing required from you.

Practical application

Same real housing dataset from the last post (500 houses, Housing Prices Regression, Kaggle). Now with a real compute_cost.

def compute_cost(x, y, w, b):
    m = x.shape[0]
    cost_sum = 0
    for i in range(m):
        f_wb = w * x[i] + b
        cost_sum += (f_wb - y[i]) ** 2
    return (1 / (2 * m)) * cost_sum

# x_sqft already in "hundreds of sqft", y_price already in "thousands of dollars"
# (same 50-house sample from the last post)

print("cost with a bad guess, w=50, b=100:", compute_cost(x_sqft, y_price, 50, 100))
print("cost near the optimal fit, w=116, b=399:", compute_cost(x_sqft, y_price, 116, 399))

Output: cost with a bad guess, w=50, b=100: 90317.6 / cost near the optimal fit, w=116, b=399: 5189.6

Almost 20 times less cost just from picking better parameters. Move the sliders yourself and try to get close to that ~5189 (notice: you can't hit zero, that really is the real minimum, real data has noise):

Loading real data...