Just a Little Extra: Loss Functions (MSE, MAE, Huber)

This post doesn't come from any specific course lab, it's a bonus we earned from hammering on the cost function so much in posts 2 and 3. Back there we squared the error without questioning the choice much, but "squaring it" is a choice, not the only option. Let's open that up.

Naming what you already built: MSE

The cost function you've been using since post 2 has an official name: MSE (Mean Squared Error).

$\text{MSE} = \frac{1}{m}\sum_{i=0}^{m-1}\left(f_{w,b}(x^{(i)}) - y^{(i)}\right)^2$

Notice it's basically the same formula from post 2, just without the extra "2" in the denominator (that 2 only existed to make gradient descent's derivative cleaner, remember?). The name changes, the substance doesn't.

The most direct alternative: MAE

What if, instead of squaring the error, we just took its absolute value? That's MAE (Mean Absolute Error):

$\text{MAE} = \frac{1}{m}\sum_{i=0}^{m-1}\left|f_{w,b}(x^{(i)}) - y^{(i)}\right|$

The difference looks small on paper, but it changes everything about the behavior. With MSE, missing by twice as much costs four times as much (remember post 2?). With MAE, missing by twice as much costs exactly twice as much, no more, no less. It's the difference between a judge who loses their mind when you miss badly and a judge who just counts points proportionally, no extra drama.

Both shapes side by side

No need to imagine the shape, here it is:

Notice: MSE is a parabola (grows faster and faster), MAE is a V (grows at a constant rate). Hold that image in your head, it explains everything that follows.

The best of both worlds: Huber

MSE is too sensitive to big errors. MAE is too harsh, even on small ones (look at the V: even tiny errors cost proportionally to their size, without the "discount" squaring gives near zero). The Huber loss tries to get the best of both: a smooth parabola right near zero, a straight line past a threshold $\delta$ (delta).

$L_\delta(e) = \begin{cases} \frac{1}{2}e^2 & \text{if } |e| \le \delta \\ \delta\left(|e| - \frac{1}{2}\delta\right) & \text{if } |e| > \delta \end{cases}$

Where $e$ is one example's error. Drag the delta slider above again and watch the orange curve: with a small delta, Huber turns nearly into MAE, and with a big delta, it turns nearly into MSE. Delta is literally the control for "past what error size do I stop giving a discount".

The impact of an outlier

This is where picking a loss function stops being theory and becomes a real decision. Take the 6-house dataset from post 2 and add one more house, at a fixed position. You control its price with the slider, dragging from a reasonable value to one way off the trend (an outlier), and watch three fitted lines live, one per loss function:

Drag it all the way to the max and watch: the blue line (MSE) runs after the red dot, twisting itself to try to "please" the outlier. The green line (MAE) barely budges. The orange one (Huber) sits in between. That's not a coincidence of this one example, it's the direct consequence of the two shapes you saw above: squaring punishes big errors without limit, absolute value doesn't.

Wrapping up

Three takeaways:

1. The loss function isn't a hidden technical detail, it's a design choice that changes the entire model's behavior, especially in the presence of real, messy data.
2. Squaring punishes big errors disproportionately, great when you trust your dataset, dangerous when you don't.
3. Huber exists exactly so you don't have to choose blind: delta is a dial that slides between the two extremes.

Practical application

Same real housing dataset from the previous posts (Housing Prices Regression, Kaggle). This time, a very realistic scenario: a typo. A 60 sqft house (tiny) got mistakenly logged as costing almost $2 million.

# 50 real houses + 1 typo: 60 sqft, $1.9 million
x_with_typo = [*x_sqft, 0.6]
y_with_typo = [*y_price, 1900]

ols   = fit_mse(x_with_typo, y_with_typo)                 # ordinary least squares fit
mae   = fit_irls(x_with_typo, y_with_typo, "mae")         # robust to the outlier
huber = fit_irls(x_with_typo, y_with_typo, "huber", delta=50)

new_house = 1.5   # 150 sqft, same scale as the previous posts

for name, (w, b) in [("MSE", ols), ("MAE", mae), ("Huber", huber)]:
    print(f"{name}: ${(w * new_house + b) * 1000:,.0f}")

Output: MSE: $608,900 / MAE: $570,700 / Huber: $569,800

Without the typo, all three give roughly the same guess for this house (around $567k to$573k). With the typo, MSE jumps to almost $609k, a jolt of over$35k just from one wrong row in the spreadsheet, while MAE and Huber barely move (under $3k of difference). Try it yourself, dragging the outlier:

Loading real data...

In a real database, with thousands of rows, typos happen. The choice of loss function decides whether one of those turns into your problem or not.