Swapping the Cost Function Like Changing Clothes

Lectures 3a and 3b. So far I've seen 4 different algorithms (perceptron, vectorized perceptron, Adaline via pseudo-inverse, Adaline via gradient) as somewhat separate things. This lecture shows they're actually the same recipe, just swapping one ingredient: the cost function.

First, the training algorithm becomes pluggable

class TrainingAlgorithm(ABC):
  @abstractmethod
  def get_w(self, X, y):
    pass

class PseudoInverse(TrainingAlgorithm):
  def get_w(self, X, y):
    return np.linalg.pinv(X) @ y

class NeuralNetwork(BaseEstimator, ClassifierMixin):
  def __init__(self, training_algorithm=PseudoInverse()):
    self.training_algorithm = training_algorithm

  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = self.training_algorithm.get_w(X, y)
    return self

  def predict(self, X):
    X = include_bias(X)
    return np.sign(X @ self.w_)

This is the Strategy design pattern: NeuralNetwork no longer knows how the weights get computed, just that there's a training_algorithm object with a get_w method. Swapping PseudoInverse() for SGD() in the constructor swaps out the entire training algorithm, without touching NeuralNetwork.

Output (PseudoInverse): accuracy 0.95, weights [-1.905, 2.656, 1.049]. Output (SGD): accuracy 0.95, essentially identical weights.

Confirms again, now with more cleanly organized code, what I already saw in the last two posts: pseudo-inverse and gradient descent solve the exact same problem.

Now the cost function becomes pluggable too

class CostFunction(ABC):
  @abstractstaticmethod
  def get_cost(y, y_pred):
    pass
  @abstractstaticmethod
  def get_gradient(X, y, y_pred):
    pass

class WidrowHoff(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.mean((y-y_pred)**2)
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y-y_pred)

SGD now also takes a cost_function, and uses self.cost_function.get_gradient(...) instead of computing the gradient by hand. WidrowHoff is exactly the delta rule from the last post: continuous error (y - y_pred, no sign()) times the input. Swapping the cost function here means swapping what "error" means, without touching the training loop.

Every cost function recovers a different algorithm

class SmoothedSurrogate(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - np.sign(y_pred))

Notice the np.sign(y_pred) inside the gradient: this goes back to measuring error after the threshold, exactly like Rosenblatt's perceptron. The name SmoothedSurrogate matches what Aggarwal calls the perceptron criterion: $L = \max(0, -y \cdot \hat{y})$, zero once the point is on the right side, growing linearly when it's wrong. No coincidence the accuracy hits 1.0: this cost function, plugged into this generic framework, is the original perceptron again, just expressed in the language of "cost function" instead of "update rule".

class LogLikehood(CostFunction):
  @staticmethod
  def get_cost(y, y_pred):
    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - expit(y_pred))

Output: accuracy 0.65, weights [-60.78, 27.44, -24.79]. Much worse than anything I've seen so far, and the weights got huge.

Two things wrong here, worth separating. First, a detail that doesn't affect the outcome: this class's get_cost uses the hinge loss formula (max(0, 1 - y·ŷ)), not an actual log-likelihood formula. That doesn't break anything in practice because get_gradient is the only thing SGD calls, get_cost never gets used during training, it's leftover residue from copying and pasting from another cell.

The second problem is real, and explains the bad accuracy: expit (the sigmoid function) only returns values between 0 and 1, but the label y is -1 or +1. For class y=-1, the error y - expit(y_pred) can never get close to zero, because expit never goes negative: even with an infinitely confident correctly-classified prediction, -1 - expit(y_pred) stays pinned near -1, never 0. I checked this by hand: expit(-1000) = 0.0, so -1 - expit(-1000) = -1.0 exactly, not 0. The gradient for half the points never vanishes, so training never settles, and the weights keep growing trying to compensate for an error that's structurally impossible to zero out.

class LogLikehood(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    return X.T @ (y - tanh(y_pred))

model = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))

Output: accuracy 1.0, weights [-9.84, 12.40, 8.37] (with max_iter=10000, ten times more iterations).

Swapping expit for tanh (same class redefined, Python just lets that run live in a notebook session), the problem disappears. Makes sense: tanh ranges from -1 to +1, exactly the labels' range. Checked it by hand again: tanh(-1000) = -1.0, so -1 - tanh(-1000) = 0.0, the error genuinely reaches zero this time. Aggarwal states this relationship directly in chapter 1: $\tanh(v) = 2 \cdot \text{sigmoid}(2v) - 1$, and it's exactly that range shift, from $(0,1)$ to $(-1,1)$, that fixes the mismatch with ±1 labels.

class HingeLoss(CostFunction):
  @staticmethod
  def get_gradient(X, y, y_pred):
    marginal_errors = (y * y_pred) < 1
    marginal_ys = np.copy(y)
    marginal_ys[~marginal_errors] = 0
    return X.T @ marginal_ys

Output: accuracy 1.0, weights [-8.70, 11.59, 7.71].

Notice marginal_ys[~marginal_errors] = 0: points that are already well classified, with room to spare (y · ŷ ≥ 1), get zeroed out and contribute nothing to the gradient. Only points inside the margin (or misclassified) participate in the update. That's literally SVM's central idea: only the points near the boundary (the "support vectors") matter for deciding where it sits. The rest of the dataset gets ignored once it's already well separated.

All four curves, side by side

The professor has a reference image saved in the notebook comparing the four penalty curves as a function of the "margin" ($y \cdot \hat{y}$: positive and large means a confident correct call, negative means a mistake). I recreated the same idea here, interactively:

Hover over any point on the x-axis and compare all four. Notice the shapes: Widrow-Hoff is a parabola, it keeps penalizing even a point that's already correctly classified with room to spare (margin > 1), because it doesn't know "correct is correct", it only knows how to measure distance to a continuous target. Perceptron and Hinge are the only two that fully zero out once a point is well classified (perceptron zeros as soon as it crosses 0, hinge requires crossing 1, with room to spare). Logistic never truly zeroes out, it only approaches zero, which is the price it pays for returning a smooth probability instead of a binary decision.

Wrapping up

Practical application

I ran all four cost functions (Widrow-Hoff, perceptron criterion, hinge, and the log-likelihood version with tanh) on Iris (setosa vs. versicolor), plus the expit version on purpose, to confirm the sigmoid problem isn't exclusive to the notebook's toy synthetic dataset.

Four out of five hit 100% (Iris has a generous enough margin for any of them to find a perfect boundary), and sigmoid falls behind again, this time on real data, not just the notebook's synthetic dataset. Confirms it wasn't a one-run coincidence: the range mismatch between expit and ±1 labels genuinely hurts convergence, on every dataset I tried it on.