Adaline: Train on the Line, Classify on the Sign

Lectures 2e and 2f, and the model's name changes to Adaline (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: what the model optimizes during training and what it computes at prediction time.

Pre-activation and post-activation: the distinction Adaline introduces

class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.linalg.pinv(X) @ y
    return self

  def predict(self, X):
    X = include_bias(X)
    return X @ self.w_

Notice: this is exactly last post's normal equation, without changing a line, except now y is -1 or +1 instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to -1 on one class's points and +1 on the other's, never applying sign().

Output: RMSE 0.4148, weights [-1.76, 1.01, 2.74].

RMSE makes sense here because predict returns a continuous number, not a class. But to actually classify, there's one last step missing:

class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.linalg.pinv(X) @ y
    return self

  def pre_activation(self, X):
    X = include_bias(X)
    return X @ self.w_

  def predict(self, X):
    return np.sign(self.pre_activation(X))

Now there are two methods: pre_activation (the continuous value, before any threshold) and predict (applies sign() on top). Aggarwal calls exactly these two things the pre-activation value and the post-activation value (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure "how far off" each prediction landed from the target) and only applies the activation (sign) when deciding the final class.

Output: accuracy 1.0, the same [-1.76, 1.01, 2.74] weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).

This is the delta rule, and I've already seen it before, under that exact name: "update the weight proportionally to the error times the input" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered two posts ago) is exactly this: the perceptron measures error after applying sign() (error in {-2,0,+2}), Adaline measures it before, on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit "unsure". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of "how wrong."

Interactive: nudging the pre-activation and watching RMSE (and accuracy) move

Instead of training automatically, drag the w0, w1, and bias sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).

Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between "finding some line that separates" (what the perceptron does) and "finding the line that separates with room to spare" (what you get by optimizing RMSE instead of just counting mistakes).

Lecture 2f: the same computation, just iterating (and the notebook calls it "SGD")

class Adaline(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.zeros(X.shape[1])
    for _ in range(self.max_iter):
      y_pred = X @ self.w_
      error = y - y_pred
      self.w_ += self.learning_rate * error @ X
    return self

One honest note about the notebook's name (aula02f adaline with SGD): the code shown here is batch gradient descent (the same X.T @ error as always), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking ("it's kind of like gradient descent, so I call it SGD"), but the technical difference is worth noting, since the names carry precise meaning.

Output: accuracy 1.0 on training, weights [-2.92, 3.32, 2.63]. Tested on 1000 new points: accuracy 0.953.

The "bad" dataset: what it actually proves

X_bad = np.concatenate((X,np.ones_like(X)))
y_bad = np.concatenate((y,np.ones_like(y)))
X_bad = np.concatenate((X_bad,np.ones_like(X)))
y_bad = np.concatenate((y_bad,np.ones_like(y)))
clf_bad = Adaline()
clf_bad.fit(X_bad, y_bad)

The professor concatenates the original dataset with two extra blocks of artificial points: everyone at (1,1), everyone labeled +1. This isn't noise, it's a deliberate bias, nudging training to "believe" the region near (1,1) is even more strongly class +1 than it really is.

Output: accuracy 0.967 on training (over the biased dataset), but only 0.811 on the same 1000 clean test points as before.

Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent on the same biased dataset, and that hypothesis didn't hold up:

Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about training data quality: biasing the training distribution (even without adding "noise" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. This echoes a lesson I already saw in the other playlist: training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.

Wrapping up

Practical application

I reproduced pseudo-inverse vs. batch gradient descent on Iris (setosa vs. versicolor), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train

On the first attempt, I used the notebook's original learning_rate=0.01 for batch gradient descent, and it diverged (weight became NaN) straight away on raw Iris data. No surprise at this point: it's the same scale lesson from the normal equation post, just rediscovered again, this time needing a much smaller rate (0.001) to not blow up.

With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, the same generous margin from Iris that already favored the perceptron with bias. The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of learning_rate), not the quality of the final solution.