Multiclass: When a Weight Becomes a Matrix

Lecture 3c, and the professor swaps the 2-class dataset for a 4-class one. The first attempt is to honestly reuse everything that already exists without changing anything, just to show, in practice, exactly where it breaks.

The dataset: four blobs, one in each corner

def createMulticlassDataset(n=40):
  X, y = make_blobs(n_samples=n,
                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],
                    n_features=2,
                    cluster_std=0.05,
                    center_box=(0,1))
  return X, y

set(y_train)

Output: {0, 1, 2, 3}. Four classes, one integer each.

First attempt: reuse what already exists (and fail on purpose)

model = NeuralNetwork()  # the usual one, sign(X @ w_)
model.fit(X, y)
y_pred = model.predict(X)
print(f"Accuracy: {accuracy_score(y, y_pred)}")

Output: accuracy 0.25.

Worth understanding exactly why 0.25, it's not just some random number: with 4 well-balanced classes, always guessing the same thing gets you right on average 1 out of every 4 times, that is, 25%. The usual NeuralNetwork uses sign(X @ w_), which only returns -1 or +1, two possible values, never 0, 1, 2, or 3. Comparing that against a label that can be 0, 1, 2, or 3 is comparing things of different natures. The model isn't "almost getting it right", it literally can't express 3 of the 4 possible answers. The 0.25 accuracy is, in practice, the same level as a blind guess.

One-hot: every class becomes its own column

y_hot = np.zeros((y_train.shape[0], len(set(y_train))), dtype=int)
for i, label in enumerate(list(set(y_train))):
  idxs = np.where(y_train == label)[0]
  y_hot[idxs, i] = 1

Instead of a scalar label (0, 1, 2, or 3), every example becomes a row with a 1 in its class's column and 0 in the others. The professor confirms this matches exactly scikit-learn's LabelBinarizer:

from sklearn.preprocessing import LabelBinarizer
lb = LabelBinarizer()
y_hot = lb.fit_transform(y_train)

Same matrix, two implementations, the same kind of "matches the professional tool" check that already showed up in this playlist.

The fix: weight becomes a matrix, sign becomes argmax

class SGD(TrainingAlgorithm):
  def get_w(self, X, y):
    self.w_ = np.random.random(size=(X.shape[1], y.shape[1]))
    for _ in range(self.max_iter):
      y_pred = X @ self.w_
      self.w_ += self.learning_rate * self.cost_function.get_gradient(X, y, y_pred)
    return self.w_

class NeuralNetwork(BaseEstimator, ClassifierMixin):
  def predict(self, X):
    X = include_bias(X)
    logits = X @ self.w_
    idxs = np.argmax(logits, axis=1)
    return np.array([self.labels[idx] for idx in idxs])

The change that fixes everything: self.w_ stops being a vector ((features,)) and becomes a matrix ((features, classes)), one column of weights per class. X @ self.w_ now returns, for every point, 4 numbers (one "how confident" score per class), not just 1. And the final prediction swaps sign() for argmax: instead of asking "positive or negative?", it asks "which of the 4 columns had the highest value?". This is exactly the multiple-output layer architecture Aggarwal describes for categorical classification: one weight per class, and the final decision is whichever one "won". The only piece missing to turn it into his full version (with softmax, turning the 4 numbers into probabilities that sum to 1) is the normalization. Here the model just compares the raw numbers, without turning them into a probability, but the argmax winner doesn't change either way.

Output: accuracy 1.0.

Interactive: the four decision regions

I rebuilt the same dataset (make_blobs, same 4 centers) and trained the weight-matrix version. Every background color is the region where that class wins the argmax.

Notice the four boundaries meeting near the middle of the chart, splitting the plane into four wedges, one per class. Every blob lands cleanly inside the right color.

Wrapping up

Practical application

I tested the same idea (one-hot + weight matrix + argmax) on Wine (load_wine, 3 grape cultivars, 13 chemical variables), with one extra detail: the variables here live on quite different scales, the same problem already seen earlier in this playlist, so I normalized before training.

X_train_s = StandardScaler().fit_transform(X_train)

Even normalized, learning_rate=0.01 (the notebook's default) still diverged with 13 variables, so I had to drop to 0.001.

The naive version can't even reach all 3 possible class values (it can only ever predict -1 or +1, never class 2), so even that 0.403 number is misleading, it counts as a "hit" any case where the label happened to already be -1 or 1. The weight-matrix version gets nearly everything right, both train and test, confirming the same trick that worked on the synthetic 4-blob dataset generalizes to a real multiclass classification problem.