[{"data":1,"prerenderedAt":607},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse":3,"post-en-neural-networks-multiclasse":4},"\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse",{"id":5,"title":6,"body":7,"cover":593,"date":594,"description":595,"extension":596,"meta":597,"navigation":317,"order":63,"path":598,"playlist":599,"seo":600,"status":601,"stem":602,"tags":603,"__hash__":606},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse.md","Multiclass: When a Weight Becomes a Matrix",{"type":8,"value":9,"toc":583},"minimark",[10,14,19,73,82,96,100,125,136,182,186,211,233,253,261,272,355,389,398,402,409,421,424,428,483,487,498,507,517,563,579],[11,12,13],"p",{},"Lecture 3c, and the professor swaps the 2-class dataset for a 4-class one. The first attempt is to honestly reuse everything that already exists without changing anything, just to show, in practice, exactly where it breaks.",[15,16,18],"h2",{"id":17},"the-dataset-four-blobs-one-in-each-corner","The dataset: four blobs, one in each corner",[20,21,26],"pre",{"className":22,"code":23,"language":24,"meta":25,"style":25},"language-python shiki shiki-themes github-light github-dark","def createMulticlassDataset(n=40):\n  X, y = make_blobs(n_samples=n,\n                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n                    n_features=2,\n                    cluster_std=0.05,\n                    center_box=(0,1))\n  return X, y\n","python","",[27,28,29,37,43,49,55,61,67],"code",{"__ignoreMap":25},[30,31,34],"span",{"class":32,"line":33},"line",1,[30,35,36],{},"def createMulticlassDataset(n=40):\n",[30,38,40],{"class":32,"line":39},2,[30,41,42],{},"  X, y = make_blobs(n_samples=n,\n",[30,44,46],{"class":32,"line":45},3,[30,47,48],{},"                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n",[30,50,52],{"class":32,"line":51},4,[30,53,54],{},"                    n_features=2,\n",[30,56,58],{"class":32,"line":57},5,[30,59,60],{},"                    cluster_std=0.05,\n",[30,62,64],{"class":32,"line":63},6,[30,65,66],{},"                    center_box=(0,1))\n",[30,68,70],{"class":32,"line":69},7,[30,71,72],{},"  return X, y\n",[20,74,76],{"className":22,"code":75,"language":24,"meta":25,"style":25},"set(y_train)\n",[27,77,78],{"__ignoreMap":25},[30,79,80],{"class":32,"line":33},[30,81,75],{},[83,84,85],"blockquote",{},[11,86,87,91,92,95],{},[88,89,90],"strong",{},"Output:"," ",[27,93,94],{},"{0, 1, 2, 3}",". Four classes, one integer each.",[15,97,99],{"id":98},"first-attempt-reuse-what-already-exists-and-fail-on-purpose","First attempt: reuse what already exists (and fail on purpose)",[20,101,103],{"className":22,"code":102,"language":24,"meta":25,"style":25},"model = NeuralNetwork()  # the usual one, sign(X @ w_)\nmodel.fit(X, y)\ny_pred = model.predict(X)\nprint(f\"Accuracy: {accuracy_score(y, y_pred)}\")\n",[27,104,105,110,115,120],{"__ignoreMap":25},[30,106,107],{"class":32,"line":33},[30,108,109],{},"model = NeuralNetwork()  # the usual one, sign(X @ w_)\n",[30,111,112],{"class":32,"line":39},[30,113,114],{},"model.fit(X, y)\n",[30,116,117],{"class":32,"line":45},[30,118,119],{},"y_pred = model.predict(X)\n",[30,121,122],{"class":32,"line":51},[30,123,124],{},"print(f\"Accuracy: {accuracy_score(y, y_pred)}\")\n",[83,126,127],{},[11,128,129,131,132,135],{},[88,130,90],{}," accuracy ",[88,133,134],{},"0.25",".",[11,137,138,139,141,142,145,146,149,150,153,154,157,158,161,162,161,165,168,169,172,173,161,175,161,177,168,179,181],{},"Worth understanding exactly why ",[27,140,134],{},", it's not just some random number: with 4 well-balanced classes, always guessing the same thing gets you right on average 1 out of every 4 times, that is, 25%. The usual ",[27,143,144],{},"NeuralNetwork"," uses ",[27,147,148],{},"sign(X @ w_)",", which only returns ",[27,151,152],{},"-1"," or ",[27,155,156],{},"+1",", two possible values, never ",[27,159,160],{},"0",", ",[27,163,164],{},"1",[27,166,167],{},"2",", or ",[27,170,171],{},"3",". Comparing that against a label that can be ",[27,174,160],{},[27,176,164],{},[27,178,167],{},[27,180,171],{}," is comparing things of different natures. The model isn't \"almost getting it right\", it literally can't express 3 of the 4 possible answers. The 0.25 accuracy is, in practice, the same level as a blind guess.",[15,183,185],{"id":184},"one-hot-every-class-becomes-its-own-column","One-hot: every class becomes its own column",[20,187,189],{"className":22,"code":188,"language":24,"meta":25,"style":25},"y_hot = np.zeros((y_train.shape[0], len(set(y_train))), dtype=int)\nfor i, label in enumerate(list(set(y_train))):\n  idxs = np.where(y_train == label)[0]\n  y_hot[idxs, i] = 1\n",[27,190,191,196,201,206],{"__ignoreMap":25},[30,192,193],{"class":32,"line":33},[30,194,195],{},"y_hot = np.zeros((y_train.shape[0], len(set(y_train))), dtype=int)\n",[30,197,198],{"class":32,"line":39},[30,199,200],{},"for i, label in enumerate(list(set(y_train))):\n",[30,202,203],{"class":32,"line":45},[30,204,205],{},"  idxs = np.where(y_train == label)[0]\n",[30,207,208],{"class":32,"line":51},[30,209,210],{},"  y_hot[idxs, i] = 1\n",[11,212,213,214,161,216,161,218,168,220,222,223,225,226,228,229,232],{},"Instead of a scalar label (",[27,215,160],{},[27,217,164],{},[27,219,167],{},[27,221,171],{},"), every example becomes a row with a ",[27,224,164],{}," in its class's column and ",[27,227,160],{}," in the others. The professor confirms this matches exactly scikit-learn's ",[27,230,231],{},"LabelBinarizer",":",[20,234,236],{"className":22,"code":235,"language":24,"meta":25,"style":25},"from sklearn.preprocessing import LabelBinarizer\nlb = LabelBinarizer()\ny_hot = lb.fit_transform(y_train)\n",[27,237,238,243,248],{"__ignoreMap":25},[30,239,240],{"class":32,"line":33},[30,241,242],{},"from sklearn.preprocessing import LabelBinarizer\n",[30,244,245],{"class":32,"line":39},[30,246,247],{},"lb = LabelBinarizer()\n",[30,249,250],{"class":32,"line":45},[30,251,252],{},"y_hot = lb.fit_transform(y_train)\n",[11,254,255,256,135],{},"Same matrix, two implementations, ",[257,258,260],"a",{"href":259},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","the same kind of \"matches the professional tool\" check that already showed up in this playlist",[15,262,264,265,268,269],{"id":263},"the-fix-weight-becomes-a-matrix-sign-becomes-argmax","The fix: weight becomes a matrix, ",[27,266,267],{},"sign"," becomes ",[27,270,271],{},"argmax",[20,273,275],{"className":22,"code":274,"language":24,"meta":25,"style":25},"class SGD(TrainingAlgorithm):\n  def get_w(self, X, y):\n    self.w_ = np.random.random(size=(X.shape[1], y.shape[1]))\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      self.w_ += self.learning_rate * self.cost_function.get_gradient(X, y, y_pred)\n    return self.w_\n\nclass NeuralNetwork(BaseEstimator, ClassifierMixin):\n  def predict(self, X):\n    X = include_bias(X)\n    logits = X @ self.w_\n    idxs = np.argmax(logits, axis=1)\n    return np.array([self.labels[idx] for idx in idxs])\n",[27,276,277,282,287,292,297,302,307,312,319,325,331,337,343,349],{"__ignoreMap":25},[30,278,279],{"class":32,"line":33},[30,280,281],{},"class SGD(TrainingAlgorithm):\n",[30,283,284],{"class":32,"line":39},[30,285,286],{},"  def get_w(self, X, y):\n",[30,288,289],{"class":32,"line":45},[30,290,291],{},"    self.w_ = np.random.random(size=(X.shape[1], y.shape[1]))\n",[30,293,294],{"class":32,"line":51},[30,295,296],{},"    for _ in range(self.max_iter):\n",[30,298,299],{"class":32,"line":57},[30,300,301],{},"      y_pred = X @ self.w_\n",[30,303,304],{"class":32,"line":63},[30,305,306],{},"      self.w_ += self.learning_rate * self.cost_function.get_gradient(X, y, y_pred)\n",[30,308,309],{"class":32,"line":69},[30,310,311],{},"    return self.w_\n",[30,313,315],{"class":32,"line":314},8,[30,316,318],{"emptyLinePlaceholder":317},true,"\n",[30,320,322],{"class":32,"line":321},9,[30,323,324],{},"class NeuralNetwork(BaseEstimator, ClassifierMixin):\n",[30,326,328],{"class":32,"line":327},10,[30,329,330],{},"  def predict(self, X):\n",[30,332,334],{"class":32,"line":333},11,[30,335,336],{},"    X = include_bias(X)\n",[30,338,340],{"class":32,"line":339},12,[30,341,342],{},"    logits = X @ self.w_\n",[30,344,346],{"class":32,"line":345},13,[30,347,348],{},"    idxs = np.argmax(logits, axis=1)\n",[30,350,352],{"class":32,"line":351},14,[30,353,354],{},"    return np.array([self.labels[idx] for idx in idxs])\n",[11,356,357,358,361,362,365,366,369,370,373,374,377,378,381,382,384,385,388],{},"The change that fixes everything: ",[27,359,360],{},"self.w_"," stops being a vector (",[27,363,364],{},"(features,)",") and becomes a ",[88,367,368],{},"matrix"," (",[27,371,372],{},"(features, classes)","), one column of weights per class. ",[27,375,376],{},"X @ self.w_"," now returns, for every point, 4 numbers (one \"how confident\" score per class), not just 1. And the final prediction swaps ",[27,379,380],{},"sign()"," for ",[27,383,271],{},": instead of asking \"positive or negative?\", it asks \"which of the 4 columns had the highest value?\". This is exactly the multiple-output layer architecture Aggarwal describes for categorical classification: one weight per class, and the final decision is whichever one \"won\". The only piece missing to turn it into his full version (with ",[27,386,387],{},"softmax",", turning the 4 numbers into probabilities that sum to 1) is the normalization. Here the model just compares the raw numbers, without turning them into a probability, but the argmax winner doesn't change either way.",[83,390,391],{},[11,392,393,131,395,135],{},[88,394,90],{},[88,396,397],{},"1.0",[15,399,401],{"id":400},"interactive-the-four-decision-regions","Interactive: the four decision regions",[11,403,404,405,408],{},"I rebuilt the same dataset (",[27,406,407],{},"make_blobs",", same 4 centers) and trained the weight-matrix version. Every background color is the region where that class wins the argmax.",[410,411],"multiclass-region-chart",{":classes":412,":points":413,":weights":414,":x-max":164,":x-min":160,":y-max":164,":y-min":160,"class0-label":415,"class1-label":416,"class2-label":417,"class3-label":418,"x-label":419,"y-label":420},"[1, 1, 3, 2, 0, 1, 2, 1, 2, 0, 0, 0, 0, 1, 2, 1, 3, 3, 0, 3, 1, 2, 3, 3, 1, 3, 3, 2, 3, 0, 2, 0, 1, 1, 2, 0, 2, 2, 0, 3]","[[0.6968, 0.1669], [0.7174, 0.2268], [0.785, 0.741], [0.2956, 0.8119], [0.2277, 0.2062], [0.7398, 0.2731], [0.1953, 0.8597], [0.8825, 0.2077], [0.242, 0.791], [0.2, 0.1123], [0.1687, 0.1914], [0.2845, 0.1767], [0.1606, 0.2001], [0.8525, 0.1792], [0.2051, 0.8126], [0.7629, 0.2536], [0.795, 0.885], [0.8166, 0.8368], [0.1879, 0.1273], [0.8749, 0.7859], [0.7797, 0.0856], [0.1146, 0.7098], [0.7403, 0.7475], [0.7904, 0.7111], [0.8883, 0.1835], [0.7816, 0.7047], [0.7808, 0.7555], [0.1283, 0.8251], [0.8054, 0.8719], [0.2016, 0.2204], [0.2135, 0.7738], [0.2253, 0.1869], [0.7806, 0.3015], [0.7977, 0.1275], [0.1934, 0.7845], [0.2509, 0.23], [0.2284, 0.7624], [0.2192, 0.9124], [0.2137, 0.1237], [0.8752, 0.7894]]","[[1.0602, -0.8069, -0.8239], [0.2232, 0.812, -0.7759], [0.2644, -0.8451, 0.8351], [-0.5478, 0.84, 0.7647]]","Class 0","Class 1","Class 2","Class 3","x0","x1",[11,422,423],{},"Notice the four boundaries meeting near the middle of the chart, splitting the plane into four wedges, one per class. Every blob lands cleanly inside the right color.",[15,425,427],{"id":426},"wrapping-up","Wrapping up",[429,430,431,445],"table",{},[432,433,434],"thead",{},[435,436,437,442],"tr",{},[438,439,441],"th",{"align":440},"left","What I already knew",[438,443,444],{"align":440},"What this lecture settled",[446,447,448,462,470],"tbody",{},[435,449,450,456],{},[451,452,453,455],"td",{"align":440},[27,454,380],{}," classifies into two classes",[451,457,458,459,461],{"align":440},"With more than two classes, ",[27,460,380],{}," structurally can't work, there are only 2 possible outputs for N classes",[435,463,464,467],{},[451,465,466],{"align":440},"One-hot encoding turns a categorical label into a vector",[451,468,469],{"align":440},"That's not just a formatting convenience, it's what lets the weight become a matrix (one column per class)",[435,471,472,477],{},[451,473,474,476],{"align":440},[27,475,271],{}," picks the largest value",[451,478,479,480,482],{"align":440},"It's the direct generalization of ",[27,481,380],{}," (which is basically \"argmax between 2 options: positive or negative\") to any number of classes",[15,484,486],{"id":485},"practical-application","Practical application",[11,488,489,490,493,494,497],{},"I tested the same idea (one-hot + weight matrix + argmax) on Wine (",[27,491,492],{},"load_wine",", 3 grape cultivars, 13 chemical variables), with one extra detail: the variables here live on quite different scales, ",[257,495,496],{"href":259},"the same problem already seen earlier in this playlist",", so I normalized before training.",[20,499,501],{"className":22,"code":500,"language":24,"meta":25,"style":25},"X_train_s = StandardScaler().fit_transform(X_train)\n",[27,502,503],{"__ignoreMap":25},[30,504,505],{"class":32,"line":33},[30,506,500],{},[11,508,509,510,513,514,135],{},"Even normalized, ",[27,511,512],{},"learning_rate=0.01"," (the notebook's default) still diverged with 13 variables, so I had to drop to ",[27,515,516],{},"0.001",[429,518,519,533],{},[432,520,521],{},[435,522,523,526,530],{},[438,524,525],{"align":440},"Approach",[438,527,529],{"align":528},"right","Train accuracy",[438,531,532],{"align":528},"Test accuracy",[446,534,535,549],{},[435,536,537,543,546],{},[451,538,539,540,542],{"align":440},"Naive (",[27,541,267],{},", scalar label)",[451,544,545],{"align":528},"0.403",[451,547,548],{"align":528},"-",[435,550,551,556,558],{},[451,552,553,554],{"align":440},"One-hot + weight matrix + ",[27,555,271],{},[451,557,397],{"align":528},[451,559,560],{"align":528},[88,561,562],{},"0.9815",[11,564,565,566,153,568,570,571,573,574,153,576,578],{},"The naive version can't even reach all 3 possible class values (it can only ever predict ",[27,567,152],{},[27,569,156],{},", never class ",[27,572,167],{},"), so even that 0.403 number is misleading, it counts as a \"hit\" any case where the label happened to already be ",[27,575,152],{},[27,577,164],{},". The weight-matrix version gets nearly everything right, both train and test, confirming the same trick that worked on the synthetic 4-blob dataset generalizes to a real multiclass classification problem.",[580,581,582],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":25,"searchDepth":39,"depth":39,"links":584},[585,586,587,588,590,591,592],{"id":17,"depth":39,"text":18},{"id":98,"depth":39,"text":99},{"id":184,"depth":39,"text":185},{"id":263,"depth":39,"text":589},"The fix: weight becomes a matrix, sign becomes argmax",{"id":400,"depth":39,"text":401},{"id":426,"depth":39,"text":427},{"id":485,"depth":39,"text":486},null,"2026-08-20","Lecture 3c: reusing the binary classifier directly on a 4-class problem lands exactly at random-guess accuracy, 25%. The fix is generalizing the weight from a vector to a matrix, one set of weights per class, and swapping sign() for argmax.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse","neural-networks",{"title":6,"description":595},"published","en\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse",[604,605,271],"multiclass","one-hot-encoding","l52gY4-RRFdZBm-iOZFY-VoFu7l55JDyblv5enLfn7k",1787338984655]