[{"data":1,"prerenderedAt":824},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias":3,"post-en-neural-networks-perceptron-com-bias":4},"\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias",{"id":5,"title":6,"body":7,"cover":809,"date":810,"description":811,"extension":812,"meta":813,"navigation":155,"order":39,"path":814,"playlist":815,"seo":816,"status":817,"stem":818,"tags":819,"__hash__":823},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias.md","Bias and Vectorization: Unlocking the Perceptron",{"type":8,"value":9,"toc":799},"minimark",[10,14,19,73,99,106,122,127,131,262,303,310,326,341,345,354,368,372,453,505,520,535,539,542,552,563,566,645,648,652,659,663,667,709,713,725,740,788,795],[11,12,13],"p",{},"Lectures 2a and 2b, and each one solves one problem: the first unlocks where the decision boundary can sit, the second unlocks how the weight gets updated.",[15,16,18],"h2",{"id":17},"the-same-limit-from-last-lecture-confirmed-again","The same limit from last lecture, confirmed again",[20,21,26],"pre",{"className":22,"code":23,"language":24,"meta":25,"style":25},"language-python shiki shiki-themes github-light github-dark","def createDataset(n=20):\n  X = np.random.rand(n,2)\n  coefs = np.array([1, 1])\n  intercept = 1\n  labels = X @ coefs - intercept\n  y = np.array(labels>0, dtype=int)*2-1\n  return X, y\n","python","",[27,28,29,37,43,49,55,61,67],"code",{"__ignoreMap":25},[30,31,34],"span",{"class":32,"line":33},"line",1,[30,35,36],{},"def createDataset(n=20):\n",[30,38,40],{"class":32,"line":39},2,[30,41,42],{},"  X = np.random.rand(n,2)\n",[30,44,46],{"class":32,"line":45},3,[30,47,48],{},"  coefs = np.array([1, 1])\n",[30,50,52],{"class":32,"line":51},4,[30,53,54],{},"  intercept = 1\n",[30,56,58],{"class":32,"line":57},5,[30,59,60],{},"  labels = X @ coefs - intercept\n",[30,62,64],{"class":32,"line":63},6,[30,65,66],{},"  y = np.array(labels>0, dtype=int)*2-1\n",[30,68,70],{"class":32,"line":69},7,[30,71,72],{},"  return X, y\n",[11,74,75,76,79,80,83,84,88,89,94,95,98],{},"Notice the subtle difference from last lecture's dataset: there's now an ",[27,77,78],{},"intercept = 1"," subtracted before applying the sign. That shifts the true separating line to ",[27,81,82],{},"x + y = 1",", which ",[85,86,87],"strong",{},"doesn't pass through the origin",". ",[90,91,93],"a",{"href":92},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","That's exactly the scenario I simulated artificially in the previous post"," using Iris: a boundary that doesn't pass through ",[27,96,97],{},"(0,0)",".",[11,100,101,102,105],{},"The professor runs the exact same no-bias ",[27,103,104],{},"PLA"," from last lecture, without changing a line, straight on this new dataset:",[107,108,109],"blockquote",{},[11,110,111,114,115,118,119,98],{},[85,112,113],{},"Output:"," accuracy ",[85,116,117],{},"0.7",", weights ",[27,120,121],{},"[2.71, 0.20]",[11,123,124,125,98],{},"Confirms, with his real code (not just my Iris simulation), the exact same limit: without bias, the boundary can only rotate around the origin, and there's no way to rotate a line through the origin until it matches ",[27,126,82],{},[15,128,130],{"id":129},"adding-bias-the-boundary-gains-the-freedom-to-slide","Adding bias: the boundary gains the freedom to slide",[20,132,134],{"className":22,"code":133,"language":24,"meta":25,"style":25},"class PLA(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=1000):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      cost = 0\n      idx = np.arange(X.shape[0])\n      np.random.shuffle(idx)\n      for i in idx:\n        logits = X[i] @ self.w_ + self.b_\n        y_pred = np.sign(logits)\n        error = y[i] - y_pred\n        if error != 0:\n          cost += error**2\n          self.w_ += error*X[i]\n          self.b_ += error\n        if cost == 0:\n          break\n    return self\n",[27,135,136,141,146,151,157,162,167,172,178,184,190,196,202,208,214,220,226,232,238,244,250,256],{"__ignoreMap":25},[30,137,138],{"class":32,"line":33},[30,139,140],{},"class PLA(BaseEstimator, ClassifierMixin):\n",[30,142,143],{"class":32,"line":39},[30,144,145],{},"  def __init__(self, max_iter=1000):\n",[30,147,148],{"class":32,"line":45},[30,149,150],{},"    self.max_iter = max_iter\n",[30,152,153],{"class":32,"line":51},[30,154,156],{"emptyLinePlaceholder":155},true,"\n",[30,158,159],{"class":32,"line":57},[30,160,161],{},"  def fit(self, X, y):\n",[30,163,164],{"class":32,"line":63},[30,165,166],{},"    self.w_ = np.random.rand(X.shape[1])\n",[30,168,169],{"class":32,"line":69},[30,170,171],{},"    self.b_ = np.random.rand()\n",[30,173,175],{"class":32,"line":174},8,[30,176,177],{},"    for _ in range(self.max_iter):\n",[30,179,181],{"class":32,"line":180},9,[30,182,183],{},"      cost = 0\n",[30,185,187],{"class":32,"line":186},10,[30,188,189],{},"      idx = np.arange(X.shape[0])\n",[30,191,193],{"class":32,"line":192},11,[30,194,195],{},"      np.random.shuffle(idx)\n",[30,197,199],{"class":32,"line":198},12,[30,200,201],{},"      for i in idx:\n",[30,203,205],{"class":32,"line":204},13,[30,206,207],{},"        logits = X[i] @ self.w_ + self.b_\n",[30,209,211],{"class":32,"line":210},14,[30,212,213],{},"        y_pred = np.sign(logits)\n",[30,215,217],{"class":32,"line":216},15,[30,218,219],{},"        error = y[i] - y_pred\n",[30,221,223],{"class":32,"line":222},16,[30,224,225],{},"        if error != 0:\n",[30,227,229],{"class":32,"line":228},17,[30,230,231],{},"          cost += error**2\n",[30,233,235],{"class":32,"line":234},18,[30,236,237],{},"          self.w_ += error*X[i]\n",[30,239,241],{"class":32,"line":240},19,[30,242,243],{},"          self.b_ += error\n",[30,245,247],{"class":32,"line":246},20,[30,248,249],{},"        if cost == 0:\n",[30,251,253],{"class":32,"line":252},21,[30,254,255],{},"          break\n",[30,257,259],{"class":32,"line":258},22,[30,260,261],{},"    return self\n",[11,263,264,265,268,269,272,273,276,277,280,281,284,285,288,289,291,292,295,296,299,300,98],{},"The change is small in code but big in what it unlocks: now ",[27,266,267],{},"logits = X[i] @ w_ + b_",", and ",[27,270,271],{},"b_"," gets updated alongside, ",[27,274,275],{},"self.b_ += error",". The most direct way to see why this is the right update: think of the bias as the weight of an extra input variable that's always worth ",[27,278,279],{},"1",". If ",[27,282,283],{},"w_"," already gets updated by ",[27,286,287],{},"error * x[i]"," for each real variable, the \"weight\" of that phantom variable worth ",[27,290,279],{}," would get updated by ",[27,293,294],{},"error * 1",", that is, just ",[27,297,298],{},"error"," on its own. It's the same old trick (adding a column of 1s), just showing up here explicitly as a separate variable instead of hidden inside ",[27,301,302],{},"X",[11,304,305,306,309],{},"Geometrically, the bias shifts the hyperplane without rotating it: now ",[27,307,308],{},"w · x + b = 0"," can sit anywhere in the plane, not just crossing the origin.",[107,311,312],{},[11,313,314,114,316,118,319,322,323,98],{},[85,315,113],{},[85,317,318],{},"1.0",[27,320,321],{},"[5.10, 3.28]",", bias ",[27,324,325],{},"-3.98",[11,327,328,329,332,333,336,337,340],{},"Converges. Notice the professor also bumped ",[27,330,331],{},"max_iter"," from 10 to 1000 in this version, ",[90,334,335],{"href":92},"the same stopping bug from last lecture"," still lives inside that ",[27,338,339],{},"if cost == 0: break",", just with 1000 epochs of chances instead of 10, the odds of never getting one fully clean pass drop a lot. It's not a fix for the bug, it's just giving luck enough room to compensate.",[15,342,344],{"id":343},"interactive-perceptron-with-bias-point-by-point","Interactive: perceptron with bias, point by point",[11,346,347,348,351,352,98],{},"Same real dataset from the notebook (the exact 20 points the professor ran), now with bias turned on. Click \"Process next point\" and notice how the boundary, this time, can ",[85,349,350],{},"slide"," away from the origin until it lines up with ",[27,353,82],{},[355,356],"perceptron-explorer",{":classes":357,":points":358,":update-bias":359,":x-max":279,":x-min":360,":y-max":279,":y-min":360,"converged-label":361,"negative-label":362,"positive-label":363,"reset-label":364,"step-label":365,"x-label":366,"y-label":367},"[1, 1, 1, 1, -1, 1, 1, -1, 1, 1, 1, -1, 1, -1, 1, -1, -1, 1, 1, 1]","[[0.78019343, 0.84822159], [0.69216878, 0.44056631], [0.60614199, 0.66202198], [0.47672573, 0.54204385], [0.7461111, 0.01893793], [0.6065733, 0.435086], [0.75295568, 0.47766191], [0.07180149, 0.08440745], [0.64647152, 0.46276951], [0.95252713, 0.83099628], [0.74947845, 0.82567423], [0.26043976, 0.72253777], [0.45985729, 0.86798639], [0.21107237, 0.04054585], [0.87159312, 0.76289216], [0.11142677, 0.30233227], [0.10352009, 0.82903887], [0.60640317, 0.65858996], [0.8663473, 0.184016], [0.91813453, 0.47377763]]","true","0","converged, zero mistakes in a full pass","class -1","class +1","Reset (new random draw)","Process next point","x0","x1",[15,369,371],{"id":370},"vectorizing-from-a-point-by-point-loop-to-a-single-computation","Vectorizing: from a point-by-point loop to a single computation",[20,373,375],{"className":22,"code":374,"language":24,"meta":25,"style":25},"class Perceptron(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=1000):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      cost = 0\n      y_pred = self.predict(X)\n      error = y - y_pred\n      self.w_ += np.dot(X.T, error)\n      self.b_ += np.sum(error)\n      cost = np.sum(error**2)\n      if cost == 0:\n        break\n    return self\n",[27,376,377,382,386,390,394,398,402,406,410,414,419,424,429,434,439,444,449],{"__ignoreMap":25},[30,378,379],{"class":32,"line":33},[30,380,381],{},"class Perceptron(BaseEstimator, ClassifierMixin):\n",[30,383,384],{"class":32,"line":39},[30,385,145],{},[30,387,388],{"class":32,"line":45},[30,389,150],{},[30,391,392],{"class":32,"line":51},[30,393,156],{"emptyLinePlaceholder":155},[30,395,396],{"class":32,"line":57},[30,397,161],{},[30,399,400],{"class":32,"line":63},[30,401,166],{},[30,403,404],{"class":32,"line":69},[30,405,171],{},[30,407,408],{"class":32,"line":174},[30,409,177],{},[30,411,412],{"class":32,"line":180},[30,413,183],{},[30,415,416],{"class":32,"line":186},[30,417,418],{},"      y_pred = self.predict(X)\n",[30,420,421],{"class":32,"line":192},[30,422,423],{},"      error = y - y_pred\n",[30,425,426],{"class":32,"line":198},[30,427,428],{},"      self.w_ += np.dot(X.T, error)\n",[30,430,431],{"class":32,"line":204},[30,432,433],{},"      self.b_ += np.sum(error)\n",[30,435,436],{"class":32,"line":210},[30,437,438],{},"      cost = np.sum(error**2)\n",[30,440,441],{"class":32,"line":216},[30,442,443],{},"      if cost == 0:\n",[30,445,446],{"class":32,"line":222},[30,447,448],{},"        break\n",[30,450,451],{"class":32,"line":228},[30,452,261],{},[11,454,455,456,459,460,463,464,467,468,463,471,474,475,478,479,482,483,487,488,492,493,496,497,500,501,504],{},"This version throws out the entire ",[27,457,458],{},"for i in idx"," loop. Instead of looking at one point at a time, it: predicts ",[85,461,462],{},"everyone at once"," (",[27,465,466],{},"y_pred = self.predict(X)","), computes ",[85,469,470],{},"everyone's error at once",[27,472,473],{},"error = y - y_pred","), and does ",[85,476,477],{},"a single update"," summing each mistaken point's contribution (",[27,480,481],{},"X.T @ error",", the same matrix-vector product that already showed up ",[90,484,486],{"href":485},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation","in the normal equation post"," and ",[90,489,491],{"href":490},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization","in Andrew Ng's specialization","). This also kills the premature-stop bug for good: since ",[27,494,495],{},"cost"," only gets computed ",[85,498,499],{},"after"," the whole dataset has already been processed that iteration, there's no way for the ",[27,502,503],{},"break"," to fire too early.",[11,506,507,508,511,512,515,516,519],{},"That's the same distinction between ",[85,509,510],{},"batch gradient descent"," (uses the whole dataset per update) and ",[85,513,514],{},"stochastic gradient descent"," (updates on every example) ",[90,517,518],{"href":490},"I already saw in the other playlist",", just applied here to the perceptron's learning rule instead of a regression.",[107,521,522],{},[11,523,524,114,527,118,529,322,532,98],{},[85,525,526],{},"Output (same 20-point dataset):",[85,528,318],{},[27,530,531],{},"[24.75, 16.55]",[27,533,534],{},"-19.04",[15,536,538],{"id":537},"the-detail-that-slips-by-100-on-training-isnt-100-guaranteed-on-new-data","The detail that slips by: 100% on training isn't 100% guaranteed on new data",[11,540,541],{},"The professor tests this same vectorized model on a much larger test set, 1000 new points generated by the same rule:",[107,543,544],{},[11,545,546,114,548,551],{},[85,547,113],{},[85,549,550],{},"0.897"," on 1000 test points.",[11,553,554,555,558,559,562],{},"Dropped from 1.0 to 0.897. That's not a sign of a bug in the code, it's a property of the perceptron itself, which Aggarwal points out in chapter 1: the perceptron's update rule only guarantees finding ",[85,556,557],{},"some"," line that separates the training points, not necessarily the ",[85,560,561],{},"best"," one (the one with the widest margin to each class's points). With only 20 training points, several different lines can separate all of them perfectly, but each of those lines gets a different slice of new points near the true boundary wrong. Aggarwal calls this the \"perceptron criterion\", in contrast with the SVM (Support Vector Machine), which solves exactly this problem by maximizing the margin on purpose.",[11,564,565],{},"I ran this same comparison (small training set, 1000-point test set) 5 times, both in the point-by-point version and the vectorized one, to see if this is specific to one version or shows up in both:",[567,568,569,587],"table",{},[570,571,572],"thead",{},[573,574,575,580,584],"tr",{},[576,577,579],"th",{"align":578},"center","Run",[576,581,583],{"align":582},"right","Point-by-point (train \u002F test)",[576,585,586],{"align":582},"Vectorized (train \u002F test)",[588,589,590,601,612,623,634],"tbody",{},[573,591,592,595,598],{},[593,594,279],"td",{"align":578},[593,596,597],{"align":582},"1.0 \u002F 0.894",[593,599,600],{"align":582},"1.0 \u002F 0.937",[573,602,603,606,609],{},[593,604,605],{"align":578},"2",[593,607,608],{"align":582},"1.0 \u002F 0.963",[593,610,611],{"align":582},"1.0 \u002F 0.881",[573,613,614,617,620],{},[593,615,616],{"align":578},"3",[593,618,619],{"align":582},"1.0 \u002F 0.967",[593,621,622],{"align":582},"1.0 \u002F 0.960",[573,624,625,628,631],{},[593,626,627],{"align":578},"4",[593,629,630],{"align":582},"1.0 \u002F 0.986",[593,632,633],{"align":582},"1.0 \u002F 0.978",[573,635,636,639,642],{},[593,637,638],{"align":578},"5",[593,640,641],{"align":582},"1.0 \u002F 0.948",[593,643,644],{"align":582},"1.0 \u002F 0.980",[11,646,647],{},"Both always hit 100% on training, and both swing quite a bit on test (from 0.88 to 0.99), with no clear pattern of which update style is more reliable. This confirms the instability doesn't come from \"updating point by point\" versus \"updating everyone at once\", it comes from the perceptron criterion itself: any separator will do, not necessarily the safest one.",[15,649,651],{"id":650},"interactive-the-vectorized-version-one-step-one-full-epoch","Interactive: the vectorized version, one step = one full epoch",[11,653,654,655,658],{},"Same dataset, same component, just switching modes: now every click processes the ",[85,656,657],{},"entire"," dataset at once, instead of point by point.",[355,660],{":classes":357,":points":358,":update-bias":359,":x-max":279,":x-min":360,":y-max":279,":y-min":360,"converged-label":361,"negative-label":362,"positive-label":363,"reset-label":364,"step-label":661,"x-label":366,"y-label":367,"mode":662},"Process next iteration","batch",[15,664,666],{"id":665},"wrapping-up","Wrapping up",[567,668,669,680],{},[570,670,671],{},[573,672,673,677],{},[576,674,676],{"align":675},"left","What I already knew",[576,678,679],{"align":675},"What this lecture settled",[588,681,682,690,701],{},[573,683,684,687],{},[593,685,686],{"align":675},"A no-bias boundary always passes through the origin",[593,688,689],{"align":675},"Adding bias is the same old trick (phantom variable worth 1), and it unlocks the boundary to slide anywhere",[573,691,692,695],{},[593,693,694],{"align":675},"Batch vs. stochastic gradient descent is a regression thing",[593,696,697,698,700],{"align":675},"The same distinction exists for the perceptron: the point-by-point loop is the \"stochastic\" version, ",[27,699,481],{}," all at once is the \"batch\" version",[573,702,703,706],{},[593,704,705],{"align":675},"1.0 training accuracy sounds like \"I'm done\"",[593,707,708],{"align":675},"It doesn't guarantee generalization: the perceptron only finds some separating line, not the safest one, and that's a limit of the criterion, not an implementation bug",[15,710,712],{"id":711},"practical-application","Practical application",[11,714,715,716,720,721,724],{},"I repeated the point-by-point vs. vectorized comparison on Iris (",[717,718,719],"em",{},"setosa"," vs. ",[717,722,723],{},"versicolor",", petal length and width), now with bias and a real 70\u002F30 train\u002Ftest split.",[20,726,728],{"className":22,"code":727,"language":24,"meta":25,"style":25},"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[27,729,730,735],{"__ignoreMap":25},[30,731,732],{"class":32,"line":33},[30,733,734],{},"from sklearn.model_selection import train_test_split\n",[30,736,737],{"class":32,"line":39},[30,738,739],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[567,741,742,758],{},[570,743,744],{},[573,745,746,749,752,755],{},[576,747,748],{"align":675},"Version",[576,750,751],{"align":578},"Epochs to converge",[576,753,754],{"align":582},"Train accuracy",[576,756,757],{"align":582},"Test accuracy",[588,759,760,774],{},[573,761,762,765,768,770],{},[593,763,764],{"align":675},"Point-by-point",[593,766,767],{"align":578},"2 to 3",[593,769,318],{"align":582},[593,771,772],{"align":582},[85,773,318],{},[573,775,776,779,782,784],{},[593,777,778],{"align":675},"Vectorized",[593,780,781],{"align":578},"7",[593,783,318],{"align":582},[593,785,786],{"align":582},[85,787,318],{},[11,789,790,791,794],{},"This time both hit 100% on test too, across the 3 seeds I tried. Makes sense: unlike the synthetic dataset above (just 20 loosely scattered points), Iris has a pretty generous margin between the two classes, so any reasonable separating line already gets the new points right. What was left was speed: the point-by-point version converged in 2 to 3 epochs, the vectorized one needed 7. That tracks with what I explained above: within a single epoch, the point-by-point version can make up to 20 weight updates (one per mistake found), while the vectorized version makes only ",[85,792,793],{},"one"," update per epoch, so it naturally needs more passes over the dataset to accumulate the same amount of adjustment.",[796,797,798],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":25,"searchDepth":39,"depth":39,"links":800},[801,802,803,804,805,806,807,808],{"id":17,"depth":39,"text":18},{"id":129,"depth":39,"text":130},{"id":343,"depth":39,"text":344},{"id":370,"depth":39,"text":371},{"id":537,"depth":39,"text":538},{"id":650,"depth":39,"text":651},{"id":665,"depth":39,"text":666},{"id":711,"depth":39,"text":712},null,"2026-08-20","Lectures 2a and 2b: the professor adds bias to the perceptron (solving exactly the limit that closed last lecture) and then vectorizes the whole weight update, swapping the point-by-point loop for a single computation over the entire dataset.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias","neural-networks",{"title":6,"description":811},"published","en\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias",[820,821,822],"perceptron","bias","vectorization","JDWpvRvHFyo_ILqbpAEKUsk7Lj8BjfF-lkNkHIBzMLo",1787338984157]