[{"data":1,"prerenderedAt":691},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron":3,"post-en-neural-networks-mcculloch-pitts-perceptron":4},"\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron",{"id":5,"title":6,"body":7,"cover":676,"date":677,"description":678,"extension":679,"meta":680,"navigation":209,"order":56,"path":681,"playlist":682,"seo":683,"status":684,"stem":685,"tags":686,"__hash__":690},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron.md","Perceptron: the First Neuron That Learns on Its Own",{"type":8,"value":9,"toc":667},"minimark",[10,14,19,30,38,42,90,123,148,167,181,185,305,319,362,369,373,392,409,416,436,449,464,474,478,489,503,517,521,566,570,585,615,618,660,663],[11,12,13],"p",{},"First lecture of the living playlist, and it starts exactly where any neural networks course should: at the simplest neuron there is.",[15,16,18],"h2",{"id":17},"before-the-code-two-papers-15-years-apart","Before the code: two papers, 15 years apart",[11,20,21,22,29],{},"Neural networks weren't born as code, they were born as a biophysics question. In 1943, Warren McCulloch and Walter Pitts published ",[23,24,28],"a",{"href":25,"rel":26},"https:\u002F\u002Fwww.cs.cmu.edu\u002F~epxing\u002FClass\u002F10715\u002Freading\u002FMcCulloch.and.Pitts.pdf",[27],"nofollow","\"A Logical Calculus of the Ideas Immanent in Nervous Activity\"",", proposing a lean mathematical model for a biological neuron: sum up the input signals, and fire a binary (all-or-nothing) signal if that sum crosses a threshold. No learning yet, it's just a fixed logic circuit, each \"neuron\" solves a logical function someone already decided ahead of time.",[11,31,32,33,37],{},"The missing leap came 15 years later, with Frank Rosenblatt in 1958: what if, instead of someone hand-picking the weights, the neuron itself learned the right weights by looking at examples? That's the Perceptron, and it's exactly what the professor's lecture 1a implements: the ",[34,35,36],"strong",{},"Perceptron Learning Algorithm"," (PLA), in its rawest form, no bias yet (that's next lecture).",[15,39,41],{"id":40},"the-dataset-two-groups-separable-by-a-line","The dataset: two groups separable by a line",[43,44,49],"pre",{"className":45,"code":46,"language":47,"meta":48,"style":48},"language-python shiki shiki-themes github-light github-dark","def createDataset(n=20):\n  X = np.random.rand(n,2)\n  coefs = np.array([1, -1])\n  labels = X @ coefs\n  y = np.array(labels>0, dtype=int)*2-1\n  return X, y\n","python","",[50,51,52,60,66,72,78,84],"code",{"__ignoreMap":48},[53,54,57],"span",{"class":55,"line":56},"line",1,[53,58,59],{},"def createDataset(n=20):\n",[53,61,63],{"class":55,"line":62},2,[53,64,65],{},"  X = np.random.rand(n,2)\n",[53,67,69],{"class":55,"line":68},3,[53,70,71],{},"  coefs = np.array([1, -1])\n",[53,73,75],{"class":55,"line":74},4,[53,76,77],{},"  labels = X @ coefs\n",[53,79,81],{"class":55,"line":80},5,[53,82,83],{},"  y = np.array(labels>0, dtype=int)*2-1\n",[53,85,87],{"class":55,"line":86},6,[53,88,89],{},"  return X, y\n",[11,91,92,93,96,97,100,101,103,104,106,107,110,111,114,115,118,119,122],{},"The professor generates random 2D points and labels each one by the sign of ",[50,94,95],{},"X @ coefs",", the dot product between the point and the vector ",[50,98,99],{},"[1, -1]",". Geometrically, ",[50,102,95],{}," is positive on one side of the line that passes through the origin and is perpendicular to ",[50,105,99],{},", and negative on the other side, so the label ",[50,108,109],{},"y"," (",[50,112,113],{},"-1"," or ",[50,116,117],{},"+1",") is ",[34,120,121],{},"linearly separable by construction",": a straight line exists that separates the two classes perfectly, because that exact line is what generated the labels.",[43,124,126],{"className":45,"code":125,"language":47,"meta":48,"style":48},"def plotHyperplan(vector):\n  xs = np.array([0,1])\n  ys = -(vector[0]*xs)\u002Fvector[1]\n  plt.plot(xs, ys)\n",[50,127,128,133,138,143],{"__ignoreMap":48},[53,129,130],{"class":55,"line":56},[53,131,132],{},"def plotHyperplan(vector):\n",[53,134,135],{"class":55,"line":62},[53,136,137],{},"  xs = np.array([0,1])\n",[53,139,140],{"class":55,"line":68},[53,141,142],{},"  ys = -(vector[0]*xs)\u002Fvector[1]\n",[53,144,145],{"class":55,"line":74},[53,146,147],{},"  plt.plot(xs, ys)\n",[11,149,150,151,154,155,158,159,162,163,166],{},"This function draws the line where ",[50,152,153],{},"vector[0]*x + vector[1]*y = 0",", the decision boundary for any weight vector ",[50,156,157],{},"w",". Same equation as always, ",[50,160,161],{},"w · x = 0"," defines a hyperplane, except here without a bias, so the hyperplane is forced to pass through the origin ",[50,164,165],{},"(0,0)",". Keep that detail in mind, it matters in a bit.",[11,168,169,170,173,174,176,177,180],{},"The professor even tests a ",[50,171,172],{},"DummyClassifier"," with fixed weights ",[50,175,99],{},", the exact same ones that generated the dataset, and of course it hits 100%: that's the answer key fed straight back in. The real question is: can these weights actually be ",[34,178,179],{},"learned"," just by looking at examples, without me handing over the answer already solved?",[15,182,184],{"id":183},"the-algorithm-nudge-the-weight-toward-the-error","The algorithm: nudge the weight toward the error",[43,186,188],{"className":45,"code":187,"language":47,"meta":48,"style":48},"class PLA(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=10):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    for _ in range(self.max_iter):\n      cost = 0\n      idx = np.arange(X.shape[0])\n      np.random.shuffle(idx)\n      for i in idx:\n        logits = X[i] @ self.w_\n        y_pred = np.sign(logits)\n        error = y[i] - y_pred\n        if error != 0:\n          cost += error**2\n          self.w_ += error*X[i]\n        if cost == 0:\n          break\n    return self\n",[50,189,190,195,200,205,211,216,221,227,233,239,245,251,257,263,269,275,281,287,293,299],{"__ignoreMap":48},[53,191,192],{"class":55,"line":56},[53,193,194],{},"class PLA(BaseEstimator, ClassifierMixin):\n",[53,196,197],{"class":55,"line":62},[53,198,199],{},"  def __init__(self, max_iter=10):\n",[53,201,202],{"class":55,"line":68},[53,203,204],{},"    self.max_iter = max_iter\n",[53,206,207],{"class":55,"line":74},[53,208,210],{"emptyLinePlaceholder":209},true,"\n",[53,212,213],{"class":55,"line":80},[53,214,215],{},"  def fit(self, X, y):\n",[53,217,218],{"class":55,"line":86},[53,219,220],{},"    self.w_ = np.random.rand(X.shape[1])\n",[53,222,224],{"class":55,"line":223},7,[53,225,226],{},"    for _ in range(self.max_iter):\n",[53,228,230],{"class":55,"line":229},8,[53,231,232],{},"      cost = 0\n",[53,234,236],{"class":55,"line":235},9,[53,237,238],{},"      idx = np.arange(X.shape[0])\n",[53,240,242],{"class":55,"line":241},10,[53,243,244],{},"      np.random.shuffle(idx)\n",[53,246,248],{"class":55,"line":247},11,[53,249,250],{},"      for i in idx:\n",[53,252,254],{"class":55,"line":253},12,[53,255,256],{},"        logits = X[i] @ self.w_\n",[53,258,260],{"class":55,"line":259},13,[53,261,262],{},"        y_pred = np.sign(logits)\n",[53,264,266],{"class":55,"line":265},14,[53,267,268],{},"        error = y[i] - y_pred\n",[53,270,272],{"class":55,"line":271},15,[53,273,274],{},"        if error != 0:\n",[53,276,278],{"class":55,"line":277},16,[53,279,280],{},"          cost += error**2\n",[53,282,284],{"class":55,"line":283},17,[53,285,286],{},"          self.w_ += error*X[i]\n",[53,288,290],{"class":55,"line":289},18,[53,291,292],{},"        if cost == 0:\n",[53,294,296],{"class":55,"line":295},19,[53,297,298],{},"          break\n",[53,300,302],{"class":55,"line":301},20,[53,303,304],{},"    return self\n",[11,306,307,310,311,314,315,318],{},[50,308,309],{},"w_"," starts random. Then, point by point, in shuffled order: compute the prediction (",[50,312,313],{},"sign(w · x)","), compare against the true label, and if it's wrong, update ",[50,316,317],{},"w_ += error * x",". That's the entire learning rule, and it's worth understanding why it works, not just memorizing the formula.",[11,320,321,322,324,325,328,329,331,332,335,336,338,339,342,343,346,347,349,350,353,354,357,358,361],{},"Think about it geometrically: ",[50,323,157],{}," is the vector normal to the decision hyperplane, it points toward the side the model considers \"class +1\". If the model missed a point that was ",[34,326,327],{},"actually"," class +1 but got classified as -1, that means ",[50,330,157],{}," is pointing \"too far away\" from that point. Adding ",[50,333,334],{},"error * x"," to ",[50,337,157],{}," (here ",[50,340,341],{},"error = +2",", since ",[50,344,345],{},"y_pred"," and ",[50,348,109],{}," live in ",[50,351,352],{},"{-1,+1}",") pushes the weight vector ",[34,355,356],{},"toward that point's direction",", making ",[50,359,360],{},"w · x"," a bit more positive next time. The opposite happens when the error goes the other way. It's a local, cheap adjustment: every mistake nudges the decision boundary a little in the direction that would have gotten that specific point right, with no actual gradient computed at all (this is exactly what Aggarwal calls the \"perceptron criterion\" in chapter 1: a heuristic update rule that looks a lot like gradient descent, but was designed directly on top of the classification error, before anyone formalized which smooth loss function it was implicitly optimizing).",[11,363,364,365,368],{},"And Rosenblatt proved, back in 1958, something strong: if the data really is linearly separable (as it is here, by construction), PLA ",[34,366,367],{},"always converges"," to a zero-error solution, in a finite number of steps. Not \"usually works\", a mathematical guarantee.",[15,370,372],{"id":371},"a-real-bug-hiding-in-the-stopping-condition","A real bug, hiding in the stopping condition",[11,374,375,376,379,380,383,384,387,388,391],{},"But looking closely at ",[50,377,378],{},"if cost == 0: break"," reveals a problem. That check sits ",[34,381,382],{},"inside"," the loop that walks through the points, not after it. That means: as soon as ",[50,385,386],{},"cost"," (which only grows on error) hits zero, the loop ",[34,389,390],{},"stops immediately",", even if there are still points left to check that epoch.",[11,393,394,395,397,398,401,402,404,405,408],{},"The catch is that at the very start of every epoch, ",[50,396,386],{}," already starts at zero. So if the ",[34,399,400],{},"first sampled point"," in that epoch happens to already be classified correctly, ",[50,403,386],{}," stays zero, and the ",[50,406,407],{},"break"," fires right there, without checking the other 19 points. The epoch ends thinking \"everything's fine\", when really only one point got checked.",[11,410,411,412,415],{},"I ran this exact code, byte for byte, with a fixed seed (",[50,413,414],{},"np.random.seed(10)","), to see the actual damage:",[417,418,419],"blockquote",{},[11,420,421,424,425,427,428,431,432,435],{},[34,422,423],{},"Output:"," counting how many points (out of 20) each epoch actually processed before the ",[50,426,407],{},": ",[50,429,430],{},"[1, 1, 1, 1, 1, 1, 1, 1, 20, 20]",". Only the last two epochs checked the whole dataset. Final accuracy: ",[34,433,434],{},"0.6",", far from the perfect separation Rosenblatt guarantees.",[11,437,438,439,441,442,445,446,448],{},"Moving just the ",[50,440,378],{}," outside the inner loop (checking zero errors ",[34,443,444],{},"after"," going through all 20 points, not after each one), same seed, same data, same initial ",[50,447,157],{},":",[417,450,451],{},[11,452,453,455,456,459,460,463],{},[34,454,423],{}," ",[50,457,458],{},"[20, 20]",", two full epochs, and done: converged with accuracy ",[34,461,462],{},"1.0",", exactly what the theorem promises for linearly separable data.",[11,465,466,467,469,470,473],{},"A real, non-hypothetical finding: the obvious intent of the code is \"stop once there's no more error this epoch\", but the way the ",[50,468,407],{}," got positioned makes it stop as soon as a ",[34,471,472],{},"single favorable point"," shows up, even if more error is lurking further down the queue. With luck (as with most seeds I tested), that doesn't change the final outcome because other epochs make up for it. But with this specific seed, the algorithm declares success prematurely, eight times in a row, and never reaches the perfect solution it should.",[15,475,477],{"id":476},"interactive-training-the-perceptron-point-by-point","Interactive: training the perceptron point by point",[11,479,480,481,484,485,488],{},"I rebuilt the same dataset (that seed ",[50,482,483],{},"10"," above, the same 20 points) in a component that runs the ",[34,486,487],{},"correct"," version of the algorithm, one point at a time. Click \"Process next point\" and notice: every time a colored point falls on the wrong side of the background region, that's a mistake, and the next click pushes the boundary toward it.",[490,491],"perceptron-explorer",{":classes":492,":points":493,":x-max":494,":x-min":495,":y-max":494,":y-min":495,"converged-label":496,"negative-label":497,"positive-label":498,"reset-label":499,"step-label":500,"x-label":501,"y-label":502},"[1, -1, 1, -1, 1, -1, -1, 1, 1, 1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1]","[[0.7713, 0.0208], [0.6336, 0.7488], [0.4985, 0.2248], [0.1981, 0.7605], [0.1691, 0.0883], [0.6854, 0.9534], [0.0039, 0.5122], [0.8126, 0.6125], [0.7218, 0.2919], [0.9178, 0.7146], [0.5425, 0.1422], [0.3733, 0.6741], [0.4418, 0.434], [0.6178, 0.5131], [0.6504, 0.601], [0.8052, 0.5216], [0.9086, 0.3192], [0.0905, 0.3007], [0.114, 0.8287], [0.0469, 0.6263]]","1","0","converged, zero mistakes in a full pass","class -1","class +1","Reset (new random draw)","Process next point","x0","x1",[11,504,505,506,508,509,512,513,516],{},"Notice that, with no bias, the boundary is always a line through the origin ",[50,507,165],{},", it can only ",[34,510,511],{},"rotate",", never ",[34,514,515],{},"slide",". It works on this dataset because the data was drawn around the origin on purpose. But what if the point cloud were shifted far away from the origin? I get to that in the Practical Application.",[15,518,520],{"id":519},"wrapping-up","Wrapping up",[522,523,524,538],"table",{},[525,526,527],"thead",{},[528,529,530,535],"tr",{},[531,532,534],"th",{"align":533},"left","What I already knew",[531,536,537],{"align":533},"What this lecture settled",[539,540,541,550,558],"tbody",{},[528,542,543,547],{},[544,545,546],"td",{"align":533},"Neural networks are about \"learning weights\"",[544,548,549],{"align":533},"McCulloch-Pitts comes before that: someone first had to propose that a neuron could be modeled mathematically at all, learning came 15 years later with Rosenblatt",[528,551,552,555],{},[544,553,554],{"align":533},"Updating a weight \"toward the error\" feels intuitive",[544,556,557],{"align":533},"It has a name (the perceptron criterion) and a mathematical convergence guarantee for linearly separable data",[528,559,560,563],{},[544,561,562],{"align":533},"The professor's code is the ground truth",[544,564,565],{"align":533},"Even reference code can hide a subtle bug in a stopping condition, and it's worth testing instead of trusting it with your eyes closed",[15,567,569],{"id":568},"practical-application","Practical application",[11,571,572,573,575,576,580,581,584],{},"I tested the same no-bias PLA (bug-fixed version, without the ",[50,574,407],{}," issue) on a real dataset: Iris, the two easiest classes to separate (",[577,578,579],"em",{},"setosa"," vs. ",[577,582,583],{},"versicolor","), using petal length and petal width as the two variables.",[43,586,588],{"className":45,"code":587,"language":47,"meta":48,"style":48},"from sklearn.datasets import load_iris\niris = load_iris()\nmask = iris.target \u003C 2\nX = iris.data[mask][:, [2, 3]]  # petal length and width\ny = np.where(iris.target[mask] == 0, -1, 1)\n",[50,589,590,595,600,605,610],{"__ignoreMap":48},[53,591,592],{"class":55,"line":56},[53,593,594],{},"from sklearn.datasets import load_iris\n",[53,596,597],{"class":55,"line":62},[53,598,599],{},"iris = load_iris()\n",[53,601,602],{"class":55,"line":68},[53,603,604],{},"mask = iris.target \u003C 2\n",[53,606,607],{"class":55,"line":74},[53,608,609],{},"X = iris.data[mask][:, [2, 3]]  # petal length and width\n",[53,611,612],{"class":55,"line":80},[53,613,614],{},"y = np.where(iris.target[mask] == 0, -1, 1)\n",[11,616,617],{},"These two classes are genuinely linearly separable (it's one of the most-cited \"actually separable\" examples in introductory ML material). But petal length and width are never negative, so the entire point cloud lives far from the origin, quite unlike the synthetic dataset above.",[522,619,620,635],{},[525,621,622],{},[528,623,624,627,631],{},[531,625,626],{"align":533},"Version",[531,628,630],{"align":629},"center","Converged in (max 50 epochs)",[531,632,634],{"align":633},"right","Final accuracy",[539,636,637,648],{},[528,638,639,642,645],{},[544,640,641],{"align":533},"No-bias PLA",[544,643,644],{"align":629},"never converged",[544,646,647],{"align":633},"0.84 to 0.93 (varies by seed)",[528,649,650,653,656],{},[544,651,652],{"align":533},"With-bias PLA",[544,654,655],{"align":629},"2 epochs",[544,657,658],{"align":633},[34,659,462],{},[11,661,662],{},"Without bias, PLA never declares convergence within the 50-epoch cap, because the line that would truly separate the two classes doesn't pass through the origin, and without bias that line is simply out of the model's reach. I ran 5 different seeds and none went past 93% accuracy. Adding a single parameter (the bias, which shifts the hyperplane instead of just rotating it around the origin), the same algorithm converges in just 2 epochs with perfect accuracy. That's the exact limit next lecture tackles head-on.",[664,665,666],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":48,"searchDepth":62,"depth":62,"links":668},[669,670,671,672,673,674,675],{"id":17,"depth":62,"text":18},{"id":40,"depth":62,"text":41},{"id":183,"depth":62,"text":184},{"id":371,"depth":62,"text":372},{"id":476,"depth":62,"text":477},{"id":519,"depth":62,"text":520},{"id":568,"depth":62,"text":569},null,"2026-08-20","Lecture 1a: the professor implements the Perceptron Learning Algorithm from scratch, no bias yet. I tell the story of two papers that came before it and find a real bug hiding in the algorithm's stopping condition.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","neural-networks",{"title":6,"description":678},"published","en\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron",[687,688,689],"perceptron","mcculloch-pitts","rosenblatt","7H0vY4nKBB5YNpqcUvWRgrVi2GCunnVyBByfnG6v2Ss",1787338982707]