[{"data":1,"prerenderedAt":5017},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks":3,"playlist-en-neural-networks":4,"playlist-posts-en-neural-networks":89},null,{"id":5,"title":6,"body":7,"cover":3,"description":79,"extension":80,"meta":81,"navigation":82,"order":83,"path":84,"seo":85,"status":86,"stem":87,"__hash__":88},"playlists\u002Fen\u002Fplaylists\u002Fneural-networks\u002Findex.md","Neural Networks",{"type":8,"value":9,"toc":75},"minimark",[10,25,43,59],[11,12,13,14,18,19,24],"p",{},"Second playlist with ",[15,16,17],"strong",{},"Dr. Francisco Boldt"," (the first was ",[20,21,23],"a",{"href":22},"\u002Fen\u002Fplaylists\u002Fpattern-recognition","Pattern Recognition","), now in his Neural Networks course. Same style as always: code by hand, live, in class, no pre-baked formula slides. If you've already read the previous playlist, you know exactly what to expect.",[11,26,27,28,32,33,37,38,42],{},"The reference book changes: here I use ",[29,30,31],"em",{},"Neural Networks and Deep Learning: A Textbook",", by Charu Aggarwal (2018), to play the role Bishop played in the previous playlist, filling in the foundation the notebook only shows in code. A lot of the material also connects straight back to what I've already covered: ",[20,34,36],{"href":35},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","the delta rule and linear regression already showed up in Pattern Recognition",", and ",[20,39,41],{"href":40},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions","loss functions already showed up in Andrew Ng's specialization",", so whenever it fits I'll pull those threads instead of reteaching from scratch.",[11,44,45,46,52,53,58],{},"The notebooks come from the course repository, ",[20,47,51],{"href":48,"rel":49},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasann",[50],"nofollow","pablobelmiro\u002Faulasann",", a fork of Dr. Boldt's own repository, ",[20,54,57],{"href":55,"rel":56},"https:\u002F\u002Fgithub.com\u002Ffboldt\u002Faulasann",[50],"fboldt\u002Faulasann",", where he publishes each lecture's code. Same convention as the previous playlist: whoever \"wrote\" the code is always him, the professor; the foundational explanation is mine.",[11,60,61,62,65,66,69,70,74],{},"One important detail this time: this course is being taught ",[15,63,64],{},"right now",", live, and the repository only has the beginning of the course as of this moment (perceptron, Adaline, cost functions, and a cliffhanger right at the edge of what a single neuron can solve). Unlike the Pattern Recognition playlist, which I only started once the whole course had already ended, this one is a ",[15,67,68],{},"living playlist",": it grows every time the professor publishes a new lecture, and I come back to keep going. We start at the very beginning: ",[20,71,73],{"href":72},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","the simplest neuron there is",".",{"title":76,"searchDepth":77,"depth":77,"links":78},"",2,[],"My notes from my Neural Networks course, lecture by lecture, with Aggarwal's book as the theoretical backbone. A living playlist, growing along with the course.","md",{},true,3,"\u002Fen\u002Fplaylists\u002Fneural-networks",{"title":6,"description":79},"published","en\u002Fplaylists\u002Fneural-networks\u002Findex","fD14SmK9Haa_ztrp627bgiWBVDETZgSpdrWnwlC_LnU",[90,763,1475,2147,2774,4126,4645],{"id":91,"title":92,"body":93,"cover":3,"date":752,"description":753,"extension":80,"meta":754,"navigation":82,"order":136,"path":72,"playlist":755,"seo":756,"status":86,"stem":757,"tags":758,"__hash__":762},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron.md","Perceptron: the First Neuron That Learns on Its Own",{"type":8,"value":94,"toc":743},[95,98,103,112,119,123,168,201,226,245,259,263,382,396,439,446,450,469,486,493,513,526,541,551,555,566,580,594,598,643,647,661,691,694,736,739],[11,96,97],{},"First lecture of the living playlist, and it starts exactly where any neural networks course should: at the simplest neuron there is.",[99,100,102],"h2",{"id":101},"before-the-code-two-papers-15-years-apart","Before the code: two papers, 15 years apart",[11,104,105,106,111],{},"Neural networks weren't born as code, they were born as a biophysics question. In 1943, Warren McCulloch and Walter Pitts published ",[20,107,110],{"href":108,"rel":109},"https:\u002F\u002Fwww.cs.cmu.edu\u002F~epxing\u002FClass\u002F10715\u002Freading\u002FMcCulloch.and.Pitts.pdf",[50],"\"A Logical Calculus of the Ideas Immanent in Nervous Activity\"",", proposing a lean mathematical model for a biological neuron: sum up the input signals, and fire a binary (all-or-nothing) signal if that sum crosses a threshold. No learning yet, it's just a fixed logic circuit, each \"neuron\" solves a logical function someone already decided ahead of time.",[11,113,114,115,118],{},"The missing leap came 15 years later, with Frank Rosenblatt in 1958: what if, instead of someone hand-picking the weights, the neuron itself learned the right weights by looking at examples? That's the Perceptron, and it's exactly what the professor's lecture 1a implements: the ",[15,116,117],{},"Perceptron Learning Algorithm"," (PLA), in its rawest form, no bias yet (that's next lecture).",[99,120,122],{"id":121},"the-dataset-two-groups-separable-by-a-line","The dataset: two groups separable by a line",[124,125,129],"pre",{"className":126,"code":127,"language":128,"meta":76,"style":76},"language-python shiki shiki-themes github-light github-dark","def createDataset(n=20):\n  X = np.random.rand(n,2)\n  coefs = np.array([1, -1])\n  labels = X @ coefs\n  y = np.array(labels>0, dtype=int)*2-1\n  return X, y\n","python",[130,131,132,140,145,150,156,162],"code",{"__ignoreMap":76},[133,134,137],"span",{"class":135,"line":136},"line",1,[133,138,139],{},"def createDataset(n=20):\n",[133,141,142],{"class":135,"line":77},[133,143,144],{},"  X = np.random.rand(n,2)\n",[133,146,147],{"class":135,"line":83},[133,148,149],{},"  coefs = np.array([1, -1])\n",[133,151,153],{"class":135,"line":152},4,[133,154,155],{},"  labels = X @ coefs\n",[133,157,159],{"class":135,"line":158},5,[133,160,161],{},"  y = np.array(labels>0, dtype=int)*2-1\n",[133,163,165],{"class":135,"line":164},6,[133,166,167],{},"  return X, y\n",[11,169,170,171,174,175,178,179,181,182,184,185,188,189,192,193,196,197,200],{},"The professor generates random 2D points and labels each one by the sign of ",[130,172,173],{},"X @ coefs",", the dot product between the point and the vector ",[130,176,177],{},"[1, -1]",". Geometrically, ",[130,180,173],{}," is positive on one side of the line that passes through the origin and is perpendicular to ",[130,183,177],{},", and negative on the other side, so the label ",[130,186,187],{},"y"," (",[130,190,191],{},"-1"," or ",[130,194,195],{},"+1",") is ",[15,198,199],{},"linearly separable by construction",": a straight line exists that separates the two classes perfectly, because that exact line is what generated the labels.",[124,202,204],{"className":126,"code":203,"language":128,"meta":76,"style":76},"def plotHyperplan(vector):\n  xs = np.array([0,1])\n  ys = -(vector[0]*xs)\u002Fvector[1]\n  plt.plot(xs, ys)\n",[130,205,206,211,216,221],{"__ignoreMap":76},[133,207,208],{"class":135,"line":136},[133,209,210],{},"def plotHyperplan(vector):\n",[133,212,213],{"class":135,"line":77},[133,214,215],{},"  xs = np.array([0,1])\n",[133,217,218],{"class":135,"line":83},[133,219,220],{},"  ys = -(vector[0]*xs)\u002Fvector[1]\n",[133,222,223],{"class":135,"line":152},[133,224,225],{},"  plt.plot(xs, ys)\n",[11,227,228,229,232,233,236,237,240,241,244],{},"This function draws the line where ",[130,230,231],{},"vector[0]*x + vector[1]*y = 0",", the decision boundary for any weight vector ",[130,234,235],{},"w",". Same equation as always, ",[130,238,239],{},"w · x = 0"," defines a hyperplane, except here without a bias, so the hyperplane is forced to pass through the origin ",[130,242,243],{},"(0,0)",". Keep that detail in mind, it matters in a bit.",[11,246,247,248,251,252,254,255,258],{},"The professor even tests a ",[130,249,250],{},"DummyClassifier"," with fixed weights ",[130,253,177],{},", the exact same ones that generated the dataset, and of course it hits 100%: that's the answer key fed straight back in. The real question is: can these weights actually be ",[15,256,257],{},"learned"," just by looking at examples, without me handing over the answer already solved?",[99,260,262],{"id":261},"the-algorithm-nudge-the-weight-toward-the-error","The algorithm: nudge the weight toward the error",[124,264,266],{"className":126,"code":265,"language":128,"meta":76,"style":76},"class PLA(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=10):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    for _ in range(self.max_iter):\n      cost = 0\n      idx = np.arange(X.shape[0])\n      np.random.shuffle(idx)\n      for i in idx:\n        logits = X[i] @ self.w_\n        y_pred = np.sign(logits)\n        error = y[i] - y_pred\n        if error != 0:\n          cost += error**2\n          self.w_ += error*X[i]\n        if cost == 0:\n          break\n    return self\n",[130,267,268,273,278,283,288,293,298,304,310,316,322,328,334,340,346,352,358,364,370,376],{"__ignoreMap":76},[133,269,270],{"class":135,"line":136},[133,271,272],{},"class PLA(BaseEstimator, ClassifierMixin):\n",[133,274,275],{"class":135,"line":77},[133,276,277],{},"  def __init__(self, max_iter=10):\n",[133,279,280],{"class":135,"line":83},[133,281,282],{},"    self.max_iter = max_iter\n",[133,284,285],{"class":135,"line":152},[133,286,287],{"emptyLinePlaceholder":82},"\n",[133,289,290],{"class":135,"line":158},[133,291,292],{},"  def fit(self, X, y):\n",[133,294,295],{"class":135,"line":164},[133,296,297],{},"    self.w_ = np.random.rand(X.shape[1])\n",[133,299,301],{"class":135,"line":300},7,[133,302,303],{},"    for _ in range(self.max_iter):\n",[133,305,307],{"class":135,"line":306},8,[133,308,309],{},"      cost = 0\n",[133,311,313],{"class":135,"line":312},9,[133,314,315],{},"      idx = np.arange(X.shape[0])\n",[133,317,319],{"class":135,"line":318},10,[133,320,321],{},"      np.random.shuffle(idx)\n",[133,323,325],{"class":135,"line":324},11,[133,326,327],{},"      for i in idx:\n",[133,329,331],{"class":135,"line":330},12,[133,332,333],{},"        logits = X[i] @ self.w_\n",[133,335,337],{"class":135,"line":336},13,[133,338,339],{},"        y_pred = np.sign(logits)\n",[133,341,343],{"class":135,"line":342},14,[133,344,345],{},"        error = y[i] - y_pred\n",[133,347,349],{"class":135,"line":348},15,[133,350,351],{},"        if error != 0:\n",[133,353,355],{"class":135,"line":354},16,[133,356,357],{},"          cost += error**2\n",[133,359,361],{"class":135,"line":360},17,[133,362,363],{},"          self.w_ += error*X[i]\n",[133,365,367],{"class":135,"line":366},18,[133,368,369],{},"        if cost == 0:\n",[133,371,373],{"class":135,"line":372},19,[133,374,375],{},"          break\n",[133,377,379],{"class":135,"line":378},20,[133,380,381],{},"    return self\n",[11,383,384,387,388,391,392,395],{},[130,385,386],{},"w_"," starts random. Then, point by point, in shuffled order: compute the prediction (",[130,389,390],{},"sign(w · x)","), compare against the true label, and if it's wrong, update ",[130,393,394],{},"w_ += error * x",". That's the entire learning rule, and it's worth understanding why it works, not just memorizing the formula.",[11,397,398,399,401,402,405,406,408,409,412,413,415,416,419,420,423,424,426,427,430,431,434,435,438],{},"Think about it geometrically: ",[130,400,235],{}," is the vector normal to the decision hyperplane, it points toward the side the model considers \"class +1\". If the model missed a point that was ",[15,403,404],{},"actually"," class +1 but got classified as -1, that means ",[130,407,235],{}," is pointing \"too far away\" from that point. Adding ",[130,410,411],{},"error * x"," to ",[130,414,235],{}," (here ",[130,417,418],{},"error = +2",", since ",[130,421,422],{},"y_pred"," and ",[130,425,187],{}," live in ",[130,428,429],{},"{-1,+1}",") pushes the weight vector ",[15,432,433],{},"toward that point's direction",", making ",[130,436,437],{},"w · x"," a bit more positive next time. The opposite happens when the error goes the other way. It's a local, cheap adjustment: every mistake nudges the decision boundary a little in the direction that would have gotten that specific point right, with no actual gradient computed at all (this is exactly what Aggarwal calls the \"perceptron criterion\" in chapter 1: a heuristic update rule that looks a lot like gradient descent, but was designed directly on top of the classification error, before anyone formalized which smooth loss function it was implicitly optimizing).",[11,440,441,442,445],{},"And Rosenblatt proved, back in 1958, something strong: if the data really is linearly separable (as it is here, by construction), PLA ",[15,443,444],{},"always converges"," to a zero-error solution, in a finite number of steps. Not \"usually works\", a mathematical guarantee.",[99,447,449],{"id":448},"a-real-bug-hiding-in-the-stopping-condition","A real bug, hiding in the stopping condition",[11,451,452,453,456,457,460,461,464,465,468],{},"But looking closely at ",[130,454,455],{},"if cost == 0: break"," reveals a problem. That check sits ",[15,458,459],{},"inside"," the loop that walks through the points, not after it. That means: as soon as ",[130,462,463],{},"cost"," (which only grows on error) hits zero, the loop ",[15,466,467],{},"stops immediately",", even if there are still points left to check that epoch.",[11,470,471,472,474,475,478,479,481,482,485],{},"The catch is that at the very start of every epoch, ",[130,473,463],{}," already starts at zero. So if the ",[15,476,477],{},"first sampled point"," in that epoch happens to already be classified correctly, ",[130,480,463],{}," stays zero, and the ",[130,483,484],{},"break"," fires right there, without checking the other 19 points. The epoch ends thinking \"everything's fine\", when really only one point got checked.",[11,487,488,489,492],{},"I ran this exact code, byte for byte, with a fixed seed (",[130,490,491],{},"np.random.seed(10)","), to see the actual damage:",[494,495,496],"blockquote",{},[11,497,498,501,502,504,505,508,509,512],{},[15,499,500],{},"Output:"," counting how many points (out of 20) each epoch actually processed before the ",[130,503,484],{},": ",[130,506,507],{},"[1, 1, 1, 1, 1, 1, 1, 1, 20, 20]",". Only the last two epochs checked the whole dataset. Final accuracy: ",[15,510,511],{},"0.6",", far from the perfect separation Rosenblatt guarantees.",[11,514,515,516,518,519,522,523,525],{},"Moving just the ",[130,517,455],{}," outside the inner loop (checking zero errors ",[15,520,521],{},"after"," going through all 20 points, not after each one), same seed, same data, same initial ",[130,524,235],{},":",[494,527,528],{},[11,529,530,532,533,536,537,540],{},[15,531,500],{}," ",[130,534,535],{},"[20, 20]",", two full epochs, and done: converged with accuracy ",[15,538,539],{},"1.0",", exactly what the theorem promises for linearly separable data.",[11,542,543,544,546,547,550],{},"A real, non-hypothetical finding: the obvious intent of the code is \"stop once there's no more error this epoch\", but the way the ",[130,545,484],{}," got positioned makes it stop as soon as a ",[15,548,549],{},"single favorable point"," shows up, even if more error is lurking further down the queue. With luck (as with most seeds I tested), that doesn't change the final outcome because other epochs make up for it. But with this specific seed, the algorithm declares success prematurely, eight times in a row, and never reaches the perfect solution it should.",[99,552,554],{"id":553},"interactive-training-the-perceptron-point-by-point","Interactive: training the perceptron point by point",[11,556,557,558,561,562,565],{},"I rebuilt the same dataset (that seed ",[130,559,560],{},"10"," above, the same 20 points) in a component that runs the ",[15,563,564],{},"correct"," version of the algorithm, one point at a time. Click \"Process next point\" and notice: every time a colored point falls on the wrong side of the background region, that's a mistake, and the next click pushes the boundary toward it.",[567,568],"perceptron-explorer",{":classes":569,":points":570,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"converged-label":573,"negative-label":574,"positive-label":575,"reset-label":576,"step-label":577,"x-label":578,"y-label":579},"[1, -1, 1, -1, 1, -1, -1, 1, 1, 1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1]","[[0.7713, 0.0208], [0.6336, 0.7488], [0.4985, 0.2248], [0.1981, 0.7605], [0.1691, 0.0883], [0.6854, 0.9534], [0.0039, 0.5122], [0.8126, 0.6125], [0.7218, 0.2919], [0.9178, 0.7146], [0.5425, 0.1422], [0.3733, 0.6741], [0.4418, 0.434], [0.6178, 0.5131], [0.6504, 0.601], [0.8052, 0.5216], [0.9086, 0.3192], [0.0905, 0.3007], [0.114, 0.8287], [0.0469, 0.6263]]","1","0","converged, zero mistakes in a full pass","class -1","class +1","Reset (new random draw)","Process next point","x0","x1",[11,581,582,583,585,586,589,590,593],{},"Notice that, with no bias, the boundary is always a line through the origin ",[130,584,243],{},", it can only ",[15,587,588],{},"rotate",", never ",[15,591,592],{},"slide",". It works on this dataset because the data was drawn around the origin on purpose. But what if the point cloud were shifted far away from the origin? I get to that in the Practical Application.",[99,595,597],{"id":596},"wrapping-up","Wrapping up",[599,600,601,615],"table",{},[602,603,604],"thead",{},[605,606,607,612],"tr",{},[608,609,611],"th",{"align":610},"left","What I already knew",[608,613,614],{"align":610},"What this lecture settled",[616,617,618,627,635],"tbody",{},[605,619,620,624],{},[621,622,623],"td",{"align":610},"Neural networks are about \"learning weights\"",[621,625,626],{"align":610},"McCulloch-Pitts comes before that: someone first had to propose that a neuron could be modeled mathematically at all, learning came 15 years later with Rosenblatt",[605,628,629,632],{},[621,630,631],{"align":610},"Updating a weight \"toward the error\" feels intuitive",[621,633,634],{"align":610},"It has a name (the perceptron criterion) and a mathematical convergence guarantee for linearly separable data",[605,636,637,640],{},[621,638,639],{"align":610},"The professor's code is the ground truth",[621,641,642],{"align":610},"Even reference code can hide a subtle bug in a stopping condition, and it's worth testing instead of trusting it with your eyes closed",[99,644,646],{"id":645},"practical-application","Practical application",[11,648,649,650,652,653,656,657,660],{},"I tested the same no-bias PLA (bug-fixed version, without the ",[130,651,484],{}," issue) on a real dataset: Iris, the two easiest classes to separate (",[29,654,655],{},"setosa"," vs. ",[29,658,659],{},"versicolor","), using petal length and petal width as the two variables.",[124,662,664],{"className":126,"code":663,"language":128,"meta":76,"style":76},"from sklearn.datasets import load_iris\niris = load_iris()\nmask = iris.target \u003C 2\nX = iris.data[mask][:, [2, 3]]  # petal length and width\ny = np.where(iris.target[mask] == 0, -1, 1)\n",[130,665,666,671,676,681,686],{"__ignoreMap":76},[133,667,668],{"class":135,"line":136},[133,669,670],{},"from sklearn.datasets import load_iris\n",[133,672,673],{"class":135,"line":77},[133,674,675],{},"iris = load_iris()\n",[133,677,678],{"class":135,"line":83},[133,679,680],{},"mask = iris.target \u003C 2\n",[133,682,683],{"class":135,"line":152},[133,684,685],{},"X = iris.data[mask][:, [2, 3]]  # petal length and width\n",[133,687,688],{"class":135,"line":158},[133,689,690],{},"y = np.where(iris.target[mask] == 0, -1, 1)\n",[11,692,693],{},"These two classes are genuinely linearly separable (it's one of the most-cited \"actually separable\" examples in introductory ML material). But petal length and width are never negative, so the entire point cloud lives far from the origin, quite unlike the synthetic dataset above.",[599,695,696,711],{},[602,697,698],{},[605,699,700,703,707],{},[608,701,702],{"align":610},"Version",[608,704,706],{"align":705},"center","Converged in (max 50 epochs)",[608,708,710],{"align":709},"right","Final accuracy",[616,712,713,724],{},[605,714,715,718,721],{},[621,716,717],{"align":610},"No-bias PLA",[621,719,720],{"align":705},"never converged",[621,722,723],{"align":709},"0.84 to 0.93 (varies by seed)",[605,725,726,729,732],{},[621,727,728],{"align":610},"With-bias PLA",[621,730,731],{"align":705},"2 epochs",[621,733,734],{"align":709},[15,735,539],{},[11,737,738],{},"Without bias, PLA never declares convergence within the 50-epoch cap, because the line that would truly separate the two classes doesn't pass through the origin, and without bias that line is simply out of the model's reach. I ran 5 different seeds and none went past 93% accuracy. Adding a single parameter (the bias, which shifts the hyperplane instead of just rotating it around the origin), the same algorithm converges in just 2 epochs with perfect accuracy. That's the exact limit next lecture tackles head-on.",[740,741,742],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":76,"searchDepth":77,"depth":77,"links":744},[745,746,747,748,749,750,751],{"id":101,"depth":77,"text":102},{"id":121,"depth":77,"text":122},{"id":261,"depth":77,"text":262},{"id":448,"depth":77,"text":449},{"id":553,"depth":77,"text":554},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"2026-08-20","Lecture 1a: the professor implements the Perceptron Learning Algorithm from scratch, no bias yet. I tell the story of two papers that came before it and find a real bug hiding in the algorithm's stopping condition.",{},"neural-networks",{"title":92,"description":753},"en\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron",[759,760,761],"perceptron","mcculloch-pitts","rosenblatt","7H0vY4nKBB5YNpqcUvWRgrVi2GCunnVyBByfnG6v2Ss",{"id":764,"title":765,"body":766,"cover":3,"date":752,"description":1466,"extension":80,"meta":1467,"navigation":82,"order":77,"path":1468,"playlist":755,"seo":1469,"status":86,"stem":1470,"tags":1471,"__hash__":1474},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias.md","Bias and Vectorization: Unlocking the Perceptron",{"type":8,"value":767,"toc":1456},[768,771,775,811,832,839,853,858,862,961,999,1006,1021,1035,1039,1047,1052,1056,1137,1184,1199,1214,1218,1221,1231,1242,1245,1316,1319,1323,1330,1334,1336,1375,1377,1385,1400,1447,1454],[11,769,770],{},"Lectures 2a and 2b, and each one solves one problem: the first unlocks where the decision boundary can sit, the second unlocks how the weight gets updated.",[99,772,774],{"id":773},"the-same-limit-from-last-lecture-confirmed-again","The same limit from last lecture, confirmed again",[124,776,778],{"className":126,"code":777,"language":128,"meta":76,"style":76},"def createDataset(n=20):\n  X = np.random.rand(n,2)\n  coefs = np.array([1, 1])\n  intercept = 1\n  labels = X @ coefs - intercept\n  y = np.array(labels>0, dtype=int)*2-1\n  return X, y\n",[130,779,780,784,788,793,798,803,807],{"__ignoreMap":76},[133,781,782],{"class":135,"line":136},[133,783,139],{},[133,785,786],{"class":135,"line":77},[133,787,144],{},[133,789,790],{"class":135,"line":83},[133,791,792],{},"  coefs = np.array([1, 1])\n",[133,794,795],{"class":135,"line":152},[133,796,797],{},"  intercept = 1\n",[133,799,800],{"class":135,"line":158},[133,801,802],{},"  labels = X @ coefs - intercept\n",[133,804,805],{"class":135,"line":164},[133,806,161],{},[133,808,809],{"class":135,"line":300},[133,810,167],{},[11,812,813,814,817,818,821,822,825,826,829,830,74],{},"Notice the subtle difference from last lecture's dataset: there's now an ",[130,815,816],{},"intercept = 1"," subtracted before applying the sign. That shifts the true separating line to ",[130,819,820],{},"x + y = 1",", which ",[15,823,824],{},"doesn't pass through the origin",". ",[20,827,828],{"href":72},"That's exactly the scenario I simulated artificially in the previous post"," using Iris: a boundary that doesn't pass through ",[130,831,243],{},[11,833,834,835,838],{},"The professor runs the exact same no-bias ",[130,836,837],{},"PLA"," from last lecture, without changing a line, straight on this new dataset:",[494,840,841],{},[11,842,843,845,846,849,850,74],{},[15,844,500],{}," accuracy ",[15,847,848],{},"0.7",", weights ",[130,851,852],{},"[2.71, 0.20]",[11,854,855,856,74],{},"Confirms, with his real code (not just my Iris simulation), the exact same limit: without bias, the boundary can only rotate around the origin, and there's no way to rotate a line through the origin until it matches ",[130,857,820],{},[99,859,861],{"id":860},"adding-bias-the-boundary-gains-the-freedom-to-slide","Adding bias: the boundary gains the freedom to slide",[124,863,865],{"className":126,"code":864,"language":128,"meta":76,"style":76},"class PLA(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=1000):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      cost = 0\n      idx = np.arange(X.shape[0])\n      np.random.shuffle(idx)\n      for i in idx:\n        logits = X[i] @ self.w_ + self.b_\n        y_pred = np.sign(logits)\n        error = y[i] - y_pred\n        if error != 0:\n          cost += error**2\n          self.w_ += error*X[i]\n          self.b_ += error\n        if cost == 0:\n          break\n    return self\n",[130,866,867,871,876,880,884,888,892,897,901,905,909,913,917,922,926,930,934,938,942,947,951,956],{"__ignoreMap":76},[133,868,869],{"class":135,"line":136},[133,870,272],{},[133,872,873],{"class":135,"line":77},[133,874,875],{},"  def __init__(self, max_iter=1000):\n",[133,877,878],{"class":135,"line":83},[133,879,282],{},[133,881,882],{"class":135,"line":152},[133,883,287],{"emptyLinePlaceholder":82},[133,885,886],{"class":135,"line":158},[133,887,292],{},[133,889,890],{"class":135,"line":164},[133,891,297],{},[133,893,894],{"class":135,"line":300},[133,895,896],{},"    self.b_ = np.random.rand()\n",[133,898,899],{"class":135,"line":306},[133,900,303],{},[133,902,903],{"class":135,"line":312},[133,904,309],{},[133,906,907],{"class":135,"line":318},[133,908,315],{},[133,910,911],{"class":135,"line":324},[133,912,321],{},[133,914,915],{"class":135,"line":330},[133,916,327],{},[133,918,919],{"class":135,"line":336},[133,920,921],{},"        logits = X[i] @ self.w_ + self.b_\n",[133,923,924],{"class":135,"line":342},[133,925,339],{},[133,927,928],{"class":135,"line":348},[133,929,345],{},[133,931,932],{"class":135,"line":354},[133,933,351],{},[133,935,936],{"class":135,"line":360},[133,937,357],{},[133,939,940],{"class":135,"line":366},[133,941,363],{},[133,943,944],{"class":135,"line":372},[133,945,946],{},"          self.b_ += error\n",[133,948,949],{"class":135,"line":378},[133,950,369],{},[133,952,954],{"class":135,"line":953},21,[133,955,375],{},[133,957,959],{"class":135,"line":958},22,[133,960,381],{},[11,962,963,964,37,967,970,971,974,975,977,978,980,981,984,985,987,988,991,992,995,996,74],{},"The change is small in code but big in what it unlocks: now ",[130,965,966],{},"logits = X[i] @ w_ + b_",[130,968,969],{},"b_"," gets updated alongside, ",[130,972,973],{},"self.b_ += error",". The most direct way to see why this is the right update: think of the bias as the weight of an extra input variable that's always worth ",[130,976,571],{},". If ",[130,979,386],{}," already gets updated by ",[130,982,983],{},"error * x[i]"," for each real variable, the \"weight\" of that phantom variable worth ",[130,986,571],{}," would get updated by ",[130,989,990],{},"error * 1",", that is, just ",[130,993,994],{},"error"," on its own. It's the same old trick (adding a column of 1s), just showing up here explicitly as a separate variable instead of hidden inside ",[130,997,998],{},"X",[11,1000,1001,1002,1005],{},"Geometrically, the bias shifts the hyperplane without rotating it: now ",[130,1003,1004],{},"w · x + b = 0"," can sit anywhere in the plane, not just crossing the origin.",[494,1007,1008],{},[11,1009,1010,845,1012,849,1014,1017,1018,74],{},[15,1011,500],{},[15,1013,539],{},[130,1015,1016],{},"[5.10, 3.28]",", bias ",[130,1019,1020],{},"-3.98",[11,1022,1023,1024,1027,1028,1031,1032,1034],{},"Converges. Notice the professor also bumped ",[130,1025,1026],{},"max_iter"," from 10 to 1000 in this version, ",[20,1029,1030],{"href":72},"the same stopping bug from last lecture"," still lives inside that ",[130,1033,455],{},", just with 1000 epochs of chances instead of 10, the odds of never getting one fully clean pass drop a lot. It's not a fix for the bug, it's just giving luck enough room to compensate.",[99,1036,1038],{"id":1037},"interactive-perceptron-with-bias-point-by-point","Interactive: perceptron with bias, point by point",[11,1040,1041,1042,1044,1045,74],{},"Same real dataset from the notebook (the exact 20 points the professor ran), now with bias turned on. Click \"Process next point\" and notice how the boundary, this time, can ",[15,1043,592],{}," away from the origin until it lines up with ",[130,1046,820],{},[567,1048],{":classes":1049,":points":1050,":update-bias":1051,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"converged-label":573,"negative-label":574,"positive-label":575,"reset-label":576,"step-label":577,"x-label":578,"y-label":579},"[1, 1, 1, 1, -1, 1, 1, -1, 1, 1, 1, -1, 1, -1, 1, -1, -1, 1, 1, 1]","[[0.78019343, 0.84822159], [0.69216878, 0.44056631], [0.60614199, 0.66202198], [0.47672573, 0.54204385], [0.7461111, 0.01893793], [0.6065733, 0.435086], [0.75295568, 0.47766191], [0.07180149, 0.08440745], [0.64647152, 0.46276951], [0.95252713, 0.83099628], [0.74947845, 0.82567423], [0.26043976, 0.72253777], [0.45985729, 0.86798639], [0.21107237, 0.04054585], [0.87159312, 0.76289216], [0.11142677, 0.30233227], [0.10352009, 0.82903887], [0.60640317, 0.65858996], [0.8663473, 0.184016], [0.91813453, 0.47377763]]","true",[99,1053,1055],{"id":1054},"vectorizing-from-a-point-by-point-loop-to-a-single-computation","Vectorizing: from a point-by-point loop to a single computation",[124,1057,1059],{"className":126,"code":1058,"language":128,"meta":76,"style":76},"class Perceptron(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=1000):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      cost = 0\n      y_pred = self.predict(X)\n      error = y - y_pred\n      self.w_ += np.dot(X.T, error)\n      self.b_ += np.sum(error)\n      cost = np.sum(error**2)\n      if cost == 0:\n        break\n    return self\n",[130,1060,1061,1066,1070,1074,1078,1082,1086,1090,1094,1098,1103,1108,1113,1118,1123,1128,1133],{"__ignoreMap":76},[133,1062,1063],{"class":135,"line":136},[133,1064,1065],{},"class Perceptron(BaseEstimator, ClassifierMixin):\n",[133,1067,1068],{"class":135,"line":77},[133,1069,875],{},[133,1071,1072],{"class":135,"line":83},[133,1073,282],{},[133,1075,1076],{"class":135,"line":152},[133,1077,287],{"emptyLinePlaceholder":82},[133,1079,1080],{"class":135,"line":158},[133,1081,292],{},[133,1083,1084],{"class":135,"line":164},[133,1085,297],{},[133,1087,1088],{"class":135,"line":300},[133,1089,896],{},[133,1091,1092],{"class":135,"line":306},[133,1093,303],{},[133,1095,1096],{"class":135,"line":312},[133,1097,309],{},[133,1099,1100],{"class":135,"line":318},[133,1101,1102],{},"      y_pred = self.predict(X)\n",[133,1104,1105],{"class":135,"line":324},[133,1106,1107],{},"      error = y - y_pred\n",[133,1109,1110],{"class":135,"line":330},[133,1111,1112],{},"      self.w_ += np.dot(X.T, error)\n",[133,1114,1115],{"class":135,"line":336},[133,1116,1117],{},"      self.b_ += np.sum(error)\n",[133,1119,1120],{"class":135,"line":342},[133,1121,1122],{},"      cost = np.sum(error**2)\n",[133,1124,1125],{"class":135,"line":348},[133,1126,1127],{},"      if cost == 0:\n",[133,1129,1130],{"class":135,"line":354},[133,1131,1132],{},"        break\n",[133,1134,1135],{"class":135,"line":360},[133,1136,381],{},[11,1138,1139,1140,1143,1144,188,1147,1150,1151,188,1154,1157,1158,1161,1162,1165,1166,423,1170,1174,1175,1177,1178,1180,1181,1183],{},"This version throws out the entire ",[130,1141,1142],{},"for i in idx"," loop. Instead of looking at one point at a time, it: predicts ",[15,1145,1146],{},"everyone at once",[130,1148,1149],{},"y_pred = self.predict(X)","), computes ",[15,1152,1153],{},"everyone's error at once",[130,1155,1156],{},"error = y - y_pred","), and does ",[15,1159,1160],{},"a single update"," summing each mistaken point's contribution (",[130,1163,1164],{},"X.T @ error",", the same matrix-vector product that already showed up ",[20,1167,1169],{"href":1168},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation","in the normal equation post",[20,1171,1173],{"href":1172},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization","in Andrew Ng's specialization","). This also kills the premature-stop bug for good: since ",[130,1176,463],{}," only gets computed ",[15,1179,521],{}," the whole dataset has already been processed that iteration, there's no way for the ",[130,1182,484],{}," to fire too early.",[11,1185,1186,1187,1190,1191,1194,1195,1198],{},"That's the same distinction between ",[15,1188,1189],{},"batch gradient descent"," (uses the whole dataset per update) and ",[15,1192,1193],{},"stochastic gradient descent"," (updates on every example) ",[20,1196,1197],{"href":1172},"I already saw in the other playlist",", just applied here to the perceptron's learning rule instead of a regression.",[494,1200,1201],{},[11,1202,1203,845,1206,849,1208,1017,1211,74],{},[15,1204,1205],{},"Output (same 20-point dataset):",[15,1207,539],{},[130,1209,1210],{},"[24.75, 16.55]",[130,1212,1213],{},"-19.04",[99,1215,1217],{"id":1216},"the-detail-that-slips-by-100-on-training-isnt-100-guaranteed-on-new-data","The detail that slips by: 100% on training isn't 100% guaranteed on new data",[11,1219,1220],{},"The professor tests this same vectorized model on a much larger test set, 1000 new points generated by the same rule:",[494,1222,1223],{},[11,1224,1225,845,1227,1230],{},[15,1226,500],{},[15,1228,1229],{},"0.897"," on 1000 test points.",[11,1232,1233,1234,1237,1238,1241],{},"Dropped from 1.0 to 0.897. That's not a sign of a bug in the code, it's a property of the perceptron itself, which Aggarwal points out in chapter 1: the perceptron's update rule only guarantees finding ",[15,1235,1236],{},"some"," line that separates the training points, not necessarily the ",[15,1239,1240],{},"best"," one (the one with the widest margin to each class's points). With only 20 training points, several different lines can separate all of them perfectly, but each of those lines gets a different slice of new points near the true boundary wrong. Aggarwal calls this the \"perceptron criterion\", in contrast with the SVM (Support Vector Machine), which solves exactly this problem by maximizing the margin on purpose.",[11,1243,1244],{},"I ran this same comparison (small training set, 1000-point test set) 5 times, both in the point-by-point version and the vectorized one, to see if this is specific to one version or shows up in both:",[599,1246,1247,1260],{},[602,1248,1249],{},[605,1250,1251,1254,1257],{},[608,1252,1253],{"align":705},"Run",[608,1255,1256],{"align":709},"Point-by-point (train \u002F test)",[608,1258,1259],{"align":709},"Vectorized (train \u002F test)",[616,1261,1262,1272,1283,1294,1305],{},[605,1263,1264,1266,1269],{},[621,1265,571],{"align":705},[621,1267,1268],{"align":709},"1.0 \u002F 0.894",[621,1270,1271],{"align":709},"1.0 \u002F 0.937",[605,1273,1274,1277,1280],{},[621,1275,1276],{"align":705},"2",[621,1278,1279],{"align":709},"1.0 \u002F 0.963",[621,1281,1282],{"align":709},"1.0 \u002F 0.881",[605,1284,1285,1288,1291],{},[621,1286,1287],{"align":705},"3",[621,1289,1290],{"align":709},"1.0 \u002F 0.967",[621,1292,1293],{"align":709},"1.0 \u002F 0.960",[605,1295,1296,1299,1302],{},[621,1297,1298],{"align":705},"4",[621,1300,1301],{"align":709},"1.0 \u002F 0.986",[621,1303,1304],{"align":709},"1.0 \u002F 0.978",[605,1306,1307,1310,1313],{},[621,1308,1309],{"align":705},"5",[621,1311,1312],{"align":709},"1.0 \u002F 0.948",[621,1314,1315],{"align":709},"1.0 \u002F 0.980",[11,1317,1318],{},"Both always hit 100% on training, and both swing quite a bit on test (from 0.88 to 0.99), with no clear pattern of which update style is more reliable. This confirms the instability doesn't come from \"updating point by point\" versus \"updating everyone at once\", it comes from the perceptron criterion itself: any separator will do, not necessarily the safest one.",[99,1320,1322],{"id":1321},"interactive-the-vectorized-version-one-step-one-full-epoch","Interactive: the vectorized version, one step = one full epoch",[11,1324,1325,1326,1329],{},"Same dataset, same component, just switching modes: now every click processes the ",[15,1327,1328],{},"entire"," dataset at once, instead of point by point.",[567,1331],{":classes":1049,":points":1050,":update-bias":1051,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"converged-label":573,"negative-label":574,"positive-label":575,"reset-label":576,"step-label":1332,"x-label":578,"y-label":579,"mode":1333},"Process next iteration","batch",[99,1335,597],{"id":596},[599,1337,1338,1346],{},[602,1339,1340],{},[605,1341,1342,1344],{},[608,1343,611],{"align":610},[608,1345,614],{"align":610},[616,1347,1348,1356,1367],{},[605,1349,1350,1353],{},[621,1351,1352],{"align":610},"A no-bias boundary always passes through the origin",[621,1354,1355],{"align":610},"Adding bias is the same old trick (phantom variable worth 1), and it unlocks the boundary to slide anywhere",[605,1357,1358,1361],{},[621,1359,1360],{"align":610},"Batch vs. stochastic gradient descent is a regression thing",[621,1362,1363,1364,1366],{"align":610},"The same distinction exists for the perceptron: the point-by-point loop is the \"stochastic\" version, ",[130,1365,1164],{}," all at once is the \"batch\" version",[605,1368,1369,1372],{},[621,1370,1371],{"align":610},"1.0 training accuracy sounds like \"I'm done\"",[621,1373,1374],{"align":610},"It doesn't guarantee generalization: the perceptron only finds some separating line, not the safest one, and that's a limit of the criterion, not an implementation bug",[99,1376,646],{"id":645},[11,1378,1379,1380,656,1382,1384],{},"I repeated the point-by-point vs. vectorized comparison on Iris (",[29,1381,655],{},[29,1383,659],{},", petal length and width), now with bias and a real 70\u002F30 train\u002Ftest split.",[124,1386,1388],{"className":126,"code":1387,"language":128,"meta":76,"style":76},"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[130,1389,1390,1395],{"__ignoreMap":76},[133,1391,1392],{"class":135,"line":136},[133,1393,1394],{},"from sklearn.model_selection import train_test_split\n",[133,1396,1397],{"class":135,"line":77},[133,1398,1399],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[599,1401,1402,1417],{},[602,1403,1404],{},[605,1405,1406,1408,1411,1414],{},[608,1407,702],{"align":610},[608,1409,1410],{"align":705},"Epochs to converge",[608,1412,1413],{"align":709},"Train accuracy",[608,1415,1416],{"align":709},"Test accuracy",[616,1418,1419,1433],{},[605,1420,1421,1424,1427,1429],{},[621,1422,1423],{"align":610},"Point-by-point",[621,1425,1426],{"align":705},"2 to 3",[621,1428,539],{"align":709},[621,1430,1431],{"align":709},[15,1432,539],{},[605,1434,1435,1438,1441,1443],{},[621,1436,1437],{"align":610},"Vectorized",[621,1439,1440],{"align":705},"7",[621,1442,539],{"align":709},[621,1444,1445],{"align":709},[15,1446,539],{},[11,1448,1449,1450,1453],{},"This time both hit 100% on test too, across the 3 seeds I tried. Makes sense: unlike the synthetic dataset above (just 20 loosely scattered points), Iris has a pretty generous margin between the two classes, so any reasonable separating line already gets the new points right. What was left was speed: the point-by-point version converged in 2 to 3 epochs, the vectorized one needed 7. That tracks with what I explained above: within a single epoch, the point-by-point version can make up to 20 weight updates (one per mistake found), while the vectorized version makes only ",[15,1451,1452],{},"one"," update per epoch, so it naturally needs more passes over the dataset to accumulate the same amount of adjustment.",[740,1455,742],{},{"title":76,"searchDepth":77,"depth":77,"links":1457},[1458,1459,1460,1461,1462,1463,1464,1465],{"id":773,"depth":77,"text":774},{"id":860,"depth":77,"text":861},{"id":1037,"depth":77,"text":1038},{"id":1054,"depth":77,"text":1055},{"id":1216,"depth":77,"text":1217},{"id":1321,"depth":77,"text":1322},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lectures 2a and 2b: the professor adds bias to the perceptron (solving exactly the limit that closed last lecture) and then vectorizes the whole weight update, swapping the point-by-point loop for a single computation over the entire dataset.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias",{"title":765,"description":1466},"en\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias",[759,1472,1473],"bias","vectorization","JDWpvRvHFyo_ILqbpAEKUsk7Lj8BjfF-lkNkHIBzMLo",{"id":1476,"title":1477,"body":1478,"cover":3,"date":752,"description":2137,"extension":80,"meta":2138,"navigation":82,"order":83,"path":2139,"playlist":755,"seo":2140,"status":86,"stem":2141,"tags":2142,"__hash__":2146},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal.md","From Classification to Regression, and the One-Line Solution",{"type":8,"value":1479,"toc":2129},[1480,1487,1493,1533,1548,1610,1751,1774,1792,1796,1855,1880,1915,1931,1950,1954,1967,1978,1980,2031,2033,2049,2058,2073,2090,2121,2127],[11,1481,1482,1483,1486],{},"Lectures 2c and 2d. The code structure barely changes, but the meaning shifts a lot: ",[130,1484,1485],{},"sign()"," goes out, a continuous prediction comes in, and at the end a closed-form computation replaces hundreds of iterations.",[99,1488,1490,1491],{"id":1489},"swapping-classification-for-regression-just-drop-the-sign","Swapping classification for regression: just drop the ",[130,1492,1485],{},[124,1494,1496],{"className":126,"code":1495,"language":128,"meta":76,"style":76},"def createRegressionDataset(n=20):\n  X = np.random.rand(n,1)\n  coef = 0.7\n  intercept = 0.2\n  noise = np.random.randn(n,1) * 0.1\n  y = X * coef + intercept + noise\n  return X, y.reshape(-1)\n",[130,1497,1498,1503,1508,1513,1518,1523,1528],{"__ignoreMap":76},[133,1499,1500],{"class":135,"line":136},[133,1501,1502],{},"def createRegressionDataset(n=20):\n",[133,1504,1505],{"class":135,"line":77},[133,1506,1507],{},"  X = np.random.rand(n,1)\n",[133,1509,1510],{"class":135,"line":83},[133,1511,1512],{},"  coef = 0.7\n",[133,1514,1515],{"class":135,"line":152},[133,1516,1517],{},"  intercept = 0.2\n",[133,1519,1520],{"class":135,"line":158},[133,1521,1522],{},"  noise = np.random.randn(n,1) * 0.1\n",[133,1524,1525],{"class":135,"line":164},[133,1526,1527],{},"  y = X * coef + intercept + noise\n",[133,1529,1530],{"class":135,"line":300},[133,1531,1532],{},"  return X, y.reshape(-1)\n",[11,1534,1535,1536,1538,1539,192,1541,1543,1544,1547],{},"Unlike the earlier datasets, ",[130,1537,187],{}," here isn't ",[130,1540,191],{},[130,1542,195],{}," anymore, it's a continuous number, ",[130,1545,1546],{},"0.7 * x + 0.2"," plus a small Gaussian noise. There's no class to get right, there's a line to find.",[124,1549,1551],{"className":126,"code":1550,"language":128,"meta":76,"style":76},"class LinearRegression(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      y_pred = self.predict(X)\n      error = y - y_pred\n      self.w_ += np.dot(X.T, error) * self.learning_rate\n      self.b_ += np.sum(error) * self.learning_rate\n    return self\n\n  def predict(self, X):\n    return X @ self.w_ + self.b_\n",[130,1552,1553,1558,1562,1566,1570,1574,1578,1582,1587,1592,1596,1600,1605],{"__ignoreMap":76},[133,1554,1555],{"class":135,"line":136},[133,1556,1557],{},"class LinearRegression(BaseEstimator, ClassifierMixin):\n",[133,1559,1560],{"class":135,"line":77},[133,1561,292],{},[133,1563,1564],{"class":135,"line":83},[133,1565,297],{},[133,1567,1568],{"class":135,"line":152},[133,1569,896],{},[133,1571,1572],{"class":135,"line":158},[133,1573,303],{},[133,1575,1576],{"class":135,"line":164},[133,1577,1102],{},[133,1579,1580],{"class":135,"line":300},[133,1581,1107],{},[133,1583,1584],{"class":135,"line":306},[133,1585,1586],{},"      self.w_ += np.dot(X.T, error) * self.learning_rate\n",[133,1588,1589],{"class":135,"line":312},[133,1590,1591],{},"      self.b_ += np.sum(error) * self.learning_rate\n",[133,1593,1594],{"class":135,"line":318},[133,1595,381],{},[133,1597,1598],{"class":135,"line":324},[133,1599,287],{"emptyLinePlaceholder":82},[133,1601,1602],{"class":135,"line":330},[133,1603,1604],{},"  def predict(self, X):\n",[133,1606,1607],{"class":135,"line":336},[133,1608,1609],{},"    return X @ self.w_ + self.b_\n",[11,1611,1612,1613,1616,1617,1620,1621,1623,1624,1627,1628,1631,1632,1634,1635,1638,1639,1735,1736,1739,1740,1742,1743,1746,1747,74],{},"Notice how this is ",[15,1614,1615],{},"almost identical"," to the vectorized perceptron ",[20,1618,1619],{"href":1468},"from last post",": same ",[130,1622,1164],{}," to update the weight, same ",[130,1625,1626],{},"sum(error)"," to update the bias. The two differences are small in code but change the whole meaning: first, ",[130,1629,1630],{},"predict"," no longer goes through ",[130,1633,1485],{},", the prediction stays continuous (linear activation, ",[20,1636,1637],{"href":72},"the same idea Aggarwal describes"," as the simplest activation function there is, ",[133,1640,1643,1683],{"className":1641},[1642],"katex",[133,1644,1647],{"className":1645},[1646],"katex-mathml",[1648,1649,1651],"math",{"xmlns":1650},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[1652,1653,1654,1678],"semantics",{},[1655,1656,1657,1662,1667,1670,1673,1676],"mrow",{},[1658,1659,1661],"mi",{"mathvariant":1660},"normal","Φ",[1663,1664,1666],"mo",{"stretchy":1665},"false","(",[1658,1668,1669],{},"v",[1663,1671,1672],{"stretchy":1665},")",[1663,1674,1675],{},"=",[1658,1677,1669],{},[1679,1680,1682],"annotation",{"encoding":1681},"application\u002Fx-tex","\\Phi(v) = v",[133,1684,1687,1725],{"className":1685,"ariaHidden":1051},[1686],"katex-html",[133,1688,1691,1696,1700,1704,1709,1713,1718,1722],{"className":1689},[1690],"base",[133,1692],{"className":1693,"style":1695},[1694],"strut","height:1em;vertical-align:-0.25em;",[133,1697,1661],{"className":1698},[1699],"mord",[133,1701,1666],{"className":1702},[1703],"mopen",[133,1705,1669],{"className":1706,"style":1708},[1699,1707],"mathnormal","margin-right:0.0359em;",[133,1710,1672],{"className":1711},[1712],"mclose",[133,1714],{"className":1715,"style":1717},[1716],"mspace","margin-right:0.2778em;",[133,1719,1675],{"className":1720},[1721],"mrel",[133,1723],{"className":1724,"style":1717},[1716],[133,1726,1728,1732],{"className":1727},[1690],[133,1729],{"className":1730,"style":1731},[1694],"height:0.4306em;",[133,1733,1669],{"className":1734,"style":1708},[1699,1707],"). Second, a ",[130,1737,1738],{},"learning_rate"," shows up multiplying the update. In the perceptron this learning rate didn't even exist, it was implicitly ",[130,1741,571],{},". Aggarwal calls this an interesting quirk of the perceptron, you can fix the rate at 1 because it only rescales the weight, not the direction of the adjustment. Here, with a continuous error instead of an error in ",[130,1744,1745],{},"{-2,0,+2}",", the adjustment's magnitude can end up too big or too small depending on the error's scale, and that's why an explicit learning rate becomes necessary to control the step size, ",[20,1748,1750],{"href":1749},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling","the exact subject that already earned a whole post in the other playlist",[494,1752,1753],{},[11,1754,1755,1757,1758,849,1761,1017,1764,1767,1768,423,1770,1773],{},[15,1756,500],{}," RMSE ",[130,1759,1760],{},"0.0948",[130,1762,1763],{},"[0.706]",[130,1765,1766],{},"0.204",". Pretty close to the true generator (",[130,1769,848],{},[130,1771,1772],{},"0.2","), the difference is just the noise baked in on purpose.",[11,1775,1776,1777,1780,1781,1784,1785,37,1788,1791],{},"One honest note: the notebook's next cell calls ",[130,1778,1779],{},"createDataset"," (not ",[130,1782,1783],{},"createRegressionDataset","), ",[130,1786,1787],{},"accuracy_score",[130,1789,1790],{},"plotHyperplan",", names that don't exist in this notebook, only in the previous lecture's. That only ran because the professor's Colab still had the previous session in memory (a variable reused lecture to lecture). Running this notebook fresh would break that cell. I don't reproduce it here, since it doesn't actually test the regression model that was just trained.",[99,1793,1795],{"id":1794},"the-normal-equation-the-same-question-solved-without-iterating","The normal equation: the same question, solved without iterating",[124,1797,1799],{"className":126,"code":1798,"language":128,"meta":76,"style":76},"def include_bias(X):\n  return np.hstack((np.ones((X.shape[0],1)), X))\n\nclass NormalEquation(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n",[130,1800,1801,1806,1811,1815,1820,1824,1829,1834,1838,1842,1846,1850],{"__ignoreMap":76},[133,1802,1803],{"class":135,"line":136},[133,1804,1805],{},"def include_bias(X):\n",[133,1807,1808],{"class":135,"line":77},[133,1809,1810],{},"  return np.hstack((np.ones((X.shape[0],1)), X))\n",[133,1812,1813],{"class":135,"line":83},[133,1814,287],{"emptyLinePlaceholder":82},[133,1816,1817],{"class":135,"line":152},[133,1818,1819],{},"class NormalEquation(BaseEstimator, ClassifierMixin):\n",[133,1821,1822],{"class":135,"line":158},[133,1823,292],{},[133,1825,1826],{"class":135,"line":164},[133,1827,1828],{},"    X = include_bias(X)\n",[133,1830,1831],{"class":135,"line":300},[133,1832,1833],{},"    self.w_ = np.linalg.pinv(X) @ y\n",[133,1835,1836],{"class":135,"line":306},[133,1837,381],{},[133,1839,1840],{"class":135,"line":312},[133,1841,287],{"emptyLinePlaceholder":82},[133,1843,1844],{"class":135,"line":318},[133,1845,1604],{},[133,1847,1848],{"class":135,"line":324},[133,1849,1828],{},[133,1851,1852],{"class":135,"line":330},[133,1853,1854],{},"    return X @ self.w_\n",[11,1856,1857,1858,1861,1862,1864,1865,1867,1868,1871,1872,1874,1875,1877,1878,74],{},"Two new things here. The first is ",[130,1859,1860],{},"include_bias",": sticks a column of ",[130,1863,571],{},"s in front of ",[130,1866,998],{},". This is literally the trick I described in words ",[20,1869,1870],{"href":1468},"in the last post",", bias as the weight of a phantom variable always worth ",[130,1873,571],{},", now written as real code instead of a separate ",[130,1876,969],{},". With that extra column, the bias just becomes another regular weight inside ",[130,1879,386],{},[11,1881,1882,1883,1886,1887,1889,1890,1893,1894,1897,1898,188,1901,1904,1905,1907,1908,1911,1912,1914],{},"The second is ",[130,1884,1885],{},"NormalEquation"," itself: no loop, no ",[130,1888,1738],{},", just ",[130,1891,1892],{},"np.linalg.pinv(X) @ y",". That's the closed-form solution ",[20,1895,1896],{"href":1168},"I already explored in detail back in Pattern Recognition",", the same question (\"which line minimizes squared error\") solved directly, in a single computation, instead of stepping down gradually. The technical difference here is the method: there, I implemented it via Gauss-Jordan elimination with pivoting. The professor uses the ",[15,1899,1900],{},"pseudo-inverse",[130,1902,1903],{},"pinv","), which solves via SVD decomposition under the hood. The advantage of the pseudo-inverse is that it never gets stuck: even if ",[130,1906,998],{}," has redundant (linearly dependent) columns and ",[130,1909,1910],{},"X^T X"," can't be inverted the traditional way, ",[130,1913,1903],{}," still returns a valid answer (the smallest-norm one among the infinitely many possible solutions). Plain Gauss-Jordan elimination, in that same case, simply breaks.",[494,1916,1917],{},[11,1918,1919,1757,1921,849,1924,1927,1928,1930],{},[15,1920,500],{},[130,1922,1923],{},"0.12364226682065012",[130,1925,1926],{},"[0.25402951 0.61056989]"," (bias and coefficient, in that order, because of ",[130,1929,1860],{},").",[11,1932,1933,1934,1937,1938,1941,1942,1945,1946,1949],{},"And the gradient descent from the previous cell, on the same dataset, landed at RMSE ",[130,1935,1936],{},"0.12364226680246916",", essentially identical weights. Two completely different computations (one iterative, one closed-form) converging to the exact same place, down to the seventh decimal. I reproduced this myself with a seeded dataset (",[130,1939,1940],{},"np.random.seed(7)",") to confirm it wasn't a one-run coincidence: GD gave weights ",[130,1943,1944],{},"[0.6478, bias 0.2041]",", normal equation gave ",[130,1947,1948],{},"[bias 0.2041, coef 0.6478]",", identical RMSE down to the ninth decimal in both.",[99,1951,1953],{"id":1952},"interactive-find-the-line-yourself","Interactive: find the line yourself",[11,1955,1956,1957,1960,1961,423,1963,1966],{},"Before watching the machine solve it, try solving it by hand. This is the same seeded dataset from above (20 real points generated by ",[130,1958,1959],{},"coef=0.7, intercept=0.2"," plus noise), drag the ",[130,1962,235],{},[130,1964,1965],{},"b"," sliders and watch the total error change live.",[1968,1969],"model-playground",{":b-max":571,":b-min":1970,":b-step":1971,":initial-b":572,":initial-w":572,":w-max":1972,":w-min":572,":w-step":1971,":x-train":1973,":y-train":1974,"dataLabel":1975,"prediction-label":1976,"x-label":1977,"y-label":187},"-0.5","0.05","1.5","[0.0763, 0.7799, 0.4384, 0.7235, 0.978, 0.5385, 0.5011, 0.0721, 0.2684, 0.4999, 0.6792, 0.8037, 0.3809, 0.0659, 0.2881, 0.9096, 0.2134, 0.4521, 0.9312, 0.0249]","[0.3089, 0.7583, 0.5343, 0.5538, 1.0497, 0.5924, 0.5121, 0.4533, 0.3834, 0.4048, 0.6349, 0.5338, 0.5716, 0.2045, 0.3274, 0.944, 0.1843, 0.57, 0.6454, 0.1512]","training points","fitted line","x",[99,1979,597],{"id":596},[599,1981,1982,1990],{},[602,1983,1984],{},[605,1985,1986,1988],{},[608,1987,611],{"align":610},[608,1989,614],{"align":610},[616,1991,1992,2005,2018],{},[605,1993,1994,1999],{},[621,1995,1996,1997],{"align":610},"The perceptron classifies with ",[130,1998,1485],{},[621,2000,2001,2002,2004],{"align":610},"Dropping ",[130,2003,1485],{}," from the exact same code structure turns classification into regression, almost without touching anything else",[605,2006,2007,2010],{},[621,2008,2009],{"align":610},"Bias is the weight of a phantom variable worth 1",[621,2011,2012,2013,2015,2016],{"align":610},"That becomes explicit code with ",[130,2014,1860],{},", instead of a separate ",[130,2017,969],{},[605,2019,2020,2023],{},[621,2021,2022],{"align":610},"I already solved the normal equation via Gauss-Jordan",[621,2024,2025,2027,2028,2030],{"align":610},[130,2026,1903],{}," (pseudo-inverse via SVD) solves the same computation and still works when ",[130,2029,1910],{}," isn't invertible",[99,2032,646],{"id":645},[11,2034,2035,2036,2040,2041,2044,2045,2048],{},"I tested gradient descent against the normal equation on the real 50-house dataset ",[20,2037,2039],{"href":2038},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","that already showed up in the other playlist",", using just ",[130,2042,2043],{},"square_feet"," to predict ",[130,2046,2047],{},"price",", with nothing normalized first.",[124,2050,2052],{"className":126,"code":2051,"language":128,"meta":76,"style":76},"w, b = fit_gd(X, y, max_iter=1000, learning_rate=0.01)  # not normalized\n",[130,2053,2054],{"__ignoreMap":76},[133,2055,2056],{"class":135,"line":136},[133,2057,2051],{},[494,2059,2060],{},[11,2061,2062,2064,2065,2068,2069,2072],{},[15,2063,500],{}," the weight becomes ",[130,2066,2067],{},"-inf"," by iteration 71. Gradient descent ",[15,2070,2071],{},"diverges"," completely.",[11,2074,2075,2076,2079,2080,2082,2083,2085,2086,2089],{},"Expected: ",[20,2077,2078],{"href":1749},"it's the same feature-scaling lesson from the other playlist",", just rediscovered here inside a neural-network context. ",[130,2081,2043],{}," lives in the hundreds and ",[130,2084,2047],{}," in the hundreds of thousands, so the gradient is huge and a ",[130,2087,2088],{},"0.01"," step blows up.",[599,2091,2092,2102],{},[602,2093,2094],{},[605,2095,2096,2099],{},[608,2097,2098],{"align":610},"Approach",[608,2100,2101],{"align":709},"RMSE",[616,2103,2104,2112],{},[605,2105,2106,2109],{},[621,2107,2108],{"align":610},"Gradient descent, normalized data",[621,2110,2111],{"align":709},"101878.42",[605,2113,2114,2117],{},[621,2115,2116],{"align":610},"Normal equation, raw data (not normalized)",[621,2118,2119],{"align":709},[15,2120,2111],{},[11,2122,2123,2124,2126],{},"Normalizing ",[130,2125,2043],{}," before running gradient descent, it converges and lands at exactly the same RMSE the normal equation finds directly on the raw data, no normalization needed. Makes sense: the normal equation solves the linear system in one shot, so the variables' scale only affects the computation's numerical stability, not whether it converges at all (unlike gradient descent, which can literally diverge if the step is too big for the data's scale).",[740,2128,742],{},{"title":76,"searchDepth":77,"depth":77,"links":2130},[2131,2133,2134,2135,2136],{"id":1489,"depth":77,"text":2132},"Swapping classification for regression: just drop the sign()",{"id":1794,"depth":77,"text":1795},{"id":1952,"depth":77,"text":1953},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lectures 2c and 2d: the professor swaps the binary sign for a continuous line (linear regression) and then replaces the whole gradient descent loop with a single closed-form computation, the normal equation via pseudo-inverse.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal",{"title":1477,"description":2137},"en\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal",[2143,2144,2145],"linear-regression","normal-equation","gradient-descent","Npkp7jgJ9TPUTQRrmQtdpLENefGbKgpSYQLH1QpMI2w",{"id":2148,"title":2149,"body":2150,"cover":3,"date":752,"description":2764,"extension":80,"meta":2765,"navigation":82,"order":152,"path":2766,"playlist":755,"seo":2767,"status":86,"stem":2768,"tags":2769,"__hash__":2773},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta.md","Adaline: Train on the Line, Classify on the Sign",{"type":8,"value":2151,"toc":2756},[2152,2165,2169,2211,2237,2249,2255,2310,2335,2347,2374,2378,2391,2397,2400,2404,2449,2465,2481,2485,2520,2540,2554,2561,2597,2609,2611,2654,2656,2664,2678,2701,2744,2754],[11,2153,2154,2155,2158,2159,423,2162,74],{},"Lectures 2e and 2f, and the model's name changes to ",[15,2156,2157],{},"Adaline"," (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: ",[15,2160,2161],{},"what the model optimizes during training",[15,2163,2164],{},"what it computes at prediction time",[99,2166,2168],{"id":2167},"pre-activation-and-post-activation-the-distinction-adaline-introduces","Pre-activation and post-activation: the distinction Adaline introduces",[124,2170,2172],{"className":126,"code":2171,"language":128,"meta":76,"style":76},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n",[130,2173,2174,2179,2183,2187,2191,2195,2199,2203,2207],{"__ignoreMap":76},[133,2175,2176],{"class":135,"line":136},[133,2177,2178],{},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n",[133,2180,2181],{"class":135,"line":77},[133,2182,292],{},[133,2184,2185],{"class":135,"line":83},[133,2186,1828],{},[133,2188,2189],{"class":135,"line":152},[133,2190,1833],{},[133,2192,2193],{"class":135,"line":158},[133,2194,381],{},[133,2196,2197],{"class":135,"line":164},[133,2198,287],{"emptyLinePlaceholder":82},[133,2200,2201],{"class":135,"line":300},[133,2202,1604],{},[133,2204,2205],{"class":135,"line":306},[133,2206,1828],{},[133,2208,2209],{"class":135,"line":312},[133,2210,1854],{},[11,2212,2213,2214,532,2217,2220,2221,2223,2224,192,2226,2228,2229,2231,2232,2234,2235,74],{},"Notice: this is ",[15,2215,2216],{},"exactly",[20,2218,2219],{"href":2139},"last post's normal equation",", without changing a line, except now ",[130,2222,187],{}," is ",[130,2225,191],{},[130,2227,195],{}," instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to ",[130,2230,191],{}," on one class's points and ",[130,2233,195],{}," on the other's, never applying ",[130,2236,1485],{},[494,2238,2239],{},[11,2240,2241,1757,2243,849,2246,74],{},[15,2242,500],{},[130,2244,2245],{},"0.4148",[130,2247,2248],{},"[-1.76, 1.01, 2.74]",[11,2250,2251,2252,2254],{},"RMSE makes sense here because ",[130,2253,1630],{}," returns a continuous number, not a class. But to actually classify, there's one last step missing:",[124,2256,2258],{"className":126,"code":2257,"language":128,"meta":76,"style":76},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def pre_activation(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n\n  def predict(self, X):\n    return np.sign(self.pre_activation(X))\n",[130,2259,2260,2264,2268,2272,2276,2280,2284,2289,2293,2297,2301,2305],{"__ignoreMap":76},[133,2261,2262],{"class":135,"line":136},[133,2263,2178],{},[133,2265,2266],{"class":135,"line":77},[133,2267,292],{},[133,2269,2270],{"class":135,"line":83},[133,2271,1828],{},[133,2273,2274],{"class":135,"line":152},[133,2275,1833],{},[133,2277,2278],{"class":135,"line":158},[133,2279,381],{},[133,2281,2282],{"class":135,"line":164},[133,2283,287],{"emptyLinePlaceholder":82},[133,2285,2286],{"class":135,"line":300},[133,2287,2288],{},"  def pre_activation(self, X):\n",[133,2290,2291],{"class":135,"line":306},[133,2292,1828],{},[133,2294,2295],{"class":135,"line":312},[133,2296,1854],{},[133,2298,2299],{"class":135,"line":318},[133,2300,287],{"emptyLinePlaceholder":82},[133,2302,2303],{"class":135,"line":324},[133,2304,1604],{},[133,2306,2307],{"class":135,"line":330},[133,2308,2309],{},"    return np.sign(self.pre_activation(X))\n",[11,2311,2312,2313,2316,2317,2319,2320,2322,2323,2326,2327,2330,2331,2334],{},"Now there are two methods: ",[130,2314,2315],{},"pre_activation"," (the continuous value, before any threshold) and ",[130,2318,1630],{}," (applies ",[130,2321,1485],{}," on top). Aggarwal calls exactly these two things the ",[15,2324,2325],{},"pre-activation value"," and the ",[15,2328,2329],{},"post-activation value"," (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure \"how far off\" each prediction landed from the target) and only applies the activation (",[130,2332,2333],{},"sign",") when deciding the final class.",[494,2336,2337],{},[11,2338,2339,845,2341,2343,2344,2346],{},[15,2340,500],{},[130,2342,539],{},", the same ",[130,2345,2248],{}," weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).",[11,2348,2349,2350,37,2353,2356,2357,2360,2361,2363,2364,2366,2367,2369,2370,2373],{},"This is the ",[15,2351,2352],{},"delta rule",[20,2354,2355],{"href":35},"I've already seen it before, under that exact name",": \"update the weight proportionally to the error times the input\" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered ",[20,2358,2359],{"href":72},"two posts ago",") is exactly this: the perceptron measures error ",[15,2362,521],{}," applying ",[130,2365,1485],{}," (error in ",[130,2368,1745],{},"), Adaline measures it ",[15,2371,2372],{},"before",", on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit \"unsure\". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of \"how wrong.\"",[99,2375,2377],{"id":2376},"interactive-nudging-the-pre-activation-and-watching-rmse-and-accuracy-move","Interactive: nudging the pre-activation and watching RMSE (and accuracy) move",[11,2379,2380,2381,2384,2385,37,2388,2390],{},"Instead of training automatically, drag the ",[130,2382,2383],{},"w0",", ",[130,2386,2387],{},"w1",[130,2389,1472],{}," sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).",[2392,2393],"adaline-explorer",{":classes":2394,":points":2395,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"negative-label":574,"positive-label":575,"readout-label":2396,"x-label":578,"y-label":579},"[-1, 1, -1, -1, -1, -1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, 1, -1]","[[0.0856, 0.2368], [0.8013, 0.5822], [0.0941, 0.4331], [0.4791, 0.1597], [0.7346, 0.1137], [0.3912, 0.5167], [0.4306, 0.5868], [0.7378, 0.9563], [0.2842, 0.6485], [0.6962, 0.2927], [0.0015, 0.9735], [0.2984, 0.314], [0.8917, 0.5852], [0.4713, 0.7733], [0.0303, 0.707], [0.3742, 0.0909], [0.6605, 0.9315], [0.2072, 0.6301], [0.2982, 0.7418], [0.7222, 0.2187]]","RMSE (continuous pre-activation vs. ±1 label): {rmse} · accuracy (post-activation, sign): {acc}%",[11,2398,2399],{},"Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between \"finding some line that separates\" (what the perceptron does) and \"finding the line that separates with room to spare\" (what you get by optimizing RMSE instead of just counting mistakes).",[99,2401,2403],{"id":2402},"lecture-2f-the-same-computation-just-iterating-and-the-notebook-calls-it-sgd","Lecture 2f: the same computation, just iterating (and the notebook calls it \"SGD\")",[124,2405,2407],{"className":126,"code":2406,"language":128,"meta":76,"style":76},"class Adaline(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.zeros(X.shape[1])\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      error = y - y_pred\n      self.w_ += self.learning_rate * error @ X\n    return self\n",[130,2408,2409,2414,2418,2422,2427,2431,2436,2440,2445],{"__ignoreMap":76},[133,2410,2411],{"class":135,"line":136},[133,2412,2413],{},"class Adaline(BaseEstimator, ClassifierMixin):\n",[133,2415,2416],{"class":135,"line":77},[133,2417,292],{},[133,2419,2420],{"class":135,"line":83},[133,2421,1828],{},[133,2423,2424],{"class":135,"line":152},[133,2425,2426],{},"    self.w_ = np.zeros(X.shape[1])\n",[133,2428,2429],{"class":135,"line":158},[133,2430,303],{},[133,2432,2433],{"class":135,"line":164},[133,2434,2435],{},"      y_pred = X @ self.w_\n",[133,2437,2438],{"class":135,"line":300},[133,2439,1107],{},[133,2441,2442],{"class":135,"line":306},[133,2443,2444],{},"      self.w_ += self.learning_rate * error @ X\n",[133,2446,2447],{"class":135,"line":312},[133,2448,381],{},[11,2450,2451,2452,2455,2456,188,2458,2464],{},"One honest note about the notebook's name (",[130,2453,2454],{},"aula02f adaline with SGD","): the code shown here is ",[15,2457,1189],{},[20,2459,2460,2461,2463],{"href":1468},"the same ",[130,2462,1164],{}," as always","), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking (\"it's kind of like gradient descent, so I call it SGD\"), but the technical difference is worth noting, since the names carry precise meaning.",[494,2466,2467],{},[11,2468,2469,845,2471,2473,2474,2477,2478,74],{},[15,2470,500],{},[130,2472,539],{}," on training, weights ",[130,2475,2476],{},"[-2.92, 3.32, 2.63]",". Tested on 1000 new points: accuracy ",[15,2479,2480],{},"0.953",[99,2482,2484],{"id":2483},"the-bad-dataset-what-it-actually-proves","The \"bad\" dataset: what it actually proves",[124,2486,2488],{"className":126,"code":2487,"language":128,"meta":76,"style":76},"X_bad = np.concatenate((X,np.ones_like(X)))\ny_bad = np.concatenate((y,np.ones_like(y)))\nX_bad = np.concatenate((X_bad,np.ones_like(X)))\ny_bad = np.concatenate((y_bad,np.ones_like(y)))\nclf_bad = Adaline()\nclf_bad.fit(X_bad, y_bad)\n",[130,2489,2490,2495,2500,2505,2510,2515],{"__ignoreMap":76},[133,2491,2492],{"class":135,"line":136},[133,2493,2494],{},"X_bad = np.concatenate((X,np.ones_like(X)))\n",[133,2496,2497],{"class":135,"line":77},[133,2498,2499],{},"y_bad = np.concatenate((y,np.ones_like(y)))\n",[133,2501,2502],{"class":135,"line":83},[133,2503,2504],{},"X_bad = np.concatenate((X_bad,np.ones_like(X)))\n",[133,2506,2507],{"class":135,"line":152},[133,2508,2509],{},"y_bad = np.concatenate((y_bad,np.ones_like(y)))\n",[133,2511,2512],{"class":135,"line":158},[133,2513,2514],{},"clf_bad = Adaline()\n",[133,2516,2517],{"class":135,"line":164},[133,2518,2519],{},"clf_bad.fit(X_bad, y_bad)\n",[11,2521,2522,2523,2526,2527,2530,2531,2533,2534,2536,2537,2539],{},"The professor concatenates the original dataset with ",[15,2524,2525],{},"two extra blocks"," of artificial points: everyone at ",[130,2528,2529],{},"(1,1)",", everyone labeled ",[130,2532,195],{},". This isn't noise, it's a deliberate bias, nudging training to \"believe\" the region near ",[130,2535,2529],{}," is even more strongly class ",[130,2538,195],{}," than it really is.",[494,2541,2542],{},[11,2543,2544,845,2546,2549,2550,2553],{},[15,2545,500],{},[130,2547,2548],{},"0.967"," on training (over the biased dataset), but only ",[15,2551,2552],{},"0.811"," on the same 1000 clean test points as before.",[11,2555,2556,2557,2560],{},"Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent ",[15,2558,2559],{},"on the same biased dataset",", and that hypothesis didn't hold up:",[599,2562,2563,2576],{},[602,2564,2565],{},[605,2566,2567,2570,2573],{},[608,2568,2569],{"align":610},"Method",[608,2571,2572],{"align":709},"Train accuracy (biased)",[608,2574,2575],{"align":709},"Test accuracy (clean)",[616,2577,2578,2588],{},[605,2579,2580,2583,2585],{},[621,2581,2582],{"align":610},"Pseudo-inverse",[621,2584,2548],{"align":709},[621,2586,2587],{"align":709},"0.783",[605,2589,2590,2593,2595],{},[621,2591,2592],{"align":610},"Batch gradient descent",[621,2594,2548],{"align":709},[621,2596,2587],{"align":709},[11,2598,2599,2600,2603,2604,2608],{},"Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about ",[15,2601,2602],{},"training data quality",": biasing the training distribution (even without adding \"noise\" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. ",[20,2605,2607],{"href":2606},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","This echoes a lesson I already saw in the other playlist",": training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.",[99,2610,597],{"id":596},[599,2612,2613,2621],{},[602,2614,2615],{},[605,2616,2617,2619],{},[608,2618,611],{"align":610},[608,2620,614],{"align":610},[616,2622,2623,2634,2646],{},[605,2624,2625,2628],{},[621,2626,2627],{"align":610},"The perceptron classifies and updates on the binary error",[621,2629,2630,2631,2633],{"align":610},"Adaline separates pre-activation (continuous, used in training) from post-activation (",[130,2632,2333],{},", used only to decide the final class)",[605,2635,2636,2639],{},[621,2637,2638],{"align":610},"The delta rule already showed up in the other playlist",[621,2640,2641,2642,2645],{"align":610},"Adaline ",[15,2643,2644],{},"is"," the delta rule applied to classification, training as if it were regression on the ±1 labels",[605,2647,2648,2651],{},[621,2649,2650],{"align":610},"Finding a line that separates seems like enough",[621,2652,2653],{"align":610},"Optimizing RMSE (not just counting mistakes) keeps finding a better boundary even after accuracy already hit 100%",[99,2655,646],{"id":645},[11,2657,2658,2659,656,2661,2663],{},"I reproduced pseudo-inverse vs. batch gradient descent on Iris (",[29,2660,655],{},[29,2662,659],{},"), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.",[124,2665,2667],{"className":126,"code":2666,"language":128,"meta":76,"style":76},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\nw_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[130,2668,2669,2673],{"__ignoreMap":76},[133,2670,2671],{"class":135,"line":136},[133,2672,1399],{},[133,2674,2675],{"class":135,"line":77},[133,2676,2677],{},"w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[11,2679,2680,2681,2684,2685,2688,2689,2692,2693,2696,2697,2700],{},"On the first attempt, I used the notebook's original ",[130,2682,2683],{},"learning_rate=0.01"," for batch gradient descent, and it ",[15,2686,2687],{},"diverged"," (weight became ",[130,2690,2691],{},"NaN",") straight away on raw Iris data. No surprise at this point: ",[20,2694,2695],{"href":2139},"it's the same scale lesson from the normal equation post",", just rediscovered again, this time needing a much smaller rate (",[130,2698,2699],{},"0.001",") to not blow up.",[599,2702,2703,2715],{},[602,2704,2705],{},[605,2706,2707,2709,2712],{},[608,2708,2569],{"align":610},[608,2710,2711],{"align":709},"RMSE (train)",[608,2713,2714],{"align":709},"Accuracy (test)",[616,2716,2717,2729],{},[605,2718,2719,2722,2725],{},[621,2720,2721],{"align":610},"Pseudo-inverse (Adaline)",[621,2723,2724],{"align":709},"0.2414",[621,2726,2727],{"align":709},[15,2728,539],{},[605,2730,2731,2738,2740],{},[621,2732,2733,2734,2737],{"align":610},"Batch gradient descent, ",[130,2735,2736],{},"learning_rate=0.001"," (Adaline)",[621,2739,2724],{"align":709},[621,2741,2742],{"align":709},[15,2743,539],{},[11,2745,2746,2747,2750,2751,2753],{},"With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, ",[20,2748,2749],{"href":1468},"the same generous margin from Iris that already favored the perceptron with bias",". The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of ",[130,2752,1738],{},"), not the quality of the final solution.",[740,2755,742],{},{"title":76,"searchDepth":77,"depth":77,"links":2757},[2758,2759,2760,2761,2762,2763],{"id":2167,"depth":77,"text":2168},{"id":2376,"depth":77,"text":2377},{"id":2402,"depth":77,"text":2403},{"id":2483,"depth":77,"text":2484},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",{"title":2149,"description":2764},"en\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",[2770,2771,2772],"adaline","delta-rule","widrow-hoff","tDtwjGKmRJJOKzWjkAyQ72-9DNH1O8sc9YdTGBFjxlk",{"id":2775,"title":2776,"body":2777,"cover":3,"date":752,"description":4116,"extension":80,"meta":4117,"navigation":82,"order":158,"path":4118,"playlist":755,"seo":4119,"status":86,"stem":4120,"tags":4121,"__hash__":4125},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo.md","Swapping the Cost Function Like Changing Clothes",{"type":8,"value":2778,"toc":4108},[2779,2786,2790,2890,2922,2948,2955,2959,3032,3063,3067,3103,3311,3347,3360,3381,3435,3466,3482,3760,3798,3809,3824,3828,3928,3937,3954,3956,4008,4010,4024,4098,4106],[11,2780,2781,2782,2785],{},"Lectures 3a and 3b. So far I've seen 4 different algorithms (perceptron, vectorized perceptron, Adaline via pseudo-inverse, Adaline via gradient) as somewhat separate things. This lecture shows they're actually the ",[15,2783,2784],{},"same recipe",", just swapping one ingredient: the cost function.",[99,2787,2789],{"id":2788},"first-the-training-algorithm-becomes-pluggable","First, the training algorithm becomes pluggable",[124,2791,2793],{"className":126,"code":2792,"language":128,"meta":76,"style":76},"class TrainingAlgorithm(ABC):\n  @abstractmethod\n  def get_w(self, X, y):\n    pass\n\nclass PseudoInverse(TrainingAlgorithm):\n  def get_w(self, X, y):\n    return np.linalg.pinv(X) @ y\n\nclass NeuralNetwork(BaseEstimator, ClassifierMixin):\n  def __init__(self, training_algorithm=PseudoInverse()):\n    self.training_algorithm = training_algorithm\n\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = self.training_algorithm.get_w(X, y)\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return np.sign(X @ self.w_)\n",[130,2794,2795,2800,2805,2810,2815,2819,2824,2828,2833,2837,2842,2847,2852,2856,2860,2864,2869,2873,2877,2881,2885],{"__ignoreMap":76},[133,2796,2797],{"class":135,"line":136},[133,2798,2799],{},"class TrainingAlgorithm(ABC):\n",[133,2801,2802],{"class":135,"line":77},[133,2803,2804],{},"  @abstractmethod\n",[133,2806,2807],{"class":135,"line":83},[133,2808,2809],{},"  def get_w(self, X, y):\n",[133,2811,2812],{"class":135,"line":152},[133,2813,2814],{},"    pass\n",[133,2816,2817],{"class":135,"line":158},[133,2818,287],{"emptyLinePlaceholder":82},[133,2820,2821],{"class":135,"line":164},[133,2822,2823],{},"class PseudoInverse(TrainingAlgorithm):\n",[133,2825,2826],{"class":135,"line":300},[133,2827,2809],{},[133,2829,2830],{"class":135,"line":306},[133,2831,2832],{},"    return np.linalg.pinv(X) @ y\n",[133,2834,2835],{"class":135,"line":312},[133,2836,287],{"emptyLinePlaceholder":82},[133,2838,2839],{"class":135,"line":318},[133,2840,2841],{},"class NeuralNetwork(BaseEstimator, ClassifierMixin):\n",[133,2843,2844],{"class":135,"line":324},[133,2845,2846],{},"  def __init__(self, training_algorithm=PseudoInverse()):\n",[133,2848,2849],{"class":135,"line":330},[133,2850,2851],{},"    self.training_algorithm = training_algorithm\n",[133,2853,2854],{"class":135,"line":336},[133,2855,287],{"emptyLinePlaceholder":82},[133,2857,2858],{"class":135,"line":342},[133,2859,292],{},[133,2861,2862],{"class":135,"line":348},[133,2863,1828],{},[133,2865,2866],{"class":135,"line":354},[133,2867,2868],{},"    self.w_ = self.training_algorithm.get_w(X, y)\n",[133,2870,2871],{"class":135,"line":360},[133,2872,381],{},[133,2874,2875],{"class":135,"line":366},[133,2876,287],{"emptyLinePlaceholder":82},[133,2878,2879],{"class":135,"line":372},[133,2880,1604],{},[133,2882,2883],{"class":135,"line":378},[133,2884,1828],{},[133,2886,2887],{"class":135,"line":953},[133,2888,2889],{},"    return np.sign(X @ self.w_)\n",[11,2891,2349,2892,2895,2896,2899,2900,2903,2904,2907,2908,2911,2912,2915,2916,2919,2920,74],{},[15,2893,2894],{},"Strategy"," design pattern: ",[130,2897,2898],{},"NeuralNetwork"," no longer knows ",[15,2901,2902],{},"how"," the weights get computed, just that there's a ",[130,2905,2906],{},"training_algorithm"," object with a ",[130,2909,2910],{},"get_w"," method. Swapping ",[130,2913,2914],{},"PseudoInverse()"," for ",[130,2917,2918],{},"SGD()"," in the constructor swaps out the entire training algorithm, without touching ",[130,2921,2898],{},[494,2923,2924],{},[11,2925,2926,845,2933,849,2936,2939,2940,845,2945,2947],{},[15,2927,2928,2929,2932],{},"Output (",[130,2930,2931],{},"PseudoInverse","):",[130,2934,2935],{},"0.95",[130,2937,2938],{},"[-1.905, 2.656, 1.049]",".\n",[15,2941,2928,2942,2932],{},[130,2943,2944],{},"SGD",[130,2946,2935],{},", essentially identical weights.",[11,2949,2950,2951,2954],{},"Confirms again, now with more cleanly organized code, what ",[20,2952,2953],{"href":2766},"I already saw in the last two posts",": pseudo-inverse and gradient descent solve the exact same problem.",[99,2956,2958],{"id":2957},"now-the-cost-function-becomes-pluggable-too","Now the cost function becomes pluggable too",[124,2960,2962],{"className":126,"code":2961,"language":128,"meta":76,"style":76},"class CostFunction(ABC):\n  @abstractstaticmethod\n  def get_cost(y, y_pred):\n    pass\n  @abstractstaticmethod\n  def get_gradient(X, y, y_pred):\n    pass\n\nclass WidrowHoff(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.mean((y-y_pred)**2)\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y-y_pred)\n",[130,2963,2964,2969,2974,2979,2983,2987,2992,2996,3000,3005,3010,3014,3019,3023,3027],{"__ignoreMap":76},[133,2965,2966],{"class":135,"line":136},[133,2967,2968],{},"class CostFunction(ABC):\n",[133,2970,2971],{"class":135,"line":77},[133,2972,2973],{},"  @abstractstaticmethod\n",[133,2975,2976],{"class":135,"line":83},[133,2977,2978],{},"  def get_cost(y, y_pred):\n",[133,2980,2981],{"class":135,"line":152},[133,2982,2814],{},[133,2984,2985],{"class":135,"line":158},[133,2986,2973],{},[133,2988,2989],{"class":135,"line":164},[133,2990,2991],{},"  def get_gradient(X, y, y_pred):\n",[133,2993,2994],{"class":135,"line":300},[133,2995,2814],{},[133,2997,2998],{"class":135,"line":306},[133,2999,287],{"emptyLinePlaceholder":82},[133,3001,3002],{"class":135,"line":312},[133,3003,3004],{},"class WidrowHoff(CostFunction):\n",[133,3006,3007],{"class":135,"line":318},[133,3008,3009],{},"  @staticmethod\n",[133,3011,3012],{"class":135,"line":324},[133,3013,2978],{},[133,3015,3016],{"class":135,"line":330},[133,3017,3018],{},"    return np.mean((y-y_pred)**2)\n",[133,3020,3021],{"class":135,"line":336},[133,3022,3009],{},[133,3024,3025],{"class":135,"line":342},[133,3026,2991],{},[133,3028,3029],{"class":135,"line":348},[133,3030,3031],{},"    return X.T @ (y-y_pred)\n",[11,3033,3034,3036,3037,3040,3041,3044,3045,2223,3048,3051,3052,3055,3056,3058,3059,3062],{},[130,3035,2944],{}," now also takes a ",[130,3038,3039],{},"cost_function",", and uses ",[130,3042,3043],{},"self.cost_function.get_gradient(...)"," instead of computing the gradient by hand. ",[130,3046,3047],{},"WidrowHoff",[20,3049,3050],{"href":2766},"exactly the delta rule from the last post",": continuous error (",[130,3053,3054],{},"y - y_pred",", no ",[130,3057,1485],{},") times the input. Swapping the cost function here means swapping ",[15,3060,3061],{},"what \"error\" means",", without touching the training loop.",[99,3064,3066],{"id":3065},"every-cost-function-recovers-a-different-algorithm","Every cost function recovers a different algorithm",[124,3068,3070],{"className":126,"code":3069,"language":128,"meta":76,"style":76},"class SmoothedSurrogate(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - np.sign(y_pred))\n",[130,3071,3072,3077,3081,3085,3090,3094,3098],{"__ignoreMap":76},[133,3073,3074],{"class":135,"line":136},[133,3075,3076],{},"class SmoothedSurrogate(CostFunction):\n",[133,3078,3079],{"class":135,"line":77},[133,3080,3009],{},[133,3082,3083],{"class":135,"line":83},[133,3084,2978],{},[133,3086,3087],{"class":135,"line":152},[133,3088,3089],{},"    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n",[133,3091,3092],{"class":135,"line":158},[133,3093,3009],{},[133,3095,3096],{"class":135,"line":164},[133,3097,2991],{},[133,3099,3100],{"class":135,"line":300},[133,3101,3102],{},"    return X.T @ (y - np.sign(y_pred))\n",[11,3104,3105,3106,3109,3110,3112,3113,3116,3117,3120,3121,504,3124,3304,3305,3307,3308,3310],{},"Notice the ",[130,3107,3108],{},"np.sign(y_pred)"," inside the gradient: this goes back to measuring error ",[15,3111,521],{}," the threshold, exactly like ",[20,3114,3115],{"href":72},"Rosenblatt's perceptron",". The name ",[130,3118,3119],{},"SmoothedSurrogate"," matches what Aggarwal calls the ",[15,3122,3123],{},"perceptron criterion",[133,3125,3127,3176],{"className":3126},[1642],[133,3128,3130],{"className":3129},[1646],[1648,3131,3132],{"xmlns":1650},[1652,3133,3134,3173],{},[1655,3135,3136,3139,3141,3144,3147,3149,3152,3155,3158,3160,3163,3171],{},[1658,3137,3138],{},"L",[1663,3140,1675],{},[1658,3142,3143],{},"max",[1663,3145,3146],{},"⁡",[1663,3148,1666],{"stretchy":1665},[3150,3151,572],"mn",{},[1663,3153,3154],{"separator":1051},",",[1663,3156,3157],{},"−",[1658,3159,187],{},[1663,3161,3162],{},"⋅",[3164,3165,3166,3168],"mover",{"accent":1051},[1658,3167,187],{},[1663,3169,3170],{},"^",[1663,3172,1672],{"stretchy":1665},[1679,3174,3175],{"encoding":1681},"L = \\max(0, -y \\cdot \\hat{y})",[133,3177,3179,3198,3239],{"className":3178,"ariaHidden":1051},[1686],[133,3180,3182,3186,3189,3192,3195],{"className":3181},[1690],[133,3183],{"className":3184,"style":3185},[1694],"height:0.6833em;",[133,3187,3138],{"className":3188},[1699,1707],[133,3190],{"className":3191,"style":1717},[1716],[133,3193,1675],{"className":3194},[1721],[133,3196],{"className":3197,"style":1717},[1716],[133,3199,3201,3204,3208,3211,3214,3218,3222,3225,3228,3232,3236],{"className":3200},[1690],[133,3202],{"className":3203,"style":1695},[1694],[133,3205,3143],{"className":3206},[3207],"mop",[133,3209,1666],{"className":3210},[1703],[133,3212,572],{"className":3213},[1699],[133,3215,3154],{"className":3216},[3217],"mpunct",[133,3219],{"className":3220,"style":3221},[1716],"margin-right:0.1667em;",[133,3223,3157],{"className":3224},[1699],[133,3226,187],{"className":3227,"style":1708},[1699,1707],[133,3229],{"className":3230,"style":3231},[1716],"margin-right:0.2222em;",[133,3233,3162],{"className":3234},[3235],"mbin",[133,3237],{"className":3238,"style":3231},[1716],[133,3240,3242,3245,3301],{"className":3241},[1690],[133,3243],{"className":3244,"style":1695},[1694],[133,3246,3249],{"className":3247},[1699,3248],"accent",[133,3250,3254,3292],{"className":3251},[3252,3253],"vlist-t","vlist-t2",[133,3255,3258,3287],{"className":3256},[3257],"vlist-r",[133,3259,3263,3274],{"className":3260,"style":3262},[3261],"vlist","height:0.6944em;",[133,3264,3266,3271],{"style":3265},"top:-3em;",[133,3267],{"className":3268,"style":3270},[3269],"pstrut","height:3em;",[133,3272,187],{"className":3273,"style":1708},[1699,1707],[133,3275,3276,3279],{"style":3265},[133,3277],{"className":3278,"style":3270},[3269],[133,3280,3284],{"className":3281,"style":3283},[3282],"accent-body","left:-0.1944em;",[133,3285,3170],{"className":3286},[1699],[133,3288,3291],{"className":3289},[3290],"vlist-s","​",[133,3293,3295],{"className":3294},[3257],[133,3296,3299],{"className":3297,"style":3298},[3261],"height:0.1944em;",[133,3300],{},[133,3302,1672],{"className":3303},[1712],", zero once the point is on the right side, growing linearly when it's wrong. No coincidence the accuracy hits ",[130,3306,539],{},": this cost function, plugged into this generic framework, ",[15,3309,2644],{}," the original perceptron again, just expressed in the language of \"cost function\" instead of \"update rule\".",[124,3312,3314],{"className":126,"code":3313,"language":128,"meta":76,"style":76},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - expit(y_pred))\n",[130,3315,3316,3321,3325,3329,3334,3338,3342],{"__ignoreMap":76},[133,3317,3318],{"class":135,"line":136},[133,3319,3320],{},"class LogLikehood(CostFunction):\n",[133,3322,3323],{"class":135,"line":77},[133,3324,3009],{},[133,3326,3327],{"class":135,"line":83},[133,3328,2978],{},[133,3330,3331],{"class":135,"line":152},[133,3332,3333],{},"    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n",[133,3335,3336],{"class":135,"line":158},[133,3337,3009],{},[133,3339,3340],{"class":135,"line":164},[133,3341,2991],{},[133,3343,3344],{"class":135,"line":300},[133,3345,3346],{},"    return X.T @ (y - expit(y_pred))\n",[494,3348,3349],{},[11,3350,3351,845,3353,849,3356,3359],{},[15,3352,500],{},[15,3354,3355],{},"0.65",[130,3357,3358],{},"[-60.78, 27.44, -24.79]",". Much worse than anything I've seen so far, and the weights got huge.",[11,3361,3362,3363,3366,3367,3370,3371,3374,3375,3377,3378,3380],{},"Two things wrong here, worth separating. First, a detail that doesn't affect the outcome: this class's ",[130,3364,3365],{},"get_cost"," uses the hinge loss formula (",[130,3368,3369],{},"max(0, 1 - y·ŷ)","), not an actual log-likelihood formula. That doesn't break anything in practice because ",[130,3372,3373],{},"get_gradient"," is the only thing ",[130,3376,2944],{}," calls, ",[130,3379,3365],{}," never gets used during training, it's leftover residue from copying and pasting from another cell.",[11,3382,3383,3384,3387,3388,423,3390,3392,3393,2223,3395,192,3397,3399,3400,3403,3404,3407,3408,3411,3412,3414,3415,3418,3419,589,3421,3423,3424,3427,3428,3431,3432,3434],{},"The second problem is real, and explains the bad accuracy: ",[130,3385,3386],{},"expit"," (the sigmoid function) only returns values between ",[130,3389,572],{},[130,3391,571],{},", but the label ",[130,3394,187],{},[130,3396,191],{},[130,3398,195],{},". For class ",[130,3401,3402],{},"y=-1",", the error ",[130,3405,3406],{},"y - expit(y_pred)"," can ",[15,3409,3410],{},"never"," get close to zero, because ",[130,3413,3386],{}," never goes negative: even with an infinitely confident correctly-classified prediction, ",[130,3416,3417],{},"-1 - expit(y_pred)"," stays pinned near ",[130,3420,191],{},[130,3422,572],{},". I checked this by hand: ",[130,3425,3426],{},"expit(-1000) = 0.0",", so ",[130,3429,3430],{},"-1 - expit(-1000) = -1.0"," exactly, not ",[130,3433,572],{},". The gradient for half the points never vanishes, so training never settles, and the weights keep growing trying to compensate for an error that's structurally impossible to zero out.",[124,3436,3438],{"className":126,"code":3437,"language":128,"meta":76,"style":76},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - tanh(y_pred))\n\nmodel = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[130,3439,3440,3444,3448,3452,3457,3461],{"__ignoreMap":76},[133,3441,3442],{"class":135,"line":136},[133,3443,3320],{},[133,3445,3446],{"class":135,"line":77},[133,3447,3009],{},[133,3449,3450],{"class":135,"line":83},[133,3451,2991],{},[133,3453,3454],{"class":135,"line":152},[133,3455,3456],{},"    return X.T @ (y - tanh(y_pred))\n",[133,3458,3459],{"class":135,"line":158},[133,3460,287],{"emptyLinePlaceholder":82},[133,3462,3463],{"class":135,"line":164},[133,3464,3465],{},"model = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[494,3467,3468],{},[11,3469,3470,845,3472,849,3474,3477,3478,3481],{},[15,3471,500],{},[15,3473,539],{},[130,3475,3476],{},"[-9.84, 12.40, 8.37]"," (with ",[130,3479,3480],{},"max_iter=10000",", ten times more iterations).",[11,3483,3484,3485,2915,3487,3490,3491,3493,3494,412,3496,3498,3499,3427,3502,3505,3506,3645,3646,412,3698,3755,3756,3759],{},"Swapping ",[130,3486,3386],{},[130,3488,3489],{},"tanh"," (same class redefined, Python just lets that run live in a notebook session), the problem disappears. Makes sense: ",[130,3492,3489],{}," ranges from ",[130,3495,191],{},[130,3497,195],{},", exactly the labels' range. Checked it by hand again: ",[130,3500,3501],{},"tanh(-1000) = -1.0",[130,3503,3504],{},"-1 - tanh(-1000) = 0.0",", the error genuinely reaches zero this time. Aggarwal states this relationship directly in chapter 1: ",[133,3507,3509,3553],{"className":3508},[1642],[133,3510,3512],{"className":3511},[1646],[1648,3513,3514],{"xmlns":1650},[1652,3515,3516,3550],{},[1655,3517,3518,3520,3522,3524,3526,3528,3530,3532,3534,3538,3540,3542,3544,3546,3548],{},[1658,3519,3489],{},[1663,3521,3146],{},[1663,3523,1666],{"stretchy":1665},[1658,3525,1669],{},[1663,3527,1672],{"stretchy":1665},[1663,3529,1675],{},[3150,3531,1276],{},[1663,3533,3162],{},[3535,3536,3537],"mtext",{},"sigmoid",[1663,3539,1666],{"stretchy":1665},[3150,3541,1276],{},[1658,3543,1669],{},[1663,3545,1672],{"stretchy":1665},[1663,3547,3157],{},[3150,3549,571],{},[1679,3551,3552],{"encoding":1681},"\\tanh(v) = 2 \\cdot \\text{sigmoid}(2v) - 1",[133,3554,3556,3583,3602,3636],{"className":3555,"ariaHidden":1051},[1686],[133,3557,3559,3562,3565,3568,3571,3574,3577,3580],{"className":3558},[1690],[133,3560],{"className":3561,"style":1695},[1694],[133,3563,3489],{"className":3564},[3207],[133,3566,1666],{"className":3567},[1703],[133,3569,1669],{"className":3570,"style":1708},[1699,1707],[133,3572,1672],{"className":3573},[1712],[133,3575],{"className":3576,"style":1717},[1716],[133,3578,1675],{"className":3579},[1721],[133,3581],{"className":3582,"style":1717},[1716],[133,3584,3586,3590,3593,3596,3599],{"className":3585},[1690],[133,3587],{"className":3588,"style":3589},[1694],"height:0.6444em;",[133,3591,1276],{"className":3592},[1699],[133,3594],{"className":3595,"style":3231},[1716],[133,3597,3162],{"className":3598},[3235],[133,3600],{"className":3601,"style":3231},[1716],[133,3603,3605,3608,3615,3618,3621,3624,3627,3630,3633],{"className":3604},[1690],[133,3606],{"className":3607,"style":1695},[1694],[133,3609,3612],{"className":3610},[1699,3611],"text",[133,3613,3537],{"className":3614},[1699],[133,3616,1666],{"className":3617},[1703],[133,3619,1276],{"className":3620},[1699],[133,3622,1669],{"className":3623,"style":1708},[1699,1707],[133,3625,1672],{"className":3626},[1712],[133,3628],{"className":3629,"style":3231},[1716],[133,3631,3157],{"className":3632},[3235],[133,3634],{"className":3635,"style":3231},[1716],[133,3637,3639,3642],{"className":3638},[1690],[133,3640],{"className":3641,"style":3589},[1694],[133,3643,571],{"className":3644},[1699],", and it's exactly that range shift, from ",[133,3647,3649,3671],{"className":3648},[1642],[133,3650,3652],{"className":3651},[1646],[1648,3653,3654],{"xmlns":1650},[1652,3655,3656,3668],{},[1655,3657,3658,3660,3662,3664,3666],{},[1663,3659,1666],{"stretchy":1665},[3150,3661,572],{},[1663,3663,3154],{"separator":1051},[3150,3665,571],{},[1663,3667,1672],{"stretchy":1665},[1679,3669,3670],{"encoding":1681},"(0,1)",[133,3672,3674],{"className":3673,"ariaHidden":1051},[1686],[133,3675,3677,3680,3683,3686,3689,3692,3695],{"className":3676},[1690],[133,3678],{"className":3679,"style":1695},[1694],[133,3681,1666],{"className":3682},[1703],[133,3684,572],{"className":3685},[1699],[133,3687,3154],{"className":3688},[3217],[133,3690],{"className":3691,"style":3221},[1716],[133,3693,571],{"className":3694},[1699],[133,3696,1672],{"className":3697},[1712],[133,3699,3701,3725],{"className":3700},[1642],[133,3702,3704],{"className":3703},[1646],[1648,3705,3706],{"xmlns":1650},[1652,3707,3708,3722],{},[1655,3709,3710,3712,3714,3716,3718,3720],{},[1663,3711,1666],{"stretchy":1665},[1663,3713,3157],{},[3150,3715,571],{},[1663,3717,3154],{"separator":1051},[3150,3719,571],{},[1663,3721,1672],{"stretchy":1665},[1679,3723,3724],{"encoding":1681},"(-1,1)",[133,3726,3728],{"className":3727,"ariaHidden":1051},[1686],[133,3729,3731,3734,3737,3740,3743,3746,3749,3752],{"className":3730},[1690],[133,3732],{"className":3733,"style":1695},[1694],[133,3735,1666],{"className":3736},[1703],[133,3738,3157],{"className":3739},[1699],[133,3741,571],{"className":3742},[1699],[133,3744,3154],{"className":3745},[3217],[133,3747],{"className":3748,"style":3221},[1716],[133,3750,571],{"className":3751},[1699],[133,3753,1672],{"className":3754},[1712],", that fixes the mismatch with ",[130,3757,3758],{},"±1"," labels.",[124,3761,3763],{"className":126,"code":3762,"language":128,"meta":76,"style":76},"class HingeLoss(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    marginal_errors = (y * y_pred) \u003C 1\n    marginal_ys = np.copy(y)\n    marginal_ys[~marginal_errors] = 0\n    return X.T @ marginal_ys\n",[130,3764,3765,3770,3774,3778,3783,3788,3793],{"__ignoreMap":76},[133,3766,3767],{"class":135,"line":136},[133,3768,3769],{},"class HingeLoss(CostFunction):\n",[133,3771,3772],{"class":135,"line":77},[133,3773,3009],{},[133,3775,3776],{"class":135,"line":83},[133,3777,2991],{},[133,3779,3780],{"class":135,"line":152},[133,3781,3782],{},"    marginal_errors = (y * y_pred) \u003C 1\n",[133,3784,3785],{"class":135,"line":158},[133,3786,3787],{},"    marginal_ys = np.copy(y)\n",[133,3789,3790],{"class":135,"line":164},[133,3791,3792],{},"    marginal_ys[~marginal_errors] = 0\n",[133,3794,3795],{"class":135,"line":300},[133,3796,3797],{},"    return X.T @ marginal_ys\n",[494,3799,3800],{},[11,3801,3802,845,3804,849,3806,74],{},[15,3803,500],{},[15,3805,539],{},[130,3807,3808],{},"[-8.70, 11.59, 7.71]",[11,3810,3811,3812,3815,3816,3819,3820,3823],{},"Notice ",[130,3813,3814],{},"marginal_ys[~marginal_errors] = 0",": points that are already well classified, with room to spare (",[130,3817,3818],{},"y · ŷ ≥ 1","), get zeroed out and ",[15,3821,3822],{},"contribute nothing"," to the gradient. Only points inside the margin (or misclassified) participate in the update. That's literally SVM's central idea: only the points near the boundary (the \"support vectors\") matter for deciding where it sits. The rest of the dataset gets ignored once it's already well separated.",[99,3825,3827],{"id":3826},"all-four-curves-side-by-side","All four curves, side by side",[11,3829,3830,3831,3927],{},"The professor has a reference image saved in the notebook comparing the four penalty curves as a function of the \"margin\" (",[133,3832,3834,3856],{"className":3833},[1642],[133,3835,3837],{"className":3836},[1646],[1648,3838,3839],{"xmlns":1650},[1652,3840,3841,3853],{},[1655,3842,3843,3845,3847],{},[1658,3844,187],{},[1663,3846,3162],{},[3164,3848,3849,3851],{"accent":1051},[1658,3850,187],{},[1663,3852,3170],{},[1679,3854,3855],{"encoding":1681},"y \\cdot \\hat{y}",[133,3857,3859,3878],{"className":3858,"ariaHidden":1051},[1686],[133,3860,3862,3866,3869,3872,3875],{"className":3861},[1690],[133,3863],{"className":3864,"style":3865},[1694],"height:0.6389em;vertical-align:-0.1944em;",[133,3867,187],{"className":3868,"style":1708},[1699,1707],[133,3870],{"className":3871,"style":3231},[1716],[133,3873,3162],{"className":3874},[3235],[133,3876],{"className":3877,"style":3231},[1716],[133,3879,3881,3885],{"className":3880},[1690],[133,3882],{"className":3883,"style":3884},[1694],"height:0.8889em;vertical-align:-0.1944em;",[133,3886,3888],{"className":3887},[1699,3248],[133,3889,3891,3919],{"className":3890},[3252,3253],[133,3892,3894,3916],{"className":3893},[3257],[133,3895,3897,3905],{"className":3896,"style":3262},[3261],[133,3898,3899,3902],{"style":3265},[133,3900],{"className":3901,"style":3270},[3269],[133,3903,187],{"className":3904,"style":1708},[1699,1707],[133,3906,3907,3910],{"style":3265},[133,3908],{"className":3909,"style":3270},[3269],[133,3911,3913],{"className":3912,"style":3283},[3282],[133,3914,3170],{"className":3915},[1699],[133,3917,3291],{"className":3918},[3290],[133,3920,3922],{"className":3921},[3257],[133,3923,3925],{"className":3924,"style":3298},[3261],[133,3926],{},": positive and large means a confident correct call, negative means a mistake). I recreated the same idea here, interactively:",[3929,3930],"margin-loss-chart",{"hinge-label":3931,"logistic-label":3932,"perceptron-label":3933,"widrow-hoff-label":3934,"x-label":3935,"y-label":3936},"Hinge (SVM)","Logistic","Perceptron","Widrow-Hoff","margin (y · ŷ)","penalty",[11,3938,3939,3940,3943,3944,3947,3948,3950,3951,3953],{},"Hover over any point on the x-axis and compare all four. Notice the shapes: Widrow-Hoff is a parabola, it keeps penalizing even a point that's already correctly classified with room to spare (margin ",[130,3941,3942],{},"> 1","), because it doesn't know \"correct is correct\", it only knows how to measure distance to a continuous target. Perceptron and Hinge are the only two that ",[15,3945,3946],{},"fully zero out"," once a point is well classified (perceptron zeros as soon as it crosses ",[130,3949,572],{},", hinge requires crossing ",[130,3952,571],{},", with room to spare). Logistic never truly zeroes out, it only approaches zero, which is the price it pays for returning a smooth probability instead of a binary decision.",[99,3955,597],{"id":596},[599,3957,3958,3966],{},[602,3959,3960],{},[605,3961,3962,3964],{},[608,3963,611],{"align":610},[608,3965,614],{"align":610},[616,3967,3968,3986,4000],{},[605,3969,3970,3973],{},[621,3971,3972],{"align":610},"Perceptron, Adaline, batch gradient descent looked like separate algorithms",[621,3974,3975,3976,3978,3979,3978,3982,3985],{"align":610},"They're the same framework (",[130,3977,2898],{}," + ",[130,3980,3981],{},"TrainingAlgorithm",[130,3983,3984],{},"CostFunction","), just swapping which cost function gets plugged in",[605,3987,3988,3991],{},[621,3989,3990],{"align":610},"Sigmoid returns a probability between 0 and 1",[621,3992,3993,3994,3996,3997,3999],{"align":610},"Using sigmoid directly against a ",[130,3995,3758],{}," label locks up the gradient, because the output range doesn't match the target's range. ",[130,3998,3489],{}," fixes it",[605,4001,4002,4005],{},[621,4003,4004],{"align":610},"SVM uses \"support vectors\"",[621,4006,4007],{"align":610},"That's not empty jargon: the hinge loss gradient literally zeros out the contribution of every point that isn't a support vector",[99,4009,646],{"id":645},[11,4011,4012,4013,4015,4016,656,4018,4020,4021,4023],{},"I ran all four cost functions (Widrow-Hoff, perceptron criterion, hinge, and the log-likelihood version with ",[130,4014,3489],{},") on Iris (",[29,4017,655],{},[29,4019,659],{},"), plus the ",[130,4022,3386],{}," version on purpose, to confirm the sigmoid problem isn't exclusive to the notebook's toy synthetic dataset.",[599,4025,4026,4037],{},[602,4027,4028],{},[605,4029,4030,4033,4035],{},[608,4031,4032],{"align":610},"Cost function",[608,4034,1413],{"align":709},[608,4036,1416],{"align":709},[616,4038,4039,4049,4059,4070,4083],{},[605,4040,4041,4043,4045],{},[621,4042,3934],{"align":610},[621,4044,539],{"align":709},[621,4046,4047],{"align":709},[15,4048,539],{},[605,4050,4051,4053,4055],{},[621,4052,3933],{"align":610},[621,4054,539],{"align":709},[621,4056,4057],{"align":709},[15,4058,539],{},[605,4060,4061,4064,4066],{},[621,4062,4063],{"align":610},"Hinge",[621,4065,539],{"align":709},[621,4067,4068],{"align":709},[15,4069,539],{},[605,4071,4072,4077,4079],{},[621,4073,4074,4075,1672],{"align":610},"Log-likelihood (",[130,4076,3489],{},[621,4078,539],{"align":709},[621,4080,4081],{"align":709},[15,4082,539],{},[605,4084,4085,4090,4093],{},[621,4086,4074,4087,4089],{"align":610},[130,4088,3386],{},", sigmoid)",[621,4091,4092],{"align":709},"0.843",[621,4094,4095],{"align":709},[15,4096,4097],{},"0.933",[11,4099,4100,4101,423,4103,4105],{},"Four out of five hit 100% (Iris has a generous enough margin for any of them to find a perfect boundary), and sigmoid falls behind again, this time on real data, not just the notebook's synthetic dataset. Confirms it wasn't a one-run coincidence: the range mismatch between ",[130,4102,3386],{},[130,4104,3758],{}," labels genuinely hurts convergence, on every dataset I tried it on.",[740,4107,742],{},{"title":76,"searchDepth":77,"depth":77,"links":4109},[4110,4111,4112,4113,4114,4115],{"id":2788,"depth":77,"text":2789},{"id":2957,"depth":77,"text":2958},{"id":3065,"depth":77,"text":3066},{"id":3826,"depth":77,"text":3827},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lectures 3a and 3b: the professor generalizes training to accept any pluggable cost function, and each choice (Widrow-Hoff, perceptron criterion, log-likelihood, hinge) recovers a different algorithm from this playlist. Along the way, I found a real mismatch between activation function and label encoding.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",{"title":2776,"description":4116},"en\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",[4122,4123,4124],"cost-functions","hinge-loss","logistic-regression","alBoIF9HT5FQ7eR5ld8q-Pe1ojBlQBX2mpc5QwQQgnk",{"id":4127,"title":4128,"body":4129,"cover":3,"date":752,"description":4636,"extension":80,"meta":4637,"navigation":82,"order":164,"path":4638,"playlist":755,"seo":4639,"status":86,"stem":4640,"tags":4641,"__hash__":4644},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse.md","Multiclass: When a Weight Becomes a Matrix",{"type":8,"value":4130,"toc":4626},[4131,4134,4138,4177,4186,4196,4200,4225,4234,4271,4275,4300,4321,4341,4347,4357,4425,4456,4464,4468,4475,4485,4488,4490,4536,4538,4549,4558,4566,4608,4624],[11,4132,4133],{},"Lecture 3c, and the professor swaps the 2-class dataset for a 4-class one. The first attempt is to honestly reuse everything that already exists without changing anything, just to show, in practice, exactly where it breaks.",[99,4135,4137],{"id":4136},"the-dataset-four-blobs-one-in-each-corner","The dataset: four blobs, one in each corner",[124,4139,4141],{"className":126,"code":4140,"language":128,"meta":76,"style":76},"def createMulticlassDataset(n=40):\n  X, y = make_blobs(n_samples=n,\n                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n                    n_features=2,\n                    cluster_std=0.05,\n                    center_box=(0,1))\n  return X, y\n",[130,4142,4143,4148,4153,4158,4163,4168,4173],{"__ignoreMap":76},[133,4144,4145],{"class":135,"line":136},[133,4146,4147],{},"def createMulticlassDataset(n=40):\n",[133,4149,4150],{"class":135,"line":77},[133,4151,4152],{},"  X, y = make_blobs(n_samples=n,\n",[133,4154,4155],{"class":135,"line":83},[133,4156,4157],{},"                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n",[133,4159,4160],{"class":135,"line":152},[133,4161,4162],{},"                    n_features=2,\n",[133,4164,4165],{"class":135,"line":158},[133,4166,4167],{},"                    cluster_std=0.05,\n",[133,4169,4170],{"class":135,"line":164},[133,4171,4172],{},"                    center_box=(0,1))\n",[133,4174,4175],{"class":135,"line":300},[133,4176,167],{},[124,4178,4180],{"className":126,"code":4179,"language":128,"meta":76,"style":76},"set(y_train)\n",[130,4181,4182],{"__ignoreMap":76},[133,4183,4184],{"class":135,"line":136},[133,4185,4179],{},[494,4187,4188],{},[11,4189,4190,532,4192,4195],{},[15,4191,500],{},[130,4193,4194],{},"{0, 1, 2, 3}",". Four classes, one integer each.",[99,4197,4199],{"id":4198},"first-attempt-reuse-what-already-exists-and-fail-on-purpose","First attempt: reuse what already exists (and fail on purpose)",[124,4201,4203],{"className":126,"code":4202,"language":128,"meta":76,"style":76},"model = NeuralNetwork()  # the usual one, sign(X @ w_)\nmodel.fit(X, y)\ny_pred = model.predict(X)\nprint(f\"Accuracy: {accuracy_score(y, y_pred)}\")\n",[130,4204,4205,4210,4215,4220],{"__ignoreMap":76},[133,4206,4207],{"class":135,"line":136},[133,4208,4209],{},"model = NeuralNetwork()  # the usual one, sign(X @ w_)\n",[133,4211,4212],{"class":135,"line":77},[133,4213,4214],{},"model.fit(X, y)\n",[133,4216,4217],{"class":135,"line":83},[133,4218,4219],{},"y_pred = model.predict(X)\n",[133,4221,4222],{"class":135,"line":152},[133,4223,4224],{},"print(f\"Accuracy: {accuracy_score(y, y_pred)}\")\n",[494,4226,4227],{},[11,4228,4229,845,4231,74],{},[15,4230,500],{},[15,4232,4233],{},"0.25",[11,4235,4236,4237,4239,4240,4242,4243,4246,4247,192,4249,4251,4252,2384,4254,2384,4256,4258,4259,4261,4262,2384,4264,2384,4266,4258,4268,4270],{},"Worth understanding exactly why ",[130,4238,4233],{},", it's not just some random number: with 4 well-balanced classes, always guessing the same thing gets you right on average 1 out of every 4 times, that is, 25%. The usual ",[130,4241,2898],{}," uses ",[130,4244,4245],{},"sign(X @ w_)",", which only returns ",[130,4248,191],{},[130,4250,195],{},", two possible values, never ",[130,4253,572],{},[130,4255,571],{},[130,4257,1276],{},", or ",[130,4260,1287],{},". Comparing that against a label that can be ",[130,4263,572],{},[130,4265,571],{},[130,4267,1276],{},[130,4269,1287],{}," is comparing things of different natures. The model isn't \"almost getting it right\", it literally can't express 3 of the 4 possible answers. The 0.25 accuracy is, in practice, the same level as a blind guess.",[99,4272,4274],{"id":4273},"one-hot-every-class-becomes-its-own-column","One-hot: every class becomes its own column",[124,4276,4278],{"className":126,"code":4277,"language":128,"meta":76,"style":76},"y_hot = np.zeros((y_train.shape[0], len(set(y_train))), dtype=int)\nfor i, label in enumerate(list(set(y_train))):\n  idxs = np.where(y_train == label)[0]\n  y_hot[idxs, i] = 1\n",[130,4279,4280,4285,4290,4295],{"__ignoreMap":76},[133,4281,4282],{"class":135,"line":136},[133,4283,4284],{},"y_hot = np.zeros((y_train.shape[0], len(set(y_train))), dtype=int)\n",[133,4286,4287],{"class":135,"line":77},[133,4288,4289],{},"for i, label in enumerate(list(set(y_train))):\n",[133,4291,4292],{"class":135,"line":83},[133,4293,4294],{},"  idxs = np.where(y_train == label)[0]\n",[133,4296,4297],{"class":135,"line":152},[133,4298,4299],{},"  y_hot[idxs, i] = 1\n",[11,4301,4302,4303,2384,4305,2384,4307,4258,4309,4311,4312,4314,4315,4317,4318,525],{},"Instead of a scalar label (",[130,4304,572],{},[130,4306,571],{},[130,4308,1276],{},[130,4310,1287],{},"), every example becomes a row with a ",[130,4313,571],{}," in its class's column and ",[130,4316,572],{}," in the others. The professor confirms this matches exactly scikit-learn's ",[130,4319,4320],{},"LabelBinarizer",[124,4322,4324],{"className":126,"code":4323,"language":128,"meta":76,"style":76},"from sklearn.preprocessing import LabelBinarizer\nlb = LabelBinarizer()\ny_hot = lb.fit_transform(y_train)\n",[130,4325,4326,4331,4336],{"__ignoreMap":76},[133,4327,4328],{"class":135,"line":136},[133,4329,4330],{},"from sklearn.preprocessing import LabelBinarizer\n",[133,4332,4333],{"class":135,"line":77},[133,4334,4335],{},"lb = LabelBinarizer()\n",[133,4337,4338],{"class":135,"line":83},[133,4339,4340],{},"y_hot = lb.fit_transform(y_train)\n",[11,4342,4343,4344,74],{},"Same matrix, two implementations, ",[20,4345,4346],{"href":2139},"the same kind of \"matches the professional tool\" check that already showed up in this playlist",[99,4348,4350,4351,4353,4354],{"id":4349},"the-fix-weight-becomes-a-matrix-sign-becomes-argmax","The fix: weight becomes a matrix, ",[130,4352,2333],{}," becomes ",[130,4355,4356],{},"argmax",[124,4358,4360],{"className":126,"code":4359,"language":128,"meta":76,"style":76},"class SGD(TrainingAlgorithm):\n  def get_w(self, X, y):\n    self.w_ = np.random.random(size=(X.shape[1], y.shape[1]))\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      self.w_ += self.learning_rate * self.cost_function.get_gradient(X, y, y_pred)\n    return self.w_\n\nclass NeuralNetwork(BaseEstimator, ClassifierMixin):\n  def predict(self, X):\n    X = include_bias(X)\n    logits = X @ self.w_\n    idxs = np.argmax(logits, axis=1)\n    return np.array([self.labels[idx] for idx in idxs])\n",[130,4361,4362,4367,4371,4376,4380,4384,4389,4394,4398,4402,4406,4410,4415,4420],{"__ignoreMap":76},[133,4363,4364],{"class":135,"line":136},[133,4365,4366],{},"class SGD(TrainingAlgorithm):\n",[133,4368,4369],{"class":135,"line":77},[133,4370,2809],{},[133,4372,4373],{"class":135,"line":83},[133,4374,4375],{},"    self.w_ = np.random.random(size=(X.shape[1], y.shape[1]))\n",[133,4377,4378],{"class":135,"line":152},[133,4379,303],{},[133,4381,4382],{"class":135,"line":158},[133,4383,2435],{},[133,4385,4386],{"class":135,"line":164},[133,4387,4388],{},"      self.w_ += self.learning_rate * self.cost_function.get_gradient(X, y, y_pred)\n",[133,4390,4391],{"class":135,"line":300},[133,4392,4393],{},"    return self.w_\n",[133,4395,4396],{"class":135,"line":306},[133,4397,287],{"emptyLinePlaceholder":82},[133,4399,4400],{"class":135,"line":312},[133,4401,2841],{},[133,4403,4404],{"class":135,"line":318},[133,4405,1604],{},[133,4407,4408],{"class":135,"line":324},[133,4409,1828],{},[133,4411,4412],{"class":135,"line":330},[133,4413,4414],{},"    logits = X @ self.w_\n",[133,4416,4417],{"class":135,"line":336},[133,4418,4419],{},"    idxs = np.argmax(logits, axis=1)\n",[133,4421,4422],{"class":135,"line":342},[133,4423,4424],{},"    return np.array([self.labels[idx] for idx in idxs])\n",[11,4426,4427,4428,4431,4432,4435,4436,188,4439,4442,4443,4446,4447,2915,4449,4451,4452,4455],{},"The change that fixes everything: ",[130,4429,4430],{},"self.w_"," stops being a vector (",[130,4433,4434],{},"(features,)",") and becomes a ",[15,4437,4438],{},"matrix",[130,4440,4441],{},"(features, classes)","), one column of weights per class. ",[130,4444,4445],{},"X @ self.w_"," now returns, for every point, 4 numbers (one \"how confident\" score per class), not just 1. And the final prediction swaps ",[130,4448,1485],{},[130,4450,4356],{},": instead of asking \"positive or negative?\", it asks \"which of the 4 columns had the highest value?\". This is exactly the multiple-output layer architecture Aggarwal describes for categorical classification: one weight per class, and the final decision is whichever one \"won\". The only piece missing to turn it into his full version (with ",[130,4453,4454],{},"softmax",", turning the 4 numbers into probabilities that sum to 1) is the normalization. Here the model just compares the raw numbers, without turning them into a probability, but the argmax winner doesn't change either way.",[494,4457,4458],{},[11,4459,4460,845,4462,74],{},[15,4461,500],{},[15,4463,539],{},[99,4465,4467],{"id":4466},"interactive-the-four-decision-regions","Interactive: the four decision regions",[11,4469,4470,4471,4474],{},"I rebuilt the same dataset (",[130,4472,4473],{},"make_blobs",", same 4 centers) and trained the weight-matrix version. Every background color is the region where that class wins the argmax.",[4476,4477],"multiclass-region-chart",{":classes":4478,":points":4479,":weights":4480,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"class0-label":4481,"class1-label":4482,"class2-label":4483,"class3-label":4484,"x-label":578,"y-label":579},"[1, 1, 3, 2, 0, 1, 2, 1, 2, 0, 0, 0, 0, 1, 2, 1, 3, 3, 0, 3, 1, 2, 3, 3, 1, 3, 3, 2, 3, 0, 2, 0, 1, 1, 2, 0, 2, 2, 0, 3]","[[0.6968, 0.1669], [0.7174, 0.2268], [0.785, 0.741], [0.2956, 0.8119], [0.2277, 0.2062], [0.7398, 0.2731], [0.1953, 0.8597], [0.8825, 0.2077], [0.242, 0.791], [0.2, 0.1123], [0.1687, 0.1914], [0.2845, 0.1767], [0.1606, 0.2001], [0.8525, 0.1792], [0.2051, 0.8126], [0.7629, 0.2536], [0.795, 0.885], [0.8166, 0.8368], [0.1879, 0.1273], [0.8749, 0.7859], [0.7797, 0.0856], [0.1146, 0.7098], [0.7403, 0.7475], [0.7904, 0.7111], [0.8883, 0.1835], [0.7816, 0.7047], [0.7808, 0.7555], [0.1283, 0.8251], [0.8054, 0.8719], [0.2016, 0.2204], [0.2135, 0.7738], [0.2253, 0.1869], [0.7806, 0.3015], [0.7977, 0.1275], [0.1934, 0.7845], [0.2509, 0.23], [0.2284, 0.7624], [0.2192, 0.9124], [0.2137, 0.1237], [0.8752, 0.7894]]","[[1.0602, -0.8069, -0.8239], [0.2232, 0.812, -0.7759], [0.2644, -0.8451, 0.8351], [-0.5478, 0.84, 0.7647]]","Class 0","Class 1","Class 2","Class 3",[11,4486,4487],{},"Notice the four boundaries meeting near the middle of the chart, splitting the plane into four wedges, one per class. Every blob lands cleanly inside the right color.",[99,4489,597],{"id":596},[599,4491,4492,4500],{},[602,4493,4494],{},[605,4495,4496,4498],{},[608,4497,611],{"align":610},[608,4499,614],{"align":610},[616,4501,4502,4515,4523],{},[605,4503,4504,4509],{},[621,4505,4506,4508],{"align":610},[130,4507,1485],{}," classifies into two classes",[621,4510,4511,4512,4514],{"align":610},"With more than two classes, ",[130,4513,1485],{}," structurally can't work, there are only 2 possible outputs for N classes",[605,4516,4517,4520],{},[621,4518,4519],{"align":610},"One-hot encoding turns a categorical label into a vector",[621,4521,4522],{"align":610},"That's not just a formatting convenience, it's what lets the weight become a matrix (one column per class)",[605,4524,4525,4530],{},[621,4526,4527,4529],{"align":610},[130,4528,4356],{}," picks the largest value",[621,4531,4532,4533,4535],{"align":610},"It's the direct generalization of ",[130,4534,1485],{}," (which is basically \"argmax between 2 options: positive or negative\") to any number of classes",[99,4537,646],{"id":645},[11,4539,4540,4541,4544,4545,4548],{},"I tested the same idea (one-hot + weight matrix + argmax) on Wine (",[130,4542,4543],{},"load_wine",", 3 grape cultivars, 13 chemical variables), with one extra detail: the variables here live on quite different scales, ",[20,4546,4547],{"href":2139},"the same problem already seen earlier in this playlist",", so I normalized before training.",[124,4550,4552],{"className":126,"code":4551,"language":128,"meta":76,"style":76},"X_train_s = StandardScaler().fit_transform(X_train)\n",[130,4553,4554],{"__ignoreMap":76},[133,4555,4556],{"class":135,"line":136},[133,4557,4551],{},[11,4559,4560,4561,4563,4564,74],{},"Even normalized, ",[130,4562,2683],{}," (the notebook's default) still diverged with 13 variables, so I had to drop to ",[130,4565,2699],{},[599,4567,4568,4578],{},[602,4569,4570],{},[605,4571,4572,4574,4576],{},[608,4573,2098],{"align":610},[608,4575,1413],{"align":709},[608,4577,1416],{"align":709},[616,4579,4580,4594],{},[605,4581,4582,4588,4591],{},[621,4583,4584,4585,4587],{"align":610},"Naive (",[130,4586,2333],{},", scalar label)",[621,4589,4590],{"align":709},"0.403",[621,4592,4593],{"align":709},"-",[605,4595,4596,4601,4603],{},[621,4597,4598,4599],{"align":610},"One-hot + weight matrix + ",[130,4600,4356],{},[621,4602,539],{"align":709},[621,4604,4605],{"align":709},[15,4606,4607],{},"0.9815",[11,4609,4610,4611,192,4613,4615,4616,4618,4619,192,4621,4623],{},"The naive version can't even reach all 3 possible class values (it can only ever predict ",[130,4612,191],{},[130,4614,195],{},", never class ",[130,4617,1276],{},"), so even that 0.403 number is misleading, it counts as a \"hit\" any case where the label happened to already be ",[130,4620,191],{},[130,4622,571],{},". The weight-matrix version gets nearly everything right, both train and test, confirming the same trick that worked on the synthetic 4-blob dataset generalizes to a real multiclass classification problem.",[740,4625,742],{},{"title":76,"searchDepth":77,"depth":77,"links":4627},[4628,4629,4630,4631,4633,4634,4635],{"id":4136,"depth":77,"text":4137},{"id":4198,"depth":77,"text":4199},{"id":4273,"depth":77,"text":4274},{"id":4349,"depth":77,"text":4632},"The fix: weight becomes a matrix, sign becomes argmax",{"id":4466,"depth":77,"text":4467},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lecture 3c: reusing the binary classifier directly on a 4-class problem lands exactly at random-guess accuracy, 25%. The fix is generalizing the weight from a vector to a matrix, one set of weights per class, and swapping sign() for argmax.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse",{"title":4128,"description":4636},"en\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse",[4642,4643,4356],"multiclass","one-hot-encoding","l52gY4-RRFdZBm-iOZFY-VoFu7l55JDyblv5enLfn7k",{"id":4646,"title":4647,"body":4648,"cover":3,"date":752,"description":5007,"extension":80,"meta":5008,"navigation":82,"order":300,"path":5009,"playlist":755,"seo":5010,"status":86,"stem":5011,"tags":5012,"__hash__":5016},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron.md","XOR: Where a Single Neuron Just Hits a Wall",{"type":8,"value":4649,"toc":4998},[4650,4653,4657,4696,4720,4724,4743,4755,4773,4785,4794,4798,4816,4829,4841,4858,4862,4868,4873,4876,4880,4887,4894,4896,4932,4934,4942,4957,4993,4996],[11,4651,4652],{},"Lecture 3d, and it's the last one in the repository as of today. Before closing out this batch of posts, the professor leaves a cliffhanger that's almost poetic: the same perceptron that solved everything up to this point, effortlessly, gets stuck hard on a 4-point problem.",[99,4654,4656],{"id":4655},"the-dataset-logic-gates-geometrically","The dataset: logic gates, geometrically",[124,4658,4660],{"className":126,"code":4659,"language":128,"meta":76,"style":76},"def createLogicalDataset(n=40, func=lambda a, b: bool(a) or bool(b)):\n  X, y = make_blobs(n_samples=n,\n                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n                    n_features=2,\n                    cluster_std=0.05,\n                    center_box=(0,1))\n  y = np.array([func(a>0.5, b>0.5) for a, b in X], dtype=int)\n  return X, y\n",[130,4661,4662,4667,4671,4675,4679,4683,4687,4692],{"__ignoreMap":76},[133,4663,4664],{"class":135,"line":136},[133,4665,4666],{},"def createLogicalDataset(n=40, func=lambda a, b: bool(a) or bool(b)):\n",[133,4668,4669],{"class":135,"line":77},[133,4670,4152],{},[133,4672,4673],{"class":135,"line":83},[133,4674,4157],{},[133,4676,4677],{"class":135,"line":152},[133,4678,4162],{},[133,4680,4681],{"class":135,"line":158},[133,4682,4167],{},[133,4684,4685],{"class":135,"line":164},[133,4686,4172],{},[133,4688,4689],{"class":135,"line":300},[133,4690,4691],{},"  y = np.array([func(a>0.5, b>0.5) for a, b in X], dtype=int)\n",[133,4693,4694],{"class":135,"line":306},[133,4695,167],{},[11,4697,4698,4701,4702,2384,4705,2384,4708,4711,4712,4715,4716,4719],{},[20,4699,4700],{"href":4638},"The same 4-blob dataset from the previous post",", except now the label comes from applying a logical function (",[130,4703,4704],{},"or",[130,4706,4707],{},"and",[130,4709,4710],{},"!=",") to \"is this point to the right of 0.5?\" and \"is this point above 0.5?\". Every blob effectively becomes a logic gate value: the bottom-left corner is ",[130,4713,4714],{},"(false, false)",", bottom-right is ",[130,4717,4718],{},"(true, false)",", and so on.",[99,4721,4723],{"id":4722},"or-and-and-no-drama","OR and AND: no drama",[124,4725,4727],{"className":126,"code":4726,"language":128,"meta":76,"style":76},"X, y = createLogicalDataset(func=lambda a, b: bool(a) or bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[130,4728,4729,4734,4739],{"__ignoreMap":76},[133,4730,4731],{"class":135,"line":136},[133,4732,4733],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) or bool(b))\n",[133,4735,4736],{"class":135,"line":77},[133,4737,4738],{},"model = NeuralNetwork()\n",[133,4740,4741],{"class":135,"line":83},[133,4742,4214],{},[494,4744,4745],{},[11,4746,4747,845,4750,849,4752,74],{},[15,4748,4749],{},"Output (OR):",[15,4751,539],{},[130,4753,4754],{},"[-1.14, 1.72, 1.57]",[124,4756,4758],{"className":126,"code":4757,"language":128,"meta":76,"style":76},"X, y = createLogicalDataset(func=lambda a, b: bool(a) and bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[130,4759,4760,4765,4769],{"__ignoreMap":76},[133,4761,4762],{"class":135,"line":136},[133,4763,4764],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) and bool(b))\n",[133,4766,4767],{"class":135,"line":77},[133,4768,4738],{},[133,4770,4771],{"class":135,"line":83},[133,4772,4214],{},[494,4774,4775],{},[11,4776,4777,845,4780,849,4782,74],{},[15,4778,4779],{},"Output (AND):",[15,4781,539],{},[130,4783,4784],{},"[-2.16, 1.55, 1.79]",[11,4786,4787,4788,4790,4791,4793],{},"Both gates are linearly separable: for OR, a single line separates the bottom-left blob (the only ",[130,4789,1665],{},") from the other three. For AND, it's the same thing, just isolating the top-right blob (the only ",[130,4792,1051],{},"). Nothing this perceptron hasn't already done throughout this whole post series.",[99,4795,4797],{"id":4796},"xor-the-same-recipe-the-same-code-and-it-gets-stuck","XOR: the same recipe, the same code, and it gets stuck",[124,4799,4801],{"className":126,"code":4800,"language":128,"meta":76,"style":76},"X, y = createLogicalDataset(func=lambda a, b: bool(a) != bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[130,4802,4803,4808,4812],{"__ignoreMap":76},[133,4804,4805],{"class":135,"line":136},[133,4806,4807],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) != bool(b))\n",[133,4809,4810],{"class":135,"line":77},[133,4811,4738],{},[133,4813,4814],{"class":135,"line":83},[133,4815,4214],{},[494,4817,4818],{},[11,4819,4820,845,4823,849,4826,74],{},[15,4821,4822],{},"Output (XOR):",[15,4824,4825],{},"0.5",[130,4827,4828],{},"[0.006, -0.011, -0.002]",[11,4830,4831,4832,4834,4835,4837,4838,1930],{},"Accuracy of 0.5 with 2 classes is exactly coin-flip level. And notice the weights: essentially zero across all three, the model basically gave up, shrinking the weight vector down to almost nothing instead of converging to anything useful. I reproduced this myself, with a seeded dataset, and hit the exact same wall: OR and AND at ",[130,4833,539],{},", XOR pinned at ",[130,4836,4825],{},", equally shriveled weights (",[130,4839,4840],{},"[-0.025, -0.066, 0.118]",[11,4842,4843,4844,4847,4848,4850,4851,4853,4854,4857],{},"The reason is geometric, and you can see it without any formula: XOR marks the two ",[15,4845,4846],{},"opposite"," diagonal corners (bottom-right and top-left) as ",[130,4849,1051],{}," and the other two corners, also opposite each other (bottom-left and top-right), as ",[130,4852,1665],{},". There's no line that separates \"the two corners on one diagonal\" from \"the two corners on the other diagonal\": any line I draw cuts through one of the diagonals, mixing both classes on either side. XOR is ",[15,4855,4856],{},"not linearly separable",", full stop, and no amount of training is going to change that, because the problem isn't the learning algorithm, it's what a single neuron, with a single straight boundary, is capable of representing at all.",[99,4859,4861],{"id":4860},"interactive-watch-the-perceptron-never-converge","Interactive: watch the perceptron never converge",[11,4863,4864,4865,4867],{},"Same seeded dataset from above, now on XOR. Click \"Process next point\" a bunch of times: notice that, unlike every other component in this playlist, this one ",[15,4866,3410],{}," shows the convergence message. There will always be at least one point on the wrong side, no matter how much I click.",[567,4869],{":classes":4870,":points":4479,":update-bias":1051,":x-max":571,":x-min":572,":y-max":571,":y-min":572,"converged-label":573,"negative-label":4871,"positive-label":4872,"reset-label":576,"step-label":577,"x-label":578,"y-label":579},"[1, 1, -1, 1, -1, 1, 1, 1, 1, -1, -1, -1, -1, 1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, -1, 1, -1, -1, 1, -1, 1, 1, 1, -1, 1, 1, -1, -1]","false (XOR)","true (XOR)",[11,4874,4875],{},"Feel free to click a lot. The boundary will keep rotating and sliding forever, trying to find a spot that doesn't exist.",[99,4877,4879],{"id":4878},"what-this-means-and-what-comes-next","What this means (and what comes next)",[11,4881,4882,4883,4886],{},"This result isn't an isolated curiosity, it's historically ",[15,4884,4885],{},"the"," limit that defined the early years of neural networks: a single neuron, with a linear decision boundary, has a ceiling on what it can express, and XOR sits above that ceiling. The way out, and this is where this playlist hasn't reached yet, is stacking neurons: an MLP (multi-layer network) can solve XOR by combining two linear boundaries in a hidden layer before deciding the final output. I saved two papers exactly for this moment, in this playlist's reference folder: Paul Werbos's doctoral thesis (1974), the earliest known derivation of the backpropagation algorithm, and Rumelhart, Hinton, and Williams's paper (Nature, 1986), the one that actually popularized the algorithm and unlocked practical MLP training.",[11,4888,4889,4890,4893],{},"The professor's repository doesn't have that notebook yet. The 3 lectures I covered here (perceptron, Adaline, cost functions, multiclass) are everything published as of right now, and the XOR cliffhanger is exactly where his course also currently stands. ",[20,4891,4892],{"href":72},"As I said back in this playlist's first post",", this is a living playlist: as soon as the professor publishes the MLP and backpropagation lecture, I'll come back here to continue where I left off, with Werbos and Rumelhart-Hinton-Williams already waiting to step in.",[99,4895,597],{"id":596},[599,4897,4898,4906],{},[602,4899,4900],{},[605,4901,4902,4904],{},[608,4903,611],{"align":610},[608,4905,614],{"align":610},[616,4907,4908,4916,4924],{},[605,4909,4910,4913],{},[621,4911,4912],{"align":610},"A perceptron separates anything given enough time",[621,4914,4915],{"align":610},"It only separates what's linearly separable: XOR never converges, no matter how much I train it",[605,4917,4918,4921],{},[621,4919,4920],{"align":610},"Weights grow when the model is wrong",[621,4922,4923],{"align":610},"On XOR, the weights shrink toward zero instead, the model gives up rather than keep chasing something that doesn't exist",[605,4925,4926,4929],{},[621,4927,4928],{"align":610},"A neural network is \"just\" one neuron with weights",[621,4930,4931],{"align":610},"A single neuron has an expressiveness ceiling: stacking layers (MLP) is what exists to break through it",[99,4933,646],{"id":645},[11,4935,4936,4937,4941],{},"There's no classic real dataset for \"XOR\" to go fetch (it's a logical function, not a real-world phenomenon), so the practical application here is showing the same linear-separability wall shows up in real data too, not just toy logic gates. I use the two-moons dataset ",[20,4938,4940],{"href":4939},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised","that already showed up in Pattern Recognition",", known to not be linearly separable.",[124,4943,4945],{"className":126,"code":4944,"language":128,"meta":76,"style":76},"from sklearn.datasets import make_moons\nX, y = make_moons(n_samples=300, noise=0.05, random_state=42)\n",[130,4946,4947,4952],{"__ignoreMap":76},[133,4948,4949],{"class":135,"line":136},[133,4950,4951],{},"from sklearn.datasets import make_moons\n",[133,4953,4954],{"class":135,"line":77},[133,4955,4956],{},"X, y = make_moons(n_samples=300, noise=0.05, random_state=42)\n",[599,4958,4959,4969],{},[602,4960,4961],{},[605,4962,4963,4966],{},[608,4964,4965],{"align":610},"Model",[608,4967,4968],{"align":709},"Accuracy",[616,4970,4971,4979],{},[605,4972,4973,4976],{},[621,4974,4975],{"align":610},"Perceptron (linear boundary)",[621,4977,4978],{"align":709},"0.867",[605,4980,4981,4989],{},[621,4982,4983,4984,4988],{"align":610},"KNN (K=5, ",[20,4985,4987],{"href":4986},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier","already seen in Pattern Recognition"," as a contrast)",[621,4990,4991],{"align":709},[15,4992,539],{},[11,4994,4995],{},"The perceptron doesn't get stuck as badly as on XOR (0.867 isn't 0.5, because the two moons have an approximately linear separation over most of the space, it's only the curvature at the tips that fools a straight line), but it lags well behind KNN, which has no such ceiling because it doesn't depend on a single straight boundary. It's the same structural limit as XOR, just in a softer version: any problem where the true boundary isn't a line will expose this limitation, sooner or later.",[740,4997,742],{},{"title":76,"searchDepth":77,"depth":77,"links":4999},[5000,5001,5002,5003,5004,5005,5006],{"id":4655,"depth":77,"text":4656},{"id":4722,"depth":77,"text":4723},{"id":4796,"depth":77,"text":4797},{"id":4860,"depth":77,"text":4861},{"id":4878,"depth":77,"text":4879},{"id":596,"depth":77,"text":597},{"id":645,"depth":77,"text":646},"Lecture 3d, the last one the professor has published so far: the same perceptron that learns OR and AND effortlessly gets stuck on XOR, pinned at 50% accuracy, weights collapsed to nearly zero. I close this first batch of the playlist exactly where the real course also stands: right at the threshold before the MLP.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron",{"title":4647,"description":5007},"en\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron",[5013,5014,5015],"xor","linear-separability","mlp","oQBoUTmug8QCiLbYCtWQrqgS5dp764djz4TYa0q3Daw",1787338982728]