[{"data":1,"prerenderedAt":2973},["ShallowReactive",2],{"lang-switch-post-\u002Fen":3,"home-playlists-en":4,"home-latest-posts-en":193,"home-latest-timeline-en":2894},null,[5,40,96,165],{"id":6,"title":7,"body":8,"cover":3,"description":30,"extension":31,"meta":32,"navigation":33,"order":34,"path":35,"seo":36,"status":37,"stem":38,"__hash__":39},"playlists\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex.md","Machine Learning Specialization (Andrew Ng)",{"type":9,"value":10,"toc":26},"minimark",[11,20,23],[12,13,14,15,19],"p",{},"This playlist is my public study notebook while going through the ",[16,17,18],"strong",{},"Machine Learning Specialization",", by Andrew Ng (DeepLearning.AI \u002F Stanford), probably the most recommended course for anyone getting started in ML.",[12,21,22],{},"The idea here isn't just \"solve the notebook and move on.\" Every lab in the course becomes a post where I retell what I understood, with an everyday-life metaphor, an example, and of course, the correct technical term, because you'll need it when you go looking for more on the topic later.",[12,24,25],{},"We start at the start: representing the simplest model there is, linear regression with one variable.",{"title":27,"searchDepth":28,"depth":28,"links":29},"",2,[],"My step-by-step notes going through Andrew Ng's Machine Learning Specialization (DeepLearning.AI \u002F Stanford), course by course, lab by lab.","md",{},true,1,"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization",{"title":7,"description":30},"published","en\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex","doASMuy-kKTfF67w2hQII9iS08pq0gbnf-FwE2AmImQ",{"id":41,"title":42,"body":43,"cover":3,"description":90,"extension":31,"meta":91,"navigation":33,"order":28,"path":92,"seo":93,"status":37,"stem":94,"__hash__":95},"playlists\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Findex.md","Pattern Recognition",{"type":9,"value":44,"toc":88},[45,58,70,80],[12,46,47,48,51,52,57],{},"This playlist is my study notebook from the Pattern Recognition course I took, taught by ",[16,49,50],{},"Dr. Francisco Boldt",". The guy is excellent, he genuinely codes the models by hand, live, in class, no pre-baked formula slides, and he's teaching me ",[53,54,56],"a",{"href":55},"\u002Fen\u002Fplaylists\u002Fneural-networks","neural networks now too",". If you landed here coming from one of his classes, you already know what I mean.",[12,59,60,61,64,65,69],{},"Unlike the ",[53,62,63],{"href":35},"Andrew Ng specialization playlist",", the lecture notebooks here are a lot leaner: barely any markdown cells, it's the professor live-coding and everyone following along. So the work of digging into the \"why\" behind each line of code is heavier here, and for that I lean on ",[66,67,68],"em",{},"Pattern Recognition and Machine Learning",", by Christopher Bishop (2006), pretty much a bible in the field, as the theoretical reference.",[12,71,72,73,79],{},"The notebooks come from the course repository, ",[53,74,78],{"href":75,"rel":76},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasml\u002Ftree\u002F2026-1",[77],"nofollow","pablobelmiro\u002Faulasml",", a fork of Dr. Boldt's own repository, where he publishes each lecture's code. In this playlist's posts, whoever \"wrote\" the code is always him, the professor; the foundational explanation, with the metaphor, the slightly-off analogy, and the buddy-sitting-next-to-you tone, that part is mine.",[12,81,82,83,87],{},"Every lecture becomes a post here. We start at the start: the same ",[53,84,86],{"href":85},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","linear regression problem from the other playlist",", except this time the code is being written live on the whiteboard.",{"title":27,"searchDepth":28,"depth":28,"links":89},[],"My notes from my Pattern Recognition course, lecture by lecture, with Bishop's book as the theoretical backbone.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition",{"title":42,"description":90},"en\u002Fplaylists\u002Fpattern-recognition\u002Findex","y_rvvoQ-pc7yj5eUqMw_dVQddrlu3I5aEN4sROdZOdk",{"id":97,"title":98,"body":99,"cover":3,"description":159,"extension":31,"meta":160,"navigation":33,"order":161,"path":55,"seo":162,"status":37,"stem":163,"__hash__":164},"playlists\u002Fen\u002Fplaylists\u002Fneural-networks\u002Findex.md","Neural Networks",{"type":9,"value":100,"toc":157},[101,110,127,141],[12,102,103,104,106,107,109],{},"Second playlist with ",[16,105,50],{}," (the first was ",[53,108,42],{"href":92},"), now in his Neural Networks course. Same style as always: code by hand, live, in class, no pre-baked formula slides. If you've already read the previous playlist, you know exactly what to expect.",[12,111,112,113,116,117,121,122,126],{},"The reference book changes: here I use ",[66,114,115],{},"Neural Networks and Deep Learning: A Textbook",", by Charu Aggarwal (2018), to play the role Bishop played in the previous playlist, filling in the foundation the notebook only shows in code. A lot of the material also connects straight back to what I've already covered: ",[53,118,120],{"href":119},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","the delta rule and linear regression already showed up in Pattern Recognition",", and ",[53,123,125],{"href":124},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions","loss functions already showed up in Andrew Ng's specialization",", so whenever it fits I'll pull those threads instead of reteaching from scratch.",[12,128,72,129,134,135,140],{},[53,130,133],{"href":131,"rel":132},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasann",[77],"pablobelmiro\u002Faulasann",", a fork of Dr. Boldt's own repository, ",[53,136,139],{"href":137,"rel":138},"https:\u002F\u002Fgithub.com\u002Ffboldt\u002Faulasann",[77],"fboldt\u002Faulasann",", where he publishes each lecture's code. Same convention as the previous playlist: whoever \"wrote\" the code is always him, the professor; the foundational explanation is mine.",[12,142,143,144,147,148,151,152,156],{},"One important detail this time: this course is being taught ",[16,145,146],{},"right now",", live, and the repository only has the beginning of the course as of this moment (perceptron, Adaline, cost functions, and a cliffhanger right at the edge of what a single neuron can solve). Unlike the Pattern Recognition playlist, which I only started once the whole course had already ended, this one is a ",[16,149,150],{},"living playlist",": it grows every time the professor publishes a new lecture, and I come back to keep going. We start at the very beginning: ",[53,153,155],{"href":154},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","the simplest neuron there is",".",{"title":27,"searchDepth":28,"depth":28,"links":158},[],"My notes from my Neural Networks course, lecture by lecture, with Aggarwal's book as the theoretical backbone. A living playlist, growing along with the course.",{},3,{"title":98,"description":159},"en\u002Fplaylists\u002Fneural-networks\u002Findex","fD14SmK9Haa_ztrp627bgiWBVDETZgSpdrWnwlC_LnU",{"id":166,"title":167,"body":168,"cover":3,"description":186,"extension":31,"meta":187,"navigation":33,"order":188,"path":189,"seo":190,"status":37,"stem":191,"__hash__":192},"playlists\u002Fen\u002Fplaylists\u002Fpapers\u002Findex.md","Papers",{"type":9,"value":169,"toc":184},[170,173,176],[12,171,172],{},"The other three playlists here are real study: lecture notebook, textbook on the side, every line of code picked apart until it hurts. This one is a different vibe. Every once in a while I read a paper or survey that gets me excited, usually skimmed diagonally, not read page by page with a magnifying glass, and I wanted a place to talk about it without all the ceremony of a lecture.",[12,174,175],{},"That's what this playlist is: I tell you what I found coolest about the paper, with a crooked metaphor, some joking around here and there, and whenever it fits, some simple Python code and a chart just to give you that view. Don't expect notebook-level rigor or section-by-section coverage, the goal here is a straight conversation about an interesting idea, not an academic summary.",[12,177,178,179,183],{},"First in the series is about a topic there's no escaping these days: ",[53,180,182],{"href":181},"\u002Fen\u002Fplaylists\u002Fpapers\u002Fagentic-reasoning-for-large-language-models","agentic reasoning in language models",", a giant survey trying to organize everything that became trendy to call an \"AI agent\".",{"title":27,"searchDepth":28,"depth":28,"links":185},[],"A diagonal read of articles and surveys I found cool, with no pretense of turning into a lecture. Simple code, a chart to give you a view, and the same conversation as always with you.",{},4,"\u002Fen\u002Fplaylists\u002Fpapers",{"title":167,"description":186},"en\u002Fplaylists\u002Fpapers\u002Findex","TwtxQvLC6nBRqD_S4n46i-MnkOFuDQKRx5oRX-hKPTo",[194,896,2291],{"id":195,"title":196,"body":197,"cover":3,"date":884,"description":885,"extension":31,"meta":886,"navigation":33,"order":188,"path":887,"playlist":888,"seo":889,"status":37,"stem":890,"tags":891,"__hash__":895},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta.md","Adaline: Train on the Line, Classify on the Sign",{"type":9,"value":198,"toc":876},[199,213,218,278,311,327,334,392,417,431,460,464,478,490,493,497,544,564,580,584,619,639,653,660,704,716,720,765,769,780,795,818,861,872],[12,200,201,202,205,206,209,210,156],{},"Lectures 2e and 2f, and the model's name changes to ",[16,203,204],{},"Adaline"," (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: ",[16,207,208],{},"what the model optimizes during training"," and ",[16,211,212],{},"what it computes at prediction time",[214,215,217],"h2",{"id":216},"pre-activation-and-post-activation-the-distinction-adaline-introduces","Pre-activation and post-activation: the distinction Adaline introduces",[219,220,224],"pre",{"className":221,"code":222,"language":223,"meta":27,"style":27},"language-python shiki shiki-themes github-light github-dark","class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n","python",[225,226,227,234,239,244,249,255,261,267,272],"code",{"__ignoreMap":27},[228,229,231],"span",{"class":230,"line":34},"line",[228,232,233],{},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n",[228,235,236],{"class":230,"line":28},[228,237,238],{},"  def fit(self, X, y):\n",[228,240,241],{"class":230,"line":161},[228,242,243],{},"    X = include_bias(X)\n",[228,245,246],{"class":230,"line":188},[228,247,248],{},"    self.w_ = np.linalg.pinv(X) @ y\n",[228,250,252],{"class":230,"line":251},5,[228,253,254],{},"    return self\n",[228,256,258],{"class":230,"line":257},6,[228,259,260],{"emptyLinePlaceholder":33},"\n",[228,262,264],{"class":230,"line":263},7,[228,265,266],{},"  def predict(self, X):\n",[228,268,270],{"class":230,"line":269},8,[228,271,243],{},[228,273,275],{"class":230,"line":274},9,[228,276,277],{},"    return X @ self.w_\n",[12,279,280,281,284,285,289,290,293,294,297,298,301,302,304,305,307,308,156],{},"Notice: this is ",[16,282,283],{},"exactly"," ",[53,286,288],{"href":287},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","last post's normal equation",", without changing a line, except now ",[225,291,292],{},"y"," is ",[225,295,296],{},"-1"," or ",[225,299,300],{},"+1"," instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to ",[225,303,296],{}," on one class's points and ",[225,306,300],{}," on the other's, never applying ",[225,309,310],{},"sign()",[312,313,314],"blockquote",{},[12,315,316,319,320,323,324,156],{},[16,317,318],{},"Output:"," RMSE ",[225,321,322],{},"0.4148",", weights ",[225,325,326],{},"[-1.76, 1.01, 2.74]",[12,328,329,330,333],{},"RMSE makes sense here because ",[225,331,332],{},"predict"," returns a continuous number, not a class. But to actually classify, there's one last step missing:",[219,335,337],{"className":221,"code":336,"language":223,"meta":27,"style":27},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def pre_activation(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n\n  def predict(self, X):\n    return np.sign(self.pre_activation(X))\n",[225,338,339,343,347,351,355,359,363,368,372,376,381,386],{"__ignoreMap":27},[228,340,341],{"class":230,"line":34},[228,342,233],{},[228,344,345],{"class":230,"line":28},[228,346,238],{},[228,348,349],{"class":230,"line":161},[228,350,243],{},[228,352,353],{"class":230,"line":188},[228,354,248],{},[228,356,357],{"class":230,"line":251},[228,358,254],{},[228,360,361],{"class":230,"line":257},[228,362,260],{"emptyLinePlaceholder":33},[228,364,365],{"class":230,"line":263},[228,366,367],{},"  def pre_activation(self, X):\n",[228,369,370],{"class":230,"line":269},[228,371,243],{},[228,373,374],{"class":230,"line":274},[228,375,277],{},[228,377,379],{"class":230,"line":378},10,[228,380,260],{"emptyLinePlaceholder":33},[228,382,384],{"class":230,"line":383},11,[228,385,266],{},[228,387,389],{"class":230,"line":388},12,[228,390,391],{},"    return np.sign(self.pre_activation(X))\n",[12,393,394,395,398,399,401,402,404,405,408,409,412,413,416],{},"Now there are two methods: ",[225,396,397],{},"pre_activation"," (the continuous value, before any threshold) and ",[225,400,332],{}," (applies ",[225,403,310],{}," on top). Aggarwal calls exactly these two things the ",[16,406,407],{},"pre-activation value"," and the ",[16,410,411],{},"post-activation value"," (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure \"how far off\" each prediction landed from the target) and only applies the activation (",[225,414,415],{},"sign",") when deciding the final class.",[312,418,419],{},[12,420,421,423,424,427,428,430],{},[16,422,318],{}," accuracy ",[225,425,426],{},"1.0",", the same ",[225,429,326],{}," weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).",[12,432,433,434,121,437,440,441,444,445,448,449,451,452,455,456,459],{},"This is the ",[16,435,436],{},"delta rule",[53,438,439],{"href":119},"I've already seen it before, under that exact name",": \"update the weight proportionally to the error times the input\" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered ",[53,442,443],{"href":154},"two posts ago",") is exactly this: the perceptron measures error ",[16,446,447],{},"after"," applying ",[225,450,310],{}," (error in ",[225,453,454],{},"{-2,0,+2}","), Adaline measures it ",[16,457,458],{},"before",", on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit \"unsure\". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of \"how wrong.\"",[214,461,463],{"id":462},"interactive-nudging-the-pre-activation-and-watching-rmse-and-accuracy-move","Interactive: nudging the pre-activation and watching RMSE (and accuracy) move",[12,465,466,467,470,471,121,474,477],{},"Instead of training automatically, drag the ",[225,468,469],{},"w0",", ",[225,472,473],{},"w1",[225,475,476],{},"bias"," sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).",[479,480],"adaline-explorer",{":classes":481,":points":482,":x-max":483,":x-min":484,":y-max":483,":y-min":484,"negative-label":485,"positive-label":486,"readout-label":487,"x-label":488,"y-label":489},"[-1, 1, -1, -1, -1, -1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, 1, -1]","[[0.0856, 0.2368], [0.8013, 0.5822], [0.0941, 0.4331], [0.4791, 0.1597], [0.7346, 0.1137], [0.3912, 0.5167], [0.4306, 0.5868], [0.7378, 0.9563], [0.2842, 0.6485], [0.6962, 0.2927], [0.0015, 0.9735], [0.2984, 0.314], [0.8917, 0.5852], [0.4713, 0.7733], [0.0303, 0.707], [0.3742, 0.0909], [0.6605, 0.9315], [0.2072, 0.6301], [0.2982, 0.7418], [0.7222, 0.2187]]","1","0","class -1","class +1","RMSE (continuous pre-activation vs. ±1 label): {rmse} · accuracy (post-activation, sign): {acc}%","x0","x1",[12,491,492],{},"Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between \"finding some line that separates\" (what the perceptron does) and \"finding the line that separates with room to spare\" (what you get by optimizing RMSE instead of just counting mistakes).",[214,494,496],{"id":495},"lecture-2f-the-same-computation-just-iterating-and-the-notebook-calls-it-sgd","Lecture 2f: the same computation, just iterating (and the notebook calls it \"SGD\")",[219,498,500],{"className":221,"code":499,"language":223,"meta":27,"style":27},"class Adaline(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.zeros(X.shape[1])\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      error = y - y_pred\n      self.w_ += self.learning_rate * error @ X\n    return self\n",[225,501,502,507,511,515,520,525,530,535,540],{"__ignoreMap":27},[228,503,504],{"class":230,"line":34},[228,505,506],{},"class Adaline(BaseEstimator, ClassifierMixin):\n",[228,508,509],{"class":230,"line":28},[228,510,238],{},[228,512,513],{"class":230,"line":161},[228,514,243],{},[228,516,517],{"class":230,"line":188},[228,518,519],{},"    self.w_ = np.zeros(X.shape[1])\n",[228,521,522],{"class":230,"line":251},[228,523,524],{},"    for _ in range(self.max_iter):\n",[228,526,527],{"class":230,"line":257},[228,528,529],{},"      y_pred = X @ self.w_\n",[228,531,532],{"class":230,"line":263},[228,533,534],{},"      error = y - y_pred\n",[228,536,537],{"class":230,"line":269},[228,538,539],{},"      self.w_ += self.learning_rate * error @ X\n",[228,541,542],{"class":230,"line":274},[228,543,254],{},[12,545,546,547,550,551,554,555,563],{},"One honest note about the notebook's name (",[225,548,549],{},"aula02f adaline with SGD","): the code shown here is ",[16,552,553],{},"batch gradient descent"," (",[53,556,558,559,562],{"href":557},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias","the same ",[225,560,561],{},"X.T @ error"," as always","), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking (\"it's kind of like gradient descent, so I call it SGD\"), but the technical difference is worth noting, since the names carry precise meaning.",[312,565,566],{},[12,567,568,423,570,572,573,576,577,156],{},[16,569,318],{},[225,571,426],{}," on training, weights ",[225,574,575],{},"[-2.92, 3.32, 2.63]",". Tested on 1000 new points: accuracy ",[16,578,579],{},"0.953",[214,581,583],{"id":582},"the-bad-dataset-what-it-actually-proves","The \"bad\" dataset: what it actually proves",[219,585,587],{"className":221,"code":586,"language":223,"meta":27,"style":27},"X_bad = np.concatenate((X,np.ones_like(X)))\ny_bad = np.concatenate((y,np.ones_like(y)))\nX_bad = np.concatenate((X_bad,np.ones_like(X)))\ny_bad = np.concatenate((y_bad,np.ones_like(y)))\nclf_bad = Adaline()\nclf_bad.fit(X_bad, y_bad)\n",[225,588,589,594,599,604,609,614],{"__ignoreMap":27},[228,590,591],{"class":230,"line":34},[228,592,593],{},"X_bad = np.concatenate((X,np.ones_like(X)))\n",[228,595,596],{"class":230,"line":28},[228,597,598],{},"y_bad = np.concatenate((y,np.ones_like(y)))\n",[228,600,601],{"class":230,"line":161},[228,602,603],{},"X_bad = np.concatenate((X_bad,np.ones_like(X)))\n",[228,605,606],{"class":230,"line":188},[228,607,608],{},"y_bad = np.concatenate((y_bad,np.ones_like(y)))\n",[228,610,611],{"class":230,"line":251},[228,612,613],{},"clf_bad = Adaline()\n",[228,615,616],{"class":230,"line":257},[228,617,618],{},"clf_bad.fit(X_bad, y_bad)\n",[12,620,621,622,625,626,629,630,632,633,635,636,638],{},"The professor concatenates the original dataset with ",[16,623,624],{},"two extra blocks"," of artificial points: everyone at ",[225,627,628],{},"(1,1)",", everyone labeled ",[225,631,300],{},". This isn't noise, it's a deliberate bias, nudging training to \"believe\" the region near ",[225,634,628],{}," is even more strongly class ",[225,637,300],{}," than it really is.",[312,640,641],{},[12,642,643,423,645,648,649,652],{},[16,644,318],{},[225,646,647],{},"0.967"," on training (over the biased dataset), but only ",[16,650,651],{},"0.811"," on the same 1000 clean test points as before.",[12,654,655,656,659],{},"Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent ",[16,657,658],{},"on the same biased dataset",", and that hypothesis didn't hold up:",[661,662,663,681],"table",{},[664,665,666],"thead",{},[667,668,669,674,678],"tr",{},[670,671,673],"th",{"align":672},"left","Method",[670,675,677],{"align":676},"right","Train accuracy (biased)",[670,679,680],{"align":676},"Test accuracy (clean)",[682,683,684,695],"tbody",{},[667,685,686,690,692],{},[687,688,689],"td",{"align":672},"Pseudo-inverse",[687,691,647],{"align":676},[687,693,694],{"align":676},"0.783",[667,696,697,700,702],{},[687,698,699],{"align":672},"Batch gradient descent",[687,701,647],{"align":676},[687,703,694],{"align":676},[12,705,706,707,710,711,715],{},"Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about ",[16,708,709],{},"training data quality",": biasing the training distribution (even without adding \"noise\" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. ",[53,712,714],{"href":713},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","This echoes a lesson I already saw in the other playlist",": training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.",[214,717,719],{"id":718},"wrapping-up","Wrapping up",[661,721,722,732],{},[664,723,724],{},[667,725,726,729],{},[670,727,728],{"align":672},"What I already knew",[670,730,731],{"align":672},"What this lecture settled",[682,733,734,745,757],{},[667,735,736,739],{},[687,737,738],{"align":672},"The perceptron classifies and updates on the binary error",[687,740,741,742,744],{"align":672},"Adaline separates pre-activation (continuous, used in training) from post-activation (",[225,743,415],{},", used only to decide the final class)",[667,746,747,750],{},[687,748,749],{"align":672},"The delta rule already showed up in the other playlist",[687,751,752,753,756],{"align":672},"Adaline ",[16,754,755],{},"is"," the delta rule applied to classification, training as if it were regression on the ±1 labels",[667,758,759,762],{},[687,760,761],{"align":672},"Finding a line that separates seems like enough",[687,763,764],{"align":672},"Optimizing RMSE (not just counting mistakes) keeps finding a better boundary even after accuracy already hit 100%",[214,766,768],{"id":767},"practical-application","Practical application",[12,770,771,772,775,776,779],{},"I reproduced pseudo-inverse vs. batch gradient descent on Iris (",[66,773,774],{},"setosa"," vs. ",[66,777,778],{},"versicolor","), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.",[219,781,783],{"className":221,"code":782,"language":223,"meta":27,"style":27},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\nw_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[225,784,785,790],{"__ignoreMap":27},[228,786,787],{"class":230,"line":34},[228,788,789],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[228,791,792],{"class":230,"line":28},[228,793,794],{},"w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[12,796,797,798,801,802,805,806,809,810,813,814,817],{},"On the first attempt, I used the notebook's original ",[225,799,800],{},"learning_rate=0.01"," for batch gradient descent, and it ",[16,803,804],{},"diverged"," (weight became ",[225,807,808],{},"NaN",") straight away on raw Iris data. No surprise at this point: ",[53,811,812],{"href":287},"it's the same scale lesson from the normal equation post",", just rediscovered again, this time needing a much smaller rate (",[225,815,816],{},"0.001",") to not blow up.",[661,819,820,832],{},[664,821,822],{},[667,823,824,826,829],{},[670,825,673],{"align":672},[670,827,828],{"align":676},"RMSE (train)",[670,830,831],{"align":676},"Accuracy (test)",[682,833,834,846],{},[667,835,836,839,842],{},[687,837,838],{"align":672},"Pseudo-inverse (Adaline)",[687,840,841],{"align":676},"0.2414",[687,843,844],{"align":676},[16,845,426],{},[667,847,848,855,857],{},[687,849,850,851,854],{"align":672},"Batch gradient descent, ",[225,852,853],{},"learning_rate=0.001"," (Adaline)",[687,856,841],{"align":676},[687,858,859],{"align":676},[16,860,426],{},[12,862,863,864,867,868,871],{},"With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, ",[53,865,866],{"href":557},"the same generous margin from Iris that already favored the perceptron with bias",". The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of ",[225,869,870],{},"learning_rate","), not the quality of the final solution.",[873,874,875],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":27,"searchDepth":28,"depth":28,"links":877},[878,879,880,881,882,883],{"id":216,"depth":28,"text":217},{"id":462,"depth":28,"text":463},{"id":495,"depth":28,"text":496},{"id":582,"depth":28,"text":583},{"id":718,"depth":28,"text":719},{"id":767,"depth":28,"text":768},"2026-08-20","Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta","neural-networks",{"title":196,"description":885},"en\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",[892,893,894],"adaline","delta-rule","widrow-hoff","tDtwjGKmRJJOKzWjkAyQ72-9DNH1O8sc9YdTGBFjxlk",{"id":897,"title":898,"body":899,"cover":3,"date":884,"description":2281,"extension":31,"meta":2282,"navigation":33,"order":251,"path":2283,"playlist":888,"seo":2284,"status":37,"stem":2285,"tags":2286,"__hash__":2290},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo.md","Swapping the Cost Function Like Changing Clothes",{"type":9,"value":900,"toc":2273},[901,908,912,1021,1053,1079,1086,1090,1163,1194,1198,1234,1470,1506,1519,1540,1595,1626,1642,1923,1961,1972,1987,1991,2091,2100,2117,2119,2171,2173,2187,2263,2271],[12,902,903,904,907],{},"Lectures 3a and 3b. So far I've seen 4 different algorithms (perceptron, vectorized perceptron, Adaline via pseudo-inverse, Adaline via gradient) as somewhat separate things. This lecture shows they're actually the ",[16,905,906],{},"same recipe",", just swapping one ingredient: the cost function.",[214,909,911],{"id":910},"first-the-training-algorithm-becomes-pluggable","First, the training algorithm becomes pluggable",[219,913,915],{"className":221,"code":914,"language":223,"meta":27,"style":27},"class TrainingAlgorithm(ABC):\n  @abstractmethod\n  def get_w(self, X, y):\n    pass\n\nclass PseudoInverse(TrainingAlgorithm):\n  def get_w(self, X, y):\n    return np.linalg.pinv(X) @ y\n\nclass NeuralNetwork(BaseEstimator, ClassifierMixin):\n  def __init__(self, training_algorithm=PseudoInverse()):\n    self.training_algorithm = training_algorithm\n\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = self.training_algorithm.get_w(X, y)\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return np.sign(X @ self.w_)\n",[225,916,917,922,927,932,937,941,946,950,955,959,964,969,974,979,984,989,995,1000,1005,1010,1015],{"__ignoreMap":27},[228,918,919],{"class":230,"line":34},[228,920,921],{},"class TrainingAlgorithm(ABC):\n",[228,923,924],{"class":230,"line":28},[228,925,926],{},"  @abstractmethod\n",[228,928,929],{"class":230,"line":161},[228,930,931],{},"  def get_w(self, X, y):\n",[228,933,934],{"class":230,"line":188},[228,935,936],{},"    pass\n",[228,938,939],{"class":230,"line":251},[228,940,260],{"emptyLinePlaceholder":33},[228,942,943],{"class":230,"line":257},[228,944,945],{},"class PseudoInverse(TrainingAlgorithm):\n",[228,947,948],{"class":230,"line":263},[228,949,931],{},[228,951,952],{"class":230,"line":269},[228,953,954],{},"    return np.linalg.pinv(X) @ y\n",[228,956,957],{"class":230,"line":274},[228,958,260],{"emptyLinePlaceholder":33},[228,960,961],{"class":230,"line":378},[228,962,963],{},"class NeuralNetwork(BaseEstimator, ClassifierMixin):\n",[228,965,966],{"class":230,"line":383},[228,967,968],{},"  def __init__(self, training_algorithm=PseudoInverse()):\n",[228,970,971],{"class":230,"line":388},[228,972,973],{},"    self.training_algorithm = training_algorithm\n",[228,975,977],{"class":230,"line":976},13,[228,978,260],{"emptyLinePlaceholder":33},[228,980,982],{"class":230,"line":981},14,[228,983,238],{},[228,985,987],{"class":230,"line":986},15,[228,988,243],{},[228,990,992],{"class":230,"line":991},16,[228,993,994],{},"    self.w_ = self.training_algorithm.get_w(X, y)\n",[228,996,998],{"class":230,"line":997},17,[228,999,254],{},[228,1001,1003],{"class":230,"line":1002},18,[228,1004,260],{"emptyLinePlaceholder":33},[228,1006,1008],{"class":230,"line":1007},19,[228,1009,266],{},[228,1011,1013],{"class":230,"line":1012},20,[228,1014,243],{},[228,1016,1018],{"class":230,"line":1017},21,[228,1019,1020],{},"    return np.sign(X @ self.w_)\n",[12,1022,433,1023,1026,1027,1030,1031,1034,1035,1038,1039,1042,1043,1046,1047,1050,1051,156],{},[16,1024,1025],{},"Strategy"," design pattern: ",[225,1028,1029],{},"NeuralNetwork"," no longer knows ",[16,1032,1033],{},"how"," the weights get computed, just that there's a ",[225,1036,1037],{},"training_algorithm"," object with a ",[225,1040,1041],{},"get_w"," method. Swapping ",[225,1044,1045],{},"PseudoInverse()"," for ",[225,1048,1049],{},"SGD()"," in the constructor swaps out the entire training algorithm, without touching ",[225,1052,1029],{},[312,1054,1055],{},[12,1056,1057,423,1064,323,1067,1070,1071,423,1076,1078],{},[16,1058,1059,1060,1063],{},"Output (",[225,1061,1062],{},"PseudoInverse","):",[225,1065,1066],{},"0.95",[225,1068,1069],{},"[-1.905, 2.656, 1.049]",".\n",[16,1072,1059,1073,1063],{},[225,1074,1075],{},"SGD",[225,1077,1066],{},", essentially identical weights.",[12,1080,1081,1082,1085],{},"Confirms again, now with more cleanly organized code, what ",[53,1083,1084],{"href":887},"I already saw in the last two posts",": pseudo-inverse and gradient descent solve the exact same problem.",[214,1087,1089],{"id":1088},"now-the-cost-function-becomes-pluggable-too","Now the cost function becomes pluggable too",[219,1091,1093],{"className":221,"code":1092,"language":223,"meta":27,"style":27},"class CostFunction(ABC):\n  @abstractstaticmethod\n  def get_cost(y, y_pred):\n    pass\n  @abstractstaticmethod\n  def get_gradient(X, y, y_pred):\n    pass\n\nclass WidrowHoff(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.mean((y-y_pred)**2)\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y-y_pred)\n",[225,1094,1095,1100,1105,1110,1114,1118,1123,1127,1131,1136,1141,1145,1150,1154,1158],{"__ignoreMap":27},[228,1096,1097],{"class":230,"line":34},[228,1098,1099],{},"class CostFunction(ABC):\n",[228,1101,1102],{"class":230,"line":28},[228,1103,1104],{},"  @abstractstaticmethod\n",[228,1106,1107],{"class":230,"line":161},[228,1108,1109],{},"  def get_cost(y, y_pred):\n",[228,1111,1112],{"class":230,"line":188},[228,1113,936],{},[228,1115,1116],{"class":230,"line":251},[228,1117,1104],{},[228,1119,1120],{"class":230,"line":257},[228,1121,1122],{},"  def get_gradient(X, y, y_pred):\n",[228,1124,1125],{"class":230,"line":263},[228,1126,936],{},[228,1128,1129],{"class":230,"line":269},[228,1130,260],{"emptyLinePlaceholder":33},[228,1132,1133],{"class":230,"line":274},[228,1134,1135],{},"class WidrowHoff(CostFunction):\n",[228,1137,1138],{"class":230,"line":378},[228,1139,1140],{},"  @staticmethod\n",[228,1142,1143],{"class":230,"line":383},[228,1144,1109],{},[228,1146,1147],{"class":230,"line":388},[228,1148,1149],{},"    return np.mean((y-y_pred)**2)\n",[228,1151,1152],{"class":230,"line":976},[228,1153,1140],{},[228,1155,1156],{"class":230,"line":981},[228,1157,1122],{},[228,1159,1160],{"class":230,"line":986},[228,1161,1162],{},"    return X.T @ (y-y_pred)\n",[12,1164,1165,1167,1168,1171,1172,1175,1176,293,1179,1182,1183,1186,1187,1189,1190,1193],{},[225,1166,1075],{}," now also takes a ",[225,1169,1170],{},"cost_function",", and uses ",[225,1173,1174],{},"self.cost_function.get_gradient(...)"," instead of computing the gradient by hand. ",[225,1177,1178],{},"WidrowHoff",[53,1180,1181],{"href":887},"exactly the delta rule from the last post",": continuous error (",[225,1184,1185],{},"y - y_pred",", no ",[225,1188,310],{},") times the input. Swapping the cost function here means swapping ",[16,1191,1192],{},"what \"error\" means",", without touching the training loop.",[214,1195,1197],{"id":1196},"every-cost-function-recovers-a-different-algorithm","Every cost function recovers a different algorithm",[219,1199,1201],{"className":221,"code":1200,"language":223,"meta":27,"style":27},"class SmoothedSurrogate(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - np.sign(y_pred))\n",[225,1202,1203,1208,1212,1216,1221,1225,1229],{"__ignoreMap":27},[228,1204,1205],{"class":230,"line":34},[228,1206,1207],{},"class SmoothedSurrogate(CostFunction):\n",[228,1209,1210],{"class":230,"line":28},[228,1211,1140],{},[228,1213,1214],{"class":230,"line":161},[228,1215,1109],{},[228,1217,1218],{"class":230,"line":188},[228,1219,1220],{},"    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n",[228,1222,1223],{"class":230,"line":251},[228,1224,1140],{},[228,1226,1227],{"class":230,"line":257},[228,1228,1122],{},[228,1230,1231],{"class":230,"line":263},[228,1232,1233],{},"    return X.T @ (y - np.sign(y_pred))\n",[12,1235,1236,1237,1240,1241,1243,1244,1247,1248,1251,1252,1255,1256,1463,1464,1466,1467,1469],{},"Notice the ",[225,1238,1239],{},"np.sign(y_pred)"," inside the gradient: this goes back to measuring error ",[16,1242,447],{}," the threshold, exactly like ",[53,1245,1246],{"href":154},"Rosenblatt's perceptron",". The name ",[225,1249,1250],{},"SmoothedSurrogate"," matches what Aggarwal calls the ",[16,1253,1254],{},"perceptron criterion",": ",[228,1257,1260,1323],{"className":1258},[1259],"katex",[228,1261,1264],{"className":1262},[1263],"katex-mathml",[1265,1266,1268],"math",{"xmlns":1267},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[1269,1270,1271,1318],"semantics",{},[1272,1273,1274,1278,1282,1285,1288,1292,1295,1299,1302,1304,1307,1315],"mrow",{},[1275,1276,1277],"mi",{},"L",[1279,1280,1281],"mo",{},"=",[1275,1283,1284],{},"max",[1279,1286,1287],{},"⁡",[1279,1289,1291],{"stretchy":1290},"false","(",[1293,1294,484],"mn",{},[1279,1296,1298],{"separator":1297},"true",",",[1279,1300,1301],{},"−",[1275,1303,292],{},[1279,1305,1306],{},"⋅",[1308,1309,1310,1312],"mover",{"accent":1297},[1275,1311,292],{},[1279,1313,1314],{},"^",[1279,1316,1317],{"stretchy":1290},")",[1319,1320,1322],"annotation",{"encoding":1321},"application\u002Fx-tex","L = \\max(0, -y \\cdot \\hat{y})",[228,1324,1327,1353,1397],{"className":1325,"ariaHidden":1297},[1326],"katex-html",[228,1328,1331,1336,1341,1346,1350],{"className":1329},[1330],"base",[228,1332],{"className":1333,"style":1335},[1334],"strut","height:0.6833em;",[228,1337,1277],{"className":1338},[1339,1340],"mord","mathnormal",[228,1342],{"className":1343,"style":1345},[1344],"mspace","margin-right:0.2778em;",[228,1347,1281],{"className":1348},[1349],"mrel",[228,1351],{"className":1352,"style":1345},[1344],[228,1354,1356,1360,1364,1368,1371,1375,1379,1382,1386,1390,1394],{"className":1355},[1330],[228,1357],{"className":1358,"style":1359},[1334],"height:1em;vertical-align:-0.25em;",[228,1361,1284],{"className":1362},[1363],"mop",[228,1365,1291],{"className":1366},[1367],"mopen",[228,1369,484],{"className":1370},[1339],[228,1372,1298],{"className":1373},[1374],"mpunct",[228,1376],{"className":1377,"style":1378},[1344],"margin-right:0.1667em;",[228,1380,1301],{"className":1381},[1339],[228,1383,292],{"className":1384,"style":1385},[1339,1340],"margin-right:0.0359em;",[228,1387],{"className":1388,"style":1389},[1344],"margin-right:0.2222em;",[228,1391,1306],{"className":1392},[1393],"mbin",[228,1395],{"className":1396,"style":1389},[1344],[228,1398,1400,1403,1459],{"className":1399},[1330],[228,1401],{"className":1402,"style":1359},[1334],[228,1404,1407],{"className":1405},[1339,1406],"accent",[228,1408,1412,1450],{"className":1409},[1410,1411],"vlist-t","vlist-t2",[228,1413,1416,1445],{"className":1414},[1415],"vlist-r",[228,1417,1421,1432],{"className":1418,"style":1420},[1419],"vlist","height:0.6944em;",[228,1422,1424,1429],{"style":1423},"top:-3em;",[228,1425],{"className":1426,"style":1428},[1427],"pstrut","height:3em;",[228,1430,292],{"className":1431,"style":1385},[1339,1340],[228,1433,1434,1437],{"style":1423},[228,1435],{"className":1436,"style":1428},[1427],[228,1438,1442],{"className":1439,"style":1441},[1440],"accent-body","left:-0.1944em;",[228,1443,1314],{"className":1444},[1339],[228,1446,1449],{"className":1447},[1448],"vlist-s","​",[228,1451,1453],{"className":1452},[1415],[228,1454,1457],{"className":1455,"style":1456},[1419],"height:0.1944em;",[228,1458],{},[228,1460,1317],{"className":1461},[1462],"mclose",", zero once the point is on the right side, growing linearly when it's wrong. No coincidence the accuracy hits ",[225,1465,426],{},": this cost function, plugged into this generic framework, ",[16,1468,755],{}," the original perceptron again, just expressed in the language of \"cost function\" instead of \"update rule\".",[219,1471,1473],{"className":221,"code":1472,"language":223,"meta":27,"style":27},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - expit(y_pred))\n",[225,1474,1475,1480,1484,1488,1493,1497,1501],{"__ignoreMap":27},[228,1476,1477],{"class":230,"line":34},[228,1478,1479],{},"class LogLikehood(CostFunction):\n",[228,1481,1482],{"class":230,"line":28},[228,1483,1140],{},[228,1485,1486],{"class":230,"line":161},[228,1487,1109],{},[228,1489,1490],{"class":230,"line":188},[228,1491,1492],{},"    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n",[228,1494,1495],{"class":230,"line":251},[228,1496,1140],{},[228,1498,1499],{"class":230,"line":257},[228,1500,1122],{},[228,1502,1503],{"class":230,"line":263},[228,1504,1505],{},"    return X.T @ (y - expit(y_pred))\n",[312,1507,1508],{},[12,1509,1510,423,1512,323,1515,1518],{},[16,1511,318],{},[16,1513,1514],{},"0.65",[225,1516,1517],{},"[-60.78, 27.44, -24.79]",". Much worse than anything I've seen so far, and the weights got huge.",[12,1520,1521,1522,1525,1526,1529,1530,1533,1534,1536,1537,1539],{},"Two things wrong here, worth separating. First, a detail that doesn't affect the outcome: this class's ",[225,1523,1524],{},"get_cost"," uses the hinge loss formula (",[225,1527,1528],{},"max(0, 1 - y·ŷ)","), not an actual log-likelihood formula. That doesn't break anything in practice because ",[225,1531,1532],{},"get_gradient"," is the only thing ",[225,1535,1075],{}," calls, ",[225,1538,1524],{}," never gets used during training, it's leftover residue from copying and pasting from another cell.",[12,1541,1542,1543,1546,1547,209,1549,1551,1552,293,1554,297,1556,1558,1559,1562,1563,1566,1567,1570,1571,1573,1574,1577,1578,1580,1581,1583,1584,1587,1588,1591,1592,1594],{},"The second problem is real, and explains the bad accuracy: ",[225,1544,1545],{},"expit"," (the sigmoid function) only returns values between ",[225,1548,484],{},[225,1550,483],{},", but the label ",[225,1553,292],{},[225,1555,296],{},[225,1557,300],{},". For class ",[225,1560,1561],{},"y=-1",", the error ",[225,1564,1565],{},"y - expit(y_pred)"," can ",[16,1568,1569],{},"never"," get close to zero, because ",[225,1572,1545],{}," never goes negative: even with an infinitely confident correctly-classified prediction, ",[225,1575,1576],{},"-1 - expit(y_pred)"," stays pinned near ",[225,1579,296],{},", never ",[225,1582,484],{},". I checked this by hand: ",[225,1585,1586],{},"expit(-1000) = 0.0",", so ",[225,1589,1590],{},"-1 - expit(-1000) = -1.0"," exactly, not ",[225,1593,484],{},". The gradient for half the points never vanishes, so training never settles, and the weights keep growing trying to compensate for an error that's structurally impossible to zero out.",[219,1596,1598],{"className":221,"code":1597,"language":223,"meta":27,"style":27},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - tanh(y_pred))\n\nmodel = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[225,1599,1600,1604,1608,1612,1617,1621],{"__ignoreMap":27},[228,1601,1602],{"class":230,"line":34},[228,1603,1479],{},[228,1605,1606],{"class":230,"line":28},[228,1607,1140],{},[228,1609,1610],{"class":230,"line":161},[228,1611,1122],{},[228,1613,1614],{"class":230,"line":188},[228,1615,1616],{},"    return X.T @ (y - tanh(y_pred))\n",[228,1618,1619],{"class":230,"line":251},[228,1620,260],{"emptyLinePlaceholder":33},[228,1622,1623],{"class":230,"line":257},[228,1624,1625],{},"model = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[312,1627,1628],{},[12,1629,1630,423,1632,323,1634,1637,1638,1641],{},[16,1631,318],{},[16,1633,426],{},[225,1635,1636],{},"[-9.84, 12.40, 8.37]"," (with ",[225,1639,1640],{},"max_iter=10000",", ten times more iterations).",[12,1643,1644,1645,1046,1647,1650,1651,1653,1654,1656,1657,1659,1660,1587,1663,1666,1667,1808,1809,1656,1861,1918,1919,1922],{},"Swapping ",[225,1646,1545],{},[225,1648,1649],{},"tanh"," (same class redefined, Python just lets that run live in a notebook session), the problem disappears. Makes sense: ",[225,1652,1649],{}," ranges from ",[225,1655,296],{}," to ",[225,1658,300],{},", exactly the labels' range. Checked it by hand again: ",[225,1661,1662],{},"tanh(-1000) = -1.0",[225,1664,1665],{},"-1 - tanh(-1000) = 0.0",", the error genuinely reaches zero this time. Aggarwal states this relationship directly in chapter 1: ",[228,1668,1670,1716],{"className":1669},[1259],[228,1671,1673],{"className":1672},[1263],[1265,1674,1675],{"xmlns":1267},[1269,1676,1677,1713],{},[1272,1678,1679,1681,1683,1685,1688,1690,1692,1695,1697,1701,1703,1705,1707,1709,1711],{},[1275,1680,1649],{},[1279,1682,1287],{},[1279,1684,1291],{"stretchy":1290},[1275,1686,1687],{},"v",[1279,1689,1317],{"stretchy":1290},[1279,1691,1281],{},[1293,1693,1694],{},"2",[1279,1696,1306],{},[1698,1699,1700],"mtext",{},"sigmoid",[1279,1702,1291],{"stretchy":1290},[1293,1704,1694],{},[1275,1706,1687],{},[1279,1708,1317],{"stretchy":1290},[1279,1710,1301],{},[1293,1712,483],{},[1319,1714,1715],{"encoding":1321},"\\tanh(v) = 2 \\cdot \\text{sigmoid}(2v) - 1",[228,1717,1719,1746,1765,1799],{"className":1718,"ariaHidden":1297},[1326],[228,1720,1722,1725,1728,1731,1734,1737,1740,1743],{"className":1721},[1330],[228,1723],{"className":1724,"style":1359},[1334],[228,1726,1649],{"className":1727},[1363],[228,1729,1291],{"className":1730},[1367],[228,1732,1687],{"className":1733,"style":1385},[1339,1340],[228,1735,1317],{"className":1736},[1462],[228,1738],{"className":1739,"style":1345},[1344],[228,1741,1281],{"className":1742},[1349],[228,1744],{"className":1745,"style":1345},[1344],[228,1747,1749,1753,1756,1759,1762],{"className":1748},[1330],[228,1750],{"className":1751,"style":1752},[1334],"height:0.6444em;",[228,1754,1694],{"className":1755},[1339],[228,1757],{"className":1758,"style":1389},[1344],[228,1760,1306],{"className":1761},[1393],[228,1763],{"className":1764,"style":1389},[1344],[228,1766,1768,1771,1778,1781,1784,1787,1790,1793,1796],{"className":1767},[1330],[228,1769],{"className":1770,"style":1359},[1334],[228,1772,1775],{"className":1773},[1339,1774],"text",[228,1776,1700],{"className":1777},[1339],[228,1779,1291],{"className":1780},[1367],[228,1782,1694],{"className":1783},[1339],[228,1785,1687],{"className":1786,"style":1385},[1339,1340],[228,1788,1317],{"className":1789},[1462],[228,1791],{"className":1792,"style":1389},[1344],[228,1794,1301],{"className":1795},[1393],[228,1797],{"className":1798,"style":1389},[1344],[228,1800,1802,1805],{"className":1801},[1330],[228,1803],{"className":1804,"style":1752},[1334],[228,1806,483],{"className":1807},[1339],", and it's exactly that range shift, from ",[228,1810,1812,1834],{"className":1811},[1259],[228,1813,1815],{"className":1814},[1263],[1265,1816,1817],{"xmlns":1267},[1269,1818,1819,1831],{},[1272,1820,1821,1823,1825,1827,1829],{},[1279,1822,1291],{"stretchy":1290},[1293,1824,484],{},[1279,1826,1298],{"separator":1297},[1293,1828,483],{},[1279,1830,1317],{"stretchy":1290},[1319,1832,1833],{"encoding":1321},"(0,1)",[228,1835,1837],{"className":1836,"ariaHidden":1297},[1326],[228,1838,1840,1843,1846,1849,1852,1855,1858],{"className":1839},[1330],[228,1841],{"className":1842,"style":1359},[1334],[228,1844,1291],{"className":1845},[1367],[228,1847,484],{"className":1848},[1339],[228,1850,1298],{"className":1851},[1374],[228,1853],{"className":1854,"style":1378},[1344],[228,1856,483],{"className":1857},[1339],[228,1859,1317],{"className":1860},[1462],[228,1862,1864,1888],{"className":1863},[1259],[228,1865,1867],{"className":1866},[1263],[1265,1868,1869],{"xmlns":1267},[1269,1870,1871,1885],{},[1272,1872,1873,1875,1877,1879,1881,1883],{},[1279,1874,1291],{"stretchy":1290},[1279,1876,1301],{},[1293,1878,483],{},[1279,1880,1298],{"separator":1297},[1293,1882,483],{},[1279,1884,1317],{"stretchy":1290},[1319,1886,1887],{"encoding":1321},"(-1,1)",[228,1889,1891],{"className":1890,"ariaHidden":1297},[1326],[228,1892,1894,1897,1900,1903,1906,1909,1912,1915],{"className":1893},[1330],[228,1895],{"className":1896,"style":1359},[1334],[228,1898,1291],{"className":1899},[1367],[228,1901,1301],{"className":1902},[1339],[228,1904,483],{"className":1905},[1339],[228,1907,1298],{"className":1908},[1374],[228,1910],{"className":1911,"style":1378},[1344],[228,1913,483],{"className":1914},[1339],[228,1916,1317],{"className":1917},[1462],", that fixes the mismatch with ",[225,1920,1921],{},"±1"," labels.",[219,1924,1926],{"className":221,"code":1925,"language":223,"meta":27,"style":27},"class HingeLoss(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    marginal_errors = (y * y_pred) \u003C 1\n    marginal_ys = np.copy(y)\n    marginal_ys[~marginal_errors] = 0\n    return X.T @ marginal_ys\n",[225,1927,1928,1933,1937,1941,1946,1951,1956],{"__ignoreMap":27},[228,1929,1930],{"class":230,"line":34},[228,1931,1932],{},"class HingeLoss(CostFunction):\n",[228,1934,1935],{"class":230,"line":28},[228,1936,1140],{},[228,1938,1939],{"class":230,"line":161},[228,1940,1122],{},[228,1942,1943],{"class":230,"line":188},[228,1944,1945],{},"    marginal_errors = (y * y_pred) \u003C 1\n",[228,1947,1948],{"class":230,"line":251},[228,1949,1950],{},"    marginal_ys = np.copy(y)\n",[228,1952,1953],{"class":230,"line":257},[228,1954,1955],{},"    marginal_ys[~marginal_errors] = 0\n",[228,1957,1958],{"class":230,"line":263},[228,1959,1960],{},"    return X.T @ marginal_ys\n",[312,1962,1963],{},[12,1964,1965,423,1967,323,1969,156],{},[16,1966,318],{},[16,1968,426],{},[225,1970,1971],{},"[-8.70, 11.59, 7.71]",[12,1973,1974,1975,1978,1979,1982,1983,1986],{},"Notice ",[225,1976,1977],{},"marginal_ys[~marginal_errors] = 0",": points that are already well classified, with room to spare (",[225,1980,1981],{},"y · ŷ ≥ 1","), get zeroed out and ",[16,1984,1985],{},"contribute nothing"," to the gradient. Only points inside the margin (or misclassified) participate in the update. That's literally SVM's central idea: only the points near the boundary (the \"support vectors\") matter for deciding where it sits. The rest of the dataset gets ignored once it's already well separated.",[214,1988,1990],{"id":1989},"all-four-curves-side-by-side","All four curves, side by side",[12,1992,1993,1994,2090],{},"The professor has a reference image saved in the notebook comparing the four penalty curves as a function of the \"margin\" (",[228,1995,1997,2019],{"className":1996},[1259],[228,1998,2000],{"className":1999},[1263],[1265,2001,2002],{"xmlns":1267},[1269,2003,2004,2016],{},[1272,2005,2006,2008,2010],{},[1275,2007,292],{},[1279,2009,1306],{},[1308,2011,2012,2014],{"accent":1297},[1275,2013,292],{},[1279,2015,1314],{},[1319,2017,2018],{"encoding":1321},"y \\cdot \\hat{y}",[228,2020,2022,2041],{"className":2021,"ariaHidden":1297},[1326],[228,2023,2025,2029,2032,2035,2038],{"className":2024},[1330],[228,2026],{"className":2027,"style":2028},[1334],"height:0.6389em;vertical-align:-0.1944em;",[228,2030,292],{"className":2031,"style":1385},[1339,1340],[228,2033],{"className":2034,"style":1389},[1344],[228,2036,1306],{"className":2037},[1393],[228,2039],{"className":2040,"style":1389},[1344],[228,2042,2044,2048],{"className":2043},[1330],[228,2045],{"className":2046,"style":2047},[1334],"height:0.8889em;vertical-align:-0.1944em;",[228,2049,2051],{"className":2050},[1339,1406],[228,2052,2054,2082],{"className":2053},[1410,1411],[228,2055,2057,2079],{"className":2056},[1415],[228,2058,2060,2068],{"className":2059,"style":1420},[1419],[228,2061,2062,2065],{"style":1423},[228,2063],{"className":2064,"style":1428},[1427],[228,2066,292],{"className":2067,"style":1385},[1339,1340],[228,2069,2070,2073],{"style":1423},[228,2071],{"className":2072,"style":1428},[1427],[228,2074,2076],{"className":2075,"style":1441},[1440],[228,2077,1314],{"className":2078},[1339],[228,2080,1449],{"className":2081},[1448],[228,2083,2085],{"className":2084},[1415],[228,2086,2088],{"className":2087,"style":1456},[1419],[228,2089],{},": positive and large means a confident correct call, negative means a mistake). I recreated the same idea here, interactively:",[2092,2093],"margin-loss-chart",{"hinge-label":2094,"logistic-label":2095,"perceptron-label":2096,"widrow-hoff-label":2097,"x-label":2098,"y-label":2099},"Hinge (SVM)","Logistic","Perceptron","Widrow-Hoff","margin (y · ŷ)","penalty",[12,2101,2102,2103,2106,2107,2110,2111,2113,2114,2116],{},"Hover over any point on the x-axis and compare all four. Notice the shapes: Widrow-Hoff is a parabola, it keeps penalizing even a point that's already correctly classified with room to spare (margin ",[225,2104,2105],{},"> 1","), because it doesn't know \"correct is correct\", it only knows how to measure distance to a continuous target. Perceptron and Hinge are the only two that ",[16,2108,2109],{},"fully zero out"," once a point is well classified (perceptron zeros as soon as it crosses ",[225,2112,484],{},", hinge requires crossing ",[225,2115,483],{},", with room to spare). Logistic never truly zeroes out, it only approaches zero, which is the price it pays for returning a smooth probability instead of a binary decision.",[214,2118,719],{"id":718},[661,2120,2121,2129],{},[664,2122,2123],{},[667,2124,2125,2127],{},[670,2126,728],{"align":672},[670,2128,731],{"align":672},[682,2130,2131,2149,2163],{},[667,2132,2133,2136],{},[687,2134,2135],{"align":672},"Perceptron, Adaline, batch gradient descent looked like separate algorithms",[687,2137,2138,2139,2141,2142,2141,2145,2148],{"align":672},"They're the same framework (",[225,2140,1029],{}," + ",[225,2143,2144],{},"TrainingAlgorithm",[225,2146,2147],{},"CostFunction","), just swapping which cost function gets plugged in",[667,2150,2151,2154],{},[687,2152,2153],{"align":672},"Sigmoid returns a probability between 0 and 1",[687,2155,2156,2157,2159,2160,2162],{"align":672},"Using sigmoid directly against a ",[225,2158,1921],{}," label locks up the gradient, because the output range doesn't match the target's range. ",[225,2161,1649],{}," fixes it",[667,2164,2165,2168],{},[687,2166,2167],{"align":672},"SVM uses \"support vectors\"",[687,2169,2170],{"align":672},"That's not empty jargon: the hinge loss gradient literally zeros out the contribution of every point that isn't a support vector",[214,2172,768],{"id":767},[12,2174,2175,2176,2178,2179,775,2181,2183,2184,2186],{},"I ran all four cost functions (Widrow-Hoff, perceptron criterion, hinge, and the log-likelihood version with ",[225,2177,1649],{},") on Iris (",[66,2180,774],{},[66,2182,778],{},"), plus the ",[225,2185,1545],{}," version on purpose, to confirm the sigmoid problem isn't exclusive to the notebook's toy synthetic dataset.",[661,2188,2189,2202],{},[664,2190,2191],{},[667,2192,2193,2196,2199],{},[670,2194,2195],{"align":672},"Cost function",[670,2197,2198],{"align":676},"Train accuracy",[670,2200,2201],{"align":676},"Test accuracy",[682,2203,2204,2214,2224,2235,2248],{},[667,2205,2206,2208,2210],{},[687,2207,2097],{"align":672},[687,2209,426],{"align":676},[687,2211,2212],{"align":676},[16,2213,426],{},[667,2215,2216,2218,2220],{},[687,2217,2096],{"align":672},[687,2219,426],{"align":676},[687,2221,2222],{"align":676},[16,2223,426],{},[667,2225,2226,2229,2231],{},[687,2227,2228],{"align":672},"Hinge",[687,2230,426],{"align":676},[687,2232,2233],{"align":676},[16,2234,426],{},[667,2236,2237,2242,2244],{},[687,2238,2239,2240,1317],{"align":672},"Log-likelihood (",[225,2241,1649],{},[687,2243,426],{"align":676},[687,2245,2246],{"align":676},[16,2247,426],{},[667,2249,2250,2255,2258],{},[687,2251,2239,2252,2254],{"align":672},[225,2253,1545],{},", sigmoid)",[687,2256,2257],{"align":676},"0.843",[687,2259,2260],{"align":676},[16,2261,2262],{},"0.933",[12,2264,2265,2266,209,2268,2270],{},"Four out of five hit 100% (Iris has a generous enough margin for any of them to find a perfect boundary), and sigmoid falls behind again, this time on real data, not just the notebook's synthetic dataset. Confirms it wasn't a one-run coincidence: the range mismatch between ",[225,2267,1545],{},[225,2269,1921],{}," labels genuinely hurts convergence, on every dataset I tried it on.",[873,2272,875],{},{"title":27,"searchDepth":28,"depth":28,"links":2274},[2275,2276,2277,2278,2279,2280],{"id":910,"depth":28,"text":911},{"id":1088,"depth":28,"text":1089},{"id":1196,"depth":28,"text":1197},{"id":1989,"depth":28,"text":1990},{"id":718,"depth":28,"text":719},{"id":767,"depth":28,"text":768},"Lectures 3a and 3b: the professor generalizes training to accept any pluggable cost function, and each choice (Widrow-Hoff, perceptron criterion, log-likelihood, hinge) recovers a different algorithm from this playlist. Along the way, I found a real mismatch between activation function and label encoding.",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",{"title":898,"description":2281},"en\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",[2287,2288,2289],"cost-functions","hinge-loss","logistic-regression","alBoIF9HT5FQ7eR5ld8q-Pe1ojBlQBX2mpc5QwQQgnk",{"id":2292,"title":2293,"body":2294,"cover":3,"date":884,"description":2885,"extension":31,"meta":2886,"navigation":33,"order":34,"path":154,"playlist":888,"seo":2887,"status":37,"stem":2888,"tags":2889,"__hash__":2893},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron.md","Perceptron: the First Neuron That Learns on Its Own",{"type":9,"value":2295,"toc":2876},[2296,2299,2303,2312,2319,2323,2358,2386,2411,2430,2444,2448,2549,2563,2604,2611,2615,2634,2651,2658,2675,2687,2700,2710,2714,2725,2733,2746,2748,2784,2786,2797,2827,2830,2871,2874],[12,2297,2298],{},"First lecture of the living playlist, and it starts exactly where any neural networks course should: at the simplest neuron there is.",[214,2300,2302],{"id":2301},"before-the-code-two-papers-15-years-apart","Before the code: two papers, 15 years apart",[12,2304,2305,2306,2311],{},"Neural networks weren't born as code, they were born as a biophysics question. In 1943, Warren McCulloch and Walter Pitts published ",[53,2307,2310],{"href":2308,"rel":2309},"https:\u002F\u002Fwww.cs.cmu.edu\u002F~epxing\u002FClass\u002F10715\u002Freading\u002FMcCulloch.and.Pitts.pdf",[77],"\"A Logical Calculus of the Ideas Immanent in Nervous Activity\"",", proposing a lean mathematical model for a biological neuron: sum up the input signals, and fire a binary (all-or-nothing) signal if that sum crosses a threshold. No learning yet, it's just a fixed logic circuit, each \"neuron\" solves a logical function someone already decided ahead of time.",[12,2313,2314,2315,2318],{},"The missing leap came 15 years later, with Frank Rosenblatt in 1958: what if, instead of someone hand-picking the weights, the neuron itself learned the right weights by looking at examples? That's the Perceptron, and it's exactly what the professor's lecture 1a implements: the ",[16,2316,2317],{},"Perceptron Learning Algorithm"," (PLA), in its rawest form, no bias yet (that's next lecture).",[214,2320,2322],{"id":2321},"the-dataset-two-groups-separable-by-a-line","The dataset: two groups separable by a line",[219,2324,2326],{"className":221,"code":2325,"language":223,"meta":27,"style":27},"def createDataset(n=20):\n  X = np.random.rand(n,2)\n  coefs = np.array([1, -1])\n  labels = X @ coefs\n  y = np.array(labels>0, dtype=int)*2-1\n  return X, y\n",[225,2327,2328,2333,2338,2343,2348,2353],{"__ignoreMap":27},[228,2329,2330],{"class":230,"line":34},[228,2331,2332],{},"def createDataset(n=20):\n",[228,2334,2335],{"class":230,"line":28},[228,2336,2337],{},"  X = np.random.rand(n,2)\n",[228,2339,2340],{"class":230,"line":161},[228,2341,2342],{},"  coefs = np.array([1, -1])\n",[228,2344,2345],{"class":230,"line":188},[228,2346,2347],{},"  labels = X @ coefs\n",[228,2349,2350],{"class":230,"line":251},[228,2351,2352],{},"  y = np.array(labels>0, dtype=int)*2-1\n",[228,2354,2355],{"class":230,"line":257},[228,2356,2357],{},"  return X, y\n",[12,2359,2360,2361,2364,2365,2368,2369,2371,2372,2374,2375,554,2377,297,2379,2381,2382,2385],{},"The professor generates random 2D points and labels each one by the sign of ",[225,2362,2363],{},"X @ coefs",", the dot product between the point and the vector ",[225,2366,2367],{},"[1, -1]",". Geometrically, ",[225,2370,2363],{}," is positive on one side of the line that passes through the origin and is perpendicular to ",[225,2373,2367],{},", and negative on the other side, so the label ",[225,2376,292],{},[225,2378,296],{},[225,2380,300],{},") is ",[16,2383,2384],{},"linearly separable by construction",": a straight line exists that separates the two classes perfectly, because that exact line is what generated the labels.",[219,2387,2389],{"className":221,"code":2388,"language":223,"meta":27,"style":27},"def plotHyperplan(vector):\n  xs = np.array([0,1])\n  ys = -(vector[0]*xs)\u002Fvector[1]\n  plt.plot(xs, ys)\n",[225,2390,2391,2396,2401,2406],{"__ignoreMap":27},[228,2392,2393],{"class":230,"line":34},[228,2394,2395],{},"def plotHyperplan(vector):\n",[228,2397,2398],{"class":230,"line":28},[228,2399,2400],{},"  xs = np.array([0,1])\n",[228,2402,2403],{"class":230,"line":161},[228,2404,2405],{},"  ys = -(vector[0]*xs)\u002Fvector[1]\n",[228,2407,2408],{"class":230,"line":188},[228,2409,2410],{},"  plt.plot(xs, ys)\n",[12,2412,2413,2414,2417,2418,2421,2422,2425,2426,2429],{},"This function draws the line where ",[225,2415,2416],{},"vector[0]*x + vector[1]*y = 0",", the decision boundary for any weight vector ",[225,2419,2420],{},"w",". Same equation as always, ",[225,2423,2424],{},"w · x = 0"," defines a hyperplane, except here without a bias, so the hyperplane is forced to pass through the origin ",[225,2427,2428],{},"(0,0)",". Keep that detail in mind, it matters in a bit.",[12,2431,2432,2433,2436,2437,2439,2440,2443],{},"The professor even tests a ",[225,2434,2435],{},"DummyClassifier"," with fixed weights ",[225,2438,2367],{},", the exact same ones that generated the dataset, and of course it hits 100%: that's the answer key fed straight back in. The real question is: can these weights actually be ",[16,2441,2442],{},"learned"," just by looking at examples, without me handing over the answer already solved?",[214,2445,2447],{"id":2446},"the-algorithm-nudge-the-weight-toward-the-error","The algorithm: nudge the weight toward the error",[219,2449,2451],{"className":221,"code":2450,"language":223,"meta":27,"style":27},"class PLA(BaseEstimator, ClassifierMixin):\n  def __init__(self, max_iter=10):\n    self.max_iter = max_iter\n\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    for _ in range(self.max_iter):\n      cost = 0\n      idx = np.arange(X.shape[0])\n      np.random.shuffle(idx)\n      for i in idx:\n        logits = X[i] @ self.w_\n        y_pred = np.sign(logits)\n        error = y[i] - y_pred\n        if error != 0:\n          cost += error**2\n          self.w_ += error*X[i]\n        if cost == 0:\n          break\n    return self\n",[225,2452,2453,2458,2463,2468,2472,2476,2481,2485,2490,2495,2500,2505,2510,2515,2520,2525,2530,2535,2540,2545],{"__ignoreMap":27},[228,2454,2455],{"class":230,"line":34},[228,2456,2457],{},"class PLA(BaseEstimator, ClassifierMixin):\n",[228,2459,2460],{"class":230,"line":28},[228,2461,2462],{},"  def __init__(self, max_iter=10):\n",[228,2464,2465],{"class":230,"line":161},[228,2466,2467],{},"    self.max_iter = max_iter\n",[228,2469,2470],{"class":230,"line":188},[228,2471,260],{"emptyLinePlaceholder":33},[228,2473,2474],{"class":230,"line":251},[228,2475,238],{},[228,2477,2478],{"class":230,"line":257},[228,2479,2480],{},"    self.w_ = np.random.rand(X.shape[1])\n",[228,2482,2483],{"class":230,"line":263},[228,2484,524],{},[228,2486,2487],{"class":230,"line":269},[228,2488,2489],{},"      cost = 0\n",[228,2491,2492],{"class":230,"line":274},[228,2493,2494],{},"      idx = np.arange(X.shape[0])\n",[228,2496,2497],{"class":230,"line":378},[228,2498,2499],{},"      np.random.shuffle(idx)\n",[228,2501,2502],{"class":230,"line":383},[228,2503,2504],{},"      for i in idx:\n",[228,2506,2507],{"class":230,"line":388},[228,2508,2509],{},"        logits = X[i] @ self.w_\n",[228,2511,2512],{"class":230,"line":976},[228,2513,2514],{},"        y_pred = np.sign(logits)\n",[228,2516,2517],{"class":230,"line":981},[228,2518,2519],{},"        error = y[i] - y_pred\n",[228,2521,2522],{"class":230,"line":986},[228,2523,2524],{},"        if error != 0:\n",[228,2526,2527],{"class":230,"line":991},[228,2528,2529],{},"          cost += error**2\n",[228,2531,2532],{"class":230,"line":997},[228,2533,2534],{},"          self.w_ += error*X[i]\n",[228,2536,2537],{"class":230,"line":1002},[228,2538,2539],{},"        if cost == 0:\n",[228,2541,2542],{"class":230,"line":1007},[228,2543,2544],{},"          break\n",[228,2546,2547],{"class":230,"line":1012},[228,2548,254],{},[12,2550,2551,2554,2555,2558,2559,2562],{},[225,2552,2553],{},"w_"," starts random. Then, point by point, in shuffled order: compute the prediction (",[225,2556,2557],{},"sign(w · x)","), compare against the true label, and if it's wrong, update ",[225,2560,2561],{},"w_ += error * x",". That's the entire learning rule, and it's worth understanding why it works, not just memorizing the formula.",[12,2564,2565,2566,2568,2569,2572,2573,2575,2576,1656,2579,2581,2582,2585,2586,209,2589,2591,2592,2595,2596,2599,2600,2603],{},"Think about it geometrically: ",[225,2567,2420],{}," is the vector normal to the decision hyperplane, it points toward the side the model considers \"class +1\". If the model missed a point that was ",[16,2570,2571],{},"actually"," class +1 but got classified as -1, that means ",[225,2574,2420],{}," is pointing \"too far away\" from that point. Adding ",[225,2577,2578],{},"error * x",[225,2580,2420],{}," (here ",[225,2583,2584],{},"error = +2",", since ",[225,2587,2588],{},"y_pred",[225,2590,292],{}," live in ",[225,2593,2594],{},"{-1,+1}",") pushes the weight vector ",[16,2597,2598],{},"toward that point's direction",", making ",[225,2601,2602],{},"w · x"," a bit more positive next time. The opposite happens when the error goes the other way. It's a local, cheap adjustment: every mistake nudges the decision boundary a little in the direction that would have gotten that specific point right, with no actual gradient computed at all (this is exactly what Aggarwal calls the \"perceptron criterion\" in chapter 1: a heuristic update rule that looks a lot like gradient descent, but was designed directly on top of the classification error, before anyone formalized which smooth loss function it was implicitly optimizing).",[12,2605,2606,2607,2610],{},"And Rosenblatt proved, back in 1958, something strong: if the data really is linearly separable (as it is here, by construction), PLA ",[16,2608,2609],{},"always converges"," to a zero-error solution, in a finite number of steps. Not \"usually works\", a mathematical guarantee.",[214,2612,2614],{"id":2613},"a-real-bug-hiding-in-the-stopping-condition","A real bug, hiding in the stopping condition",[12,2616,2617,2618,2621,2622,2625,2626,2629,2630,2633],{},"But looking closely at ",[225,2619,2620],{},"if cost == 0: break"," reveals a problem. That check sits ",[16,2623,2624],{},"inside"," the loop that walks through the points, not after it. That means: as soon as ",[225,2627,2628],{},"cost"," (which only grows on error) hits zero, the loop ",[16,2631,2632],{},"stops immediately",", even if there are still points left to check that epoch.",[12,2635,2636,2637,2639,2640,2643,2644,2646,2647,2650],{},"The catch is that at the very start of every epoch, ",[225,2638,2628],{}," already starts at zero. So if the ",[16,2641,2642],{},"first sampled point"," in that epoch happens to already be classified correctly, ",[225,2645,2628],{}," stays zero, and the ",[225,2648,2649],{},"break"," fires right there, without checking the other 19 points. The epoch ends thinking \"everything's fine\", when really only one point got checked.",[12,2652,2653,2654,2657],{},"I ran this exact code, byte for byte, with a fixed seed (",[225,2655,2656],{},"np.random.seed(10)","), to see the actual damage:",[312,2659,2660],{},[12,2661,2662,2664,2665,1255,2667,2670,2671,2674],{},[16,2663,318],{}," counting how many points (out of 20) each epoch actually processed before the ",[225,2666,2649],{},[225,2668,2669],{},"[1, 1, 1, 1, 1, 1, 1, 1, 20, 20]",". Only the last two epochs checked the whole dataset. Final accuracy: ",[16,2672,2673],{},"0.6",", far from the perfect separation Rosenblatt guarantees.",[12,2676,2677,2678,2680,2681,2683,2684,2686],{},"Moving just the ",[225,2679,2620],{}," outside the inner loop (checking zero errors ",[16,2682,447],{}," going through all 20 points, not after each one), same seed, same data, same initial ",[225,2685,2420],{},":",[312,2688,2689],{},[12,2690,2691,284,2693,2696,2697,2699],{},[16,2692,318],{},[225,2694,2695],{},"[20, 20]",", two full epochs, and done: converged with accuracy ",[16,2698,426],{},", exactly what the theorem promises for linearly separable data.",[12,2701,2702,2703,2705,2706,2709],{},"A real, non-hypothetical finding: the obvious intent of the code is \"stop once there's no more error this epoch\", but the way the ",[225,2704,2649],{}," got positioned makes it stop as soon as a ",[16,2707,2708],{},"single favorable point"," shows up, even if more error is lurking further down the queue. With luck (as with most seeds I tested), that doesn't change the final outcome because other epochs make up for it. But with this specific seed, the algorithm declares success prematurely, eight times in a row, and never reaches the perfect solution it should.",[214,2711,2713],{"id":2712},"interactive-training-the-perceptron-point-by-point","Interactive: training the perceptron point by point",[12,2715,2716,2717,2720,2721,2724],{},"I rebuilt the same dataset (that seed ",[225,2718,2719],{},"10"," above, the same 20 points) in a component that runs the ",[16,2722,2723],{},"correct"," version of the algorithm, one point at a time. Click \"Process next point\" and notice: every time a colored point falls on the wrong side of the background region, that's a mistake, and the next click pushes the boundary toward it.",[2726,2727],"perceptron-explorer",{":classes":2728,":points":2729,":x-max":483,":x-min":484,":y-max":483,":y-min":484,"converged-label":2730,"negative-label":485,"positive-label":486,"reset-label":2731,"step-label":2732,"x-label":488,"y-label":489},"[1, -1, 1, -1, 1, -1, -1, 1, 1, 1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1]","[[0.7713, 0.0208], [0.6336, 0.7488], [0.4985, 0.2248], [0.1981, 0.7605], [0.1691, 0.0883], [0.6854, 0.9534], [0.0039, 0.5122], [0.8126, 0.6125], [0.7218, 0.2919], [0.9178, 0.7146], [0.5425, 0.1422], [0.3733, 0.6741], [0.4418, 0.434], [0.6178, 0.5131], [0.6504, 0.601], [0.8052, 0.5216], [0.9086, 0.3192], [0.0905, 0.3007], [0.114, 0.8287], [0.0469, 0.6263]]","converged, zero mistakes in a full pass","Reset (new random draw)","Process next point",[12,2734,2735,2736,2738,2739,1580,2742,2745],{},"Notice that, with no bias, the boundary is always a line through the origin ",[225,2737,2428],{},", it can only ",[16,2740,2741],{},"rotate",[16,2743,2744],{},"slide",". It works on this dataset because the data was drawn around the origin on purpose. But what if the point cloud were shifted far away from the origin? I get to that in the Practical Application.",[214,2747,719],{"id":718},[661,2749,2750,2758],{},[664,2751,2752],{},[667,2753,2754,2756],{},[670,2755,728],{"align":672},[670,2757,731],{"align":672},[682,2759,2760,2768,2776],{},[667,2761,2762,2765],{},[687,2763,2764],{"align":672},"Neural networks are about \"learning weights\"",[687,2766,2767],{"align":672},"McCulloch-Pitts comes before that: someone first had to propose that a neuron could be modeled mathematically at all, learning came 15 years later with Rosenblatt",[667,2769,2770,2773],{},[687,2771,2772],{"align":672},"Updating a weight \"toward the error\" feels intuitive",[687,2774,2775],{"align":672},"It has a name (the perceptron criterion) and a mathematical convergence guarantee for linearly separable data",[667,2777,2778,2781],{},[687,2779,2780],{"align":672},"The professor's code is the ground truth",[687,2782,2783],{"align":672},"Even reference code can hide a subtle bug in a stopping condition, and it's worth testing instead of trusting it with your eyes closed",[214,2785,768],{"id":767},[12,2787,2788,2789,2791,2792,775,2794,2796],{},"I tested the same no-bias PLA (bug-fixed version, without the ",[225,2790,2649],{}," issue) on a real dataset: Iris, the two easiest classes to separate (",[66,2793,774],{},[66,2795,778],{},"), using petal length and petal width as the two variables.",[219,2798,2800],{"className":221,"code":2799,"language":223,"meta":27,"style":27},"from sklearn.datasets import load_iris\niris = load_iris()\nmask = iris.target \u003C 2\nX = iris.data[mask][:, [2, 3]]  # petal length and width\ny = np.where(iris.target[mask] == 0, -1, 1)\n",[225,2801,2802,2807,2812,2817,2822],{"__ignoreMap":27},[228,2803,2804],{"class":230,"line":34},[228,2805,2806],{},"from sklearn.datasets import load_iris\n",[228,2808,2809],{"class":230,"line":28},[228,2810,2811],{},"iris = load_iris()\n",[228,2813,2814],{"class":230,"line":161},[228,2815,2816],{},"mask = iris.target \u003C 2\n",[228,2818,2819],{"class":230,"line":188},[228,2820,2821],{},"X = iris.data[mask][:, [2, 3]]  # petal length and width\n",[228,2823,2824],{"class":230,"line":251},[228,2825,2826],{},"y = np.where(iris.target[mask] == 0, -1, 1)\n",[12,2828,2829],{},"These two classes are genuinely linearly separable (it's one of the most-cited \"actually separable\" examples in introductory ML material). But petal length and width are never negative, so the entire point cloud lives far from the origin, quite unlike the synthetic dataset above.",[661,2831,2832,2846],{},[664,2833,2834],{},[667,2835,2836,2839,2843],{},[670,2837,2838],{"align":672},"Version",[670,2840,2842],{"align":2841},"center","Converged in (max 50 epochs)",[670,2844,2845],{"align":676},"Final accuracy",[682,2847,2848,2859],{},[667,2849,2850,2853,2856],{},[687,2851,2852],{"align":672},"No-bias PLA",[687,2854,2855],{"align":2841},"never converged",[687,2857,2858],{"align":676},"0.84 to 0.93 (varies by seed)",[667,2860,2861,2864,2867],{},[687,2862,2863],{"align":672},"With-bias PLA",[687,2865,2866],{"align":2841},"2 epochs",[687,2868,2869],{"align":676},[16,2870,426],{},[12,2872,2873],{},"Without bias, PLA never declares convergence within the 50-epoch cap, because the line that would truly separate the two classes doesn't pass through the origin, and without bias that line is simply out of the model's reach. I ran 5 different seeds and none went past 93% accuracy. Adding a single parameter (the bias, which shifts the hyperplane instead of just rotating it around the origin), the same algorithm converges in just 2 epochs with perfect accuracy. That's the exact limit next lecture tackles head-on.",[873,2875,875],{},{"title":27,"searchDepth":28,"depth":28,"links":2877},[2878,2879,2880,2881,2882,2883,2884],{"id":2301,"depth":28,"text":2302},{"id":2321,"depth":28,"text":2322},{"id":2446,"depth":28,"text":2447},{"id":2613,"depth":28,"text":2614},{"id":2712,"depth":28,"text":2713},{"id":718,"depth":28,"text":719},{"id":767,"depth":28,"text":768},"Lecture 1a: the professor implements the Perceptron Learning Algorithm from scratch, no bias yet. I tell the story of two papers that came before it and find a real bug hiding in the algorithm's stopping condition.",{},{"title":2293,"description":2885},"en\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron",[2890,2891,2892],"perceptron","mcculloch-pitts","rosenblatt","7H0vY4nKBB5YNpqcUvWRgrVi2GCunnVyBByfnG6v2Ss",[2895,2933],{"id":2896,"title":2897,"body":2898,"date":2918,"description":2919,"extension":31,"meta":2920,"navigation":33,"path":2921,"photos":2922,"seo":2927,"status":37,"stem":2928,"tags":2929,"__hash__":2932},"timeline\u002Fen\u002Ftimeline\u002F2026-08-19-estudando-go.md","Getting shaken up by Go",{"type":9,"value":2899,"toc":2916},[2900,2907,2910,2913],[12,2901,2902,2903,2906],{},"I've been studying a lot of Go these past few weeks, and I mean that literally: I brought ",[66,2904,2905],{},"The Go Programming Language"," (the classic Donovan and Kernighan book, Brazilian edition) to the beach. Between waves, I got through a fair amount of it.",[12,2908,2909],{},"I'm getting a real shake-up trying to adapt to the \"way of Go\". After years used to Python, my brain has to relearn a much more explicit way of thinking: errors handled by hand instead of exceptions, static typing, concurrency that's part of the language itself instead of a framework bolted on top. Nothing absurd, just a different way of solving the same problem.",[12,2911,2912],{},"And the enthusiasm is real. I'm feeling a strong pull to gradually migrate toward Golang, mainly for its robustness: the compiler catches a ton of silly mistakes before the code even runs, and the performance is on another level. Unfortunately the Machine Learning and AI ecosystem in Go is still pretty thin on mature libraries, so I can't switch everything over at once. But I think in the long run the language has real potential to become an important piece in that space, especially for serving models in production.",[12,2914,2915],{},"And yes, the cat decided to show up in one of the photos too. He doesn't study Go with me, he just supervises.",{"title":27,"searchDepth":28,"depth":28,"links":2917},[],"2026-08-19","I've been studying a lot of Go these past few weeks, and I mean that literally: I brought The Go Programming Language (the classic Donovan and Kernighan book, Brazilian edition) to the beach. Between waves, I got through a fair amount of it.",{},"\u002Fen\u002Ftimeline\u002F2026-08-19-estudando-go",[2923,2924,2925,2926],"\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_book.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_cup.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fgolang_learn.jpeg","\u002Fimages\u002Ftimeline\u002F20260819\u002Fmy_cat.jpeg",{"title":2897,"description":2919},"en\u002Ftimeline\u002F2026-08-19-estudando-go",[2930,2931],"golang","go","AQlTyWKLXttO1qD9BnNoCctqX-52ZcX0X1DOuF5zWzY",{"id":2934,"title":2935,"body":2936,"date":2957,"description":2958,"extension":31,"meta":2959,"navigation":33,"path":2960,"photos":2961,"seo":2965,"status":37,"stem":2966,"tags":2967,"__hash__":2972},"timeline\u002Fen\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro.md","Chapter 2 of the SLM book, but with Dom Casmurro",{"type":9,"value":2937,"toc":2955},[2938,2945,2952],[12,2939,2940,2941,2944],{},"Started Chapter 2 of ",[66,2942,2943],{},"How to Build and Fine-Tune a Small Language Model",", by Prof. Paul Liu, and this time the exercise is building a small GPT from scratch, along the lines of Karpathy's \"build a GPT\", around 10 million parameters.",[12,2946,2947,2948,2951],{},"Instead of using the book's own example dataset though, I swapped it for a text I genuinely love: ",[66,2949,2950],{},"Dom Casmurro",", by Machado de Assis. Same code, same architecture, but now tokenizing and training on Bentinho and Capitu instead of some generic corpus.",[12,2953,2954],{},"More updates coming soon once I wrap up this study.",{"title":27,"searchDepth":28,"depth":28,"links":2956},[],"2026-08-18","Started Chapter 2 of How to Build and Fine-Tune a Small Language Model, by Prof. Paul Liu, and this time the exercise is building a small GPT from scratch, along the lines of Karpathy's \"build a GPT\", around 10 million parameters.",{},"\u002Fen\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro",[2962,2963,2964],"\u002Fimages\u002Ftimeline\u002F20260818\u002Fslms_book.jpeg","\u002Fimages\u002Ftimeline\u002F20260818\u002Fcapitulo_2_implementacao.jpeg","\u002Fimages\u002Ftimeline\u002F20260818\u002Ftrecho_livro.jpeg",{"title":2935,"description":2958},"en\u002Ftimeline\u002F2026-08-18-slm-dom-casmurro",[2968,2969,2970,2971],"slm","gpt","nlp","dom-casmurro","qR1jn7cqvL9YLI9xWeuqbHfPou4oBgnKlDZ1xLA-T4E",1787338982067]