[{"data":1,"prerenderedAt":724},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta":3,"post-en-neural-networks-adaline-regra-delta":4},"\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",{"id":5,"title":6,"body":7,"cover":709,"date":710,"description":711,"extension":712,"meta":713,"navigation":79,"order":64,"path":714,"playlist":715,"seo":716,"status":717,"stem":718,"tags":719,"__hash__":723},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta.md","Adaline: Train on the Line, Classify on the Sign",{"type":8,"value":9,"toc":701},"minimark",[10,27,32,98,132,148,155,213,238,252,284,288,302,314,317,321,368,388,404,408,443,463,477,484,528,540,544,589,593,605,620,643,686,697],[11,12,13,14,18,19,22,23,26],"p",{},"Lectures 2e and 2f, and the model's name changes to ",[15,16,17],"strong",{},"Adaline"," (ADAptive LInear NEuron), by Bernard Widrow and Ted Hoff, 1960, just two years after Rosenblatt. Their core idea is subtle, but it splits apart two things I'd been treating as one so far: ",[15,20,21],{},"what the model optimizes during training"," and ",[15,24,25],{},"what it computes at prediction time",".",[28,29,31],"h2",{"id":30},"pre-activation-and-post-activation-the-distinction-adaline-introduces","Pre-activation and post-activation: the distinction Adaline introduces",[33,34,39],"pre",{"className":35,"code":36,"language":37,"meta":38,"style":38},"language-python shiki shiki-themes github-light github-dark","class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n","python","",[40,41,42,50,56,62,68,74,81,87,92],"code",{"__ignoreMap":38},[43,44,47],"span",{"class":45,"line":46},"line",1,[43,48,49],{},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n",[43,51,53],{"class":45,"line":52},2,[43,54,55],{},"  def fit(self, X, y):\n",[43,57,59],{"class":45,"line":58},3,[43,60,61],{},"    X = include_bias(X)\n",[43,63,65],{"class":45,"line":64},4,[43,66,67],{},"    self.w_ = np.linalg.pinv(X) @ y\n",[43,69,71],{"class":45,"line":70},5,[43,72,73],{},"    return self\n",[43,75,77],{"class":45,"line":76},6,[43,78,80],{"emptyLinePlaceholder":79},true,"\n",[43,82,84],{"class":45,"line":83},7,[43,85,86],{},"  def predict(self, X):\n",[43,88,90],{"class":45,"line":89},8,[43,91,61],{},[43,93,95],{"class":45,"line":94},9,[43,96,97],{},"    return X @ self.w_\n",[11,99,100,101,104,105,110,111,114,115,118,119,122,123,125,126,128,129,26],{},"Notice: this is ",[15,102,103],{},"exactly"," ",[106,107,109],"a",{"href":108},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","last post's normal equation",", without changing a line, except now ",[40,112,113],{},"y"," is ",[40,116,117],{},"-1"," or ",[40,120,121],{},"+1"," instead of continuous. The professor is treating classification as if it were regression: fitting the line to land as close as possible to ",[40,124,117],{}," on one class's points and ",[40,127,121],{}," on the other's, never applying ",[40,130,131],{},"sign()",[133,134,135],"blockquote",{},[11,136,137,140,141,144,145,26],{},[15,138,139],{},"Output:"," RMSE ",[40,142,143],{},"0.4148",", weights ",[40,146,147],{},"[-1.76, 1.01, 2.74]",[11,149,150,151,154],{},"RMSE makes sense here because ",[40,152,153],{},"predict"," returns a continuous number, not a class. But to actually classify, there's one last step missing:",[33,156,158],{"className":35,"code":157,"language":37,"meta":38,"style":38},"class AdalinePseudoInverse(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def pre_activation(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n\n  def predict(self, X):\n    return np.sign(self.pre_activation(X))\n",[40,159,160,164,168,172,176,180,184,189,193,197,202,207],{"__ignoreMap":38},[43,161,162],{"class":45,"line":46},[43,163,49],{},[43,165,166],{"class":45,"line":52},[43,167,55],{},[43,169,170],{"class":45,"line":58},[43,171,61],{},[43,173,174],{"class":45,"line":64},[43,175,67],{},[43,177,178],{"class":45,"line":70},[43,179,73],{},[43,181,182],{"class":45,"line":76},[43,183,80],{"emptyLinePlaceholder":79},[43,185,186],{"class":45,"line":83},[43,187,188],{},"  def pre_activation(self, X):\n",[43,190,191],{"class":45,"line":89},[43,192,61],{},[43,194,195],{"class":45,"line":94},[43,196,97],{},[43,198,200],{"class":45,"line":199},10,[43,201,80],{"emptyLinePlaceholder":79},[43,203,205],{"class":45,"line":204},11,[43,206,86],{},[43,208,210],{"class":45,"line":209},12,[43,211,212],{},"    return np.sign(self.pre_activation(X))\n",[11,214,215,216,219,220,222,223,225,226,229,230,233,234,237],{},"Now there are two methods: ",[40,217,218],{},"pre_activation"," (the continuous value, before any threshold) and ",[40,221,153],{}," (applies ",[40,224,131],{}," on top). Aggarwal calls exactly these two things the ",[15,227,228],{},"pre-activation value"," and the ",[15,231,232],{},"post-activation value"," (chapter 1): everything a neuron computes happens in two steps, first the weighted sum, then the activation function on top of it. Adaline trains on the pre-activation (it's continuous, so you can measure \"how far off\" each prediction landed from the target) and only applies the activation (",[40,235,236],{},"sign",") when deciding the final class.",[133,239,240],{},[11,241,242,244,245,248,249,251],{},[15,243,139],{}," accuracy ",[40,246,247],{},"1.0",", the same ",[40,250,147],{}," weights as before (it's the same computation, just evaluated by accuracy instead of RMSE this time).",[11,253,254,255,258,259,263,264,268,269,272,273,275,276,279,280,283],{},"This is the ",[15,256,257],{},"delta rule",", and ",[106,260,262],{"href":261},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","I've already seen it before, under that exact name",": \"update the weight proportionally to the error times the input\" is Widrow-Hoff's signature. The difference from Rosenblatt's perceptron (which I covered ",[106,265,267],{"href":266},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","two posts ago",") is exactly this: the perceptron measures error ",[15,270,271],{},"after"," applying ",[40,274,131],{}," (error in ",[40,277,278],{},"{-2,0,+2}","), Adaline measures it ",[15,281,282],{},"before",", on the continuous pre-activation. That sounds like a small detail, but it changes everything: a continuous error gives a real, smooth gradient that points toward the better direction even when a prediction is already on the right side but still a bit \"unsure\". The perceptron's binary error, by contrast, only fires on an outright misclassification, with no notion of \"how wrong.\"",[28,285,287],{"id":286},"interactive-nudging-the-pre-activation-and-watching-rmse-and-accuracy-move","Interactive: nudging the pre-activation and watching RMSE (and accuracy) move",[11,289,290,291,294,295,258,298,301],{},"Instead of training automatically, drag the ",[40,292,293],{},"w0",", ",[40,296,297],{},"w1",[40,299,300],{},"bias"," sliders by hand and watch two readouts at once: RMSE (continuous, changes smoothly with every drag) and accuracy (discrete, only jumps when a point crosses the decision boundary).",[303,304],"adaline-explorer",{":classes":305,":points":306,":x-max":307,":x-min":308,":y-max":307,":y-min":308,"negative-label":309,"positive-label":310,"readout-label":311,"x-label":312,"y-label":313},"[-1, 1, -1, -1, -1, -1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, 1, -1]","[[0.0856, 0.2368], [0.8013, 0.5822], [0.0941, 0.4331], [0.4791, 0.1597], [0.7346, 0.1137], [0.3912, 0.5167], [0.4306, 0.5868], [0.7378, 0.9563], [0.2842, 0.6485], [0.6962, 0.2927], [0.0015, 0.9735], [0.2984, 0.314], [0.8917, 0.5852], [0.4713, 0.7733], [0.0303, 0.707], [0.3742, 0.0909], [0.6605, 0.9315], [0.2072, 0.6301], [0.2982, 0.7418], [0.7222, 0.2187]]","1","0","class -1","class +1","RMSE (continuous pre-activation vs. ±1 label): {rmse} · accuracy (post-activation, sign): {acc}%","x0","x1",[11,315,316],{},"Notice how RMSE almost always keeps changing a little even after accuracy already hit 100%: you can push the boundary further into the empty gap between the two classes (RMSE drops more) without gaining or losing a single point (accuracy stays put). That's precisely the difference between \"finding some line that separates\" (what the perceptron does) and \"finding the line that separates with room to spare\" (what you get by optimizing RMSE instead of just counting mistakes).",[28,318,320],{"id":319},"lecture-2f-the-same-computation-just-iterating-and-the-notebook-calls-it-sgd","Lecture 2f: the same computation, just iterating (and the notebook calls it \"SGD\")",[33,322,324],{"className":35,"code":323,"language":37,"meta":38,"style":38},"class Adaline(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.zeros(X.shape[1])\n    for _ in range(self.max_iter):\n      y_pred = X @ self.w_\n      error = y - y_pred\n      self.w_ += self.learning_rate * error @ X\n    return self\n",[40,325,326,331,335,339,344,349,354,359,364],{"__ignoreMap":38},[43,327,328],{"class":45,"line":46},[43,329,330],{},"class Adaline(BaseEstimator, ClassifierMixin):\n",[43,332,333],{"class":45,"line":52},[43,334,55],{},[43,336,337],{"class":45,"line":58},[43,338,61],{},[43,340,341],{"class":45,"line":64},[43,342,343],{},"    self.w_ = np.zeros(X.shape[1])\n",[43,345,346],{"class":45,"line":70},[43,347,348],{},"    for _ in range(self.max_iter):\n",[43,350,351],{"class":45,"line":76},[43,352,353],{},"      y_pred = X @ self.w_\n",[43,355,356],{"class":45,"line":83},[43,357,358],{},"      error = y - y_pred\n",[43,360,361],{"class":45,"line":89},[43,362,363],{},"      self.w_ += self.learning_rate * error @ X\n",[43,365,366],{"class":45,"line":94},[43,367,73],{},[11,369,370,371,374,375,378,379,387],{},"One honest note about the notebook's name (",[40,372,373],{},"aula02f adaline with SGD","): the code shown here is ",[15,376,377],{},"batch gradient descent"," (",[106,380,382,383,386],{"href":381},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias","the same ",[40,384,385],{},"X.T @ error"," as always","), computing the error over the entire dataset every iteration, not real SGD (which would update on one example at a time, in shuffled order). It's a common informal way of talking (\"it's kind of like gradient descent, so I call it SGD\"), but the technical difference is worth noting, since the names carry precise meaning.",[133,389,390],{},[11,391,392,244,394,396,397,400,401,26],{},[15,393,139],{},[40,395,247],{}," on training, weights ",[40,398,399],{},"[-2.92, 3.32, 2.63]",". Tested on 1000 new points: accuracy ",[15,402,403],{},"0.953",[28,405,407],{"id":406},"the-bad-dataset-what-it-actually-proves","The \"bad\" dataset: what it actually proves",[33,409,411],{"className":35,"code":410,"language":37,"meta":38,"style":38},"X_bad = np.concatenate((X,np.ones_like(X)))\ny_bad = np.concatenate((y,np.ones_like(y)))\nX_bad = np.concatenate((X_bad,np.ones_like(X)))\ny_bad = np.concatenate((y_bad,np.ones_like(y)))\nclf_bad = Adaline()\nclf_bad.fit(X_bad, y_bad)\n",[40,412,413,418,423,428,433,438],{"__ignoreMap":38},[43,414,415],{"class":45,"line":46},[43,416,417],{},"X_bad = np.concatenate((X,np.ones_like(X)))\n",[43,419,420],{"class":45,"line":52},[43,421,422],{},"y_bad = np.concatenate((y,np.ones_like(y)))\n",[43,424,425],{"class":45,"line":58},[43,426,427],{},"X_bad = np.concatenate((X_bad,np.ones_like(X)))\n",[43,429,430],{"class":45,"line":64},[43,431,432],{},"y_bad = np.concatenate((y_bad,np.ones_like(y)))\n",[43,434,435],{"class":45,"line":70},[43,436,437],{},"clf_bad = Adaline()\n",[43,439,440],{"class":45,"line":76},[43,441,442],{},"clf_bad.fit(X_bad, y_bad)\n",[11,444,445,446,449,450,453,454,456,457,459,460,462],{},"The professor concatenates the original dataset with ",[15,447,448],{},"two extra blocks"," of artificial points: everyone at ",[40,451,452],{},"(1,1)",", everyone labeled ",[40,455,121],{},". This isn't noise, it's a deliberate bias, nudging training to \"believe\" the region near ",[40,458,452],{}," is even more strongly class ",[40,461,121],{}," than it really is.",[133,464,465],{},[11,466,467,244,469,472,473,476],{},[15,468,139],{},[40,470,471],{},"0.967"," on training (over the biased dataset), but only ",[15,474,475],{},"0.811"," on the same 1000 clean test points as before.",[11,478,479,480,483],{},"Dropped from 0.953 to 0.811. Before writing this post, my working hypothesis was that this cell would show the normal equation (pseudo-inverse) breaking on this problematic dataset, with SGD holding up better. I reproduced the experiment myself, comparing the pseudo-inverse against batch gradient descent ",[15,481,482],{},"on the same biased dataset",", and that hypothesis didn't hold up:",[485,486,487,505],"table",{},[488,489,490],"thead",{},[491,492,493,498,502],"tr",{},[494,495,497],"th",{"align":496},"left","Method",[494,499,501],{"align":500},"right","Train accuracy (biased)",[494,503,504],{"align":500},"Test accuracy (clean)",[506,507,508,519],"tbody",{},[491,509,510,514,516],{},[511,512,513],"td",{"align":496},"Pseudo-inverse",[511,515,471],{"align":500},[511,517,518],{"align":500},"0.783",[491,520,521,524,526],{},[511,522,523],{"align":496},"Batch gradient descent",[511,525,471],{"align":500},[511,527,518],{"align":500},[11,529,530,531,534,535,539],{},"Both drop by the exact same amount, with essentially identical weights between them. The real lesson of this cell isn't about which training algorithm is more robust, it's about ",[15,532,533],{},"training data quality",": biasing the training distribution (even without adding \"noise\" in the random-error sense) shifts the learned boundary somewhere that no longer represents the real distribution, and it hurts both methods equally, because both are solving the exact same optimization problem under the hood. ",[106,536,538],{"href":537},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","This echoes a lesson I already saw in the other playlist",": training on a distribution that doesn't match the real world is a data problem, not an algorithm problem.",[28,541,543],{"id":542},"wrapping-up","Wrapping up",[485,545,546,556],{},[488,547,548],{},[491,549,550,553],{},[494,551,552],{"align":496},"What I already knew",[494,554,555],{"align":496},"What this lecture settled",[506,557,558,569,581],{},[491,559,560,563],{},[511,561,562],{"align":496},"The perceptron classifies and updates on the binary error",[511,564,565,566,568],{"align":496},"Adaline separates pre-activation (continuous, used in training) from post-activation (",[40,567,236],{},", used only to decide the final class)",[491,570,571,574],{},[511,572,573],{"align":496},"The delta rule already showed up in the other playlist",[511,575,576,577,580],{"align":496},"Adaline ",[15,578,579],{},"is"," the delta rule applied to classification, training as if it were regression on the ±1 labels",[491,582,583,586],{},[511,584,585],{"align":496},"Finding a line that separates seems like enough",[511,587,588],{"align":496},"Optimizing RMSE (not just counting mistakes) keeps finding a better boundary even after accuracy already hit 100%",[28,590,592],{"id":591},"practical-application","Practical application",[11,594,595,596,600,601,604],{},"I reproduced pseudo-inverse vs. batch gradient descent on Iris (",[597,598,599],"em",{},"setosa"," vs. ",[597,602,603],{},"versicolor","), by now a familiar face in this playlist, but this time training as Adaline, on the continuous ±1 target, not as a perceptron.",[33,606,608],{"className":35,"code":607,"language":37,"meta":38,"style":38},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\nw_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[40,609,610,615],{"__ignoreMap":38},[43,611,612],{"class":45,"line":46},[43,613,614],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",[43,616,617],{"class":45,"line":52},[43,618,619],{},"w_pinv = np.linalg.pinv(include_bias(X_train)) @ y_train\n",[11,621,622,623,626,627,630,631,634,635,638,639,642],{},"On the first attempt, I used the notebook's original ",[40,624,625],{},"learning_rate=0.01"," for batch gradient descent, and it ",[15,628,629],{},"diverged"," (weight became ",[40,632,633],{},"NaN",") straight away on raw Iris data. No surprise at this point: ",[106,636,637],{"href":108},"it's the same scale lesson from the normal equation post",", just rediscovered again, this time needing a much smaller rate (",[40,640,641],{},"0.001",") to not blow up.",[485,644,645,657],{},[488,646,647],{},[491,648,649,651,654],{},[494,650,497],{"align":496},[494,652,653],{"align":500},"RMSE (train)",[494,655,656],{"align":500},"Accuracy (test)",[506,658,659,671],{},[491,660,661,664,667],{},[511,662,663],{"align":496},"Pseudo-inverse (Adaline)",[511,665,666],{"align":500},"0.2414",[511,668,669],{"align":500},[15,670,247],{},[491,672,673,680,682],{},[511,674,675,676,679],{"align":496},"Batch gradient descent, ",[40,677,678],{},"learning_rate=0.001"," (Adaline)",[511,681,666],{"align":500},[511,683,684],{"align":500},[15,685,247],{},[11,687,688,689,692,693,696],{},"With the rate adjusted, same result from both methods again, RMSE identical to the fourth decimal, and both accuracies hitting 100%, ",[106,690,691],{"href":381},"the same generous margin from Iris that already favored the perceptron with bias",". The real difference between the methods, on this easy dataset, remains just convergence speed (and sensitivity to the choice of ",[40,694,695],{},"learning_rate","), not the quality of the final solution.",[698,699,700],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":38,"searchDepth":52,"depth":52,"links":702},[703,704,705,706,707,708],{"id":30,"depth":52,"text":31},{"id":286,"depth":52,"text":287},{"id":319,"depth":52,"text":320},{"id":406,"depth":52,"text":407},{"id":542,"depth":52,"text":543},{"id":591,"depth":52,"text":592},null,"2026-08-20","Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta","neural-networks",{"title":6,"description":711},"published","en\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta",[720,721,722],"adaline","delta-rule","widrow-hoff","tDtwjGKmRJJOKzWjkAyQ72-9DNH1O8sc9YdTGBFjxlk",1787338982669]