[{"data":1,"prerenderedAt":755},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal":3,"post-en-neural-networks-regressao-e-equacao-normal":4},"\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal",{"id":5,"title":6,"body":7,"cover":740,"date":741,"description":742,"extension":743,"meta":744,"navigation":161,"order":51,"path":745,"playlist":746,"seo":747,"status":748,"stem":749,"tags":750,"__hash__":754},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal.md","From Classification to Regression, and the One-Line Solution",{"type":8,"value":9,"toc":732},"minimark",[10,19,26,79,98,175,324,353,372,376,435,463,500,516,535,539,553,565,569,629,633,649,658,673,690,722,728],[11,12,13,14,18],"p",{},"Lectures 2c and 2d. The code structure barely changes, but the meaning shifts a lot: ",[15,16,17],"code",{},"sign()"," goes out, a continuous prediction comes in, and at the end a closed-form computation replaces hundreds of iterations.",[20,21,23,24],"h2",{"id":22},"swapping-classification-for-regression-just-drop-the-sign","Swapping classification for regression: just drop the ",[15,25,17],{},[27,28,33],"pre",{"className":29,"code":30,"language":31,"meta":32,"style":32},"language-python shiki shiki-themes github-light github-dark","def createRegressionDataset(n=20):\n  X = np.random.rand(n,1)\n  coef = 0.7\n  intercept = 0.2\n  noise = np.random.randn(n,1) * 0.1\n  y = X * coef + intercept + noise\n  return X, y.reshape(-1)\n","python","",[15,34,35,43,49,55,61,67,73],{"__ignoreMap":32},[36,37,40],"span",{"class":38,"line":39},"line",1,[36,41,42],{},"def createRegressionDataset(n=20):\n",[36,44,46],{"class":38,"line":45},2,[36,47,48],{},"  X = np.random.rand(n,1)\n",[36,50,52],{"class":38,"line":51},3,[36,53,54],{},"  coef = 0.7\n",[36,56,58],{"class":38,"line":57},4,[36,59,60],{},"  intercept = 0.2\n",[36,62,64],{"class":38,"line":63},5,[36,65,66],{},"  noise = np.random.randn(n,1) * 0.1\n",[36,68,70],{"class":38,"line":69},6,[36,71,72],{},"  y = X * coef + intercept + noise\n",[36,74,76],{"class":38,"line":75},7,[36,77,78],{},"  return X, y.reshape(-1)\n",[11,80,81,82,85,86,89,90,93,94,97],{},"Unlike the earlier datasets, ",[15,83,84],{},"y"," here isn't ",[15,87,88],{},"-1"," or ",[15,91,92],{},"+1"," anymore, it's a continuous number, ",[15,95,96],{},"0.7 * x + 0.2"," plus a small Gaussian noise. There's no class to get right, there's a line to find.",[27,99,101],{"className":29,"code":100,"language":31,"meta":32,"style":32},"class LinearRegression(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    self.w_ = np.random.rand(X.shape[1])\n    self.b_ = np.random.rand()\n    for _ in range(self.max_iter):\n      y_pred = self.predict(X)\n      error = y - y_pred\n      self.w_ += np.dot(X.T, error) * self.learning_rate\n      self.b_ += np.sum(error) * self.learning_rate\n    return self\n\n  def predict(self, X):\n    return X @ self.w_ + self.b_\n",[15,102,103,108,113,118,123,128,133,138,144,150,156,163,169],{"__ignoreMap":32},[36,104,105],{"class":38,"line":39},[36,106,107],{},"class LinearRegression(BaseEstimator, ClassifierMixin):\n",[36,109,110],{"class":38,"line":45},[36,111,112],{},"  def fit(self, X, y):\n",[36,114,115],{"class":38,"line":51},[36,116,117],{},"    self.w_ = np.random.rand(X.shape[1])\n",[36,119,120],{"class":38,"line":57},[36,121,122],{},"    self.b_ = np.random.rand()\n",[36,124,125],{"class":38,"line":63},[36,126,127],{},"    for _ in range(self.max_iter):\n",[36,129,130],{"class":38,"line":69},[36,131,132],{},"      y_pred = self.predict(X)\n",[36,134,135],{"class":38,"line":75},[36,136,137],{},"      error = y - y_pred\n",[36,139,141],{"class":38,"line":140},8,[36,142,143],{},"      self.w_ += np.dot(X.T, error) * self.learning_rate\n",[36,145,147],{"class":38,"line":146},9,[36,148,149],{},"      self.b_ += np.sum(error) * self.learning_rate\n",[36,151,153],{"class":38,"line":152},10,[36,154,155],{},"    return self\n",[36,157,159],{"class":38,"line":158},11,[36,160,162],{"emptyLinePlaceholder":161},true,"\n",[36,164,166],{"class":38,"line":165},12,[36,167,168],{},"  def predict(self, X):\n",[36,170,172],{"class":38,"line":171},13,[36,173,174],{},"    return X @ self.w_ + self.b_\n",[11,176,177,178,182,183,188,189,192,193,196,197,200,201,203,204,208,209,306,307,310,311,314,315,318,319,323],{},"Notice how this is ",[179,180,181],"strong",{},"almost identical"," to the vectorized perceptron ",[184,185,187],"a",{"href":186},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fperceptron-com-bias","from last post",": same ",[15,190,191],{},"X.T @ error"," to update the weight, same ",[15,194,195],{},"sum(error)"," to update the bias. The two differences are small in code but change the whole meaning: first, ",[15,198,199],{},"predict"," no longer goes through ",[15,202,17],{},", the prediction stays continuous (linear activation, ",[184,205,207],{"href":206},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","the same idea Aggarwal describes"," as the simplest activation function there is, ",[36,210,213,253],{"className":211},[212],"katex",[36,214,217],{"className":215},[216],"katex-mathml",[218,219,221],"math",{"xmlns":220},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[222,223,224,248],"semantics",{},[225,226,227,232,237,240,243,246],"mrow",{},[228,229,231],"mi",{"mathvariant":230},"normal","Φ",[233,234,236],"mo",{"stretchy":235},"false","(",[228,238,239],{},"v",[233,241,242],{"stretchy":235},")",[233,244,245],{},"=",[228,247,239],{},[249,250,252],"annotation",{"encoding":251},"application\u002Fx-tex","\\Phi(v) = v",[36,254,258,296],{"className":255,"ariaHidden":257},[256],"katex-html","true",[36,259,262,267,271,275,280,284,289,293],{"className":260},[261],"base",[36,263],{"className":264,"style":266},[265],"strut","height:1em;vertical-align:-0.25em;",[36,268,231],{"className":269},[270],"mord",[36,272,236],{"className":273},[274],"mopen",[36,276,239],{"className":277,"style":279},[270,278],"mathnormal","margin-right:0.0359em;",[36,281,242],{"className":282},[283],"mclose",[36,285],{"className":286,"style":288},[287],"mspace","margin-right:0.2778em;",[36,290,245],{"className":291},[292],"mrel",[36,294],{"className":295,"style":288},[287],[36,297,299,303],{"className":298},[261],[36,300],{"className":301,"style":302},[265],"height:0.4306em;",[36,304,239],{"className":305,"style":279},[270,278],"). Second, a ",[15,308,309],{},"learning_rate"," shows up multiplying the update. In the perceptron this learning rate didn't even exist, it was implicitly ",[15,312,313],{},"1",". Aggarwal calls this an interesting quirk of the perceptron, you can fix the rate at 1 because it only rescales the weight, not the direction of the adjustment. Here, with a continuous error instead of an error in ",[15,316,317],{},"{-2,0,+2}",", the adjustment's magnitude can end up too big or too small depending on the error's scale, and that's why an explicit learning rate becomes necessary to control the step size, ",[184,320,322],{"href":321},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling","the exact subject that already earned a whole post in the other playlist",".",[325,326,327],"blockquote",{},[11,328,329,332,333,336,337,340,341,344,345,348,349,352],{},[179,330,331],{},"Output:"," RMSE ",[15,334,335],{},"0.0948",", weights ",[15,338,339],{},"[0.706]",", bias ",[15,342,343],{},"0.204",". Pretty close to the true generator (",[15,346,347],{},"0.7"," and ",[15,350,351],{},"0.2","), the difference is just the noise baked in on purpose.",[11,354,355,356,359,360,363,364,367,368,371],{},"One honest note: the notebook's next cell calls ",[15,357,358],{},"createDataset"," (not ",[15,361,362],{},"createRegressionDataset","), ",[15,365,366],{},"accuracy_score",", and ",[15,369,370],{},"plotHyperplan",", names that don't exist in this notebook, only in the previous lecture's. That only ran because the professor's Colab still had the previous session in memory (a variable reused lecture to lecture). Running this notebook fresh would break that cell. I don't reproduce it here, since it doesn't actually test the regression model that was just trained.",[20,373,375],{"id":374},"the-normal-equation-the-same-question-solved-without-iterating","The normal equation: the same question, solved without iterating",[27,377,379],{"className":29,"code":378,"language":31,"meta":32,"style":32},"def include_bias(X):\n  return np.hstack((np.ones((X.shape[0],1)), X))\n\nclass NormalEquation(BaseEstimator, ClassifierMixin):\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = np.linalg.pinv(X) @ y\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return X @ self.w_\n",[15,380,381,386,391,395,400,404,409,414,418,422,426,430],{"__ignoreMap":32},[36,382,383],{"class":38,"line":39},[36,384,385],{},"def include_bias(X):\n",[36,387,388],{"class":38,"line":45},[36,389,390],{},"  return np.hstack((np.ones((X.shape[0],1)), X))\n",[36,392,393],{"class":38,"line":51},[36,394,162],{"emptyLinePlaceholder":161},[36,396,397],{"class":38,"line":57},[36,398,399],{},"class NormalEquation(BaseEstimator, ClassifierMixin):\n",[36,401,402],{"class":38,"line":63},[36,403,112],{},[36,405,406],{"class":38,"line":69},[36,407,408],{},"    X = include_bias(X)\n",[36,410,411],{"class":38,"line":75},[36,412,413],{},"    self.w_ = np.linalg.pinv(X) @ y\n",[36,415,416],{"class":38,"line":140},[36,417,155],{},[36,419,420],{"class":38,"line":146},[36,421,162],{"emptyLinePlaceholder":161},[36,423,424],{"class":38,"line":152},[36,425,168],{},[36,427,428],{"class":38,"line":158},[36,429,408],{},[36,431,432],{"class":38,"line":165},[36,433,434],{},"    return X @ self.w_\n",[11,436,437,438,441,442,444,445,448,449,452,453,455,456,459,460,323],{},"Two new things here. The first is ",[15,439,440],{},"include_bias",": sticks a column of ",[15,443,313],{},"s in front of ",[15,446,447],{},"X",". This is literally the trick I described in words ",[184,450,451],{"href":186},"in the last post",", bias as the weight of a phantom variable always worth ",[15,454,313],{},", now written as real code instead of a separate ",[15,457,458],{},"b_",". With that extra column, the bias just becomes another regular weight inside ",[15,461,462],{},"w_",[11,464,465,466,469,470,472,473,476,477,481,482,485,486,489,490,492,493,496,497,499],{},"The second is ",[15,467,468],{},"NormalEquation"," itself: no loop, no ",[15,471,309],{},", just ",[15,474,475],{},"np.linalg.pinv(X) @ y",". That's the closed-form solution ",[184,478,480],{"href":479},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation","I already explored in detail back in Pattern Recognition",", the same question (\"which line minimizes squared error\") solved directly, in a single computation, instead of stepping down gradually. The technical difference here is the method: there, I implemented it via Gauss-Jordan elimination with pivoting. The professor uses the ",[179,483,484],{},"pseudo-inverse"," (",[15,487,488],{},"pinv","), which solves via SVD decomposition under the hood. The advantage of the pseudo-inverse is that it never gets stuck: even if ",[15,491,447],{}," has redundant (linearly dependent) columns and ",[15,494,495],{},"X^T X"," can't be inverted the traditional way, ",[15,498,488],{}," still returns a valid answer (the smallest-norm one among the infinitely many possible solutions). Plain Gauss-Jordan elimination, in that same case, simply breaks.",[325,501,502],{},[11,503,504,332,506,336,509,512,513,515],{},[179,505,331],{},[15,507,508],{},"0.12364226682065012",[15,510,511],{},"[0.25402951 0.61056989]"," (bias and coefficient, in that order, because of ",[15,514,440],{},").",[11,517,518,519,522,523,526,527,530,531,534],{},"And the gradient descent from the previous cell, on the same dataset, landed at RMSE ",[15,520,521],{},"0.12364226680246916",", essentially identical weights. Two completely different computations (one iterative, one closed-form) converging to the exact same place, down to the seventh decimal. I reproduced this myself with a seeded dataset (",[15,524,525],{},"np.random.seed(7)",") to confirm it wasn't a one-run coincidence: GD gave weights ",[15,528,529],{},"[0.6478, bias 0.2041]",", normal equation gave ",[15,532,533],{},"[bias 0.2041, coef 0.6478]",", identical RMSE down to the ninth decimal in both.",[20,536,538],{"id":537},"interactive-find-the-line-yourself","Interactive: find the line yourself",[11,540,541,542,545,546,348,549,552],{},"Before watching the machine solve it, try solving it by hand. This is the same seeded dataset from above (20 real points generated by ",[15,543,544],{},"coef=0.7, intercept=0.2"," plus noise), drag the ",[15,547,548],{},"w",[15,550,551],{},"b"," sliders and watch the total error change live.",[554,555],"model-playground",{":b-max":313,":b-min":556,":b-step":557,":initial-b":558,":initial-w":558,":w-max":559,":w-min":558,":w-step":557,":x-train":560,":y-train":561,"dataLabel":562,"prediction-label":563,"x-label":564,"y-label":84},"-0.5","0.05","0","1.5","[0.0763, 0.7799, 0.4384, 0.7235, 0.978, 0.5385, 0.5011, 0.0721, 0.2684, 0.4999, 0.6792, 0.8037, 0.3809, 0.0659, 0.2881, 0.9096, 0.2134, 0.4521, 0.9312, 0.0249]","[0.3089, 0.7583, 0.5343, 0.5538, 1.0497, 0.5924, 0.5121, 0.4533, 0.3834, 0.4048, 0.6349, 0.5338, 0.5716, 0.2045, 0.3274, 0.944, 0.1843, 0.57, 0.6454, 0.1512]","training points","fitted line","x",[20,566,568],{"id":567},"wrapping-up","Wrapping up",[570,571,572,586],"table",{},[573,574,575],"thead",{},[576,577,578,583],"tr",{},[579,580,582],"th",{"align":581},"left","What I already knew",[579,584,585],{"align":581},"What this lecture settled",[587,588,589,603,616],"tbody",{},[576,590,591,597],{},[592,593,594,595],"td",{"align":581},"The perceptron classifies with ",[15,596,17],{},[592,598,599,600,602],{"align":581},"Dropping ",[15,601,17],{}," from the exact same code structure turns classification into regression, almost without touching anything else",[576,604,605,608],{},[592,606,607],{"align":581},"Bias is the weight of a phantom variable worth 1",[592,609,610,611,613,614],{"align":581},"That becomes explicit code with ",[15,612,440],{},", instead of a separate ",[15,615,458],{},[576,617,618,621],{},[592,619,620],{"align":581},"I already solved the normal equation via Gauss-Jordan",[592,622,623,625,626,628],{"align":581},[15,624,488],{}," (pseudo-inverse via SVD) solves the same computation and still works when ",[15,627,495],{}," isn't invertible",[20,630,632],{"id":631},"practical-application","Practical application",[11,634,635,636,640,641,644,645,648],{},"I tested gradient descent against the normal equation on the real 50-house dataset ",[184,637,639],{"href":638},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","that already showed up in the other playlist",", using just ",[15,642,643],{},"square_feet"," to predict ",[15,646,647],{},"price",", with nothing normalized first.",[27,650,652],{"className":29,"code":651,"language":31,"meta":32,"style":32},"w, b = fit_gd(X, y, max_iter=1000, learning_rate=0.01)  # not normalized\n",[15,653,654],{"__ignoreMap":32},[36,655,656],{"class":38,"line":39},[36,657,651],{},[325,659,660],{},[11,661,662,664,665,668,669,672],{},[179,663,331],{}," the weight becomes ",[15,666,667],{},"-inf"," by iteration 71. Gradient descent ",[179,670,671],{},"diverges"," completely.",[11,674,675,676,679,680,682,683,685,686,689],{},"Expected: ",[184,677,678],{"href":321},"it's the same feature-scaling lesson from the other playlist",", just rediscovered here inside a neural-network context. ",[15,681,643],{}," lives in the hundreds and ",[15,684,647],{}," in the hundreds of thousands, so the gradient is huge and a ",[15,687,688],{},"0.01"," step blows up.",[570,691,692,703],{},[573,693,694],{},[576,695,696,699],{},[579,697,698],{"align":581},"Approach",[579,700,702],{"align":701},"right","RMSE",[587,704,705,713],{},[576,706,707,710],{},[592,708,709],{"align":581},"Gradient descent, normalized data",[592,711,712],{"align":701},"101878.42",[576,714,715,718],{},[592,716,717],{"align":581},"Normal equation, raw data (not normalized)",[592,719,720],{"align":701},[179,721,712],{},[11,723,724,725,727],{},"Normalizing ",[15,726,643],{}," before running gradient descent, it converges and lands at exactly the same RMSE the normal equation finds directly on the raw data, no normalization needed. Makes sense: the normal equation solves the linear system in one shot, so the variables' scale only affects the computation's numerical stability, not whether it converges at all (unlike gradient descent, which can literally diverge if the step is too big for the data's scale).",[729,730,731],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":32,"searchDepth":45,"depth":45,"links":733},[734,736,737,738,739],{"id":22,"depth":45,"text":735},"Swapping classification for regression: just drop the sign()",{"id":374,"depth":45,"text":375},{"id":537,"depth":45,"text":538},{"id":567,"depth":45,"text":568},{"id":631,"depth":45,"text":632},null,"2026-08-20","Lectures 2c and 2d: the professor swaps the binary sign for a continuous line (linear regression) and then replaces the whole gradient descent loop with a single closed-form computation, the normal equation via pseudo-inverse.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal","neural-networks",{"title":6,"description":742},"published","en\u002Fplaylists\u002Fneural-networks\u002Fregressao-e-equacao-normal",[751,752,753],"linear-regression","normal-equation","gradient-descent","Npkp7jgJ9TPUTQRrmQtdpLENefGbKgpSYQLH1QpMI2w",1787338984117]