[{"data":1,"prerenderedAt":2255},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation":3,"post-en-pattern-recognition-normal-equation":4},"\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation",{"id":5,"title":6,"body":7,"cover":2241,"date":2242,"description":2243,"extension":2244,"meta":2245,"navigation":1645,"order":1237,"path":2246,"playlist":2247,"seo":2248,"status":2249,"stem":2250,"tags":2251,"__hash__":2254},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation.md","The Normal Equation: Solving Regression in One Shot",{"type":8,"value":9,"toc":2230},"minimark",[10,19,24,27,35,601,608,866,873,877,1212,1219,1241,1264,1268,1279,1356,1372,1379,1607,1664,1741,1745,1752,1791,1804,1810,1814,1821,1925,1928,1965,1970,1974,2175,2179,2191,2210,2219,2226],[11,12,13,18],"p",{},[14,15,17],"a",{"href":16},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","The previous post"," left a hook: you can solve linear regression in a single computation, no gradient descent needed. That's exactly what lectures 2b and 2c show, and along the way the professor compares the result against five other very different kinds of regressor.",[20,21,23],"h2",{"id":22},"zeroing-the-derivative-instead-of-walking-toward-it","Zeroing the derivative instead of walking toward it",[11,25,26],{},"Gradient descent works because, at every step, it walks a little in the direction that reduces the error. But for linear regression, the error function (the same sum of squares I'd already been using) has a special property: it's a parabola with respect to the weights, a smooth bowl with no false valleys. And a smooth bowl has exactly one point where the derivative is zero, its bottom. Instead of walking there step by step, you can compute that point directly.",[11,28,29,30,34],{},"Bishop writes the linear model in a more general form than mine, with a matrix Φ (the ",[31,32,33],"strong",{},"design matrix",", one row per patient, one column per input variable, plus a column of 1's for the bias term). The error function is",[11,36,37],{},[38,39,42,158],"span",{"className":40},[41],"katex",[38,43,46],{"className":44},[45],"katex-mathml",[47,48,50],"math",{"xmlns":49},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[51,52,53,153],"semantics",{},[54,55,56,66,71,75,78,81,91,109,112,119,122,131,135,137,144,146],"mrow",{},[57,58,59,63],"msub",{},[60,61,62],"mi",{},"E",[60,64,65],{},"D",[67,68,70],"mo",{"stretchy":69},"false","(",[60,72,74],{"mathvariant":73},"bold","w",[67,76,77],{"stretchy":69},")",[67,79,80],{},"=",[82,83,84,88],"mfrac",{},[85,86,87],"mn",{},"1",[85,89,90],{},"2",[92,93,94,97,106],"msubsup",{},[67,95,96],{},"∑",[54,98,99,102,104],{},[60,100,101],{},"n",[67,103,80],{},[85,105,87],{},[60,107,108],{},"N",[67,110,111],{"stretchy":69},"{",[57,113,114,117],{},[60,115,116],{},"t",[60,118,101],{},[67,120,121],{},"−",[123,124,125,127],"msup",{},[60,126,74],{"mathvariant":73},[60,128,130],{"mathvariant":129},"sans-serif","T",[60,132,134],{"mathvariant":133},"bold-italic","ϕ",[67,136,70],{"stretchy":69},[57,138,139,142],{},[60,140,141],{"mathvariant":73},"x",[60,143,101],{},[67,145,77],{"stretchy":69},[123,147,148,151],{},[67,149,150],{"stretchy":69},"}",[85,152,90],{},[154,155,157],"annotation",{"encoding":156},"application\u002Fx-tex","E_D(\\mathbf{w}) = \\frac{1}{2}\\sum_{n=1}^{N}\\{t_n - \\mathbf{w}^{\\mathsf{T}}\\boldsymbol{\\phi}(\\mathbf{x}_n)\\}^2",[38,159,163,257,473],{"className":160,"ariaHidden":162},[161],"katex-html","true",[38,164,167,172,232,236,241,245,250,254],{"className":165},[166],"base",[38,168],{"className":169,"style":171},[170],"strut","height:1em;vertical-align:-0.25em;",[38,173,176,181],{"className":174},[175],"mord",[38,177,62],{"className":178,"style":180},[175,179],"mathnormal","margin-right:0.0576em;",[38,182,185],{"className":183},[184],"msupsub",[38,186,190,223],{"className":187},[188,189],"vlist-t","vlist-t2",[38,191,194,218],{"className":192},[193],"vlist-r",[38,195,199],{"className":196,"style":198},[197],"vlist","height:0.3283em;",[38,200,202,207],{"style":201},"top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;",[38,203],{"className":204,"style":206},[205],"pstrut","height:2.7em;",[38,208,214],{"className":209},[210,211,212,213],"sizing","reset-size6","size3","mtight",[38,215,65],{"className":216,"style":217},[175,179,213],"margin-right:0.0278em;",[38,219,222],{"className":220},[221],"vlist-s","​",[38,224,226],{"className":225},[193],[38,227,230],{"className":228,"style":229},[197],"height:0.15em;",[38,231],{},[38,233,70],{"className":234},[235],"mopen",[38,237,74],{"className":238,"style":240},[175,239],"mathbf","margin-right:0.016em;",[38,242,77],{"className":243},[244],"mclose",[38,246],{"className":247,"style":249},[248],"mspace","margin-right:0.2778em;",[38,251,80],{"className":252},[253],"mrel",[38,255],{"className":256,"style":249},[248],[38,258,260,264,341,345,417,420,462,466,470],{"className":259},[166],[38,261],{"className":262,"style":263},[170],"height:1.3262em;vertical-align:-0.345em;",[38,265,267,271,338],{"className":266},[175],[38,268],{"className":269},[235,270],"nulldelimiter",[38,272,274],{"className":273},[82],[38,275,277,329],{"className":276},[188,189],[38,278,280,326],{"className":279},[193],[38,281,284,300,311],{"className":282,"style":283},[197],"height:0.8451em;",[38,285,287,291],{"style":286},"top:-2.655em;",[38,288],{"className":289,"style":290},[205],"height:3em;",[38,292,294],{"className":293},[210,211,212,213],[38,295,297],{"className":296},[175,213],[38,298,90],{"className":299},[175,213],[38,301,303,306],{"style":302},"top:-3.23em;",[38,304],{"className":305,"style":290},[205],[38,307],{"className":308,"style":310},[309],"frac-line","border-bottom-width:0.04em;",[38,312,314,317],{"style":313},"top:-3.394em;",[38,315],{"className":316,"style":290},[205],[38,318,320],{"className":319},[210,211,212,213],[38,321,323],{"className":322},[175,213],[38,324,87],{"className":325},[175,213],[38,327,222],{"className":328},[221],[38,330,332],{"className":331},[193],[38,333,336],{"className":334,"style":335},[197],"height:0.345em;",[38,337],{},[38,339],{"className":340},[244,270],[38,342],{"className":343,"style":344},[248],"margin-right:0.1667em;",[38,346,349,355],{"className":347},[348],"mop",[38,350,96],{"className":351,"style":354},[348,352,353],"op-symbol","small-op","position:relative;top:0em;",[38,356,358],{"className":357},[184],[38,359,361,408],{"className":360},[188,189],[38,362,364,405],{"className":363},[193],[38,365,368,389],{"className":366,"style":367},[197],"height:0.9812em;",[38,369,371,374],{"style":370},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[38,372],{"className":373,"style":206},[205],[38,375,377],{"className":376},[210,211,212,213],[38,378,380,383,386],{"className":379},[175,213],[38,381,101],{"className":382},[175,179,213],[38,384,80],{"className":385},[253,213],[38,387,87],{"className":388},[175,213],[38,390,392,395],{"style":391},"top:-3.2029em;margin-right:0.05em;",[38,393],{"className":394,"style":206},[205],[38,396,398],{"className":397},[210,211,212,213],[38,399,401],{"className":400},[175,213],[38,402,108],{"className":403,"style":404},[175,179,213],"margin-right:0.109em;",[38,406,222],{"className":407},[221],[38,409,411],{"className":410},[193],[38,412,415],{"className":413,"style":414},[197],"height:0.2997em;",[38,416],{},[38,418,111],{"className":419},[235],[38,421,423,426],{"className":422},[175],[38,424,116],{"className":425},[175,179],[38,427,429],{"className":428},[184],[38,430,432,454],{"className":431},[188,189],[38,433,435,451],{"className":434},[193],[38,436,439],{"className":437,"style":438},[197],"height:0.1514em;",[38,440,442,445],{"style":441},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[38,443],{"className":444,"style":206},[205],[38,446,448],{"className":447},[210,211,212,213],[38,449,101],{"className":450},[175,179,213],[38,452,222],{"className":453},[221],[38,455,457],{"className":456},[193],[38,458,460],{"className":459,"style":229},[197],[38,461],{},[38,463],{"className":464,"style":465},[248],"margin-right:0.2222em;",[38,467,121],{"className":468},[469],"mbin",[38,471],{"className":472,"style":465},[248],[38,474,476,480,515,525,528,568,571],{"className":475},[166],[38,477],{"className":478,"style":479},[170],"height:1.0991em;vertical-align:-0.25em;",[38,481,483,486],{"className":482},[175],[38,484,74],{"className":485,"style":240},[175,239],[38,487,489],{"className":488},[184],[38,490,492],{"className":491},[188],[38,493,495],{"className":494},[193],[38,496,499],{"className":497,"style":498},[197],"height:0.8491em;",[38,500,502,505],{"style":501},"top:-3.063em;margin-right:0.05em;",[38,503],{"className":504,"style":206},[205],[38,506,508],{"className":507},[210,211,212,213],[38,509,511],{"className":510},[175,213],[38,512,130],{"className":513},[175,514,213],"mathsf",[38,516,518],{"className":517},[175],[38,519,521],{"className":520},[175],[38,522,134],{"className":523},[175,524],"boldsymbol",[38,526,70],{"className":527},[235],[38,529,531,534],{"className":530},[175],[38,532,141],{"className":533},[175,239],[38,535,537],{"className":536},[184],[38,538,540,560],{"className":539},[188,189],[38,541,543,557],{"className":542},[193],[38,544,546],{"className":545,"style":438},[197],[38,547,548,551],{"style":441},[38,549],{"className":550,"style":206},[205],[38,552,554],{"className":553},[210,211,212,213],[38,555,101],{"className":556},[175,179,213],[38,558,222],{"className":559},[221],[38,561,563],{"className":562},[193],[38,564,566],{"className":565,"style":229},[197],[38,567],{},[38,569,77],{"className":570},[244],[38,572,574,577],{"className":573},[244],[38,575,150],{"className":576},[244],[38,578,580],{"className":579},[184],[38,581,583],{"className":582},[188],[38,584,586],{"className":585},[193],[38,587,590],{"className":588,"style":589},[197],"height:0.8141em;",[38,591,592,595],{"style":501},[38,593],{"className":594,"style":206},[205],[38,596,598],{"className":597},[210,211,212,213],[38,599,90],{"className":600},[175,213],[11,602,603,604,607],{},"Setting this function's gradient to zero and solving for ",[605,606,74],"code",{}," leaves",[11,609,610],{},[38,611,613,664],{"className":612},[41],[38,614,616],{"className":615},[45],[47,617,618],{"xmlns":49},[51,619,620,661],{},[54,621,622,630,632,634,641,643,653,659],{},[57,623,624,626],{},[60,625,74],{"mathvariant":73},[627,628,629],"mtext",{},"ML",[67,631,80],{},[67,633,70],{"stretchy":69},[123,635,636,639],{},[60,637,638],{"mathvariant":73},"Φ",[60,640,130],{"mathvariant":129},[60,642,638],{"mathvariant":73},[123,644,645,647],{},[67,646,77],{"stretchy":69},[54,648,649,651],{},[67,650,121],{},[85,652,87],{},[123,654,655,657],{},[60,656,638],{"mathvariant":73},[60,658,130],{"mathvariant":129},[60,660,116],{"mathvariant":73},[154,662,663],{"encoding":156},"\\mathbf{w}_{\\text{ML}} = (\\boldsymbol{\\Phi}^{\\mathsf{T}}\\boldsymbol{\\Phi})^{-1}\\boldsymbol{\\Phi}^{\\mathsf{T}}\\mathbf{t}",[38,665,667,731],{"className":666,"ariaHidden":162},[161],[38,668,670,674,722,725,728],{"className":669},[166],[38,671],{"className":672,"style":673},[170],"height:0.5944em;vertical-align:-0.15em;",[38,675,677,680],{"className":676},[175],[38,678,74],{"className":679,"style":240},[175,239],[38,681,683],{"className":682},[184],[38,684,686,714],{"className":685},[188,189],[38,687,689,711],{"className":688},[193],[38,690,692],{"className":691,"style":198},[197],[38,693,695,698],{"style":694},"top:-2.55em;margin-left:-0.016em;margin-right:0.05em;",[38,696],{"className":697,"style":206},[205],[38,699,701],{"className":700},[210,211,212,213],[38,702,704],{"className":703},[175,213],[38,705,708],{"className":706},[175,707,213],"text",[38,709,629],{"className":710},[175,213],[38,712,222],{"className":713},[221],[38,715,717],{"className":716},[193],[38,718,720],{"className":719,"style":229},[197],[38,721],{},[38,723],{"className":724,"style":249},[248],[38,726,80],{"className":727},[253],[38,729],{"className":730,"style":249},[248],[38,732,734,738,741,781,790,825,863],{"className":733},[166],[38,735],{"className":736,"style":737},[170],"height:1.1751em;vertical-align:-0.25em;",[38,739,70],{"className":740},[235],[38,742,744,753],{"className":743},[175],[38,745,747],{"className":746},[175],[38,748,750],{"className":749},[175],[38,751,638],{"className":752},[175,239],[38,754,756],{"className":755},[184],[38,757,759],{"className":758},[188],[38,760,762],{"className":761},[193],[38,763,766],{"className":764,"style":765},[197],"height:0.9251em;",[38,767,769,772],{"style":768},"top:-3.139em;margin-right:0.05em;",[38,770],{"className":771,"style":206},[205],[38,773,775],{"className":774},[210,211,212,213],[38,776,778],{"className":777},[175,213],[38,779,130],{"className":780},[175,514,213],[38,782,784],{"className":783},[175],[38,785,787],{"className":786},[175],[38,788,638],{"className":789},[175,239],[38,791,793,796],{"className":792},[244],[38,794,77],{"className":795},[244],[38,797,799],{"className":798},[184],[38,800,802],{"className":801},[188],[38,803,805],{"className":804},[193],[38,806,808],{"className":807,"style":589},[197],[38,809,810,813],{"style":501},[38,811],{"className":812,"style":206},[205],[38,814,816],{"className":815},[210,211,212,213],[38,817,819,822],{"className":818},[175,213],[38,820,121],{"className":821},[175,213],[38,823,87],{"className":824},[175,213],[38,826,828,837],{"className":827},[175],[38,829,831],{"className":830},[175],[38,832,834],{"className":833},[175],[38,835,638],{"className":836},[175,239],[38,838,840],{"className":839},[184],[38,841,843],{"className":842},[188],[38,844,846],{"className":845},[193],[38,847,849],{"className":848,"style":765},[197],[38,850,851,854],{"style":768},[38,852],{"className":853,"style":206},[205],[38,855,857],{"className":856},[210,211,212,213],[38,858,860],{"className":859},[175,213],[38,861,130],{"className":862},[175,514,213],[38,864,116],{"className":865},[175,239],[11,867,868,869,872],{},"This is the ",[31,870,871],{},"normal equation",". One matrix, two multiplications, and an inversion, and the optimal fit falls right out, no learning rate, no choosing a number of iterations, none of the care gradient descent demands.",[20,874,876],{"id":875},"the-trick-of-folding-bias-in","The trick of folding bias in",[11,878,879,880,883,884,886,887,962,963,1071,1072,1141,1142,1211],{},"Notice the formula above has no separate ",[605,881,882],{},"+ b",", the bias lives inside ",[605,885,74],{}," itself (Bishop calls it ",[38,888,890,909],{"className":889},[41],[38,891,893],{"className":892},[45],[47,894,895],{"xmlns":49},[51,896,897,906],{},[54,898,899],{},[57,900,901,903],{},[60,902,74],{},[85,904,905],{},"0",[154,907,908],{"encoding":156},"w_0",[38,910,912],{"className":911,"ariaHidden":162},[161],[38,913,915,919],{"className":914},[166],[38,916],{"className":917,"style":918},[170],"height:0.5806em;vertical-align:-0.15em;",[38,920,922,926],{"className":921},[175],[38,923,74],{"className":924,"style":925},[175,179],"margin-right:0.0269em;",[38,927,929],{"className":928},[184],[38,930,932,954],{"className":931},[188,189],[38,933,935,951],{"className":934},[193],[38,936,939],{"className":937,"style":938},[197],"height:0.3011em;",[38,940,942,945],{"style":941},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[38,943],{"className":944,"style":206},[205],[38,946,948],{"className":947},[210,211,212,213],[38,949,905],{"className":950},[175,213],[38,952,222],{"className":953},[221],[38,955,957],{"className":956},[193],[38,958,960],{"className":959,"style":229},[197],[38,961],{},"). That only works because he defines a fake \"basis function,\" ",[38,964,966,994],{"className":965},[41],[38,967,969],{"className":968},[45],[47,970,971],{"xmlns":49},[51,972,973,991],{},[54,974,975,981,983,985,987,989],{},[57,976,977,979],{},[60,978,134],{},[85,980,905],{},[67,982,70],{"stretchy":69},[60,984,141],{"mathvariant":73},[67,986,77],{"stretchy":69},[67,988,80],{},[85,990,87],{},[154,992,993],{"encoding":156},"\\phi_0(\\mathbf{x}) = 1",[38,995,997,1061],{"className":996,"ariaHidden":162},[161],[38,998,1000,1003,1043,1046,1049,1052,1055,1058],{"className":999},[166],[38,1001],{"className":1002,"style":171},[170],[38,1004,1006,1009],{"className":1005},[175],[38,1007,134],{"className":1008},[175,179],[38,1010,1012],{"className":1011},[184],[38,1013,1015,1035],{"className":1014},[188,189],[38,1016,1018,1032],{"className":1017},[193],[38,1019,1021],{"className":1020,"style":938},[197],[38,1022,1023,1026],{"style":441},[38,1024],{"className":1025,"style":206},[205],[38,1027,1029],{"className":1028},[210,211,212,213],[38,1030,905],{"className":1031},[175,213],[38,1033,222],{"className":1034},[221],[38,1036,1038],{"className":1037},[193],[38,1039,1041],{"className":1040,"style":229},[197],[38,1042],{},[38,1044,70],{"className":1045},[235],[38,1047,141],{"className":1048},[175,239],[38,1050,77],{"className":1051},[244],[38,1053],{"className":1054,"style":249},[248],[38,1056,80],{"className":1057},[253],[38,1059],{"className":1060,"style":249},[248],[38,1062,1064,1068],{"className":1063},[166],[38,1065],{"className":1066,"style":1067},[170],"height:0.6444em;",[38,1069,87],{"className":1070},[175],", an entire column of 1's, so multiplying it by ",[38,1073,1075,1092],{"className":1074},[41],[38,1076,1078],{"className":1077},[45],[47,1079,1080],{"xmlns":49},[51,1081,1082,1090],{},[54,1083,1084],{},[57,1085,1086,1088],{},[60,1087,74],{},[85,1089,905],{},[154,1091,908],{"encoding":156},[38,1093,1095],{"className":1094,"ariaHidden":162},[161],[38,1096,1098,1101],{"className":1097},[166],[38,1099],{"className":1100,"style":918},[170],[38,1102,1104,1107],{"className":1103},[175],[38,1105,74],{"className":1106,"style":925},[175,179],[38,1108,1110],{"className":1109},[184],[38,1111,1113,1133],{"className":1112},[188,189],[38,1114,1116,1130],{"className":1115},[193],[38,1117,1119],{"className":1118,"style":938},[197],[38,1120,1121,1124],{"style":941},[38,1122],{"className":1123,"style":206},[205],[38,1125,1127],{"className":1126},[210,211,212,213],[38,1128,905],{"className":1129},[175,213],[38,1131,222],{"className":1132},[221],[38,1134,1136],{"className":1135},[193],[38,1137,1139],{"className":1138,"style":229},[197],[38,1140],{}," gives exactly ",[38,1143,1145,1162],{"className":1144},[41],[38,1146,1148],{"className":1147},[45],[47,1149,1150],{"xmlns":49},[51,1151,1152,1160],{},[54,1153,1154],{},[57,1155,1156,1158],{},[60,1157,74],{},[85,1159,905],{},[154,1161,908],{"encoding":156},[38,1163,1165],{"className":1164,"ariaHidden":162},[161],[38,1166,1168,1171],{"className":1167},[166],[38,1169],{"className":1170,"style":918},[170],[38,1172,1174,1177],{"className":1173},[175],[38,1175,74],{"className":1176,"style":925},[175,179],[38,1178,1180],{"className":1179},[184],[38,1181,1183,1203],{"className":1182},[188,189],[38,1184,1186,1200],{"className":1185},[193],[38,1187,1189],{"className":1188,"style":938},[197],[38,1190,1191,1194],{"style":941},[38,1192],{"className":1193,"style":206},[205],[38,1195,1197],{"className":1196},[210,211,212,213],[38,1198,905],{"className":1199},[175,213],[38,1201,222],{"className":1202},[221],[38,1204,1206],{"className":1205},[193],[38,1207,1209],{"className":1208,"style":229},[197],[38,1210],{}," for every patient, the same effect as adding a fixed bias.",[11,1213,1214,1215,1218],{},"The code does this by hand, gluing a column of 1's onto the front of ",[605,1216,1217],{},"X",":",[1220,1221,1226],"pre",{"className":1222,"code":1223,"language":1224,"meta":1225,"style":1225},"language-python shiki shiki-themes github-light github-dark","def include_bias(X):\n    return np.hstack((np.ones((X.shape[0], 1)), X))\n","python","",[605,1227,1228,1235],{"__ignoreMap":1225},[38,1229,1232],{"class":1230,"line":1231},"line",1,[38,1233,1234],{},"def include_bias(X):\n",[38,1236,1238],{"class":1230,"line":1237},2,[38,1239,1240],{},"    return np.hstack((np.ones((X.shape[0], 1)), X))\n",[11,1242,1243,1244,1247,1248,1251,1252,1255,1256,1259,1260,1263],{},"That changes the class's shape: instead of storing ",[605,1245,1246],{},"coefs_"," and ",[605,1249,1250],{},"intercept_"," as two separate things (",[14,1253,1254],{"href":16},"like in the previous post","), there's now a single vector ",[605,1257,1258],{},"w_",", where ",[605,1261,1262],{},"w_[0]"," is the bias and the rest are each variable's coefficient.",[20,1265,1267],{"id":1266},"three-versions-one-real-comparison","Three versions, one real comparison",[11,1269,1270,1271,1274,1275,1278],{},"The notebook runs three variations of the same ",[605,1272,1273],{},"LinearRegressor"," class, all on the same diabetes dataset (now using scikit-learn's real ",[605,1276,1277],{},"train_test_split",", so the numbers don't match the previous post's exactly, which used my own hand-rolled split):",[1280,1281,1282,1300],"table",{},[1283,1284,1285],"thead",{},[1286,1287,1288,1293,1296],"tr",{},[1289,1290,1292],"th",{"align":1291},"left","Version",[1289,1294,1295],{"align":1291},"How it solves",[1289,1297,1299],{"align":1298},"right","MSE (train)",[1301,1302,1303,1321,1338],"tbody",{},[1286,1304,1305,1309,1318],{},[1306,1307,1308],"td",{"align":1291},"Gradient, separate bias",[1306,1310,1311,1314,1315,1317],{"align":1291},[605,1312,1313],{},"coefs_ += X.T@error*0.001",", ",[605,1316,1250],{}," on the side",[1306,1319,1320],{"align":1298},"3142.25",[1286,1322,1323,1326,1335],{},[1306,1324,1325],{"align":1291},"Gradient, bias folded in",[1306,1327,1328,1329,1247,1332],{"align":1291},"same idea, but with ",[605,1330,1331],{},"include_bias",[605,1333,1334],{},"learning_rate=0.005",[1306,1336,1337],{"align":1298},"2898.90",[1286,1339,1340,1345,1351],{},[1306,1341,1342],{"align":1291},[31,1343,1344],{},"Normal equation",[1306,1346,1347,1350],{"align":1291},[605,1348,1349],{},"w_ = np.linalg.pinv(X) @ y",", no iterating",[1306,1352,1353],{"align":1298},[31,1354,1355],{},"2868.55",[11,1357,1358,1359,1362,1363,1366,1367,1371],{},"The normal equation beats both, without me having to pick a ",[605,1360,1361],{},"learning_rate"," or ",[605,1364,1365],{},"max_iter",". That's not a coincidence: gradient descent is a way to ",[1368,1369,1370],"em",{},"approximate"," this exact same answer by iterating, and with enough iterations and a good learning rate it converges to the same place. The normal equation just skips straight to the end.",[20,1373,1375,1376],{"id":1374},"the-pseudo-inverse-is-pinv","The pseudo-inverse is ",[605,1377,1378],{},"pinv",[11,1380,1381,1537,1538,1541,1542,1602,1603,1606],{},[38,1382,1384,1423],{"className":1383},[41],[38,1385,1387],{"className":1386},[45],[47,1388,1389],{"xmlns":49},[51,1390,1391,1420],{},[54,1392,1393,1395,1402,1404,1414],{},[67,1394,70],{"stretchy":69},[123,1396,1397,1400],{},[60,1398,638],{"mathvariant":1399},"normal",[60,1401,130],{"mathvariant":129},[60,1403,638],{"mathvariant":1399},[123,1405,1406,1408],{},[67,1407,77],{"stretchy":69},[54,1409,1410,1412],{},[67,1411,121],{},[85,1413,87],{},[123,1415,1416,1418],{},[60,1417,638],{"mathvariant":1399},[60,1419,130],{"mathvariant":129},[154,1421,1422],{"encoding":156},"(\\Phi^{\\mathsf{T}}\\Phi)^{-1}\\Phi^{\\mathsf{T}}",[38,1424,1426],{"className":1425,"ariaHidden":162},[161],[38,1427,1429,1432,1435,1467,1470,1505],{"className":1428},[166],[38,1430],{"className":1431,"style":479},[170],[38,1433,70],{"className":1434},[235],[38,1436,1438,1441],{"className":1437},[175],[38,1439,638],{"className":1440},[175],[38,1442,1444],{"className":1443},[184],[38,1445,1447],{"className":1446},[188],[38,1448,1450],{"className":1449},[193],[38,1451,1453],{"className":1452,"style":498},[197],[38,1454,1455,1458],{"style":501},[38,1456],{"className":1457,"style":206},[205],[38,1459,1461],{"className":1460},[210,211,212,213],[38,1462,1464],{"className":1463},[175,213],[38,1465,130],{"className":1466},[175,514,213],[38,1468,638],{"className":1469},[175],[38,1471,1473,1476],{"className":1472},[244],[38,1474,77],{"className":1475},[244],[38,1477,1479],{"className":1478},[184],[38,1480,1482],{"className":1481},[188],[38,1483,1485],{"className":1484},[193],[38,1486,1488],{"className":1487,"style":589},[197],[38,1489,1490,1493],{"style":501},[38,1491],{"className":1492,"style":206},[205],[38,1494,1496],{"className":1495},[210,211,212,213],[38,1497,1499,1502],{"className":1498},[175,213],[38,1500,121],{"className":1501},[175,213],[38,1503,87],{"className":1504},[175,213],[38,1506,1508,1511],{"className":1507},[175],[38,1509,638],{"className":1510},[175],[38,1512,1514],{"className":1513},[184],[38,1515,1517],{"className":1516},[188],[38,1518,1520],{"className":1519},[193],[38,1521,1523],{"className":1522,"style":498},[197],[38,1524,1525,1528],{"style":501},[38,1526],{"className":1527,"style":206},[205],[38,1529,1531],{"className":1530},[210,211,212,213],[38,1532,1534],{"className":1533},[175,213],[38,1535,130],{"className":1536},[175,514,213]," has its own name: the ",[31,1539,1540],{},"Moore-Penrose pseudo-inverse",", denoted ",[38,1543,1545,1564],{"className":1544},[41],[38,1546,1548],{"className":1547},[45],[47,1549,1550],{"xmlns":49},[51,1551,1552,1561],{},[54,1553,1554],{},[123,1555,1556,1558],{},[60,1557,638],{"mathvariant":1399},[67,1559,1560],{},"†",[154,1562,1563],{"encoding":156},"\\Phi^\\dagger",[38,1565,1567],{"className":1566,"ariaHidden":162},[161],[38,1568,1570,1573],{"className":1569},[166],[38,1571],{"className":1572,"style":498},[170],[38,1574,1576,1579],{"className":1575},[175],[38,1577,638],{"className":1578},[175],[38,1580,1582],{"className":1581},[184],[38,1583,1585],{"className":1584},[188],[38,1586,1588],{"className":1587},[193],[38,1589,1591],{"className":1590,"style":498},[197],[38,1592,1593,1596],{"style":501},[38,1594],{"className":1595,"style":206},[205],[38,1597,1599],{"className":1598},[210,211,212,213],[38,1600,1560],{"className":1601},[469,213],". It's a generalization of \"matrix inverse\" to matrices that aren't square (which is always the case here: Φ has one row per patient and one column per variable, almost never equal). ",[605,1604,1605],{},"np.linalg.pinv(X)"," computes exactly this, so the whole class shrinks to:",[1220,1608,1610],{"className":1222,"code":1609,"language":1224,"meta":1225,"style":1225},"class LinearRegressor(BaseEstimator, RegressorMixin):\n    def fit(self, X, y):\n        X = include_bias(X)\n        self.w_ = np.linalg.pinv(X) @ y\n        return self\n\n    def predict(self, X):\n        X = include_bias(X)\n        return (X @ self.w_).reshape(X.shape[0],)\n",[605,1611,1612,1617,1622,1628,1634,1640,1647,1653,1658],{"__ignoreMap":1225},[38,1613,1614],{"class":1230,"line":1231},[38,1615,1616],{},"class LinearRegressor(BaseEstimator, RegressorMixin):\n",[38,1618,1619],{"class":1230,"line":1237},[38,1620,1621],{},"    def fit(self, X, y):\n",[38,1623,1625],{"class":1230,"line":1624},3,[38,1626,1627],{},"        X = include_bias(X)\n",[38,1629,1631],{"class":1230,"line":1630},4,[38,1632,1633],{},"        self.w_ = np.linalg.pinv(X) @ y\n",[38,1635,1637],{"class":1230,"line":1636},5,[38,1638,1639],{},"        return self\n",[38,1641,1643],{"class":1230,"line":1642},6,[38,1644,1646],{"emptyLinePlaceholder":1645},true,"\n",[38,1648,1650],{"class":1230,"line":1649},7,[38,1651,1652],{},"    def predict(self, X):\n",[38,1654,1656],{"class":1230,"line":1655},8,[38,1657,1627],{},[38,1659,1661],{"class":1230,"line":1660},9,[38,1662,1663],{},"        return (X @ self.w_).reshape(X.shape[0],)\n",[11,1665,1666,1667,1672,1673,1740],{},"An entire linear regression class, solved in one line. And Bishop already flags the risk: if two input columns are too similar to each other (",[1668,1669,1671],"glossary-term",{"definition":1670},"when two or more input variables carry nearly the same information, which makes the Φ matrix close to singular and the normal equation's solution numerically unstable","collinearity","), ",[38,1674,1676,1696],{"className":1675},[41],[38,1677,1679],{"className":1678},[45],[47,1680,1681],{"xmlns":49},[51,1682,1683,1693],{},[54,1684,1685,1691],{},[123,1686,1687,1689],{},[60,1688,638],{"mathvariant":1399},[60,1690,130],{"mathvariant":129},[60,1692,638],{"mathvariant":1399},[154,1694,1695],{"encoding":156},"\\Phi^{\\mathsf{T}}\\Phi",[38,1697,1699],{"className":1698,"ariaHidden":162},[161],[38,1700,1702,1705,1737],{"className":1701},[166],[38,1703],{"className":1704,"style":498},[170],[38,1706,1708,1711],{"className":1707},[175],[38,1709,638],{"className":1710},[175],[38,1712,1714],{"className":1713},[184],[38,1715,1717],{"className":1716},[188],[38,1718,1720],{"className":1719},[193],[38,1721,1723],{"className":1722,"style":498},[197],[38,1724,1725,1728],{"style":501},[38,1726],{"className":1727,"style":206},[205],[38,1729,1731],{"className":1730},[210,211,212,213],[38,1732,1734],{"className":1733},[175,213],[38,1735,130],{"className":1736},[175,514,213],[38,1738,638],{"className":1739},[175]," gets close to singular and the computation becomes numerically unstable. That'll come back once the course reaches feature selection.",[20,1742,1744],{"id":1743},"matches-scikit-learn-to-the-decimal","Matches scikit-learn to the decimal",[11,1746,1747,1748,1751],{},"The proof the math is right: I ran the same pseudo-inverse approach and scikit-learn's ready-made ",[605,1749,1750],{},"LinearRegression()"," side by side.",[1220,1753,1755],{"className":1222,"code":1754,"language":1224,"meta":1225,"style":1225},"regressor = LinearRegressor()  # the pinv-based class above\nregressor.fit(X_train, y_train)\nprint(mean_squared_error(y_train, regressor.predict(X_train)))\n\nfrom sklearn.linear_model import LinearRegression\nsk_regressor = LinearRegression().fit(X_train, y_train)\nprint(mean_squared_error(y_train, sk_regressor.predict(X_train)))\n",[605,1756,1757,1762,1767,1772,1776,1781,1786],{"__ignoreMap":1225},[38,1758,1759],{"class":1230,"line":1231},[38,1760,1761],{},"regressor = LinearRegressor()  # the pinv-based class above\n",[38,1763,1764],{"class":1230,"line":1237},[38,1765,1766],{},"regressor.fit(X_train, y_train)\n",[38,1768,1769],{"class":1230,"line":1624},[38,1770,1771],{},"print(mean_squared_error(y_train, regressor.predict(X_train)))\n",[38,1773,1774],{"class":1230,"line":1630},[38,1775,1646],{"emptyLinePlaceholder":1645},[38,1777,1778],{"class":1230,"line":1636},[38,1779,1780],{},"from sklearn.linear_model import LinearRegression\n",[38,1782,1783],{"class":1230,"line":1642},[38,1784,1785],{},"sk_regressor = LinearRegression().fit(X_train, y_train)\n",[38,1787,1788],{"class":1230,"line":1649},[38,1789,1790],{},"print(mean_squared_error(y_train, sk_regressor.predict(X_train)))\n",[1792,1793,1794],"blockquote",{},[11,1795,1796,1799,1800,1803],{},[31,1797,1798],{},"Output:"," my version: train MSE 2868.5497028355776. scikit-learn's ",[605,1801,1802],{},"LinearRegression",": train MSE 2868.549702835577.",[11,1805,1806,1807,1809],{},"The difference only shows up in the last decimal place, floating-point noise, not a difference in method. ",[605,1808,1750],{}," computes exactly this under the hood.",[20,1811,1813],{"id":1812},"the-regressor-zoo","The regressor zoo",[11,1815,1816,1817,1820],{},"Lecture 2c takes this same normal-equation regressor and also measures MSE on the ",[31,1818,1819],{},"test"," set (the 20% held out from training), then compares it against five very different kinds of model, all with scikit-learn's default parameters:",[1280,1822,1823,1836],{},[1283,1824,1825],{},[1286,1826,1827,1830,1833],{},[1289,1828,1829],{"align":1291},"Model",[1289,1831,1832],{"align":1298},"Train MSE",[1289,1834,1835],{"align":1298},"Test MSE",[1301,1837,1838,1854,1869,1886,1898,1912],{},[1286,1839,1840,1846,1851],{},[1306,1841,1842,1843,77],{"align":1291},"Decision tree (",[605,1844,1845],{},"DecisionTreeRegressor",[1306,1847,1848],{"align":1298},[31,1849,1850],{},"0.00",[1306,1852,1853],{"align":1298},"4872.20",[1286,1855,1856,1863,1866],{},[1306,1857,1858,1859,1862],{"align":1291},"KNN (",[605,1860,1861],{},"KNeighborsRegressor",", k=5)",[1306,1864,1865],{"align":1298},"2528.59",[1306,1867,1868],{"align":1298},"3019.08",[1286,1870,1871,1878,1881],{},[1306,1872,1873,1874,1877],{"align":1291},"Random forest (",[605,1875,1876],{},"RandomForestRegressor",", depth 3)",[1306,1879,1880],{"align":1298},"2530.82",[1306,1882,1883],{"align":1298},[31,1884,1885],{},"2785.98",[1286,1887,1888,1893,1895],{},[1306,1889,1890,1891],{"align":1291},"Normal equation \u002F ",[605,1892,1802],{},[1306,1894,1355],{"align":1298},[1306,1896,1897],{"align":1298},"2900.19",[1286,1899,1900,1906,1909],{},[1306,1901,1902,1905],{"align":1291},[605,1903,1904],{},"SGDRegressor"," (10000 iterations)",[1306,1907,1908],{"align":1298},"2950.64",[1306,1910,1911],{"align":1298},"2863.35",[1286,1913,1914,1919,1922],{},[1306,1915,1916],{"align":1291},[605,1917,1918],{},"LinearSVR",[1306,1920,1921],{"align":1298},"8224.56",[1306,1923,1924],{"align":1298},"6775.88",[11,1926,1927],{},"I'm not going to explain how each of these models works internally yet (KNN, decision trees, and random forests each get their own lecture later in the course, and that's where I'll come back to them properly). But three lessons come out of the table alone:",[1929,1930,1931,1943,1949],"ol",{},[1932,1933,1934,1937,1938,1942],"li",{},[31,1935,1936],{},"Low training MSE means nothing on its own."," The decision tree zeroed out its training error (it literally memorized every patient) and was the worst of all of them on the test set. That's ",[1668,1939,1941],{"definition":1940},"when the model memorizes the training data instead of learning the general pattern, and as a result does worse on new data","overfitting"," in its purest form, the same phenomenon Bishop showed back in chapter 1 with the degree-9 polynomial.",[1932,1944,1945,1948],{},[31,1946,1947],{},"The test-set winner wasn't the model that fit training best."," The random forest fits training almost as well as the full tree (2530.82, quite close to the tree's absurd zero), but without going as far, and that's exactly why it generalizes better: 2785.98 on test, the lowest test MSE in the whole table. A forest is many trees trained on different slices of the data, with the final prediction being the average across all of them, and that average cancels out a lot of the excess each individual tree commits.",[1932,1950,1951,1956,1957,1961,1962,1964],{},[31,1952,1953,1955],{},[605,1954,1904],{}," beat the exact normal equation on the test set"," (2863.35 versus 2900.19), even with a slightly worse training MSE. That's not a coincidence: as I already saw ",[14,1958,1960],{"href":1959},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn","in the specialization playlist",", scikit-learn's ",[605,1963,1904],{}," ships with L2 regularization on by default. Here that regularization, without me asking for it, ended up helping it generalize a bit better.",[11,1966,1967,1969],{},[605,1968,1918],{}," came out visibly worse than everything else, on both train and test, but that's more about its default hyperparameters not suiting this dataset than about the method itself, a story for another day.",[20,1971,1973],{"id":1972},"wrapping-up","Wrapping up",[1280,1975,1976,1986],{},[1283,1977,1978],{},[1286,1979,1980,1983],{},[1289,1981,1982],{"align":1291},"What I already knew",[1289,1984,1985],{"align":1291},"What these two lectures settled",[1301,1987,1988,1996,2167],{},[1286,1989,1990,1993],{},[1306,1991,1992],{"align":1291},"Gradient descent finds the optimal fit by iterating",[1306,1994,1995],{"align":1291},"Linear regression has a closed-form solution: the normal equation gets there in one computation",[1286,1997,1998,2003],{},[1306,1999,2000,2002],{"align":1291},[605,2001,1750],{}," \"just works\"",[1306,2004,2005,2006,2166],{"align":1291},"Under the hood, it computes exactly ",[38,2007,2009,2049],{"className":2008},[41],[38,2010,2012],{"className":2011},[45],[47,2013,2014],{"xmlns":49},[51,2015,2016,2046],{},[54,2017,2018,2020,2026,2028,2038,2044],{},[67,2019,70],{"stretchy":69},[123,2021,2022,2024],{},[60,2023,638],{"mathvariant":1399},[60,2025,130],{"mathvariant":129},[60,2027,638],{"mathvariant":1399},[123,2029,2030,2032],{},[67,2031,77],{"stretchy":69},[54,2033,2034,2036],{},[67,2035,121],{},[85,2037,87],{},[123,2039,2040,2042],{},[60,2041,638],{"mathvariant":1399},[60,2043,130],{"mathvariant":129},[60,2045,116],{"mathvariant":73},[154,2047,2048],{"encoding":156},"(\\Phi^{\\mathsf{T}}\\Phi)^{-1}\\Phi^{\\mathsf{T}}\\mathbf{t}",[38,2050,2052],{"className":2051,"ariaHidden":162},[161],[38,2053,2055,2058,2061,2093,2096,2131,2163],{"className":2054},[166],[38,2056],{"className":2057,"style":479},[170],[38,2059,70],{"className":2060},[235],[38,2062,2064,2067],{"className":2063},[175],[38,2065,638],{"className":2066},[175],[38,2068,2070],{"className":2069},[184],[38,2071,2073],{"className":2072},[188],[38,2074,2076],{"className":2075},[193],[38,2077,2079],{"className":2078,"style":498},[197],[38,2080,2081,2084],{"style":501},[38,2082],{"className":2083,"style":206},[205],[38,2085,2087],{"className":2086},[210,211,212,213],[38,2088,2090],{"className":2089},[175,213],[38,2091,130],{"className":2092},[175,514,213],[38,2094,638],{"className":2095},[175],[38,2097,2099,2102],{"className":2098},[244],[38,2100,77],{"className":2101},[244],[38,2103,2105],{"className":2104},[184],[38,2106,2108],{"className":2107},[188],[38,2109,2111],{"className":2110},[193],[38,2112,2114],{"className":2113,"style":589},[197],[38,2115,2116,2119],{"style":501},[38,2117],{"className":2118,"style":206},[205],[38,2120,2122],{"className":2121},[210,211,212,213],[38,2123,2125,2128],{"className":2124},[175,213],[38,2126,121],{"className":2127},[175,213],[38,2129,87],{"className":2130},[175,213],[38,2132,2134,2137],{"className":2133},[175],[38,2135,638],{"className":2136},[175],[38,2138,2140],{"className":2139},[184],[38,2141,2143],{"className":2142},[188],[38,2144,2146],{"className":2145},[193],[38,2147,2149],{"className":2148,"style":498},[197],[38,2150,2151,2154],{"style":501},[38,2152],{"className":2153,"style":206},[205],[38,2155,2157],{"className":2156},[210,211,212,213],[38,2158,2160],{"className":2159},[175,213],[38,2161,130],{"className":2162},[175,514,213],[38,2164,116],{"className":2165},[175,239]," via the pseudo-inverse",[1286,2168,2169,2172],{},[1306,2170,2171],{"align":1291},"Low training MSE is a good sign",[1306,2173,2174],{"align":1291},"Only when test MSE agrees. Low train, high test is the signature of overfitting",[20,2176,2178],{"id":2177},"practical-application","Practical application",[11,2180,2181,2182,2184,2185,2187,2188,2190],{},"I use the normal-equation model (the same ",[605,2183,1378],{},"-based ",[605,2186,1273],{}," from above) and look at the ",[31,2189,1819],{}," set, the 89 patients the model never saw during fitting, to visualize what that 2900.19 MSE actually means case by case.",[1220,2192,2194],{"className":1222,"code":2193,"language":1224,"meta":1225,"style":1225},"regressor = LinearRegressor()\nregressor.fit(X_train, y_train)\ny_pred_test = regressor.predict(X_test)\n",[605,2195,2196,2201,2205],{"__ignoreMap":1225},[38,2197,2198],{"class":1230,"line":1231},[38,2199,2200],{},"regressor = LinearRegressor()\n",[38,2202,2203],{"class":1230,"line":1237},[38,2204,1766],{},[38,2206,2207],{"class":1230,"line":1624},[38,2208,2209],{},"y_pred_test = regressor.predict(X_test)\n",[2211,2212],"predicted-vs-actual-scatter",{":actual":2213,":predicted":2214,"point-label":2215,"reference-label":2216,"x-label":2217,"y-label":2218},"[37, 42, 48, 48, 52, 52, 60, 61, 63, 63, 64, 67, 68, 69, 70, 72, 72, 72, 77, 84, 84, 87, 89, 90, 90, 90, 91, 94, 94, 95, 96, 96, 98, 99, 101, 102, 107, 108, 110, 111, 113, 118, 122, 128, 129, 135, 136, 140, 140, 151, 153, 156, 158, 164, 168, 168, 170, 171, 172, 180, 181, 184, 186, 187, 190, 200, 202, 202, 214, 219, 220, 222, 230, 232, 233, 233, 237, 242, 248, 252, 258, 263, 264, 272, 275, 281, 295, 297, 310]","[81.6, 124.4, 71.6, 48.0, 206.5, 61.5, 77.6, 146.0, 59.1, 54.4, 88.5, 130.7, 107.5, 103.5, 179.5, 109.2, 94.4, 55.9, 180.4, 119.7, 92.2, 115.0, 79.9, 51.6, 133.0, 171.0, 157.4, 94.1, 90.2, 154.6, 54.8, 108.6, 79.0, 53.5, 182.0, 105.6, 109.0, 107.7, 164.9, 123.8, 86.4, 159.4, 187.9, 70.4, 154.9, 100.1, 152.5, 174.8, 124.7, 210.7, 115.0, 165.5, 63.0, 184.5, 120.3, 155.0, 196.4, 175.6, 148.0, 173.8, 171.1, 166.3, 192.7, 140.6, 140.9, 71.7, 134.0, 144.8, 130.7, 139.5, 208.9, 199.4, 291.4, 189.0, 190.4, 206.0, 158.6, 258.2, 202.2, 168.4, 167.6, 188.3, 250.8, 181.3, 218.6, 234.2, 219.5, 206.6, 207.4]","Patient","Perfect prediction","actual progression (test)","predicted progression (test)",[11,2220,2221,2222,2225],{},"Notice the cloud looks a lot like ",[14,2223,2224],{"href":16},"the one from the previous post"," (which was on training data), without getting visibly worse on test. That confirms numerically what the table already showed: train MSE 2868.55 versus test MSE 2900.19, a small gap. The model didn't memorize training, it genuinely generalized, it's just that being a plain linear model, it still misses quite a bit case by case, the same ceiling I already saw in the previous post.",[2227,2228,2229],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":1225,"searchDepth":1237,"depth":1237,"links":2231},[2232,2233,2234,2235,2237,2238,2239,2240],{"id":22,"depth":1237,"text":23},{"id":875,"depth":1237,"text":876},{"id":1266,"depth":1237,"text":1267},{"id":1374,"depth":1237,"text":2236},"The pseudo-inverse is pinv",{"id":1743,"depth":1237,"text":1744},{"id":1812,"depth":1237,"text":1813},{"id":1972,"depth":1237,"text":1973},{"id":2177,"depth":1237,"text":2178},null,"2026-08-19","Lectures 2b and 2c: instead of iterating gradient descent, the professor solves linear regression with a single closed-form computation using the pseudo-inverse, matches scikit-learn's LinearRegression exactly, and compares it against five other kinds of regressor. I explain the why behind each step.","md",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation","pattern-recognition",{"title":6,"description":2243},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation",[2252,2253,1941],"normal-equation","least-squares","tRNTg9y2lr39prMEd05AN5P3fShFnu5_NkLDaxMhaqk",1787338984261]