[{"data":1,"prerenderedAt":33736},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization":3,"playlist-en-machine-learning-specialization":4,"playlist-posts-en-machine-learning-specialization":39},null,{"id":5,"title":6,"body":7,"cover":3,"description":29,"extension":30,"meta":31,"navigation":32,"order":33,"path":34,"seo":35,"status":36,"stem":37,"__hash__":38},"playlists\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex.md","Machine Learning Specialization (Andrew Ng)",{"type":8,"value":9,"toc":25},"minimark",[10,19,22],[11,12,13,14,18],"p",{},"This playlist is my public study notebook while going through the ",[15,16,17],"strong",{},"Machine Learning Specialization",", by Andrew Ng (DeepLearning.AI \u002F Stanford), probably the most recommended course for anyone getting started in ML.",[11,20,21],{},"The idea here isn't just \"solve the notebook and move on.\" Every lab in the course becomes a post where I retell what I understood, with an everyday-life metaphor, an example, and of course, the correct technical term, because you'll need it when you go looking for more on the topic later.",[11,23,24],{},"We start at the start: representing the simplest model there is, linear regression with one variable.",{"title":26,"searchDepth":27,"depth":27,"links":28},"",2,[],"My step-by-step notes going through Andrew Ng's Machine Learning Specialization (DeepLearning.AI \u002F Stanford), course by course, lab by lab.","md",{},true,1,"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization",{"title":6,"description":29},"published","en\u002Fplaylists\u002Fmachine-learning-specialization\u002Findex","doASMuy-kKTfF67w2hQII9iS08pq0gbnf-FwE2AmImQ",[40,6988,10243,14673,16421,18480,26924,30174],{"id":41,"title":42,"body":43,"cover":3,"date":6976,"description":6977,"extension":30,"meta":6978,"navigation":32,"order":33,"path":6979,"playlist":6980,"seo":6981,"status":36,"stem":6982,"tags":6983,"__hash__":6987},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation.md","Optional Lab: Model Representation",{"type":8,"value":44,"toc":6952},[45,54,61,69,77,293,298,498,513,517,520,1210,1213,1619,1623,1873,1876,2087,2532,2596,2605,2610,2687,2702,2717,2736,2768,2811,2882,2886,2893,3220,3249,3259,3340,3344,3347,3355,3358,3362,3609,3623,3775,3883,4001,4077,4081,4091,4187,4197,4201,4265,4376,4379,4586,4593,5466,5530,5715,5719,5825,5880,6071,6077,6119,6128,6189,6193,6505,6508,6593,6713,6717,6727,6731,6768,6787,6793,6797,6800,6806,6809,6870,6873,6878,6885,6889,6936,6945,6948],[46,47,48],"blockquote",{},[11,49,50,51,53],{},"Based on Optional Lab 02 (Week 1, Course 1) of the ",[15,52,17],{},", by Andrew Ng (DeepLearning.AI \u002F Stanford). But forget the notebook for a second, let's actually understand what's going on.",[11,55,56],{},[57,58],"img",{"alt":59,"src":60},"An \"IQ bell curve\" meme comparing linear regression and deep learning: both the least and the most knowledgeable people prefer linear regression, only the folks in the middle think deep learning is fancier","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Flab02-model-representation\u002Fmeme-regressao-linear.jpg",[11,62,63,64,68],{},"Here's the thing: everyone getting into ML hears about neural nets, transformers, LLMs, all the hyped-up stuff. Then you open the first lab of the most recommended course in the field and find... a line. ",[65,66,67],"code",{},"y = ax + b"," from middle school, just with the variable names swapped around.",[11,70,71,72,76],{},"That's not the course stalling before the \"real\" material. This ",[73,74,75],"em",{},"is"," the real material. Linear regression is the base Lego brick: every fancier model you'll study later (neural nets, logistic regression, whatever comes next) is, underneath, a pile of these bricks stacked up. Understanding this one down to the bone taught me way more than memorizing a neural net formula without knowing where it came from.",[11,78,79],{},[80,81,84,149],"span",{"className":82},[83],"katex",[80,85,88],{"className":86},[87],"katex-mathml",[89,90,92],"math",{"xmlns":91},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[93,94,95,144],"semantics",{},[96,97,98,118,122,125,128,131,133,137,139,142],"mrow",{},[99,100,101,105],"msub",{},[102,103,104],"mi",{},"f",[96,106,107,110,115],{},[102,108,109],{},"w",[111,112,114],"mo",{"separator":113},"true",",",[102,116,117],{},"b",[111,119,121],{"stretchy":120},"false","(",[102,123,124],{},"x",[111,126,127],{"stretchy":120},")",[111,129,130],{},"=",[102,132,109],{},[134,135,136],"mtext",{}," ",[102,138,124],{},[111,140,141],{},"+",[102,143,117],{},[145,146,148],"annotation",{"encoding":147},"application\u002Fx-tex","f_{w,b}(x) = w\\,x + b",[80,150,153,255,283],{"className":151,"ariaHidden":113},[152],"katex-html",[80,154,157,162,232,236,239,243,248,252],{"className":155},[156],"base",[80,158],{"className":159,"style":161},[160],"strut","height:1.0361em;vertical-align:-0.2861em;",[80,163,166,171],{"className":164},[165],"mord",[80,167,104],{"className":168,"style":170},[165,169],"mathnormal","margin-right:0.1076em;",[80,172,175],{"className":173},[174],"msupsub",[80,176,180,223],{"className":177},[178,179],"vlist-t","vlist-t2",[80,181,184,218],{"className":182},[183],"vlist-r",[80,185,189],{"className":186,"style":188},[187],"vlist","height:0.3361em;",[80,190,192,197],{"style":191},"top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;",[80,193],{"className":194,"style":196},[195],"pstrut","height:2.7em;",[80,198,204],{"className":199},[200,201,202,203],"sizing","reset-size6","size3","mtight",[80,205,207,211,215],{"className":206},[165,203],[80,208,109],{"className":209,"style":210},[165,169,203],"margin-right:0.0269em;",[80,212,114],{"className":213},[214,203],"mpunct",[80,216,117],{"className":217},[165,169,203],[80,219,222],{"className":220},[221],"vlist-s","​",[80,224,226],{"className":225},[183],[80,227,230],{"className":228,"style":229},[187],"height:0.2861em;",[80,231],{},[80,233,121],{"className":234},[235],"mopen",[80,237,124],{"className":238},[165,169],[80,240,127],{"className":241},[242],"mclose",[80,244],{"className":245,"style":247},[246],"mspace","margin-right:0.2778em;",[80,249,130],{"className":250},[251],"mrel",[80,253],{"className":254,"style":247},[246],[80,256,258,262,265,269,272,276,280],{"className":257},[156],[80,259],{"className":260,"style":261},[160],"height:0.6667em;vertical-align:-0.0833em;",[80,263,109],{"className":264,"style":210},[165,169],[80,266],{"className":267,"style":268},[246],"margin-right:0.1667em;",[80,270,124],{"className":271},[165,169],[80,273],{"className":274,"style":275},[246],"margin-right:0.2222em;",[80,277,141],{"className":278},[279],"mbin",[80,281],{"className":282,"style":275},[246],[80,284,286,290],{"className":285},[156],[80,287],{"className":288,"style":289},[160],"height:0.6944em;",[80,291,117],{"className":292},[165,169],[294,295,297],"h2",{"id":296},"what-youll-walk-away-knowing","What you'll walk away knowing",[299,300,301,305,339,342,495],"ol",{},[302,303,304],"li",{},"Why we represent training data as a NumPy array instead of a plain Python list.",[302,306,307,308,338],{},"How to count how many training examples you have (",[80,309,311,325],{"className":310},[83],[80,312,314],{"className":313},[87],[89,315,316],{"xmlns":91},[93,317,318,323],{},[96,319,320],{},[102,321,322],{},"m",[145,324,322],{"encoding":147},[80,326,328],{"className":327,"ariaHidden":113},[152],[80,329,331,335],{"className":330},[156],[80,332],{"className":333,"style":334},[160],"height:0.4306em;",[80,336,322],{"className":337},[165,169],") and grab a specific one.",[302,340,341],{},"Why looking at your data before modeling isn't a \"nice to have\", it's a survival rule.",[302,343,344,345,494],{},"How to build the model ",[80,346,348,388],{"className":347},[83],[80,349,351],{"className":350},[87],[89,352,353],{"xmlns":91},[93,354,355,385],{},[96,356,357,369,371,373,375,377,379,381,383],{},[99,358,359,361],{},[102,360,104],{},[96,362,363,365,367],{},[102,364,109],{},[111,366,114],{"separator":113},[102,368,117],{},[111,370,121],{"stretchy":120},[102,372,124],{},[111,374,127],{"stretchy":120},[111,376,130],{},[102,378,109],{},[102,380,124],{},[111,382,141],{},[102,384,117],{},[145,386,387],{"encoding":147},"f_{w,b}(x) = wx + b",[80,389,391,464,485],{"className":390,"ariaHidden":113},[152],[80,392,394,397,446,449,452,455,458,461],{"className":393},[156],[80,395],{"className":396,"style":161},[160],[80,398,400,403],{"className":399},[165],[80,401,104],{"className":402,"style":170},[165,169],[80,404,406],{"className":405},[174],[80,407,409,438],{"className":408},[178,179],[80,410,412,435],{"className":411},[183],[80,413,415],{"className":414,"style":188},[187],[80,416,417,420],{"style":191},[80,418],{"className":419,"style":196},[195],[80,421,423],{"className":422},[200,201,202,203],[80,424,426,429,432],{"className":425},[165,203],[80,427,109],{"className":428,"style":210},[165,169,203],[80,430,114],{"className":431},[214,203],[80,433,117],{"className":434},[165,169,203],[80,436,222],{"className":437},[221],[80,439,441],{"className":440},[183],[80,442,444],{"className":443,"style":229},[187],[80,445],{},[80,447,121],{"className":448},[235],[80,450,124],{"className":451},[165,169],[80,453,127],{"className":454},[242],[80,456],{"className":457,"style":247},[246],[80,459,130],{"className":460},[251],[80,462],{"className":463,"style":247},[246],[80,465,467,470,473,476,479,482],{"className":466},[156],[80,468],{"className":469,"style":261},[160],[80,471,109],{"className":472,"style":210},[165,169],[80,474,124],{"className":475},[165,169],[80,477],{"className":478,"style":275},[246],[80,480,141],{"className":481},[279],[80,483],{"className":484,"style":275},[246],[80,486,488,491],{"className":487},[156],[80,489],{"className":490,"style":289},[160],[80,492,117],{"className":493},[165,169]," in code.",[302,496,497],{},"How to play with the parameters until you nail the perfect fit, by hand, no magic formula, and feel firsthand why that doesn't scale.",[11,499,500,501,504,505,508,509,512],{},"Every supervised model has three pieces: ",[15,502,503],{},"the data",", ",[15,506,507],{},"the model function",", and ",[15,510,511],{},"the parameters",". This post puts all three together in the simplest version that exists. Hold on to that structure: it comes back, just bigger, in pretty much everything else this course covers.",[294,514,516],{"id":515},"the-fields-dictionary","The field's dictionary",[11,518,519],{},"ML has an annoying habit of inventing symbols for simple things. Here's the \"dictionary\" the course uses, since it carries through everything that follows:",[521,522,523,540],"table",{},[524,525,526],"thead",{},[527,528,529,534,537],"tr",{},[530,531,533],"th",{"align":532},"left","Notation",[530,535,536],{"align":532},"What it is",[530,538,539],{"align":532},"In Python",[541,542,543,583,624,665,709,907,946,985,1024],"tbody",{},[527,544,545,577,580],{},[546,547,548],"td",{"align":532},[80,549,551,565],{"className":550},[83],[80,552,554],{"className":553},[87],[89,555,556],{"xmlns":91},[93,557,558,563],{},[96,559,560],{},[102,561,562],{},"a",[145,564,562],{"encoding":147},[80,566,568],{"className":567,"ariaHidden":113},[152],[80,569,571,574],{"className":570},[156],[80,572],{"className":573,"style":334},[160],[80,575,562],{"className":576},[165,169],[546,578,579],{"align":532},"a plain number (scalar)",[546,581,582],{"align":532},"-",[527,584,585,619,622],{},[546,586,587],{"align":532},[80,588,590,605],{"className":589},[83],[80,591,593],{"className":592},[87],[89,594,595],{"xmlns":91},[93,596,597,602],{},[96,598,599],{},[102,600,562],{"mathvariant":601},"bold",[145,603,604],{"encoding":147},"\\mathbf{a}",[80,606,608],{"className":607,"ariaHidden":113},[152],[80,609,611,615],{"className":610},[156],[80,612],{"className":613,"style":614},[160],"height:0.4444em;",[80,616,562],{"className":617},[165,618],"mathbf",[546,620,621],{"align":532},"a list of numbers (vector, bold letter)",[546,623,582],{"align":532},[527,625,626,657,660],{},[546,627,628],{"align":532},[80,629,631,645],{"className":630},[83],[80,632,634],{"className":633},[87],[89,635,636],{"xmlns":91},[93,637,638,642],{},[96,639,640],{},[102,641,124],{"mathvariant":601},[145,643,644],{"encoding":147},"\\mathbf{x}",[80,646,648],{"className":647,"ariaHidden":113},[152],[80,649,651,654],{"className":650},[156],[80,652],{"className":653,"style":614},[160],[80,655,124],{"className":656},[165,618],[546,658,659],{"align":532},"the input values of every training example",[546,661,662],{"align":532},[65,663,664],{},"x_train",[527,666,667,701,704],{},[546,668,669],{"align":532},[80,670,672,687],{"className":671},[83],[80,673,675],{"className":674},[87],[89,676,677],{"xmlns":91},[93,678,679,684],{},[96,680,681],{},[102,682,683],{"mathvariant":601},"y",[145,685,686],{"encoding":147},"\\mathbf{y}",[80,688,690],{"className":689,"ariaHidden":113},[152],[80,691,693,697],{"className":692},[156],[80,694],{"className":695,"style":696},[160],"height:0.6389em;vertical-align:-0.1944em;",[80,698,683],{"className":699,"style":700},[165,618],"margin-right:0.016em;",[546,702,703],{"align":532},"the values we're trying to predict, for every example",[546,705,706],{"align":532},[65,707,708],{},"y_train",[527,710,711,867,899],{},[546,712,713,504,791],{"align":532},[80,714,716,742],{"className":715},[83],[80,717,719],{"className":718},[87],[89,720,721],{"xmlns":91},[93,722,723,739],{},[96,724,725],{},[726,727,728,730],"msup",{},[102,729,124],{},[96,731,732,734,737],{},[111,733,121],{"stretchy":120},[102,735,736],{},"i",[111,738,127],{"stretchy":120},[145,740,741],{"encoding":147},"x^{(i)}",[80,743,745],{"className":744,"ariaHidden":113},[152],[80,746,748,752],{"className":747},[156],[80,749],{"className":750,"style":751},[160],"height:0.888em;",[80,753,755,758],{"className":754},[165],[80,756,124],{"className":757},[165,169],[80,759,761],{"className":760},[174],[80,762,764],{"className":763},[178],[80,765,767],{"className":766},[183],[80,768,770],{"className":769,"style":751},[187],[80,771,773,776],{"style":772},"top:-3.063em;margin-right:0.05em;",[80,774],{"className":775,"style":196},[195],[80,777,779],{"className":778},[200,201,202,203],[80,780,782,785,788],{"className":781},[165,203],[80,783,121],{"className":784},[235,203],[80,786,736],{"className":787},[165,169,203],[80,789,127],{"className":790},[242,203],[80,792,794,818],{"className":793},[83],[80,795,797],{"className":796},[87],[89,798,799],{"xmlns":91},[93,800,801,815],{},[96,802,803],{},[726,804,805,807],{},[102,806,683],{},[96,808,809,811,813],{},[111,810,121],{"stretchy":120},[102,812,736],{},[111,814,127],{"stretchy":120},[145,816,817],{"encoding":147},"y^{(i)}",[80,819,821],{"className":820,"ariaHidden":113},[152],[80,822,824,828],{"className":823},[156],[80,825],{"className":826,"style":827},[160],"height:1.0824em;vertical-align:-0.1944em;",[80,829,831,835],{"className":830},[165],[80,832,683],{"className":833,"style":834},[165,169],"margin-right:0.0359em;",[80,836,838],{"className":837},[174],[80,839,841],{"className":840},[178],[80,842,844],{"className":843},[183],[80,845,847],{"className":846,"style":751},[187],[80,848,849,852],{"style":772},[80,850],{"className":851,"style":196},[195],[80,853,855],{"className":854},[200,201,202,203],[80,856,858,861,864],{"className":857},[165,203],[80,859,121],{"className":860},[235,203],[80,862,736],{"className":863},[165,169,203],[80,865,127],{"className":866},[242,203],[546,868,869,870],{"align":532},"the pair of values for example number ",[80,871,873,886],{"className":872},[83],[80,874,876],{"className":875},[87],[89,877,878],{"xmlns":91},[93,879,880,884],{},[96,881,882],{},[102,883,736],{},[145,885,736],{"encoding":147},[80,887,889],{"className":888,"ariaHidden":113},[152],[80,890,892,896],{"className":891},[156],[80,893],{"className":894,"style":895},[160],"height:0.6595em;",[80,897,736],{"className":898},[165,169],[546,900,901,504,904],{"align":532},[65,902,903],{},"x_i",[65,905,906],{},"y_i",[527,908,909,939,942],{},[546,910,911],{"align":532},[80,912,914,927],{"className":913},[83],[80,915,917],{"className":916},[87],[89,918,919],{"xmlns":91},[93,920,921,925],{},[96,922,923],{},[102,924,322],{},[145,926,322],{"encoding":147},[80,928,930],{"className":929,"ariaHidden":113},[152],[80,931,933,936],{"className":932},[156],[80,934],{"className":935,"style":334},[160],[80,937,322],{"className":938},[165,169],[546,940,941],{"align":532},"how many training examples you have",[546,943,944],{"align":532},[65,945,322],{},[527,947,948,978,981],{},[546,949,950],{"align":532},[80,951,953,966],{"className":952},[83],[80,954,956],{"className":955},[87],[89,957,958],{"xmlns":91},[93,959,960,964],{},[96,961,962],{},[102,963,109],{},[145,965,109],{"encoding":147},[80,967,969],{"className":968,"ariaHidden":113},[152],[80,970,972,975],{"className":971},[156],[80,973],{"className":974,"style":334},[160],[80,976,109],{"className":977,"style":210},[165,169],[546,979,980],{"align":532},"weight (the line's slope)",[546,982,983],{"align":532},[65,984,109],{},[527,986,987,1017,1020],{},[546,988,989],{"align":532},[80,990,992,1005],{"className":991},[83],[80,993,995],{"className":994},[87],[89,996,997],{"xmlns":91},[93,998,999,1003],{},[96,1000,1001],{},[102,1002,117],{},[145,1004,117],{"encoding":147},[80,1006,1008],{"className":1007,"ariaHidden":113},[152],[80,1009,1011,1014],{"className":1010},[156],[80,1012],{"className":1013,"style":289},[160],[80,1015,117],{"className":1016},[165,169],[546,1018,1019],{"align":532},"bias (where the line starts)",[546,1021,1022],{"align":532},[65,1023,117],{},[527,1025,1026,1174,1205],{},[546,1027,1028],{"align":532},[80,1029,1031,1071],{"className":1030},[83],[80,1032,1034],{"className":1033},[87],[89,1035,1036],{"xmlns":91},[93,1037,1038,1068],{},[96,1039,1040,1052,1054,1066],{},[99,1041,1042,1044],{},[102,1043,104],{},[96,1045,1046,1048,1050],{},[102,1047,109],{},[111,1049,114],{"separator":113},[102,1051,117],{},[111,1053,121],{"stretchy":120},[726,1055,1056,1058],{},[102,1057,124],{},[96,1059,1060,1062,1064],{},[111,1061,121],{"stretchy":120},[102,1063,736],{},[111,1065,127],{"stretchy":120},[111,1067,127],{"stretchy":120},[145,1069,1070],{"encoding":147},"f_{w,b}(x^{(i)})",[80,1072,1074],{"className":1073,"ariaHidden":113},[152],[80,1075,1077,1081,1130,1133,1171],{"className":1076},[156],[80,1078],{"className":1079,"style":1080},[160],"height:1.1741em;vertical-align:-0.2861em;",[80,1082,1084,1087],{"className":1083},[165],[80,1085,104],{"className":1086,"style":170},[165,169],[80,1088,1090],{"className":1089},[174],[80,1091,1093,1122],{"className":1092},[178,179],[80,1094,1096,1119],{"className":1095},[183],[80,1097,1099],{"className":1098,"style":188},[187],[80,1100,1101,1104],{"style":191},[80,1102],{"className":1103,"style":196},[195],[80,1105,1107],{"className":1106},[200,201,202,203],[80,1108,1110,1113,1116],{"className":1109},[165,203],[80,1111,109],{"className":1112,"style":210},[165,169,203],[80,1114,114],{"className":1115},[214,203],[80,1117,117],{"className":1118},[165,169,203],[80,1120,222],{"className":1121},[221],[80,1123,1125],{"className":1124},[183],[80,1126,1128],{"className":1127,"style":229},[187],[80,1129],{},[80,1131,121],{"className":1132},[235],[80,1134,1136,1139],{"className":1135},[165],[80,1137,124],{"className":1138},[165,169],[80,1140,1142],{"className":1141},[174],[80,1143,1145],{"className":1144},[178],[80,1146,1148],{"className":1147},[183],[80,1149,1151],{"className":1150,"style":751},[187],[80,1152,1153,1156],{"style":772},[80,1154],{"className":1155,"style":196},[195],[80,1157,1159],{"className":1158},[200,201,202,203],[80,1160,1162,1165,1168],{"className":1161},[165,203],[80,1163,121],{"className":1164},[235,203],[80,1166,736],{"className":1167},[165,169,203],[80,1169,127],{"className":1170},[242,203],[80,1172,127],{"className":1173},[242],[546,1175,1176,1177],{"align":532},"what the model predicts for example ",[80,1178,1180,1193],{"className":1179},[83],[80,1181,1183],{"className":1182},[87],[89,1184,1185],{"xmlns":91},[93,1186,1187,1191],{},[96,1188,1189],{},[102,1190,736],{},[145,1192,736],{"encoding":147},[80,1194,1196],{"className":1195,"ariaHidden":113},[152],[80,1197,1199,1202],{"className":1198},[156],[80,1200],{"className":1201,"style":895},[160],[80,1203,736],{"className":1204},[165,169],[546,1206,1207],{"align":532},[65,1208,1209],{},"f_wb",[11,1211,1212],{},"Two mix-ups that trip up every beginner:",[1214,1215,1216,1438],"ul",{},[302,1217,1218,1291,1292,1295,1296,1299,1300,1376,1377,1437],{},[80,1219,1221,1244],{"className":1220},[83],[80,1222,1224],{"className":1223},[87],[89,1225,1226],{"xmlns":91},[93,1227,1228,1242],{},[96,1229,1230],{},[726,1231,1232,1234],{},[102,1233,124],{},[96,1235,1236,1238,1240],{},[111,1237,121],{"stretchy":120},[102,1239,736],{},[111,1241,127],{"stretchy":120},[145,1243,741],{"encoding":147},[80,1245,1247],{"className":1246,"ariaHidden":113},[152],[80,1248,1250,1253],{"className":1249},[156],[80,1251],{"className":1252,"style":751},[160],[80,1254,1256,1259],{"className":1255},[165],[80,1257,124],{"className":1258},[165,169],[80,1260,1262],{"className":1261},[174],[80,1263,1265],{"className":1264},[178],[80,1266,1268],{"className":1267},[183],[80,1269,1271],{"className":1270,"style":751},[187],[80,1272,1273,1276],{"style":772},[80,1274],{"className":1275,"style":196},[195],[80,1277,1279],{"className":1278},[200,201,202,203],[80,1280,1282,1285,1288],{"className":1281},[165,203],[80,1283,121],{"className":1284},[235,203],[80,1286,736],{"className":1287},[165,169,203],[80,1289,127],{"className":1290},[242,203]," is ",[15,1293,1294],{},"not"," a power. That ",[65,1297,1298],{},"(i)"," in parentheses just means \"which row in the table\", an index. ",[80,1301,1303,1329],{"className":1302},[83],[80,1304,1306],{"className":1305},[87],[89,1307,1308],{"xmlns":91},[93,1309,1310,1326],{},[96,1311,1312],{},[726,1313,1314,1316],{},[102,1315,124],{},[96,1317,1318,1320,1324],{},[111,1319,121],{"stretchy":120},[1321,1322,1323],"mn",{},"2",[111,1325,127],{"stretchy":120},[145,1327,1328],{"encoding":147},"x^{(2)}",[80,1330,1332],{"className":1331,"ariaHidden":113},[152],[80,1333,1335,1338],{"className":1334},[156],[80,1336],{"className":1337,"style":751},[160],[80,1339,1341,1344],{"className":1340},[165],[80,1342,124],{"className":1343},[165,169],[80,1345,1347],{"className":1346},[174],[80,1348,1350],{"className":1349},[178],[80,1351,1353],{"className":1352},[183],[80,1354,1356],{"className":1355,"style":751},[187],[80,1357,1358,1361],{"style":772},[80,1359],{"className":1360,"style":196},[195],[80,1362,1364],{"className":1363},[200,201,202,203],[80,1365,1367,1370,1373],{"className":1366},[165,203],[80,1368,121],{"className":1369},[235,203],[80,1371,1323],{"className":1372},[165,203],[80,1374,127],{"className":1375},[242,203]," is the third example (remember, we count from zero), while ",[80,1378,1380,1398],{"className":1379},[83],[80,1381,1383],{"className":1382},[87],[89,1384,1385],{"xmlns":91},[93,1386,1387,1395],{},[96,1388,1389],{},[726,1390,1391,1393],{},[102,1392,124],{},[1321,1394,1323],{},[145,1396,1397],{"encoding":147},"x^2",[80,1399,1401],{"className":1400,"ariaHidden":113},[152],[80,1402,1404,1408],{"className":1403},[156],[80,1405],{"className":1406,"style":1407},[160],"height:0.8141em;",[80,1409,1411,1414],{"className":1410},[165],[80,1412,124],{"className":1413},[165,169],[80,1415,1417],{"className":1416},[174],[80,1418,1420],{"className":1419},[178],[80,1421,1423],{"className":1422},[183],[80,1424,1426],{"className":1425,"style":1407},[187],[80,1427,1428,1431],{"style":772},[80,1429],{"className":1430,"style":196},[195],[80,1432,1434],{"className":1433},[200,201,202,203],[80,1435,1323],{"className":1436},[165,203]," would mean \"x squared\", a completely different thing.",[302,1439,1440,1468,1469,1472,1473,1502,1503,1506,1507,1535,1536,1564,1565,1618],{},[80,1441,1443,1456],{"className":1442},[83],[80,1444,1446],{"className":1445},[87],[89,1447,1448],{"xmlns":91},[93,1449,1450,1454],{},[96,1451,1452],{},[102,1453,322],{},[145,1455,322],{"encoding":147},[80,1457,1459],{"className":1458,"ariaHidden":113},[152],[80,1460,1462,1465],{"className":1461},[156],[80,1463],{"className":1464,"style":334},[160],[80,1466,322],{"className":1467},[165,169]," is the number of ",[15,1470,1471],{},"rows"," (examples). Later on ",[80,1474,1476,1490],{"className":1475},[83],[80,1477,1479],{"className":1478},[87],[89,1480,1481],{"xmlns":91},[93,1482,1483,1488],{},[96,1484,1485],{},[102,1486,1487],{},"n",[145,1489,1487],{"encoding":147},[80,1491,1493],{"className":1492,"ariaHidden":113},[152],[80,1494,1496,1499],{"className":1495},[156],[80,1497],{"className":1498,"style":334},[160],[80,1500,1487],{"className":1501},[165,169]," shows up, the number of ",[15,1504,1505],{},"columns"," (features). Think of a spreadsheet: ",[80,1508,1510,1523],{"className":1509},[83],[80,1511,1513],{"className":1512},[87],[89,1514,1515],{"xmlns":91},[93,1516,1517,1521],{},[96,1518,1519],{},[102,1520,322],{},[145,1522,322],{"encoding":147},[80,1524,1526],{"className":1525,"ariaHidden":113},[152],[80,1527,1529,1532],{"className":1528},[156],[80,1530],{"className":1531,"style":334},[160],[80,1533,322],{"className":1534},[165,169]," is how many houses you've logged, ",[80,1537,1539,1552],{"className":1538},[83],[80,1540,1542],{"className":1541},[87],[89,1543,1544],{"xmlns":91},[93,1545,1546,1550],{},[96,1547,1548],{},[102,1549,1487],{},[145,1551,1487],{"encoding":147},[80,1553,1555],{"className":1554,"ariaHidden":113},[152],[80,1556,1558,1561],{"className":1557},[156],[80,1559],{"className":1560,"style":334},[160],[80,1562,1487],{"className":1563},[165,169]," is how many columns of information you tracked about each one (size, bedroom count, neighborhood...). Here we've only got one column, so ",[80,1566,1568,1587],{"className":1567},[83],[80,1569,1571],{"className":1570},[87],[89,1572,1573],{"xmlns":91},[93,1574,1575,1584],{},[96,1576,1577,1579,1581],{},[102,1578,1487],{},[111,1580,130],{},[1321,1582,1583],{},"1",[145,1585,1586],{"encoding":147},"n = 1",[80,1588,1590,1608],{"className":1589,"ariaHidden":113},[152],[80,1591,1593,1596,1599,1602,1605],{"className":1592},[156],[80,1594],{"className":1595,"style":334},[160],[80,1597,1487],{"className":1598},[165,169],[80,1600],{"className":1601,"style":247},[246],[80,1603,130],{"className":1604},[251],[80,1606],{"className":1607,"style":247},[246],[80,1609,1611,1615],{"className":1610},[156],[80,1612],{"className":1613,"style":1614},[160],"height:0.6444em;",[80,1616,1583],{"className":1617},[165],".",[294,1620,1622],{"id":1621},"the-problem-whats-this-house-worth","The problem: what's this house worth?",[11,1624,1625,1626,1629,1630,1872],{},"Scenario: you're a real estate agent with exactly two closed sales so far. One house at ",[15,1627,1628],{},"1000 sqft"," sold for ",[15,1631,1632,1871],{},[80,1633,1635,1721],{"className":1634},[83],[80,1636,1638],{"className":1637},[87],[89,1639,1640],{"xmlns":91},[93,1641,1642,1718],{},[96,1643,1644,1647,1650,1653,1655,1657,1659,1661,1664,1667,1669,1672,1674,1677,1679,1681,1684,1687,1690,1692,1694,1696,1698,1700,1702,1705,1707,1709,1711,1714,1716],{},[1321,1645,1646],{},"300",[102,1648,1649],{},"k",[111,1651,1652],{},"∗",[111,1654,1652],{},[111,1656,114],{"separator":113},[102,1658,562],{},[102,1660,1487],{},[102,1662,1663],{},"d",[102,1665,1666],{},"o",[102,1668,1487],{},[102,1670,1671],{},"e",[102,1673,562],{},[102,1675,1676],{},"t",[111,1678,1652],{},[111,1680,1652],{},[1321,1682,1683],{},"2000",[102,1685,1686],{},"s",[102,1688,1689],{},"q",[102,1691,104],{},[102,1693,1676],{},[111,1695,1652],{},[111,1697,1652],{},[102,1699,1686],{},[102,1701,1666],{},[102,1703,1704],{},"l",[102,1706,1663],{},[102,1708,104],{},[102,1710,1666],{},[102,1712,1713],{},"r",[111,1715,1652],{},[111,1717,1652],{},[145,1719,1720],{"encoding":147},"300k**, and one at **2000 sqft** sold for **",[80,1722,1724,1746,1793,1826,1861],{"className":1723,"ariaHidden":113},[152],[80,1725,1727,1730,1733,1737,1740,1743],{"className":1726},[156],[80,1728],{"className":1729,"style":289},[160],[80,1731,1646],{"className":1732},[165],[80,1734,1649],{"className":1735,"style":1736},[165,169],"margin-right:0.0315em;",[80,1738],{"className":1739,"style":275},[246],[80,1741,1652],{"className":1742},[279],[80,1744],{"className":1745,"style":275},[246],[80,1747,1749,1753,1756,1759,1762,1766,1769,1772,1775,1778,1781,1784,1787,1790],{"className":1748},[156],[80,1750],{"className":1751,"style":1752},[160],"height:0.8889em;vertical-align:-0.1944em;",[80,1754,1652],{"className":1755},[165],[80,1757,114],{"className":1758},[214],[80,1760],{"className":1761,"style":268},[246],[80,1763,1765],{"className":1764},[165,169],"an",[80,1767,1663],{"className":1768},[165,169],[80,1770,1666],{"className":1771},[165,169],[80,1773,1487],{"className":1774},[165,169],[80,1776,1671],{"className":1777},[165,169],[80,1779,562],{"className":1780},[165,169],[80,1782,1676],{"className":1783},[165,169],[80,1785],{"className":1786,"style":275},[246],[80,1788,1652],{"className":1789},[279],[80,1791],{"className":1792,"style":275},[246],[80,1794,1796,1799,1802,1805,1808,1811,1814,1817,1820,1823],{"className":1795},[156],[80,1797],{"className":1798,"style":1752},[160],[80,1800,1652],{"className":1801},[165],[80,1803,1683],{"className":1804},[165],[80,1806,1686],{"className":1807},[165,169],[80,1809,1689],{"className":1810,"style":834},[165,169],[80,1812,104],{"className":1813,"style":170},[165,169],[80,1815,1676],{"className":1816},[165,169],[80,1818],{"className":1819,"style":275},[246],[80,1821,1652],{"className":1822},[279],[80,1824],{"className":1825,"style":275},[246],[80,1827,1829,1832,1835,1839,1843,1847,1852,1855,1858],{"className":1828},[156],[80,1830],{"className":1831,"style":1752},[160],[80,1833,1652],{"className":1834},[165],[80,1836,1838],{"className":1837},[165,169],"so",[80,1840,1704],{"className":1841,"style":1842},[165,169],"margin-right:0.0197em;",[80,1844,1846],{"className":1845,"style":170},[165,169],"df",[80,1848,1851],{"className":1849,"style":1850},[165,169],"margin-right:0.0278em;","or",[80,1853],{"className":1854,"style":275},[246],[80,1856,1652],{"className":1857},[279],[80,1859],{"className":1860,"style":275},[246],[80,1862,1864,1868],{"className":1863},[156],[80,1865],{"className":1866,"style":1867},[160],"height:0.4653em;",[80,1869,1652],{"className":1870},[165],"500k",". A client shows up with a 1200 sqft house asking what it's worth. You don't have a crystal ball, but you have two data points, and it was surprising how much I managed to pull out of that.",[11,1874,1875],{},"To avoid writing a pile of zeros, size becomes \"thousands of sqft\" and price becomes \"thousands of dollars\":",[521,1877,1878,2065],{},[524,1879,1880],{},[527,1881,1882,1913,1989],{},[530,1883,1885],{"align":1884},"center",[80,1886,1888,1901],{"className":1887},[83],[80,1889,1891],{"className":1890},[87],[89,1892,1893],{"xmlns":91},[93,1894,1895,1899],{},[96,1896,1897],{},[102,1898,736],{},[145,1900,736],{"encoding":147},[80,1902,1904],{"className":1903,"ariaHidden":113},[152],[80,1905,1907,1910],{"className":1906},[156],[80,1908],{"className":1909,"style":895},[160],[80,1911,736],{"className":1912},[165,169],[530,1914,1915,1916],{"align":1884},"Size (1000 sqft) → ",[80,1917,1919,1942],{"className":1918},[83],[80,1920,1922],{"className":1921},[87],[89,1923,1924],{"xmlns":91},[93,1925,1926,1940],{},[96,1927,1928],{},[726,1929,1930,1932],{},[102,1931,124],{},[96,1933,1934,1936,1938],{},[111,1935,121],{"stretchy":120},[102,1937,736],{},[111,1939,127],{"stretchy":120},[145,1941,741],{"encoding":147},[80,1943,1945],{"className":1944,"ariaHidden":113},[152],[80,1946,1948,1951],{"className":1947},[156],[80,1949],{"className":1950,"style":751},[160],[80,1952,1954,1957],{"className":1953},[165],[80,1955,124],{"className":1956},[165,169],[80,1958,1960],{"className":1959},[174],[80,1961,1963],{"className":1962},[178],[80,1964,1966],{"className":1965},[183],[80,1967,1969],{"className":1968,"style":751},[187],[80,1970,1971,1974],{"style":772},[80,1972],{"className":1973,"style":196},[195],[80,1975,1977],{"className":1976},[200,201,202,203],[80,1978,1980,1983,1986],{"className":1979},[165,203],[80,1981,121],{"className":1982},[235,203],[80,1984,736],{"className":1985},[165,169,203],[80,1987,127],{"className":1988},[242,203],[530,1990,1991,1992],{"align":1884},"Price (1000 dollars) → ",[80,1993,1995,2018],{"className":1994},[83],[80,1996,1998],{"className":1997},[87],[89,1999,2000],{"xmlns":91},[93,2001,2002,2016],{},[96,2003,2004],{},[726,2005,2006,2008],{},[102,2007,683],{},[96,2009,2010,2012,2014],{},[111,2011,121],{"stretchy":120},[102,2013,736],{},[111,2015,127],{"stretchy":120},[145,2017,817],{"encoding":147},[80,2019,2021],{"className":2020,"ariaHidden":113},[152],[80,2022,2024,2027],{"className":2023},[156],[80,2025],{"className":2026,"style":827},[160],[80,2028,2030,2033],{"className":2029},[165],[80,2031,683],{"className":2032,"style":834},[165,169],[80,2034,2036],{"className":2035},[174],[80,2037,2039],{"className":2038},[178],[80,2040,2042],{"className":2041},[183],[80,2043,2045],{"className":2044,"style":751},[187],[80,2046,2047,2050],{"style":772},[80,2048],{"className":2049,"style":196},[195],[80,2051,2053],{"className":2052},[200,201,202,203],[80,2054,2056,2059,2062],{"className":2055},[165,203],[80,2057,121],{"className":2058},[235,203],[80,2060,736],{"className":2061},[165,169,203],[80,2063,127],{"className":2064},[242,203],[541,2066,2067,2077],{},[527,2068,2069,2072,2075],{},[546,2070,2071],{"align":1884},"0",[546,2073,2074],{"align":1884},"1.0",[546,2076,1646],{"align":1884},[527,2078,2079,2081,2084],{},[546,2080,1583],{"align":1884},[546,2082,2083],{"align":1884},"2.0",[546,2085,2086],{"align":1884},"500",[11,2088,2089,2090,2141,2142,2338,2339,1618],{},"So: ",[80,2091,2093,2111],{"className":2092},[83],[80,2094,2096],{"className":2095},[87],[89,2097,2098],{"xmlns":91},[93,2099,2100,2108],{},[96,2101,2102,2104,2106],{},[102,2103,322],{},[111,2105,130],{},[1321,2107,1323],{},[145,2109,2110],{"encoding":147},"m = 2",[80,2112,2114,2132],{"className":2113,"ariaHidden":113},[152],[80,2115,2117,2120,2123,2126,2129],{"className":2116},[156],[80,2118],{"className":2119,"style":334},[160],[80,2121,322],{"className":2122},[165,169],[80,2124],{"className":2125,"style":247},[246],[80,2127,130],{"className":2128},[251],[80,2130],{"className":2131,"style":247},[246],[80,2133,2135,2138],{"className":2134},[156],[80,2136],{"className":2137,"style":1614},[160],[80,2139,1323],{"className":2140},[165]," examples, ",[80,2143,2145,2203],{"className":2144},[83],[80,2146,2148],{"className":2147},[87],[89,2149,2150],{"xmlns":91},[93,2151,2152,2200],{},[96,2153,2154,2156,2168,2170,2182,2184,2186,2188,2190,2192,2195,2198],{},[111,2155,121],{"stretchy":120},[726,2157,2158,2160],{},[102,2159,124],{},[96,2161,2162,2164,2166],{},[111,2163,121],{"stretchy":120},[1321,2165,2071],{},[111,2167,127],{"stretchy":120},[111,2169,114],{"separator":113},[726,2171,2172,2174],{},[102,2173,683],{},[96,2175,2176,2178,2180],{},[111,2177,121],{"stretchy":120},[1321,2179,2071],{},[111,2181,127],{"stretchy":120},[111,2183,127],{"stretchy":120},[111,2185,130],{},[111,2187,121],{"stretchy":120},[1321,2189,2074],{},[111,2191,114],{"separator":113},[134,2193,2194],{}," ",[1321,2196,2197],{},"300.0",[111,2199,127],{"stretchy":120},[145,2201,2202],{"encoding":147},"(x^{(0)}, y^{(0)}) = (1.0,\\ 300.0)",[80,2204,2206,2310],{"className":2205,"ariaHidden":113},[152],[80,2207,2209,2213,2216,2254,2257,2260,2298,2301,2304,2307],{"className":2208},[156],[80,2210],{"className":2211,"style":2212},[160],"height:1.138em;vertical-align:-0.25em;",[80,2214,121],{"className":2215},[235],[80,2217,2219,2222],{"className":2218},[165],[80,2220,124],{"className":2221},[165,169],[80,2223,2225],{"className":2224},[174],[80,2226,2228],{"className":2227},[178],[80,2229,2231],{"className":2230},[183],[80,2232,2234],{"className":2233,"style":751},[187],[80,2235,2236,2239],{"style":772},[80,2237],{"className":2238,"style":196},[195],[80,2240,2242],{"className":2241},[200,201,202,203],[80,2243,2245,2248,2251],{"className":2244},[165,203],[80,2246,121],{"className":2247},[235,203],[80,2249,2071],{"className":2250},[165,203],[80,2252,127],{"className":2253},[242,203],[80,2255,114],{"className":2256},[214],[80,2258],{"className":2259,"style":268},[246],[80,2261,2263,2266],{"className":2262},[165],[80,2264,683],{"className":2265,"style":834},[165,169],[80,2267,2269],{"className":2268},[174],[80,2270,2272],{"className":2271},[178],[80,2273,2275],{"className":2274},[183],[80,2276,2278],{"className":2277,"style":751},[187],[80,2279,2280,2283],{"style":772},[80,2281],{"className":2282,"style":196},[195],[80,2284,2286],{"className":2285},[200,201,202,203],[80,2287,2289,2292,2295],{"className":2288},[165,203],[80,2290,121],{"className":2291},[235,203],[80,2293,2071],{"className":2294},[165,203],[80,2296,127],{"className":2297},[242,203],[80,2299,127],{"className":2300},[242],[80,2302],{"className":2303,"style":247},[246],[80,2305,130],{"className":2306},[251],[80,2308],{"className":2309,"style":247},[246],[80,2311,2313,2317,2320,2323,2326,2329,2332,2335],{"className":2312},[156],[80,2314],{"className":2315,"style":2316},[160],"height:1em;vertical-align:-0.25em;",[80,2318,121],{"className":2319},[235],[80,2321,2074],{"className":2322},[165],[80,2324,114],{"className":2325},[214],[80,2327,2194],{"className":2328},[246],[80,2330],{"className":2331,"style":268},[246],[80,2333,2197],{"className":2334},[165],[80,2336,127],{"className":2337},[242]," and ",[80,2340,2342,2399],{"className":2341},[83],[80,2343,2345],{"className":2344},[87],[89,2346,2347],{"xmlns":91},[93,2348,2349,2396],{},[96,2350,2351,2353,2365,2367,2379,2381,2383,2385,2387,2389,2391,2394],{},[111,2352,121],{"stretchy":120},[726,2354,2355,2357],{},[102,2356,124],{},[96,2358,2359,2361,2363],{},[111,2360,121],{"stretchy":120},[1321,2362,1583],{},[111,2364,127],{"stretchy":120},[111,2366,114],{"separator":113},[726,2368,2369,2371],{},[102,2370,683],{},[96,2372,2373,2375,2377],{},[111,2374,121],{"stretchy":120},[1321,2376,1583],{},[111,2378,127],{"stretchy":120},[111,2380,127],{"stretchy":120},[111,2382,130],{},[111,2384,121],{"stretchy":120},[1321,2386,2083],{},[111,2388,114],{"separator":113},[134,2390,2194],{},[1321,2392,2393],{},"500.0",[111,2395,127],{"stretchy":120},[145,2397,2398],{"encoding":147},"(x^{(1)}, y^{(1)}) = (2.0,\\ 500.0)",[80,2400,2402,2505],{"className":2401,"ariaHidden":113},[152],[80,2403,2405,2408,2411,2449,2452,2455,2493,2496,2499,2502],{"className":2404},[156],[80,2406],{"className":2407,"style":2212},[160],[80,2409,121],{"className":2410},[235],[80,2412,2414,2417],{"className":2413},[165],[80,2415,124],{"className":2416},[165,169],[80,2418,2420],{"className":2419},[174],[80,2421,2423],{"className":2422},[178],[80,2424,2426],{"className":2425},[183],[80,2427,2429],{"className":2428,"style":751},[187],[80,2430,2431,2434],{"style":772},[80,2432],{"className":2433,"style":196},[195],[80,2435,2437],{"className":2436},[200,201,202,203],[80,2438,2440,2443,2446],{"className":2439},[165,203],[80,2441,121],{"className":2442},[235,203],[80,2444,1583],{"className":2445},[165,203],[80,2447,127],{"className":2448},[242,203],[80,2450,114],{"className":2451},[214],[80,2453],{"className":2454,"style":268},[246],[80,2456,2458,2461],{"className":2457},[165],[80,2459,683],{"className":2460,"style":834},[165,169],[80,2462,2464],{"className":2463},[174],[80,2465,2467],{"className":2466},[178],[80,2468,2470],{"className":2469},[183],[80,2471,2473],{"className":2472,"style":751},[187],[80,2474,2475,2478],{"style":772},[80,2476],{"className":2477,"style":196},[195],[80,2479,2481],{"className":2480},[200,201,202,203],[80,2482,2484,2487,2490],{"className":2483},[165,203],[80,2485,121],{"className":2486},[235,203],[80,2488,1583],{"className":2489},[165,203],[80,2491,127],{"className":2492},[242,203],[80,2494,127],{"className":2495},[242],[80,2497],{"className":2498,"style":247},[246],[80,2500,130],{"className":2501},[251],[80,2503],{"className":2504,"style":247},[246],[80,2506,2508,2511,2514,2517,2520,2523,2526,2529],{"className":2507},[156],[80,2509],{"className":2510,"style":2316},[160],[80,2512,121],{"className":2513},[235],[80,2515,2083],{"className":2516},[165],[80,2518,114],{"className":2519},[214],[80,2521,2194],{"className":2522},[246],[80,2524],{"className":2525,"style":268},[246],[80,2527,2393],{"className":2528},[165],[80,2530,127],{"className":2531},[242],[11,2533,2534,2535,2338,2563,2591,2592,2595],{},"My plan was to fit a line through (or close to) those points. Once we find that line (i.e. once we find ",[80,2536,2538,2551],{"className":2537},[83],[80,2539,2541],{"className":2540},[87],[89,2542,2543],{"xmlns":91},[93,2544,2545,2549],{},[96,2546,2547],{},[102,2548,109],{},[145,2550,109],{"encoding":147},[80,2552,2554],{"className":2553,"ariaHidden":113},[152],[80,2555,2557,2560],{"className":2556},[156],[80,2558],{"className":2559,"style":334},[160],[80,2561,109],{"className":2562,"style":210},[165,169],[80,2564,2566,2579],{"className":2565},[83],[80,2567,2569],{"className":2568},[87],[89,2570,2571],{"xmlns":91},[93,2572,2573,2577],{},[96,2574,2575],{},[102,2576,117],{},[145,2578,117],{"encoding":147},[80,2580,2582],{"className":2581,"ariaHidden":113},[152],[80,2583,2585,2588],{"className":2584},[156],[80,2586],{"className":2587,"style":289},[160],[80,2589,117],{"className":2590},[165,169],") we can estimate the price of ",[15,2593,2594],{},"any"," new house, including one we've never seen, like that 1200 sqft one.",[46,2597,2598],{},[11,2599,2600,2601,2604],{},"Smaller scales are a habit, not just tidiness: later in the course, once the numbers get genuinely large, normalizing scales (",[73,2602,2603],{},"feature scaling",") becomes essential for training to not stall or blow up. Start getting your eye used to it now.",[2606,2607,2609],"h3",{"id":2608},"put-it-in-an-array","Put it in an array",[2611,2612,2616],"pre",{"className":2613,"code":2614,"language":2615,"meta":26,"style":26},"language-python shiki shiki-themes github-light github-dark","# =====================================================================\n# TRAINING DATA\n# =====================================================================\n# x_train -> INPUT variable (feature): house size, in thousands of sqft\n# y_train -> TARGET variable:          house price, in thousands of dollars\n# Order matters: x_train[0] and y_train[0] describe the SAME house.\n\nx_train = np.array([1.0, 2.0])       # Creates a 1-D NumPy array with the two sizes.\ny_train = np.array([300.0, 500.0])   # Creates a 1-D NumPy array with the matching prices.\n\nprint(f\"x_train = {x_train}\")\nprint(f\"y_train = {y_train}\")\n","python",[65,2617,2618,2624,2629,2634,2640,2646,2652,2658,2664,2670,2675,2681],{"__ignoreMap":26},[80,2619,2621],{"class":2620,"line":33},"line",[80,2622,2623],{},"# =====================================================================\n",[80,2625,2626],{"class":2620,"line":27},[80,2627,2628],{},"# TRAINING DATA\n",[80,2630,2632],{"class":2620,"line":2631},3,[80,2633,2623],{},[80,2635,2637],{"class":2620,"line":2636},4,[80,2638,2639],{},"# x_train -> INPUT variable (feature): house size, in thousands of sqft\n",[80,2641,2643],{"class":2620,"line":2642},5,[80,2644,2645],{},"# y_train -> TARGET variable:          house price, in thousands of dollars\n",[80,2647,2649],{"class":2620,"line":2648},6,[80,2650,2651],{},"# Order matters: x_train[0] and y_train[0] describe the SAME house.\n",[80,2653,2655],{"class":2620,"line":2654},7,[80,2656,2657],{"emptyLinePlaceholder":32},"\n",[80,2659,2661],{"class":2620,"line":2660},8,[80,2662,2663],{},"x_train = np.array([1.0, 2.0])       # Creates a 1-D NumPy array with the two sizes.\n",[80,2665,2667],{"class":2620,"line":2666},9,[80,2668,2669],{},"y_train = np.array([300.0, 500.0])   # Creates a 1-D NumPy array with the matching prices.\n",[80,2671,2673],{"class":2620,"line":2672},10,[80,2674,2657],{"emptyLinePlaceholder":32},[80,2676,2678],{"class":2620,"line":2677},11,[80,2679,2680],{},"print(f\"x_train = {x_train}\")\n",[80,2682,2684],{"class":2620,"line":2683},12,[80,2685,2686],{},"print(f\"y_train = {y_train}\")\n",[46,2688,2689,2694],{},[11,2690,2691],{},[15,2692,2693],{},"Output:",[2611,2695,2700],{"className":2696,"code":2698,"language":2699},[2697],"language-text","x_train = [1. 2.]\ny_train = [300. 500.]\n","text",[65,2701,2698],{"__ignoreMap":26},[11,2703,2704,2705,2708,2709,2712,2713,2716],{},"Notice I used ",[65,2706,2707],{},"np.array",", not a plain Python list (",[65,2710,2711],{},"[1.0, 2.0]"," on its own). That's not fancy-library snobbery: a NumPy array keeps the numbers packed tightly in memory and runs math on all of them at once (this is called ",[15,2714,2715],{},"vectorization","), instead of looping item by item the way a list would. With 2 numbers it makes zero difference, but with 2 million it's the difference between running in seconds or freezing your machine. The entire course, and honestly most serious ML code, starts from NumPy because of exactly this.",[46,2718,2719],{},[11,2720,2721,2724,2725,2727,2728,2731,2732,2735],{},[15,2722,2723],{},"On f-strings:"," the ",[65,2726,104],{}," before the quotes means Python evaluates whatever's inside ",[65,2729,2730],{},"{ }"," and drops it into the text. ",[65,2733,2734],{},"{value:.2f}"," formats with 2 decimal places. You'll see this everywhere from here on out.",[2606,2737,2739,2740,127],{"id":2738},"how-many-examples-do-i-have-mmm","How many examples do I have? (",[80,2741,2743,2756],{"className":2742},[83],[80,2744,2746],{"className":2745},[87],[89,2747,2748],{"xmlns":91},[93,2749,2750,2754],{},[96,2751,2752],{},[102,2753,322],{},[145,2755,322],{"encoding":147},[80,2757,2759],{"className":2758,"ariaHidden":113},[152],[80,2760,2762,2765],{"className":2761},[156],[80,2763],{"className":2764,"style":334},[160],[80,2766,322],{"className":2767},[165,169],[2611,2769,2771],{"className":2613,"code":2770,"language":2615,"meta":26,"style":26},"# Via .shape -- returns a TUPLE with the size of each dimension.\nprint(f\"x_train.shape: {x_train.shape}\")   # (2,) for a 1-D vector with 2 elements.\nm = x_train.shape[0]\nprint(f\"Number of training examples is: {m}\")\n\n# Via len() -- works on NumPy arrays just like it does on lists.\nm = len(x_train)\nprint(f\"Number of training examples is: {m}\")\n",[65,2772,2773,2778,2783,2788,2793,2797,2802,2807],{"__ignoreMap":26},[80,2774,2775],{"class":2620,"line":33},[80,2776,2777],{},"# Via .shape -- returns a TUPLE with the size of each dimension.\n",[80,2779,2780],{"class":2620,"line":27},[80,2781,2782],{},"print(f\"x_train.shape: {x_train.shape}\")   # (2,) for a 1-D vector with 2 elements.\n",[80,2784,2785],{"class":2620,"line":2631},[80,2786,2787],{},"m = x_train.shape[0]\n",[80,2789,2790],{"class":2620,"line":2636},[80,2791,2792],{},"print(f\"Number of training examples is: {m}\")\n",[80,2794,2795],{"class":2620,"line":2642},[80,2796,2657],{"emptyLinePlaceholder":32},[80,2798,2799],{"class":2620,"line":2648},[80,2800,2801],{},"# Via len() -- works on NumPy arrays just like it does on lists.\n",[80,2803,2804],{"class":2620,"line":2654},[80,2805,2806],{},"m = len(x_train)\n",[80,2808,2809],{"class":2620,"line":2660},[80,2810,2792],{},[11,2812,2813,2814,2817,2818,2870,2871,2873,2874,2877,2878,2881],{},"Both give the same result here, but I picked up the ",[65,2815,2816],{},".shape"," habit early, and it's worth picking up too: once your data has multiple columns (that ",[80,2819,2821,2840],{"className":2820},[83],[80,2822,2824],{"className":2823},[87],[89,2825,2826],{"xmlns":91},[93,2827,2828,2837],{},[96,2829,2830,2832,2835],{},[102,2831,322],{},[111,2833,2834],{},"×",[102,2836,1487],{},[145,2838,2839],{"encoding":147},"m \\times n",[80,2841,2843,2861],{"className":2842,"ariaHidden":113},[152],[80,2844,2846,2849,2852,2855,2858],{"className":2845},[156],[80,2847],{"className":2848,"style":261},[160],[80,2850,322],{"className":2851},[165,169],[80,2853],{"className":2854,"style":275},[246],[80,2856,2834],{"className":2857},[279],[80,2859],{"className":2860,"style":275},[246],[80,2862,2864,2867],{"className":2863},[156],[80,2865],{"className":2866,"style":334},[160],[80,2868,1487],{"className":2869},[165,169]," shape mentioned above), ",[65,2872,2816],{}," tells you rows ",[73,2875,2876],{},"and"," columns up front, while ",[65,2879,2880],{},"len()"," only gives you the row count and leaves you guessing about the rest.",[2606,2883,2885],{"id":2884},"grabbing-a-specific-example","Grabbing a specific example",[11,2887,2888,2889,2892],{},"Python counts from ",[15,2890,2891],{},"zero",", always:",[521,2894,2895,3076],{},[524,2896,2897],{},[527,2898,2899,2930,3074],{},[530,2900,2901,2902],{"align":1884},"Index ",[80,2903,2905,2918],{"className":2904},[83],[80,2906,2908],{"className":2907},[87],[89,2909,2910],{"xmlns":91},[93,2911,2912,2916],{},[96,2913,2914],{},[102,2915,736],{},[145,2917,736],{"encoding":147},[80,2919,2921],{"className":2920,"ariaHidden":113},[152],[80,2922,2924,2927],{"className":2923},[156],[80,2925],{"className":2926,"style":895},[160],[80,2928,736],{"className":2929},[165,169],[530,2931,2932],{"align":1884},[80,2933,2935,2977],{"className":2934},[83],[80,2936,2938],{"className":2937},[87],[89,2939,2940],{"xmlns":91},[93,2941,2942,2974],{},[96,2943,2944,2946,2958,2960,2972],{},[111,2945,121],{"stretchy":120},[726,2947,2948,2950],{},[102,2949,124],{},[96,2951,2952,2954,2956],{},[111,2953,121],{"stretchy":120},[102,2955,736],{},[111,2957,127],{"stretchy":120},[111,2959,114],{"separator":113},[726,2961,2962,2964],{},[102,2963,683],{},[96,2965,2966,2968,2970],{},[111,2967,121],{"stretchy":120},[102,2969,736],{},[111,2971,127],{"stretchy":120},[111,2973,127],{"stretchy":120},[145,2975,2976],{"encoding":147},"(x^{(i)}, y^{(i)})",[80,2978,2980],{"className":2979,"ariaHidden":113},[152],[80,2981,2983,2986,2989,3027,3030,3033,3071],{"className":2982},[156],[80,2984],{"className":2985,"style":2212},[160],[80,2987,121],{"className":2988},[235],[80,2990,2992,2995],{"className":2991},[165],[80,2993,124],{"className":2994},[165,169],[80,2996,2998],{"className":2997},[174],[80,2999,3001],{"className":3000},[178],[80,3002,3004],{"className":3003},[183],[80,3005,3007],{"className":3006,"style":751},[187],[80,3008,3009,3012],{"style":772},[80,3010],{"className":3011,"style":196},[195],[80,3013,3015],{"className":3014},[200,201,202,203],[80,3016,3018,3021,3024],{"className":3017},[165,203],[80,3019,121],{"className":3020},[235,203],[80,3022,736],{"className":3023},[165,169,203],[80,3025,127],{"className":3026},[242,203],[80,3028,114],{"className":3029},[214],[80,3031],{"className":3032,"style":268},[246],[80,3034,3036,3039],{"className":3035},[165],[80,3037,683],{"className":3038,"style":834},[165,169],[80,3040,3042],{"className":3041},[174],[80,3043,3045],{"className":3044},[178],[80,3046,3048],{"className":3047},[183],[80,3049,3051],{"className":3050,"style":751},[187],[80,3052,3053,3056],{"style":772},[80,3054],{"className":3055,"style":196},[195],[80,3057,3059],{"className":3058},[200,201,202,203],[80,3060,3062,3065,3068],{"className":3061},[165,203],[80,3063,121],{"className":3064},[235,203],[80,3066,736],{"className":3067},[165,169,203],[80,3069,127],{"className":3070},[242,203],[80,3072,127],{"className":3073},[242],[530,3075,539],{"align":532},[541,3077,3078,3149],{},[527,3079,3080,3082,3141],{},[546,3081,2071],{"align":1884},[546,3083,3084],{"align":1884},[80,3085,3087,3111],{"className":3086},[83],[80,3088,3090],{"className":3089},[87],[89,3091,3092],{"xmlns":91},[93,3093,3094,3108],{},[96,3095,3096,3098,3100,3102,3104,3106],{},[111,3097,121],{"stretchy":120},[1321,3099,2074],{},[111,3101,114],{"separator":113},[134,3103,2194],{},[1321,3105,2197],{},[111,3107,127],{"stretchy":120},[145,3109,3110],{"encoding":147},"(1.0,\\ 300.0)",[80,3112,3114],{"className":3113,"ariaHidden":113},[152],[80,3115,3117,3120,3123,3126,3129,3132,3135,3138],{"className":3116},[156],[80,3118],{"className":3119,"style":2316},[160],[80,3121,121],{"className":3122},[235],[80,3124,2074],{"className":3125},[165],[80,3127,114],{"className":3128},[214],[80,3130,2194],{"className":3131},[246],[80,3133],{"className":3134,"style":268},[246],[80,3136,2197],{"className":3137},[165],[80,3139,127],{"className":3140},[242],[546,3142,3143,504,3146],{"align":532},[65,3144,3145],{},"x_train[0]",[65,3147,3148],{},"y_train[0]",[527,3150,3151,3153,3212],{},[546,3152,1583],{"align":1884},[546,3154,3155],{"align":1884},[80,3156,3158,3182],{"className":3157},[83],[80,3159,3161],{"className":3160},[87],[89,3162,3163],{"xmlns":91},[93,3164,3165,3179],{},[96,3166,3167,3169,3171,3173,3175,3177],{},[111,3168,121],{"stretchy":120},[1321,3170,2083],{},[111,3172,114],{"separator":113},[134,3174,2194],{},[1321,3176,2393],{},[111,3178,127],{"stretchy":120},[145,3180,3181],{"encoding":147},"(2.0,\\ 500.0)",[80,3183,3185],{"className":3184,"ariaHidden":113},[152],[80,3186,3188,3191,3194,3197,3200,3203,3206,3209],{"className":3187},[156],[80,3189],{"className":3190,"style":2316},[160],[80,3192,121],{"className":3193},[235],[80,3195,2083],{"className":3196},[165],[80,3198,114],{"className":3199},[214],[80,3201,2194],{"className":3202},[246],[80,3204],{"className":3205,"style":268},[246],[80,3207,2393],{"className":3208},[165],[80,3210,127],{"className":3211},[242],[546,3213,3214,504,3217],{"align":532},[65,3215,3216],{},"x_train[1]",[65,3218,3219],{},"y_train[1]",[2611,3221,3223],{"className":2613,"code":3222,"language":2615,"meta":26,"style":26},"i = 0                    # Index of the example we want to inspect.\nx_i = x_train[i]\ny_i = y_train[i]\n\nprint(f\"(x^({i}), y^({i})) = ({x_i}, {y_i})\")\n",[65,3224,3225,3230,3235,3240,3244],{"__ignoreMap":26},[80,3226,3227],{"class":2620,"line":33},[80,3228,3229],{},"i = 0                    # Index of the example we want to inspect.\n",[80,3231,3232],{"class":2620,"line":27},[80,3233,3234],{},"x_i = x_train[i]\n",[80,3236,3237],{"class":2620,"line":2631},[80,3238,3239],{},"y_i = y_train[i]\n",[80,3241,3242],{"class":2620,"line":2636},[80,3243,2657],{"emptyLinePlaceholder":32},[80,3245,3246],{"class":2620,"line":2642},[80,3247,3248],{},"print(f\"(x^({i}), y^({i})) = ({x_i}, {y_i})\")\n",[46,3250,3251],{},[11,3252,3253,3255,3256],{},[15,3254,2693],{}," ",[65,3257,3258],{},"(x^(0), y^(0)) = (1.0, 300.0)",[11,3260,3261,3262,3336,3337,3339],{},"An annoying detail that gets even sharp people: in lecture, Andrew Ng sometimes counts starting at 1 (",[80,3263,3265,3289],{"className":3264},[83],[80,3266,3268],{"className":3267},[87],[89,3269,3270],{"xmlns":91},[93,3271,3272,3286],{},[96,3273,3274],{},[726,3275,3276,3278],{},[102,3277,124],{},[96,3279,3280,3282,3284],{},[111,3281,121],{"stretchy":120},[1321,3283,1583],{},[111,3285,127],{"stretchy":120},[145,3287,3288],{"encoding":147},"x^{(1)}",[80,3290,3292],{"className":3291,"ariaHidden":113},[152],[80,3293,3295,3298],{"className":3294},[156],[80,3296],{"className":3297,"style":751},[160],[80,3299,3301,3304],{"className":3300},[165],[80,3302,124],{"className":3303},[165,169],[80,3305,3307],{"className":3306},[174],[80,3308,3310],{"className":3309},[178],[80,3311,3313],{"className":3312},[183],[80,3314,3316],{"className":3315,"style":751},[187],[80,3317,3318,3321],{"style":772},[80,3319],{"className":3320,"style":196},[195],[80,3322,3324],{"className":3323},[200,201,202,203],[80,3325,3327,3330,3333],{"className":3326},[165,203],[80,3328,121],{"className":3329},[235,203],[80,3331,1583],{"className":3332},[165,203],[80,3334,127],{"className":3335},[242,203]," being the first example). In code, the first one is always ",[65,3338,3145],{},". Same house, just a different counting convention. It's not a bug in your head if you get an \"off-by-one\" moment, it's just how the field talks.",[294,3341,3343],{"id":3342},"look-at-your-data-before-you-model-it","Look at your data before you model it",[11,3345,3346],{},"This isn't generic \"best practices\" advice, it's a survival rule. If you jump straight into fitting a line without looking at the shape of your data, you won't notice when the relationship isn't linear at all (in which case linear regression is the wrong tool for the job). A scatter plot settles this in two seconds. Hover the points:",[3348,3349],"model-prediction-chart",{":x-train":3350,":y-train":3351,"dataLabel":3352,"x-label":3353,"y-label":3354},"[1, 2]","[300, 500]","Training data","Size (1000 sqft)","Price (1000 dollars)",[11,3356,3357],{},"With just 2 points, \"linear or not\" is fairly obvious, but the habit of looking first is what matters here, because with 200 or 2000 points it'll save you hours of tuning the wrong model.",[294,3359,3361],{"id":3360},"the-model-function-what-w-and-b-actually-mean","The model function: what w and b actually mean",[11,3363,3364],{},[80,3365,3367,3429],{"className":3366},[83],[80,3368,3370],{"className":3369},[87],[89,3371,3372],{"xmlns":91},[93,3373,3374,3426],{},[96,3375,3376,3388,3390,3402,3404,3406,3408,3410,3422,3424],{},[99,3377,3378,3380],{},[102,3379,104],{},[96,3381,3382,3384,3386],{},[102,3383,109],{},[111,3385,114],{"separator":113},[102,3387,117],{},[111,3389,121],{"stretchy":120},[726,3391,3392,3394],{},[102,3393,124],{},[96,3395,3396,3398,3400],{},[111,3397,121],{"stretchy":120},[102,3399,736],{},[111,3401,127],{"stretchy":120},[111,3403,127],{"stretchy":120},[111,3405,130],{},[102,3407,109],{},[134,3409,136],{},[726,3411,3412,3414],{},[102,3413,124],{},[96,3415,3416,3418,3420],{},[111,3417,121],{"stretchy":120},[102,3419,736],{},[111,3421,127],{"stretchy":120},[111,3423,141],{},[102,3425,117],{},[145,3427,3428],{"encoding":147},"f_{w,b}(x^{(i)}) = w\\,x^{(i)} + b",[80,3430,3432,3540,3600],{"className":3431,"ariaHidden":113},[152],[80,3433,3435,3438,3487,3490,3528,3531,3534,3537],{"className":3434},[156],[80,3436],{"className":3437,"style":1080},[160],[80,3439,3441,3444],{"className":3440},[165],[80,3442,104],{"className":3443,"style":170},[165,169],[80,3445,3447],{"className":3446},[174],[80,3448,3450,3479],{"className":3449},[178,179],[80,3451,3453,3476],{"className":3452},[183],[80,3454,3456],{"className":3455,"style":188},[187],[80,3457,3458,3461],{"style":191},[80,3459],{"className":3460,"style":196},[195],[80,3462,3464],{"className":3463},[200,201,202,203],[80,3465,3467,3470,3473],{"className":3466},[165,203],[80,3468,109],{"className":3469,"style":210},[165,169,203],[80,3471,114],{"className":3472},[214,203],[80,3474,117],{"className":3475},[165,169,203],[80,3477,222],{"className":3478},[221],[80,3480,3482],{"className":3481},[183],[80,3483,3485],{"className":3484,"style":229},[187],[80,3486],{},[80,3488,121],{"className":3489},[235],[80,3491,3493,3496],{"className":3492},[165],[80,3494,124],{"className":3495},[165,169],[80,3497,3499],{"className":3498},[174],[80,3500,3502],{"className":3501},[178],[80,3503,3505],{"className":3504},[183],[80,3506,3508],{"className":3507,"style":751},[187],[80,3509,3510,3513],{"style":772},[80,3511],{"className":3512,"style":196},[195],[80,3514,3516],{"className":3515},[200,201,202,203],[80,3517,3519,3522,3525],{"className":3518},[165,203],[80,3520,121],{"className":3521},[235,203],[80,3523,736],{"className":3524},[165,169,203],[80,3526,127],{"className":3527},[242,203],[80,3529,127],{"className":3530},[242],[80,3532],{"className":3533,"style":247},[246],[80,3535,130],{"className":3536},[251],[80,3538],{"className":3539,"style":247},[246],[80,3541,3543,3547,3550,3553,3591,3594,3597],{"className":3542},[156],[80,3544],{"className":3545,"style":3546},[160],"height:0.9713em;vertical-align:-0.0833em;",[80,3548,109],{"className":3549,"style":210},[165,169],[80,3551],{"className":3552,"style":268},[246],[80,3554,3556,3559],{"className":3555},[165],[80,3557,124],{"className":3558},[165,169],[80,3560,3562],{"className":3561},[174],[80,3563,3565],{"className":3564},[178],[80,3566,3568],{"className":3567},[183],[80,3569,3571],{"className":3570,"style":751},[187],[80,3572,3573,3576],{"style":772},[80,3574],{"className":3575,"style":196},[195],[80,3577,3579],{"className":3578},[200,201,202,203],[80,3580,3582,3585,3588],{"className":3581},[165,203],[80,3583,121],{"className":3584},[235,203],[80,3586,736],{"className":3587},[165,169,203],[80,3589,127],{"className":3590},[242,203],[80,3592],{"className":3593,"style":275},[246],[80,3595,141],{"className":3596},[279],[80,3598],{"className":3599,"style":275},[246],[80,3601,3603,3606],{"className":3602},[156],[80,3604],{"className":3605,"style":289},[160],[80,3607,117],{"className":3608},[165,169],[11,3610,3611,3612,3615,3616,3619,3620,1618],{},"Okay, it's a line's equation. But \"slope\" and \"intercept\" are dry terms, let me give you a better picture: think of a ride-hailing app. Every ride has a ",[15,3613,3614],{},"flat pickup fee"," (you pay this just for getting in the car, no matter the distance) and a ",[15,3617,3618],{},"rate per kilometer\u002Fmile driven",". The final price is ",[65,3621,3622],{},"flat_fee + rate_per_km * distance",[11,3624,3625,3626,3774],{},"That maps exactly onto ",[80,3627,3629,3668],{"className":3628},[83],[80,3630,3632],{"className":3631},[87],[89,3633,3634],{"xmlns":91},[93,3635,3636,3666],{},[96,3637,3638,3650,3652,3654,3656,3658,3660,3662,3664],{},[99,3639,3640,3642],{},[102,3641,104],{},[96,3643,3644,3646,3648],{},[102,3645,109],{},[111,3647,114],{"separator":113},[102,3649,117],{},[111,3651,121],{"stretchy":120},[102,3653,124],{},[111,3655,127],{"stretchy":120},[111,3657,130],{},[102,3659,109],{},[102,3661,124],{},[111,3663,141],{},[102,3665,117],{},[145,3667,387],{"encoding":147},[80,3669,3671,3744,3765],{"className":3670,"ariaHidden":113},[152],[80,3672,3674,3677,3726,3729,3732,3735,3738,3741],{"className":3673},[156],[80,3675],{"className":3676,"style":161},[160],[80,3678,3680,3683],{"className":3679},[165],[80,3681,104],{"className":3682,"style":170},[165,169],[80,3684,3686],{"className":3685},[174],[80,3687,3689,3718],{"className":3688},[178,179],[80,3690,3692,3715],{"className":3691},[183],[80,3693,3695],{"className":3694,"style":188},[187],[80,3696,3697,3700],{"style":191},[80,3698],{"className":3699,"style":196},[195],[80,3701,3703],{"className":3702},[200,201,202,203],[80,3704,3706,3709,3712],{"className":3705},[165,203],[80,3707,109],{"className":3708,"style":210},[165,169,203],[80,3710,114],{"className":3711},[214,203],[80,3713,117],{"className":3714},[165,169,203],[80,3716,222],{"className":3717},[221],[80,3719,3721],{"className":3720},[183],[80,3722,3724],{"className":3723,"style":229},[187],[80,3725],{},[80,3727,121],{"className":3728},[235],[80,3730,124],{"className":3731},[165,169],[80,3733,127],{"className":3734},[242],[80,3736],{"className":3737,"style":247},[246],[80,3739,130],{"className":3740},[251],[80,3742],{"className":3743,"style":247},[246],[80,3745,3747,3750,3753,3756,3759,3762],{"className":3746},[156],[80,3748],{"className":3749,"style":261},[160],[80,3751,109],{"className":3752,"style":210},[165,169],[80,3754,124],{"className":3755},[165,169],[80,3757],{"className":3758,"style":275},[246],[80,3760,141],{"className":3761},[279],[80,3763],{"className":3764,"style":275},[246],[80,3766,3768,3771],{"className":3767},[156],[80,3769],{"className":3770,"style":289},[160],[80,3772,117],{"className":3773},[165,169],":",[521,3776,3777,3793],{},[524,3778,3779],{},[527,3780,3781,3784,3787,3790],{},[530,3782,3783],{"align":1884},"Parameter",[530,3785,3786],{"align":532},"In the ride app",[530,3788,3789],{"align":532},"Here",[530,3791,3792],{"align":532},"Geometric meaning",[541,3794,3795,3839],{},[527,3796,3797,3827,3830,3833],{},[546,3798,3799],{"align":1884},[80,3800,3802,3815],{"className":3801},[83],[80,3803,3805],{"className":3804},[87],[89,3806,3807],{"xmlns":91},[93,3808,3809,3813],{},[96,3810,3811],{},[102,3812,109],{},[145,3814,109],{"encoding":147},[80,3816,3818],{"className":3817,"ariaHidden":113},[152],[80,3819,3821,3824],{"className":3820},[156],[80,3822],{"className":3823,"style":334},[160],[80,3825,109],{"className":3826,"style":210},[165,169],[546,3828,3829],{"align":532},"rate per km\u002Fmile driven",[546,3831,3832],{"align":532},"how much price rises per extra 1000 sqft",[546,3834,3835,3836],{"align":532},"the line's ",[15,3837,3838],{},"slope",[527,3840,3841,3871,3873,3876],{},[546,3842,3843],{"align":1884},[80,3844,3846,3859],{"className":3845},[83],[80,3847,3849],{"className":3848},[87],[89,3850,3851],{"xmlns":91},[93,3852,3853,3857],{},[96,3854,3855],{},[102,3856,117],{},[145,3858,117],{"encoding":147},[80,3860,3862],{"className":3861,"ariaHidden":113},[152],[80,3863,3865,3868],{"className":3864},[156],[80,3866],{"className":3867,"style":289},[160],[80,3869,117],{"className":3870},[165,169],[546,3872,3614],{"align":532},[546,3874,3875],{"align":532},"base price, when size is zero",[546,3877,3878,3879,3882],{"align":532},"the ",[15,3880,3881],{},"intercept"," (where the line crosses the vertical axis)",[11,3884,3885,3886,3914,3915,3943,3944,2338,3972,4000],{},"If ",[80,3887,3889,3902],{"className":3888},[83],[80,3890,3892],{"className":3891},[87],[89,3893,3894],{"xmlns":91},[93,3895,3896,3900],{},[96,3897,3898],{},[102,3899,109],{},[145,3901,109],{"encoding":147},[80,3903,3905],{"className":3904,"ariaHidden":113},[152],[80,3906,3908,3911],{"className":3907},[156],[80,3909],{"className":3910,"style":334},[160],[80,3912,109],{"className":3913,"style":210},[165,169]," is large, every km (or every 1000 sqft) weighs more on the final price: the line climbs fast. If ",[80,3916,3918,3931],{"className":3917},[83],[80,3919,3921],{"className":3920},[87],[89,3922,3923],{"xmlns":91},[93,3924,3925,3929],{},[96,3926,3927],{},[102,3928,117],{},[145,3930,117],{"encoding":147},[80,3932,3934],{"className":3933,"ariaHidden":113},[152],[80,3935,3937,3940],{"className":3936},[156],[80,3938],{"className":3939,"style":289},[160],[80,3941,117],{"className":3942},[165,169]," is large, you're already \"paying a lot\" before moving an inch, and the line starts higher up on the vertical axis. Different combinations of ",[80,3945,3947,3960],{"className":3946},[83],[80,3948,3950],{"className":3949},[87],[89,3951,3952],{"xmlns":91},[93,3953,3954,3958],{},[96,3955,3956],{},[102,3957,109],{},[145,3959,109],{"encoding":147},[80,3961,3963],{"className":3962,"ariaHidden":113},[152],[80,3964,3966,3969],{"className":3965},[156],[80,3967],{"className":3968,"style":334},[160],[80,3970,109],{"className":3971,"style":210},[165,169],[80,3973,3975,3988],{"className":3974},[83],[80,3976,3978],{"className":3977},[87],[89,3979,3980],{"xmlns":91},[93,3981,3982,3986],{},[96,3983,3984],{},[102,3985,117],{},[145,3987,117],{"encoding":147},[80,3989,3991],{"className":3990,"ariaHidden":113},[152],[80,3992,3994,3997],{"className":3993},[156],[80,3995],{"className":3996,"style":289},[160],[80,3998,117],{"className":3999},[165,169]," draw completely different lines.",[11,4002,4003,4059,4060,4063,4064,2338,4068,4072,4073,4076],{},[15,4004,4005,4006,4058],{},"That's when it clicked for me: training the model is exactly about finding the pair ",[80,4007,4009,4031],{"className":4008},[83],[80,4010,4012],{"className":4011},[87],[89,4013,4014],{"xmlns":91},[93,4015,4016,4028],{},[96,4017,4018,4020,4022,4024,4026],{},[111,4019,121],{"stretchy":120},[102,4021,109],{},[111,4023,114],{"separator":113},[102,4025,117],{},[111,4027,127],{"stretchy":120},[145,4029,4030],{"encoding":147},"(w, b)",[80,4032,4034],{"className":4033,"ariaHidden":113},[152],[80,4035,4037,4040,4043,4046,4049,4052,4055],{"className":4036},[156],[80,4038],{"className":4039,"style":2316},[160],[80,4041,121],{"className":4042},[235],[80,4044,109],{"className":4045,"style":210},[165,169],[80,4047,114],{"className":4048},[214],[80,4050],{"className":4051,"style":268},[246],[80,4053,117],{"className":4054},[165,169],[80,4056,127],{"className":4057},[242]," that best describes the data you have."," We haven't yet seen ",[73,4061,4062],{},"how"," to automate that search (that's the topic of the next two posts: ",[562,4065,4067],{"href":4066},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab03-cost-function","cost function",[562,4069,4071],{"href":4070},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab04-gradient-descent","gradient descent","). For now, we'll do it ",[15,4074,4075],{},"by hand",", guessing values, so you feel firsthand the problem those next posts solve.",[2606,4078,4080],{"id":4079},"turning-the-formula-into-a-function","Turning the formula into a function",[11,4082,4083,4084,2338,4087,4090],{},"With 2 points, I could compute ",[65,4085,4086],{},"w * x[0] + b",[65,4088,4089],{},"w * x[1] + b"," by hand. With a thousand points that turns into useless busywork, and that exact kind of repetition is what we wrap into a function:",[2611,4092,4094],{"className":2613,"code":4093,"language":2615,"meta":26,"style":26},"def compute_model_output(x, w, b):\n    \"\"\"\n    Computes the prediction of a linear model f_wb(x) = w*x + b.\n\n    Args:\n      x (ndarray (m,)) : input data, m examples (the feature)\n      w, b (scalar)    : model parameters (weight and bias)\n\n    Returns:\n      f_wb (ndarray (m,)) : model prediction for every example in x\n    \"\"\"\n    m = x.shape[0]              # 1) How many examples are in the input array.\n    f_wb = np.zeros(m)          # 2) Output container: array of m zeros.\n\n    for i in range(m):          # 3) Loops through i = 0, 1, ..., m-1.\n        f_wb[i] = w * x[i] + b  #    Applies the line equation to example i.\n\n    return f_wb                 # Returns the full array of predictions.\n",[65,4095,4096,4101,4106,4111,4115,4120,4125,4130,4134,4139,4144,4148,4153,4159,4164,4170,4176,4181],{"__ignoreMap":26},[80,4097,4098],{"class":2620,"line":33},[80,4099,4100],{},"def compute_model_output(x, w, b):\n",[80,4102,4103],{"class":2620,"line":27},[80,4104,4105],{},"    \"\"\"\n",[80,4107,4108],{"class":2620,"line":2631},[80,4109,4110],{},"    Computes the prediction of a linear model f_wb(x) = w*x + b.\n",[80,4112,4113],{"class":2620,"line":2636},[80,4114,2657],{"emptyLinePlaceholder":32},[80,4116,4117],{"class":2620,"line":2642},[80,4118,4119],{},"    Args:\n",[80,4121,4122],{"class":2620,"line":2648},[80,4123,4124],{},"      x (ndarray (m,)) : input data, m examples (the feature)\n",[80,4126,4127],{"class":2620,"line":2654},[80,4128,4129],{},"      w, b (scalar)    : model parameters (weight and bias)\n",[80,4131,4132],{"class":2620,"line":2660},[80,4133,2657],{"emptyLinePlaceholder":32},[80,4135,4136],{"class":2620,"line":2666},[80,4137,4138],{},"    Returns:\n",[80,4140,4141],{"class":2620,"line":2672},[80,4142,4143],{},"      f_wb (ndarray (m,)) : model prediction for every example in x\n",[80,4145,4146],{"class":2620,"line":2677},[80,4147,4105],{},[80,4149,4150],{"class":2620,"line":2683},[80,4151,4152],{},"    m = x.shape[0]              # 1) How many examples are in the input array.\n",[80,4154,4156],{"class":2620,"line":4155},13,[80,4157,4158],{},"    f_wb = np.zeros(m)          # 2) Output container: array of m zeros.\n",[80,4160,4162],{"class":2620,"line":4161},14,[80,4163,2657],{"emptyLinePlaceholder":32},[80,4165,4167],{"class":2620,"line":4166},15,[80,4168,4169],{},"    for i in range(m):          # 3) Loops through i = 0, 1, ..., m-1.\n",[80,4171,4173],{"class":2620,"line":4172},16,[80,4174,4175],{},"        f_wb[i] = w * x[i] + b  #    Applies the line equation to example i.\n",[80,4177,4179],{"class":2620,"line":4178},17,[80,4180,2657],{"emptyLinePlaceholder":32},[80,4182,4184],{"class":2620,"line":4183},18,[80,4185,4186],{},"    return f_wb                 # Returns the full array of predictions.\n",[11,4188,4189,4190,504,4192,504,4194,4196],{},"I wrapped this in a function with a clear contract (takes ",[65,4191,124],{},[65,4193,109],{},[65,4195,117],{},", returns an array of predictions), and that means it works for 2 examples, 2 thousand, or 2 million, without you rewriting anything. It's a small engineering detail, but it's exactly the kind of detail that separates a \"script that runs once\" from code that can carry a real system.",[294,4198,4200],{"id":4199},"your-turn-find-the-perfect-fit","Your turn: find the perfect fit",[11,4202,4203,4204,2338,4232,4260,4261,4264],{},"Below is a genuinely interactive chart: drag the ",[80,4205,4207,4220],{"className":4206},[83],[80,4208,4210],{"className":4209},[87],[89,4211,4212],{"xmlns":91},[93,4213,4214,4218],{},[96,4215,4216],{},[102,4217,109],{},[145,4219,109],{"encoding":147},[80,4221,4223],{"className":4222,"ariaHidden":113},[152],[80,4224,4226,4229],{"className":4225},[156],[80,4227],{"className":4228,"style":334},[160],[80,4230,109],{"className":4231,"style":210},[165,169],[80,4233,4235,4248],{"className":4234},[83],[80,4236,4238],{"className":4237},[87],[89,4239,4240],{"xmlns":91},[93,4241,4242,4246],{},[96,4243,4244],{},[102,4245,117],{},[145,4247,117],{"encoding":147},[80,4249,4251],{"className":4250,"ariaHidden":113},[152],[80,4252,4254,4257],{"className":4253},[156],[80,4255],{"className":4256,"style":289},[160],[80,4258,117],{"className":4259},[165,169]," sliders and watch three things at once: the blue line (your model's prediction), the dashed gray segments (the gap between each prediction and the real value, that's each example's ",[15,4262,4263],{},"error","), and the \"Total error\" number under the chart.",[11,4266,4267,4268,504,4320,4371,4372,4375],{},"It starts at ",[80,4269,4271,4290],{"className":4270},[83],[80,4272,4274],{"className":4273},[87],[89,4275,4276],{"xmlns":91},[93,4277,4278,4287],{},[96,4279,4280,4282,4284],{},[102,4281,109],{},[111,4283,130],{},[1321,4285,4286],{},"100",[145,4288,4289],{"encoding":147},"w=100",[80,4291,4293,4311],{"className":4292,"ariaHidden":113},[152],[80,4294,4296,4299,4302,4305,4308],{"className":4295},[156],[80,4297],{"className":4298,"style":334},[160],[80,4300,109],{"className":4301,"style":210},[165,169],[80,4303],{"className":4304,"style":247},[246],[80,4306,130],{"className":4307},[251],[80,4309],{"className":4310,"style":247},[246],[80,4312,4314,4317],{"className":4313},[156],[80,4315],{"className":4316,"style":1614},[160],[80,4318,4286],{"className":4319},[165],[80,4321,4323,4341],{"className":4322},[83],[80,4324,4326],{"className":4325},[87],[89,4327,4328],{"xmlns":91},[93,4329,4330,4338],{},[96,4331,4332,4334,4336],{},[102,4333,117],{},[111,4335,130],{},[1321,4337,4286],{},[145,4339,4340],{"encoding":147},"b=100",[80,4342,4344,4362],{"className":4343,"ariaHidden":113},[152],[80,4345,4347,4350,4353,4356,4359],{"className":4346},[156],[80,4348],{"className":4349,"style":289},[160],[80,4351,117],{"className":4352},[165,169],[80,4354],{"className":4355,"style":247},[246],[80,4357,130],{"className":4358},[251],[80,4360],{"className":4361,"style":247},[246],[80,4363,4365,4368],{"className":4364},[156],[80,4366],{"className":4367,"style":1614},[160],[80,4369,4286],{"className":4370},[165]," (the original notebook's first guess), notice how far the line sits from the red diamonds, and how large the total error is. ",[15,4373,4374],{},"Your challenge:"," move the sliders until the total error hits zero.",[11,4377,4378],{},"Two hints, if you want to think it through before dragging blindly:",[299,4380,4381,4415],{},[302,4382,4383,4414],{},[15,4384,4385,4386],{},"The slope ",[80,4387,4389,4402],{"className":4388},[83],[80,4390,4392],{"className":4391},[87],[89,4393,4394],{"xmlns":91},[93,4395,4396,4400],{},[96,4397,4398],{},[102,4399,109],{},[145,4401,109],{"encoding":147},[80,4403,4405],{"className":4404,"ariaHidden":113},[152],[80,4406,4408,4411],{"className":4407},[156],[80,4409],{"className":4410,"style":334},[160],[80,4412,109],{"className":4413,"style":210},[165,169],": price rises from 300 to 500 (a rise of 200) while size rises from 1.0 to 2.0 (a run of 1.0). Slope is \"how much it rose divided by how much it ran\".",[302,4416,4417,4448,4449,4477,4478,4557,4558,1618],{},[15,4418,4419,4420],{},"The intercept ",[80,4421,4423,4436],{"className":4422},[83],[80,4424,4426],{"className":4425},[87],[89,4427,4428],{"xmlns":91},[93,4429,4430,4434],{},[96,4431,4432],{},[102,4433,117],{},[145,4435,117],{"encoding":147},[80,4437,4439],{"className":4438,"ariaHidden":113},[152],[80,4440,4442,4445],{"className":4441},[156],[80,4443],{"className":4444,"style":289},[160],[80,4446,117],{"className":4447},[165,169],": once you know ",[80,4450,4452,4465],{"className":4451},[83],[80,4453,4455],{"className":4454},[87],[89,4456,4457],{"xmlns":91},[93,4458,4459,4463],{},[96,4460,4461],{},[102,4462,109],{},[145,4464,109],{"encoding":147},[80,4466,4468],{"className":4467,"ariaHidden":113},[152],[80,4469,4471,4474],{"className":4470},[156],[80,4472],{"className":4473,"style":334},[160],[80,4475,109],{"className":4476,"style":210},[165,169],", plug one of the points into ",[80,4479,4481,4505],{"className":4480},[83],[80,4482,4484],{"className":4483},[87],[89,4485,4486],{"xmlns":91},[93,4487,4488,4502],{},[96,4489,4490,4492,4494,4496,4498,4500],{},[102,4491,683],{},[111,4493,130],{},[102,4495,109],{},[102,4497,124],{},[111,4499,141],{},[102,4501,117],{},[145,4503,4504],{"encoding":147},"y = wx + b",[80,4506,4508,4527,4548],{"className":4507,"ariaHidden":113},[152],[80,4509,4511,4515,4518,4521,4524],{"className":4510},[156],[80,4512],{"className":4513,"style":4514},[160],"height:0.625em;vertical-align:-0.1944em;",[80,4516,683],{"className":4517,"style":834},[165,169],[80,4519],{"className":4520,"style":247},[246],[80,4522,130],{"className":4523},[251],[80,4525],{"className":4526,"style":247},[246],[80,4528,4530,4533,4536,4539,4542,4545],{"className":4529},[156],[80,4531],{"className":4532,"style":261},[160],[80,4534,109],{"className":4535,"style":210},[165,169],[80,4537,124],{"className":4538},[165,169],[80,4540],{"className":4541,"style":275},[246],[80,4543,141],{"className":4544},[279],[80,4546],{"className":4547,"style":275},[246],[80,4549,4551,4554],{"className":4550},[156],[80,4552],{"className":4553,"style":289},[160],[80,4555,117],{"className":4556},[165,169]," and solve for ",[80,4559,4561,4574],{"className":4560},[83],[80,4562,4564],{"className":4563},[87],[89,4565,4566],{"xmlns":91},[93,4567,4568,4572],{},[96,4569,4570],{},[102,4571,117],{},[145,4573,117],{"encoding":147},[80,4575,4577],{"className":4576,"ariaHidden":113},[152],[80,4578,4580,4583],{"className":4579},[156],[80,4581],{"className":4582,"style":289},[160],[80,4584,117],{"className":4585},[165,169],[4587,4588],"model-playground",{":initial-b":4286,":initial-w":4286,":x-train":3350,":y-train":3351,"dataLabel":4589,"error-label":4590,"prediction-label":4591,"success-message":4592,"x-label":3353,"y-label":3354},"Actual values","Total error","Your prediction","Nice, zero error. This line passes exactly through both points.",[4594,4595,4596,4603,4606,5094,5097,5361],"details",{},[4597,4598,4599,4602],"summary",{},[117,4600,4601],{},"Stuck? Here's the math"," (click to reveal)",[11,4604,4605],{},"Slope:",[11,4607,4608],{},[80,4609,4611,4712],{"className":4610},[83],[80,4612,4614],{"className":4613},[87],[89,4615,4616],{"xmlns":91},[93,4617,4618,4709],{},[96,4619,4620,4622,4624,4684,4686,4704,4706],{},[102,4621,109],{},[111,4623,130],{},[4625,4626,4627,4656],"mfrac",{},[96,4628,4629,4641,4644],{},[726,4630,4631,4633],{},[102,4632,683],{},[96,4634,4635,4637,4639],{},[111,4636,121],{"stretchy":120},[1321,4638,1583],{},[111,4640,127],{"stretchy":120},[111,4642,4643],{},"−",[726,4645,4646,4648],{},[102,4647,683],{},[96,4649,4650,4652,4654],{},[111,4651,121],{"stretchy":120},[1321,4653,2071],{},[111,4655,127],{"stretchy":120},[96,4657,4658,4670,4672],{},[726,4659,4660,4662],{},[102,4661,124],{},[96,4663,4664,4666,4668],{},[111,4665,121],{"stretchy":120},[1321,4667,1583],{},[111,4669,127],{"stretchy":120},[111,4671,4643],{},[726,4673,4674,4676],{},[102,4675,124],{},[96,4677,4678,4680,4682],{},[111,4679,121],{"stretchy":120},[1321,4681,2071],{},[111,4683,127],{"stretchy":120},[111,4685,130],{},[4625,4687,4688,4696],{},[96,4689,4690,4692,4694],{},[1321,4691,2086],{},[111,4693,4643],{},[1321,4695,1646],{},[96,4697,4698,4700,4702],{},[1321,4699,2083],{},[111,4701,4643],{},[1321,4703,2074],{},[111,4705,130],{},[1321,4707,4708],{},"200",[145,4710,4711],{"encoding":147},"w = \\frac{y^{(1)} - y^{(0)}}{x^{(1)} - x^{(0)}} = \\frac{500 - 300}{2.0 - 1.0} = 200",[80,4713,4715,4733,4985,5085],{"className":4714,"ariaHidden":113},[152],[80,4716,4718,4721,4724,4727,4730],{"className":4717},[156],[80,4719],{"className":4720,"style":334},[160],[80,4722,109],{"className":4723,"style":210},[165,169],[80,4725],{"className":4726,"style":247},[246],[80,4728,130],{"className":4729},[251],[80,4731],{"className":4732,"style":247},[246],[80,4734,4736,4740,4976,4979,4982],{"className":4735},[156],[80,4737],{"className":4738,"style":4739},[160],"height:1.5666em;vertical-align:-0.4438em;",[80,4741,4743,4747,4973],{"className":4742},[165],[80,4744],{"className":4745},[235,4746],"nulldelimiter",[80,4748,4750],{"className":4749},[4625],[80,4751,4753,4964],{"className":4752},[178,179],[80,4754,4756,4961],{"className":4755},[183],[80,4757,4760,4857,4868],{"className":4758,"style":4759},[187],"height:1.1228em;",[80,4761,4763,4767],{"style":4762},"top:-2.6146em;",[80,4764],{"className":4765,"style":4766},[195],"height:3em;",[80,4768,4770],{"className":4769},[200,201,202,203],[80,4771,4773,4816,4819],{"className":4772},[165,203],[80,4774,4776,4779],{"className":4775},[165,203],[80,4777,124],{"className":4778},[165,169,203],[80,4780,4782],{"className":4781},[174],[80,4783,4785],{"className":4784},[178],[80,4786,4788],{"className":4787},[183],[80,4789,4792],{"className":4790,"style":4791},[187],"height:0.822em;",[80,4793,4795,4799],{"style":4794},"top:-2.822em;margin-right:0.0714em;",[80,4796],{"className":4797,"style":4798},[195],"height:2.5357em;",[80,4800,4804],{"className":4801},[200,4802,4803,203],"reset-size3","size1",[80,4805,4807,4810,4813],{"className":4806},[165,203],[80,4808,121],{"className":4809},[235,203],[80,4811,1583],{"className":4812},[165,203],[80,4814,127],{"className":4815},[242,203],[80,4817,4643],{"className":4818},[279,203],[80,4820,4822,4825],{"className":4821},[165,203],[80,4823,124],{"className":4824},[165,169,203],[80,4826,4828],{"className":4827},[174],[80,4829,4831],{"className":4830},[178],[80,4832,4834],{"className":4833},[183],[80,4835,4837],{"className":4836,"style":4791},[187],[80,4838,4839,4842],{"style":4794},[80,4840],{"className":4841,"style":4798},[195],[80,4843,4845],{"className":4844},[200,4802,4803,203],[80,4846,4848,4851,4854],{"className":4847},[165,203],[80,4849,121],{"className":4850},[235,203],[80,4852,2071],{"className":4853},[165,203],[80,4855,127],{"className":4856},[242,203],[80,4858,4860,4863],{"style":4859},"top:-3.23em;",[80,4861],{"className":4862,"style":4766},[195],[80,4864],{"className":4865,"style":4867},[4866],"frac-line","border-bottom-width:0.04em;",[80,4869,4871,4874],{"style":4870},"top:-3.4461em;",[80,4872],{"className":4873,"style":4766},[195],[80,4875,4877],{"className":4876},[200,201,202,203],[80,4878,4880,4920,4923],{"className":4879},[165,203],[80,4881,4883,4886],{"className":4882},[165,203],[80,4884,683],{"className":4885,"style":834},[165,169,203],[80,4887,4889],{"className":4888},[174],[80,4890,4892],{"className":4891},[178],[80,4893,4895],{"className":4894},[183],[80,4896,4899],{"className":4897,"style":4898},[187],"height:0.9667em;",[80,4900,4902,4905],{"style":4901},"top:-2.9667em;margin-right:0.0714em;",[80,4903],{"className":4904,"style":4798},[195],[80,4906,4908],{"className":4907},[200,4802,4803,203],[80,4909,4911,4914,4917],{"className":4910},[165,203],[80,4912,121],{"className":4913},[235,203],[80,4915,1583],{"className":4916},[165,203],[80,4918,127],{"className":4919},[242,203],[80,4921,4643],{"className":4922},[279,203],[80,4924,4926,4929],{"className":4925},[165,203],[80,4927,683],{"className":4928,"style":834},[165,169,203],[80,4930,4932],{"className":4931},[174],[80,4933,4935],{"className":4934},[178],[80,4936,4938],{"className":4937},[183],[80,4939,4941],{"className":4940,"style":4898},[187],[80,4942,4943,4946],{"style":4901},[80,4944],{"className":4945,"style":4798},[195],[80,4947,4949],{"className":4948},[200,4802,4803,203],[80,4950,4952,4955,4958],{"className":4951},[165,203],[80,4953,121],{"className":4954},[235,203],[80,4956,2071],{"className":4957},[165,203],[80,4959,127],{"className":4960},[242,203],[80,4962,222],{"className":4963},[221],[80,4965,4967],{"className":4966},[183],[80,4968,4971],{"className":4969,"style":4970},[187],"height:0.4438em;",[80,4972],{},[80,4974],{"className":4975},[242,4746],[80,4977],{"className":4978,"style":247},[246],[80,4980,130],{"className":4981},[251],[80,4983],{"className":4984,"style":247},[246],[80,4986,4988,4992,5076,5079,5082],{"className":4987},[156],[80,4989],{"className":4990,"style":4991},[160],"height:1.2484em;vertical-align:-0.4033em;",[80,4993,4995,4998,5073],{"className":4994},[165],[80,4996],{"className":4997},[235,4746],[80,4999,5001],{"className":5000},[4625],[80,5002,5004,5064],{"className":5003},[178,179],[80,5005,5007,5061],{"className":5006},[183],[80,5008,5011,5032,5040],{"className":5009,"style":5010},[187],"height:0.8451em;",[80,5012,5014,5017],{"style":5013},"top:-2.655em;",[80,5015],{"className":5016,"style":4766},[195],[80,5018,5020],{"className":5019},[200,201,202,203],[80,5021,5023,5026,5029],{"className":5022},[165,203],[80,5024,2083],{"className":5025},[165,203],[80,5027,4643],{"className":5028},[279,203],[80,5030,2074],{"className":5031},[165,203],[80,5033,5034,5037],{"style":4859},[80,5035],{"className":5036,"style":4766},[195],[80,5038],{"className":5039,"style":4867},[4866],[80,5041,5043,5046],{"style":5042},"top:-3.394em;",[80,5044],{"className":5045,"style":4766},[195],[80,5047,5049],{"className":5048},[200,201,202,203],[80,5050,5052,5055,5058],{"className":5051},[165,203],[80,5053,2086],{"className":5054},[165,203],[80,5056,4643],{"className":5057},[279,203],[80,5059,1646],{"className":5060},[165,203],[80,5062,222],{"className":5063},[221],[80,5065,5067],{"className":5066},[183],[80,5068,5071],{"className":5069,"style":5070},[187],"height:0.4033em;",[80,5072],{},[80,5074],{"className":5075},[242,4746],[80,5077],{"className":5078,"style":247},[246],[80,5080,130],{"className":5081},[251],[80,5083],{"className":5084,"style":247},[246],[80,5086,5088,5091],{"className":5087},[156],[80,5089],{"className":5090,"style":1614},[160],[80,5092,4708],{"className":5093},[165],[11,5095,5096],{},"Intercept, using the first point:",[11,5098,5099],{},[80,5100,5102,5164],{"className":5101},[83],[80,5103,5105],{"className":5104},[87],[89,5106,5107],{"xmlns":91},[93,5108,5109,5161],{},[96,5110,5111,5113,5115,5127,5129,5131,5133,5145,5147,5149,5151,5153,5155,5157,5159],{},[102,5112,117],{},[111,5114,130],{},[726,5116,5117,5119],{},[102,5118,683],{},[96,5120,5121,5123,5125],{},[111,5122,121],{"stretchy":120},[1321,5124,2071],{},[111,5126,127],{"stretchy":120},[111,5128,4643],{},[102,5130,109],{},[134,5132,136],{},[726,5134,5135,5137],{},[102,5136,124],{},[96,5138,5139,5141,5143],{},[111,5140,121],{"stretchy":120},[1321,5142,2071],{},[111,5144,127],{"stretchy":120},[111,5146,130],{},[1321,5148,1646],{},[111,5150,4643],{},[1321,5152,4708],{},[111,5154,2834],{},[1321,5156,2074],{},[111,5158,130],{},[1321,5160,4286],{},[145,5162,5163],{"encoding":147},"b = y^{(0)} - w\\,x^{(0)} = 300 - 200 \\times 1.0 = 100",[80,5165,5167,5185,5238,5297,5316,5334,5352],{"className":5166,"ariaHidden":113},[152],[80,5168,5170,5173,5176,5179,5182],{"className":5169},[156],[80,5171],{"className":5172,"style":289},[160],[80,5174,117],{"className":5175},[165,169],[80,5177],{"className":5178,"style":247},[246],[80,5180,130],{"className":5181},[251],[80,5183],{"className":5184,"style":247},[246],[80,5186,5188,5191,5229,5232,5235],{"className":5187},[156],[80,5189],{"className":5190,"style":827},[160],[80,5192,5194,5197],{"className":5193},[165],[80,5195,683],{"className":5196,"style":834},[165,169],[80,5198,5200],{"className":5199},[174],[80,5201,5203],{"className":5202},[178],[80,5204,5206],{"className":5205},[183],[80,5207,5209],{"className":5208,"style":751},[187],[80,5210,5211,5214],{"style":772},[80,5212],{"className":5213,"style":196},[195],[80,5215,5217],{"className":5216},[200,201,202,203],[80,5218,5220,5223,5226],{"className":5219},[165,203],[80,5221,121],{"className":5222},[235,203],[80,5224,2071],{"className":5225},[165,203],[80,5227,127],{"className":5228},[242,203],[80,5230],{"className":5231,"style":275},[246],[80,5233,4643],{"className":5234},[279],[80,5236],{"className":5237,"style":275},[246],[80,5239,5241,5244,5247,5250,5288,5291,5294],{"className":5240},[156],[80,5242],{"className":5243,"style":751},[160],[80,5245,109],{"className":5246,"style":210},[165,169],[80,5248],{"className":5249,"style":268},[246],[80,5251,5253,5256],{"className":5252},[165],[80,5254,124],{"className":5255},[165,169],[80,5257,5259],{"className":5258},[174],[80,5260,5262],{"className":5261},[178],[80,5263,5265],{"className":5264},[183],[80,5266,5268],{"className":5267,"style":751},[187],[80,5269,5270,5273],{"style":772},[80,5271],{"className":5272,"style":196},[195],[80,5274,5276],{"className":5275},[200,201,202,203],[80,5277,5279,5282,5285],{"className":5278},[165,203],[80,5280,121],{"className":5281},[235,203],[80,5283,2071],{"className":5284},[165,203],[80,5286,127],{"className":5287},[242,203],[80,5289],{"className":5290,"style":247},[246],[80,5292,130],{"className":5293},[251],[80,5295],{"className":5296,"style":247},[246],[80,5298,5300,5304,5307,5310,5313],{"className":5299},[156],[80,5301],{"className":5302,"style":5303},[160],"height:0.7278em;vertical-align:-0.0833em;",[80,5305,1646],{"className":5306},[165],[80,5308],{"className":5309,"style":275},[246],[80,5311,4643],{"className":5312},[279],[80,5314],{"className":5315,"style":275},[246],[80,5317,5319,5322,5325,5328,5331],{"className":5318},[156],[80,5320],{"className":5321,"style":5303},[160],[80,5323,4708],{"className":5324},[165],[80,5326],{"className":5327,"style":275},[246],[80,5329,2834],{"className":5330},[279],[80,5332],{"className":5333,"style":275},[246],[80,5335,5337,5340,5343,5346,5349],{"className":5336},[156],[80,5338],{"className":5339,"style":1614},[160],[80,5341,2074],{"className":5342},[165],[80,5344],{"className":5345,"style":247},[246],[80,5347,130],{"className":5348},[251],[80,5350],{"className":5351,"style":247},[246],[80,5353,5355,5358],{"className":5354},[156],[80,5356],{"className":5357,"style":1614},[160],[80,5359,4286],{"className":5360},[165],[11,5362,2089,5363,504,5414,5465],{},[80,5364,5366,5384],{"className":5365},[83],[80,5367,5369],{"className":5368},[87],[89,5370,5371],{"xmlns":91},[93,5372,5373,5381],{},[96,5374,5375,5377,5379],{},[102,5376,109],{},[111,5378,130],{},[1321,5380,4708],{},[145,5382,5383],{"encoding":147},"w = 200",[80,5385,5387,5405],{"className":5386,"ariaHidden":113},[152],[80,5388,5390,5393,5396,5399,5402],{"className":5389},[156],[80,5391],{"className":5392,"style":334},[160],[80,5394,109],{"className":5395,"style":210},[165,169],[80,5397],{"className":5398,"style":247},[246],[80,5400,130],{"className":5401},[251],[80,5403],{"className":5404,"style":247},[246],[80,5406,5408,5411],{"className":5407},[156],[80,5409],{"className":5410,"style":1614},[160],[80,5412,4708],{"className":5413},[165],[80,5415,5417,5435],{"className":5416},[83],[80,5418,5420],{"className":5419},[87],[89,5421,5422],{"xmlns":91},[93,5423,5424,5432],{},[96,5425,5426,5428,5430],{},[102,5427,117],{},[111,5429,130],{},[1321,5431,4286],{},[145,5433,5434],{"encoding":147},"b = 100",[80,5436,5438,5456],{"className":5437,"ariaHidden":113},[152],[80,5439,5441,5444,5447,5450,5453],{"className":5440},[156],[80,5442],{"className":5443,"style":289},[160],[80,5445,117],{"className":5446},[165,169],[80,5448],{"className":5449,"style":247},[246],[80,5451,130],{"className":5452},[251],[80,5454],{"className":5455,"style":247},[246],[80,5457,5459,5462],{"className":5458},[156],[80,5460],{"className":5461,"style":1614},[160],[80,5463,4286],{"className":5464},[165],". Set the sliders above and watch the error hit zero.",[11,5467,5468,5469,2338,5497,5525,5526,5529],{},"Nice, you found ",[80,5470,5472,5485],{"className":5471},[83],[80,5473,5475],{"className":5474},[87],[89,5476,5477],{"xmlns":91},[93,5478,5479,5483],{},[96,5480,5481],{},[102,5482,109],{},[145,5484,109],{"encoding":147},[80,5486,5488],{"className":5487,"ariaHidden":113},[152],[80,5489,5491,5494],{"className":5490},[156],[80,5492],{"className":5493,"style":334},[160],[80,5495,109],{"className":5496,"style":210},[165,169],[80,5498,5500,5513],{"className":5499},[83],[80,5501,5503],{"className":5502},[87],[89,5504,5505],{"xmlns":91},[93,5506,5507,5511],{},[96,5508,5509],{},[102,5510,117],{},[145,5512,117],{"encoding":147},[80,5514,5516],{"className":5515,"ariaHidden":113},[152],[80,5517,5519,5522],{"className":5518},[156],[80,5520],{"className":5521,"style":289},[160],[80,5523,117],{"className":5524},[165,169]," by hand, but notice: you could only do this because you had ",[15,5527,5528],{},"exactly 2 points and 2 parameters"," to fit, so plain algebra solved it. Throw in more real-world data (with noise, similar houses selling for slightly different prices) and no single line passes through everyone. Zero error stops being achievable at all.",[11,5531,5532,5533,5536,5537,5589,5590,504,5592,5651,5652,5655,5656,5707,5708,5710,5711,5714],{},"That raises a new question: ",[15,5534,5535],{},"which line is the \"least wrong\"?"," I needed a way to measure \"how wrong\" a line is, numerically, in a way you can compare across different ",[80,5538,5540,5562],{"className":5539},[83],[80,5541,5543],{"className":5542},[87],[89,5544,5545],{"xmlns":91},[93,5546,5547,5559],{},[96,5548,5549,5551,5553,5555,5557],{},[111,5550,121],{"stretchy":120},[102,5552,109],{},[111,5554,114],{"separator":113},[102,5556,117],{},[111,5558,127],{"stretchy":120},[145,5560,5561],{"encoding":147},"(w,b)",[80,5563,5565],{"className":5564,"ariaHidden":113},[152],[80,5566,5568,5571,5574,5577,5580,5583,5586],{"className":5567},[156],[80,5569],{"className":5570,"style":2316},[160],[80,5572,121],{"className":5573},[235],[80,5575,109],{"className":5576,"style":210},[165,169],[80,5578,114],{"className":5579},[214],[80,5581],{"className":5582,"style":268},[246],[80,5584,117],{"className":5585},[165,169],[80,5587,127],{"className":5588},[242]," pairs. That's the ",[15,5591,4067],{},[80,5593,5595,5620],{"className":5594},[83],[80,5596,5598],{"className":5597},[87],[89,5599,5600],{"xmlns":91},[93,5601,5602,5617],{},[96,5603,5604,5607,5609,5611,5613,5615],{},[102,5605,5606],{},"J",[111,5608,121],{"stretchy":120},[102,5610,109],{},[111,5612,114],{"separator":113},[102,5614,117],{},[111,5616,127],{"stretchy":120},[145,5618,5619],{"encoding":147},"J(w,b)",[80,5621,5623],{"className":5622,"ariaHidden":113},[152],[80,5624,5626,5629,5633,5636,5639,5642,5645,5648],{"className":5625},[156],[80,5627],{"className":5628,"style":2316},[160],[80,5630,5606],{"className":5631,"style":5632},[165,169],"margin-right:0.0962em;",[80,5634,121],{"className":5635},[235],[80,5637,109],{"className":5638,"style":210},[165,169],[80,5640,114],{"className":5641},[214],[80,5643],{"className":5644,"style":268},[246],[80,5646,117],{"className":5647},[165,169],[80,5649,127],{"className":5650},[242]," (the \"Total error\" you just saw in the playground is a simplified stand-in for it). And I needed a way to automatically ",[73,5653,5654],{},"search"," for the ",[80,5657,5659,5680],{"className":5658},[83],[80,5660,5662],{"className":5661},[87],[89,5663,5664],{"xmlns":91},[93,5665,5666,5678],{},[96,5667,5668,5670,5672,5674,5676],{},[111,5669,121],{"stretchy":120},[102,5671,109],{},[111,5673,114],{"separator":113},[102,5675,117],{},[111,5677,127],{"stretchy":120},[145,5679,5561],{"encoding":147},[80,5681,5683],{"className":5682,"ariaHidden":113},[152],[80,5684,5686,5689,5692,5695,5698,5701,5704],{"className":5685},[156],[80,5687],{"className":5688,"style":2316},[160],[80,5690,121],{"className":5691},[235],[80,5693,109],{"className":5694,"style":210},[165,169],[80,5696,114],{"className":5697},[214],[80,5699],{"className":5700,"style":268},[246],[80,5702,117],{"className":5703},[165,169],[80,5705,127],{"className":5706},[242]," that minimizes that error, instead of dragging sliders for the rest of your life. That's ",[15,5709,4071],{},": imagine you're blindfolded on top of a hill, and the only way down is to feel with your foot which direction slopes downward the fastest and take a step. Repeat that enough times and, if everything goes right, you end up at the bottom, at the point of lowest error. That's basically what the ",[562,5712,5713],{"href":4070},"next post"," breaks down.",[294,5716,5718],{"id":5717},"making-the-prediction-we-wanted-all-along","Making the prediction we wanted all along",[11,5720,5721,504,5771,5821,5822],{},[80,5722,5724,5741],{"className":5723},[83],[80,5725,5727],{"className":5726},[87],[89,5728,5729],{"xmlns":91},[93,5730,5731,5739],{},[96,5732,5733,5735,5737],{},[102,5734,109],{},[111,5736,130],{},[1321,5738,4708],{},[145,5740,5383],{"encoding":147},[80,5742,5744,5762],{"className":5743,"ariaHidden":113},[152],[80,5745,5747,5750,5753,5756,5759],{"className":5746},[156],[80,5748],{"className":5749,"style":334},[160],[80,5751,109],{"className":5752,"style":210},[165,169],[80,5754],{"className":5755,"style":247},[246],[80,5757,130],{"className":5758},[251],[80,5760],{"className":5761,"style":247},[246],[80,5763,5765,5768],{"className":5764},[156],[80,5766],{"className":5767,"style":1614},[160],[80,5769,4708],{"className":5770},[165],[80,5772,5774,5791],{"className":5773},[83],[80,5775,5777],{"className":5776},[87],[89,5778,5779],{"xmlns":91},[93,5780,5781,5789],{},[96,5782,5783,5785,5787],{},[102,5784,117],{},[111,5786,130],{},[1321,5788,4286],{},[145,5790,5434],{"encoding":147},[80,5792,5794,5812],{"className":5793,"ariaHidden":113},[152],[80,5795,5797,5800,5803,5806,5809],{"className":5796},[156],[80,5798],{"className":5799,"style":289},[160],[80,5801,117],{"className":5802},[165,169],[80,5804],{"className":5805,"style":247},[246],[80,5807,130],{"className":5808},[251],[80,5810],{"className":5811,"style":247},[246],[80,5813,5815,5818],{"className":5814},[156],[80,5816],{"className":5817,"style":1614},[160],[80,5819,4286],{"className":5820},[165],", fitted. Now for real: ",[15,5823,5824],{},"what's the 1200 sqft house worth?",[11,5826,5827,5828,1618],{},"Remember the scale: 1200 sqft is ",[80,5829,5831,5850],{"className":5830},[83],[80,5832,5834],{"className":5833},[87],[89,5835,5836],{"xmlns":91},[93,5837,5838,5847],{},[96,5839,5840,5842,5844],{},[102,5841,124],{},[111,5843,130],{},[1321,5845,5846],{},"1.2",[145,5848,5849],{"encoding":147},"x = 1.2",[80,5851,5853,5871],{"className":5852,"ariaHidden":113},[152],[80,5854,5856,5859,5862,5865,5868],{"className":5855},[156],[80,5857],{"className":5858,"style":334},[160],[80,5860,124],{"className":5861},[165,169],[80,5863],{"className":5864,"style":247},[246],[80,5866,130],{"className":5867},[251],[80,5869],{"className":5870,"style":247},[246],[80,5872,5874,5877],{"className":5873},[156],[80,5875],{"className":5876,"style":1614},[160],[80,5878,5846],{"className":5879},[165],[11,5881,5882],{},[80,5883,5885,5932],{"className":5884},[83],[80,5886,5888],{"className":5887},[87],[89,5889,5890],{"xmlns":91},[93,5891,5892,5929],{},[96,5893,5894,5906,5908,5910,5912,5914,5916,5918,5920,5922,5924,5926],{},[99,5895,5896,5898],{},[102,5897,104],{},[96,5899,5900,5902,5904],{},[102,5901,109],{},[111,5903,114],{"separator":113},[102,5905,117],{},[111,5907,121],{"stretchy":120},[1321,5909,5846],{},[111,5911,127],{"stretchy":120},[111,5913,130],{},[1321,5915,4708],{},[111,5917,2834],{},[1321,5919,5846],{},[111,5921,141],{},[1321,5923,4286],{},[111,5925,130],{},[1321,5927,5928],{},"340",[145,5930,5931],{"encoding":147},"f_{w,b}(1.2) = 200 \\times 1.2 + 100 = 340",[80,5933,5935,6008,6026,6044,6062],{"className":5934,"ariaHidden":113},[152],[80,5936,5938,5941,5990,5993,5996,5999,6002,6005],{"className":5937},[156],[80,5939],{"className":5940,"style":161},[160],[80,5942,5944,5947],{"className":5943},[165],[80,5945,104],{"className":5946,"style":170},[165,169],[80,5948,5950],{"className":5949},[174],[80,5951,5953,5982],{"className":5952},[178,179],[80,5954,5956,5979],{"className":5955},[183],[80,5957,5959],{"className":5958,"style":188},[187],[80,5960,5961,5964],{"style":191},[80,5962],{"className":5963,"style":196},[195],[80,5965,5967],{"className":5966},[200,201,202,203],[80,5968,5970,5973,5976],{"className":5969},[165,203],[80,5971,109],{"className":5972,"style":210},[165,169,203],[80,5974,114],{"className":5975},[214,203],[80,5977,117],{"className":5978},[165,169,203],[80,5980,222],{"className":5981},[221],[80,5983,5985],{"className":5984},[183],[80,5986,5988],{"className":5987,"style":229},[187],[80,5989],{},[80,5991,121],{"className":5992},[235],[80,5994,5846],{"className":5995},[165],[80,5997,127],{"className":5998},[242],[80,6000],{"className":6001,"style":247},[246],[80,6003,130],{"className":6004},[251],[80,6006],{"className":6007,"style":247},[246],[80,6009,6011,6014,6017,6020,6023],{"className":6010},[156],[80,6012],{"className":6013,"style":5303},[160],[80,6015,4708],{"className":6016},[165],[80,6018],{"className":6019,"style":275},[246],[80,6021,2834],{"className":6022},[279],[80,6024],{"className":6025,"style":275},[246],[80,6027,6029,6032,6035,6038,6041],{"className":6028},[156],[80,6030],{"className":6031,"style":5303},[160],[80,6033,5846],{"className":6034},[165],[80,6036],{"className":6037,"style":275},[246],[80,6039,141],{"className":6040},[279],[80,6042],{"className":6043,"style":275},[246],[80,6045,6047,6050,6053,6056,6059],{"className":6046},[156],[80,6048],{"className":6049,"style":1614},[160],[80,6051,4286],{"className":6052},[165],[80,6054],{"className":6055,"style":247},[246],[80,6057,130],{"className":6058},[251],[80,6060],{"className":6061,"style":247},[246],[80,6063,6065,6068],{"className":6064},[156],[80,6066],{"className":6067,"style":1614},[160],[80,6069,5928],{"className":6070},[165],[11,6072,6073,6074,1618],{},"340, in the problem's scale, is ",[15,6075,6076],{},"$340,000",[2611,6078,6080],{"className":2613,"code":6079,"language":2615,"meta":26,"style":26},"w = 200      # Fitted weight (found in the challenge).\nb = 100      # Fitted bias (found in the challenge).\n\nx_i = 1.2    # New input: 1200 sqft = 1.2, since x is in thousands of sqft.\n\ncost_1200sqft = w * x_i + b     # Applies the model. Don't confuse with \"cost function\", similar name, different thing.\n\nprint(f\"${cost_1200sqft:.0f} thousand dollars\")\n",[65,6081,6082,6087,6092,6096,6101,6105,6110,6114],{"__ignoreMap":26},[80,6083,6084],{"class":2620,"line":33},[80,6085,6086],{},"w = 200      # Fitted weight (found in the challenge).\n",[80,6088,6089],{"class":2620,"line":27},[80,6090,6091],{},"b = 100      # Fitted bias (found in the challenge).\n",[80,6093,6094],{"class":2620,"line":2631},[80,6095,2657],{"emptyLinePlaceholder":32},[80,6097,6098],{"class":2620,"line":2636},[80,6099,6100],{},"x_i = 1.2    # New input: 1200 sqft = 1.2, since x is in thousands of sqft.\n",[80,6102,6103],{"class":2620,"line":2642},[80,6104,2657],{"emptyLinePlaceholder":32},[80,6106,6107],{"class":2620,"line":2648},[80,6108,6109],{},"cost_1200sqft = w * x_i + b     # Applies the model. Don't confuse with \"cost function\", similar name, different thing.\n",[80,6111,6112],{"class":2620,"line":2654},[80,6113,2657],{"emptyLinePlaceholder":32},[80,6115,6116],{"class":2620,"line":2660},[80,6117,6118],{},"print(f\"${cost_1200sqft:.0f} thousand dollars\")\n",[46,6120,6121],{},[11,6122,6123,3255,6125],{},[15,6124,2693],{},[65,6126,6127],{},"$340 thousand dollars",[11,6129,6130,6131,3255,6181,6184,6185,6188],{},"Pay attention to what just happened: ",[80,6132,6134,6151],{"className":6133},[83],[80,6135,6137],{"className":6136},[87],[89,6138,6139],{"xmlns":91},[93,6140,6141,6149],{},[96,6142,6143,6145,6147],{},[102,6144,124],{},[111,6146,130],{},[1321,6148,5846],{},[145,6150,5849],{"encoding":147},[80,6152,6154,6172],{"className":6153,"ariaHidden":113},[152],[80,6155,6157,6160,6163,6166,6169],{"className":6156},[156],[80,6158],{"className":6159,"style":334},[160],[80,6161,124],{"className":6162},[165,169],[80,6164],{"className":6165,"style":247},[246],[80,6167,130],{"className":6168},[251],[80,6170],{"className":6171,"style":247},[246],[80,6173,6175,6178],{"className":6174},[156],[80,6176],{"className":6177,"style":1614},[160],[80,6179,5846],{"className":6180},[165],[15,6182,6183],{},"never showed up"," in the training data. We only had houses at 1.0 and 2.0. The model generalized to a case it had never seen, and that's exactly why we train models in the first place, not to memorize the 2 points we already knew (any table printout does that). It's to predict what we ",[15,6186,6187],{},"don't"," know.",[294,6190,6192],{"id":6191},"wrapping-up","Wrapping up",[521,6194,6195,6208],{},[524,6196,6197],{},[527,6198,6199,6202,6205],{},[530,6200,6201],{"align":532},"Step",[530,6203,6204],{"align":532},"What happened",[530,6206,6207],{"align":532},"Key code",[541,6209,6210,6225,6243,6256,6420,6490],{},[527,6211,6212,6217,6220],{},[546,6213,6214],{"align":532},[15,6215,6216],{},"1. Data",[546,6218,6219],{"align":532},"represent examples as NumPy arrays",[546,6221,6222],{"align":532},[65,6223,6224],{},"np.array([...])",[527,6226,6227,6232,6235],{},[546,6228,6229],{"align":532},[15,6230,6231],{},"2. Inspection",[546,6233,6234],{"align":532},"count examples and grab a specific one",[546,6236,6237,504,6240],{"align":532},[65,6238,6239],{},".shape[0]",[65,6241,6242],{},"x_train[i]",[527,6244,6245,6250,6253],{},[546,6246,6247],{"align":532},[15,6248,6249],{},"3. Visualization",[546,6251,6252],{"align":532},"look at the data before modeling",[546,6254,6255],{"align":532},"interactive chart",[527,6257,6258,6263,6415],{},[546,6259,6260],{"align":532},[15,6261,6262],{},"4. Model",[546,6264,6265,6266,6414],{"align":532},"define ",[80,6267,6269,6308],{"className":6268},[83],[80,6270,6272],{"className":6271},[87],[89,6273,6274],{"xmlns":91},[93,6275,6276,6306],{},[96,6277,6278,6290,6292,6294,6296,6298,6300,6302,6304],{},[99,6279,6280,6282],{},[102,6281,104],{},[96,6283,6284,6286,6288],{},[102,6285,109],{},[111,6287,114],{"separator":113},[102,6289,117],{},[111,6291,121],{"stretchy":120},[102,6293,124],{},[111,6295,127],{"stretchy":120},[111,6297,130],{},[102,6299,109],{},[102,6301,124],{},[111,6303,141],{},[102,6305,117],{},[145,6307,387],{"encoding":147},[80,6309,6311,6384,6405],{"className":6310,"ariaHidden":113},[152],[80,6312,6314,6317,6366,6369,6372,6375,6378,6381],{"className":6313},[156],[80,6315],{"className":6316,"style":161},[160],[80,6318,6320,6323],{"className":6319},[165],[80,6321,104],{"className":6322,"style":170},[165,169],[80,6324,6326],{"className":6325},[174],[80,6327,6329,6358],{"className":6328},[178,179],[80,6330,6332,6355],{"className":6331},[183],[80,6333,6335],{"className":6334,"style":188},[187],[80,6336,6337,6340],{"style":191},[80,6338],{"className":6339,"style":196},[195],[80,6341,6343],{"className":6342},[200,201,202,203],[80,6344,6346,6349,6352],{"className":6345},[165,203],[80,6347,109],{"className":6348,"style":210},[165,169,203],[80,6350,114],{"className":6351},[214,203],[80,6353,117],{"className":6354},[165,169,203],[80,6356,222],{"className":6357},[221],[80,6359,6361],{"className":6360},[183],[80,6362,6364],{"className":6363,"style":229},[187],[80,6365],{},[80,6367,121],{"className":6368},[235],[80,6370,124],{"className":6371},[165,169],[80,6373,127],{"className":6374},[242],[80,6376],{"className":6377,"style":247},[246],[80,6379,130],{"className":6380},[251],[80,6382],{"className":6383,"style":247},[246],[80,6385,6387,6390,6393,6396,6399,6402],{"className":6386},[156],[80,6388],{"className":6389,"style":261},[160],[80,6391,109],{"className":6392,"style":210},[165,169],[80,6394,124],{"className":6395},[165,169],[80,6397],{"className":6398,"style":275},[246],[80,6400,141],{"className":6401},[279],[80,6403],{"className":6404,"style":275},[246],[80,6406,6408,6411],{"className":6407},[156],[80,6409],{"className":6410,"style":289},[160],[80,6412,117],{"className":6413},[165,169]," and implement it",[546,6416,6417],{"align":532},[65,6418,6419],{},"compute_model_output()",[527,6421,6422,6427,6487],{},[546,6423,6424],{"align":532},[15,6425,6426],{},"5. Parameters",[546,6428,6429,6430,2338,6458,6486],{"align":532},"find ",[80,6431,6433,6446],{"className":6432},[83],[80,6434,6436],{"className":6435},[87],[89,6437,6438],{"xmlns":91},[93,6439,6440,6444],{},[96,6441,6442],{},[102,6443,109],{},[145,6445,109],{"encoding":147},[80,6447,6449],{"className":6448,"ariaHidden":113},[152],[80,6450,6452,6455],{"className":6451},[156],[80,6453],{"className":6454,"style":334},[160],[80,6456,109],{"className":6457,"style":210},[165,169],[80,6459,6461,6474],{"className":6460},[83],[80,6462,6464],{"className":6463},[87],[89,6465,6466],{"xmlns":91},[93,6467,6468,6472],{},[96,6469,6470],{},[102,6471,117],{},[145,6473,117],{"encoding":147},[80,6475,6477],{"className":6476,"ariaHidden":113},[152],[80,6478,6480,6483],{"className":6479},[156],[80,6481],{"className":6482,"style":289},[160],[80,6484,117],{"className":6485},[165,169]," by hand, tuning until error hits zero",[546,6488,6489],{"align":532},"interactive playground",[527,6491,6492,6497,6500],{},[546,6493,6494],{"align":532},[15,6495,6496],{},"6. Prediction",[546,6498,6499],{"align":532},"apply the model to a new input",[546,6501,6502],{"align":532},[65,6503,6504],{},"w * x_i + b",[11,6506,6507],{},"Three takeaways:",[299,6509,6510,6524,6587],{},[302,6511,6512,6515,6516,6519,6520,6523],{},[15,6513,6514],{},"Linear regression models a relationship"," between an input (",[73,6517,6518],{},"feature",") and an output (",[73,6521,6522],{},"target","). Here: size → price. Tomorrow it could be anything else: distance driven → ride price, file size → processing time, whatever.",[302,6525,6526,6529,6530,2338,6558,6586],{},[15,6527,6528],{},"The model only has two knobs",": ",[80,6531,6533,6546],{"className":6532},[83],[80,6534,6536],{"className":6535},[87],[89,6537,6538],{"xmlns":91},[93,6539,6540,6544],{},[96,6541,6542],{},[102,6543,109],{},[145,6545,109],{"encoding":147},[80,6547,6549],{"className":6548,"ariaHidden":113},[152],[80,6550,6552,6555],{"className":6551},[156],[80,6553],{"className":6554,"style":334},[160],[80,6556,109],{"className":6557,"style":210},[165,169],[80,6559,6561,6574],{"className":6560},[83],[80,6562,6564],{"className":6563},[87],[89,6565,6566],{"xmlns":91},[93,6567,6568,6572],{},[96,6569,6570],{},[102,6571,117],{},[145,6573,117],{"encoding":147},[80,6575,6577],{"className":6576,"ariaHidden":113},[152],[80,6578,6580,6583],{"className":6579},[156],[80,6581],{"className":6582,"style":289},[160],[80,6584,117],{"className":6585},[165,169],". That's all that changes when the model \"learns.\" No hidden magic.",[302,6588,6589,6592],{},[15,6590,6591],{},"The entire point is generalization",": predicting something the model has never seen. If it only reproduced the training data, it'd be a disguised lookup table, not a model.",[11,6594,6595,6600,6601,6652,6653,6709,6710,6712],{},[15,6596,6597,3774],{},[562,6598,6599],{"href":4066},"Coming up next"," you found ",[80,6602,6604,6625],{"className":6603},[83],[80,6605,6607],{"className":6606},[87],[89,6608,6609],{"xmlns":91},[93,6610,6611,6623],{},[96,6612,6613,6615,6617,6619,6621],{},[111,6614,121],{"stretchy":120},[102,6616,109],{},[111,6618,114],{"separator":113},[102,6620,117],{},[111,6622,127],{"stretchy":120},[145,6624,5561],{"encoding":147},[80,6626,6628],{"className":6627,"ariaHidden":113},[152],[80,6629,6631,6634,6637,6640,6643,6646,6649],{"className":6630},[156],[80,6632],{"className":6633,"style":2316},[160],[80,6635,121],{"className":6636},[235],[80,6638,109],{"className":6639,"style":210},[165,169],[80,6641,114],{"className":6642},[214],[80,6644],{"className":6645,"style":268},[246],[80,6647,117],{"className":6648},[165,169],[80,6650,127],{"className":6651},[242]," by dragging sliders because you only had 2 points. That doesn't scale to real data. We formalize the cost function ",[80,6654,6656,6679],{"className":6655},[83],[80,6657,6659],{"className":6658},[87],[89,6660,6661],{"xmlns":91},[93,6662,6663,6677],{},[96,6664,6665,6667,6669,6671,6673,6675],{},[102,6666,5606],{},[111,6668,121],{"stretchy":120},[102,6670,109],{},[111,6672,114],{"separator":113},[102,6674,117],{},[111,6676,127],{"stretchy":120},[145,6678,5619],{"encoding":147},[80,6680,6682],{"className":6681,"ariaHidden":113},[152],[80,6683,6685,6688,6691,6694,6697,6700,6703,6706],{"className":6684},[156],[80,6686],{"className":6687,"style":2316},[160],[80,6689,5606],{"className":6690,"style":5632},[165,169],[80,6692,121],{"className":6693},[235],[80,6695,109],{"className":6696,"style":210},[165,169],[80,6698,114],{"className":6699},[214],[80,6701],{"className":6702,"style":268},[246],[80,6704,117],{"className":6705},[165,169],[80,6707,127],{"className":6708},[242],", the \"grade\" that measures how bad a line is, and start laying the groundwork for ",[562,6711,4071],{"href":4070},", which will do that search for you, automatically, on data that no longer fits inside a mental math problem.",[294,6714,6716],{"id":6715},"practical-application","Practical application",[11,6718,6719,6720,6726],{},"Enough toy data. Let's apply exactly what you learned here to a real housing dataset: 500 houses, with real size, bedroom count, distance to downtown, and price (",[562,6721,6725],{"href":6722,"rel":6723},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fdenkuznetz\u002Fhousing-prices-regression",[6724],"nofollow","Housing Prices Regression, Kaggle","). No new concept, just what this post already taught, now on top of data with real noise.",[2606,6728,6730],{"id":6729},"exploring-the-features","Exploring the features",[2611,6732,6734],{"className":2613,"code":6733,"language":2615,"meta":26,"style":26},"import pandas as pd\n\ndf = pd.read_csv(\"real_estate_dataset.csv\")\n\nprint(df[[\"Square_Feet\", \"Num_Bedrooms\", \"Location_Score\", \"Distance_to_Center\", \"Price\"]].describe())\n\nprint(df.corr(numeric_only=True)[\"Price\"].sort_values(ascending=False))\n",[65,6735,6736,6741,6745,6750,6754,6759,6763],{"__ignoreMap":26},[80,6737,6738],{"class":2620,"line":33},[80,6739,6740],{},"import pandas as pd\n",[80,6742,6743],{"class":2620,"line":27},[80,6744,2657],{"emptyLinePlaceholder":32},[80,6746,6747],{"class":2620,"line":2631},[80,6748,6749],{},"df = pd.read_csv(\"real_estate_dataset.csv\")\n",[80,6751,6752],{"class":2620,"line":2636},[80,6753,2657],{"emptyLinePlaceholder":32},[80,6755,6756],{"class":2620,"line":2642},[80,6757,6758],{},"print(df[[\"Square_Feet\", \"Num_Bedrooms\", \"Location_Score\", \"Distance_to_Center\", \"Price\"]].describe())\n",[80,6760,6761],{"class":2620,"line":2648},[80,6762,2657],{"emptyLinePlaceholder":32},[80,6764,6765],{"class":2620,"line":2654},[80,6766,6767],{},"print(df.corr(numeric_only=True)[\"Price\"].sort_values(ascending=False))\n",[46,6769,6770],{},[11,6771,6772,3255,6775,504,6778,504,6781,504,6784,1618],{},[15,6773,6774],{},"Output (correlation with price):",[65,6776,6777],{},"Square_Feet 0.65",[65,6779,6780],{},"Num_Bedrooms 0.57",[65,6782,6783],{},"Distance_to_Center 0.26",[65,6785,6786],{},"Location_Score 0.05",[11,6788,6789,6792],{},[65,6790,6791],{},"Square_Feet"," is by far the feature that moves most closely with price here. Matches the story the whole post just told: house size is the right place to start.",[2606,6794,6796],{"id":6795},"the-feature-matrix-in-3d","The \"feature matrix\" in 3D",[11,6798,6799],{},"Before reducing to a single variable, I rotated the three dimensions that matter most, all at once: size, bedrooms, and price.",[6801,6802],"housing-scatter3d",{"x-label":6803,"y-label":6804,"z-label":6805},"size (sqft)","price (dollars)","# of bedrooms",[11,6807,6808],{},"Notice how the height (price) climbs visibly alongside size, more than alongside bedroom count. Same correlation as above, except now you're seeing it with your own eyes instead of reading a number.",[2606,6810,6812,6813,2338,6841,6869],{"id":6811},"finding-www-and-bbb-by-hand-with-real-data","Finding ",[80,6814,6816,6829],{"className":6815},[83],[80,6817,6819],{"className":6818},[87],[89,6820,6821],{"xmlns":91},[93,6822,6823,6827],{},[96,6824,6825],{},[102,6826,109],{},[145,6828,109],{"encoding":147},[80,6830,6832],{"className":6831,"ariaHidden":113},[152],[80,6833,6835,6838],{"className":6834},[156],[80,6836],{"className":6837,"style":334},[160],[80,6839,109],{"className":6840,"style":210},[165,169],[80,6842,6844,6857],{"className":6843},[83],[80,6845,6847],{"className":6846},[87],[89,6848,6849],{"xmlns":91},[93,6850,6851,6855],{},[96,6852,6853],{},[102,6854,117],{},[145,6856,117],{"encoding":147},[80,6858,6860],{"className":6859,"ariaHidden":113},[152],[80,6861,6863,6866],{"className":6862},[156],[80,6864],{"className":6865,"style":289},[160],[80,6867,117],{"className":6868},[165,169]," by hand, with real data",[11,6871,6872],{},"Same playground from the post, same mechanics, just with the 50 real houses in place of the 2 perfect points:",[6874,6875],"housing-model-playground",{"dataLabel":6876,"error-label":4590,"prediction-label":4591,"x-label":6877,"y-label":3354},"Real houses","Size (100 sqft)",[11,6879,6880,6881,6884],{},"Notice this time the error ",[15,6882,6883],{},"doesn't hit zero",", no matter how you move the sliders. That's not a bug, it's exactly what the entire next post is about: with real, noisy data, there's no perfect line, only the \"least wrong\" one.",[2606,6886,6888],{"id":6887},"predicting-the-price-of-a-new-house","Predicting the price of a new house",[2611,6890,6892],{"className":2613,"code":6891,"language":2615,"meta":26,"style":26},"# Using the approximate fit we found playing with the widget above:\nw = 116      # dollars (in thousands) added per extra 100 sqft of size\nb = 399      # base price, in thousands of dollars\n\nnew_house = 220 \u002F 100     # 220 sqft, on the same scale as the chart\n\npredicted_price = w * new_house + b   # in thousands of dollars\n\nprint(f\"${predicted_price * 1000:,.0f}\")\n",[65,6893,6894,6899,6904,6909,6913,6918,6922,6927,6931],{"__ignoreMap":26},[80,6895,6896],{"class":2620,"line":33},[80,6897,6898],{},"# Using the approximate fit we found playing with the widget above:\n",[80,6900,6901],{"class":2620,"line":27},[80,6902,6903],{},"w = 116      # dollars (in thousands) added per extra 100 sqft of size\n",[80,6905,6906],{"class":2620,"line":2631},[80,6907,6908],{},"b = 399      # base price, in thousands of dollars\n",[80,6910,6911],{"class":2620,"line":2636},[80,6912,2657],{"emptyLinePlaceholder":32},[80,6914,6915],{"class":2620,"line":2642},[80,6916,6917],{},"new_house = 220 \u002F 100     # 220 sqft, on the same scale as the chart\n",[80,6919,6920],{"class":2620,"line":2648},[80,6921,2657],{"emptyLinePlaceholder":32},[80,6923,6924],{"class":2620,"line":2654},[80,6925,6926],{},"predicted_price = w * new_house + b   # in thousands of dollars\n",[80,6928,6929],{"class":2620,"line":2660},[80,6930,2657],{"emptyLinePlaceholder":32},[80,6932,6933],{"class":2620,"line":2666},[80,6934,6935],{},"print(f\"${predicted_price * 1000:,.0f}\")\n",[46,6937,6938],{},[11,6939,6940,3255,6942],{},[15,6941,2693],{},[65,6943,6944],{},"$654,200",[11,6946,6947],{},"Same formula from the post, same math, just now validated against a dataset you (or any reader) can download and check for yourself.",[6949,6950,6951],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":26,"searchDepth":27,"depth":27,"links":6953},[6954,6955,6956,6962,6963,6966,6967,6968,6969],{"id":296,"depth":27,"text":297},{"id":515,"depth":27,"text":516},{"id":1621,"depth":27,"text":1622,"children":6957},[6958,6959,6961],{"id":2608,"depth":2631,"text":2609},{"id":2738,"depth":2631,"text":6960},"How many examples do I have? (mmm)",{"id":2884,"depth":2631,"text":2885},{"id":3342,"depth":27,"text":3343},{"id":3360,"depth":27,"text":3361,"children":6964},[6965],{"id":4079,"depth":2631,"text":4080},{"id":4199,"depth":27,"text":4200},{"id":5717,"depth":27,"text":5718},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716,"children":6970},[6971,6972,6973,6975],{"id":6729,"depth":2631,"text":6730},{"id":6795,"depth":2631,"text":6796},{"id":6811,"depth":2631,"text":6974},"Finding www and bbb by hand, with real data",{"id":6887,"depth":2631,"text":6888},"2026-08-18","The simplest ML model there is, f(x) = wx + b, taken apart: why a line, what w and b actually mean, and why 'finding the parameters by hand' doesn't scale.",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","machine-learning-specialization",{"title":42,"description":6977},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation",[6984,6985,6986],"linear-regression","numpy","fundamentals","JFfvR2d_XNJhNLc1APmuqnqAElHigx9jDkzZiP9SVx8",{"id":6989,"title":6990,"body":6991,"cover":3,"date":6976,"description":10235,"extension":30,"meta":10236,"navigation":32,"order":27,"path":4066,"playlist":6980,"seo":10237,"status":36,"stem":10238,"tags":10239,"__hash__":10242},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab03-cost-function.md","Optional Lab: the Cost Function",{"type":8,"value":6992,"toc":10225},[6993,7099,7163,7666,7669,7673,7680,7684,7909,8498,8501,8644,8677,8681,8808,8822,8826,8965,9013,9019,9022,9262,9317,9321,9324,9381,9436,9498,9504,9618,9630,9634,9642,9816,9821,9828,9830,10118,10125,10127,10140,10204,10217,10220,10223],[11,6994,6995,6600,6998,2338,7048,7098],{},[562,6996,6997],{"href":6979},"In the last post",[80,6999,7001,7018],{"className":7000},[83],[80,7002,7004],{"className":7003},[87],[89,7005,7006],{"xmlns":91},[93,7007,7008,7016],{},[96,7009,7010,7012,7014],{},[102,7011,109],{},[111,7013,130],{},[1321,7015,4708],{},[145,7017,5383],{"encoding":147},[80,7019,7021,7039],{"className":7020,"ariaHidden":113},[152],[80,7022,7024,7027,7030,7033,7036],{"className":7023},[156],[80,7025],{"className":7026,"style":334},[160],[80,7028,109],{"className":7029,"style":210},[165,169],[80,7031],{"className":7032,"style":247},[246],[80,7034,130],{"className":7035},[251],[80,7037],{"className":7038,"style":247},[246],[80,7040,7042,7045],{"className":7041},[156],[80,7043],{"className":7044,"style":1614},[160],[80,7046,4708],{"className":7047},[165],[80,7049,7051,7068],{"className":7050},[83],[80,7052,7054],{"className":7053},[87],[89,7055,7056],{"xmlns":91},[93,7057,7058,7066],{},[96,7059,7060,7062,7064],{},[102,7061,117],{},[111,7063,130],{},[1321,7065,4286],{},[145,7067,5434],{"encoding":147},[80,7069,7071,7089],{"className":7070,"ariaHidden":113},[152],[80,7072,7074,7077,7080,7083,7086],{"className":7073},[156],[80,7075],{"className":7076,"style":289},[160],[80,7078,117],{"className":7079},[165,169],[80,7081],{"className":7082,"style":247},[246],[80,7084,130],{"className":7085},[251],[80,7087],{"className":7088,"style":247},[246],[80,7090,7092,7095],{"className":7091},[156],[80,7093],{"className":7094,"style":1614},[160],[80,7096,4286],{"className":7097},[165]," by dragging a slider until the error hit zero. Cool, it worked. But wait. How did you know you'd found the \"best\" fit? You watched the total error number drop to zero and trusted it.",[11,7100,7101,7102,7106,7107,1618],{},"This post is about giving that number a name, a last name, and a formula. It's called the ",[7103,7104,4067],"glossary-term",{"definition":7105},"a formula that summarizes, in a single number, how wrong a model is on the training data",", written ",[80,7108,7110,7133],{"className":7109},[83],[80,7111,7113],{"className":7112},[87],[89,7114,7115],{"xmlns":91},[93,7116,7117,7131],{},[96,7118,7119,7121,7123,7125,7127,7129],{},[102,7120,5606],{},[111,7122,121],{"stretchy":120},[102,7124,109],{},[111,7126,114],{"separator":113},[102,7128,117],{},[111,7130,127],{"stretchy":120},[145,7132,5619],{"encoding":147},[80,7134,7136],{"className":7135,"ariaHidden":113},[152],[80,7137,7139,7142,7145,7148,7151,7154,7157,7160],{"className":7138},[156],[80,7140],{"className":7141,"style":2316},[160],[80,7143,5606],{"className":7144,"style":5632},[165,169],[80,7146,121],{"className":7147},[235],[80,7149,109],{"className":7150,"style":210},[165,169],[80,7152,114],{"className":7153},[214],[80,7155],{"className":7156,"style":268},[246],[80,7158,117],{"className":7159},[165,169],[80,7161,127],{"className":7162},[242],[11,7164,7165],{},[80,7166,7168,7278],{"className":7167},[83],[80,7169,7171],{"className":7170},[87],[89,7172,7173],{"xmlns":91},[93,7174,7175,7275],{},[96,7176,7177,7179,7181,7183,7185,7187,7189,7191,7201,7223],{},[102,7178,5606],{},[111,7180,121],{"stretchy":120},[102,7182,109],{},[111,7184,114],{"separator":113},[102,7186,117],{},[111,7188,127],{"stretchy":120},[111,7190,130],{},[4625,7192,7193,7195],{},[1321,7194,1583],{},[96,7196,7197,7199],{},[1321,7198,1323],{},[102,7200,322],{},[7202,7203,7204,7207,7215],"msubsup",{},[111,7205,7206],{},"∑",[96,7208,7209,7211,7213],{},[102,7210,736],{},[111,7212,130],{},[1321,7214,2071],{},[96,7216,7217,7219,7221],{},[102,7218,322],{},[111,7220,4643],{},[1321,7222,1583],{},[726,7224,7225,7273],{},[96,7226,7227,7229,7241,7243,7255,7257,7259,7271],{},[111,7228,121],{"fence":113},[99,7230,7231,7233],{},[102,7232,104],{},[96,7234,7235,7237,7239],{},[102,7236,109],{},[111,7238,114],{"separator":113},[102,7240,117],{},[111,7242,121],{"stretchy":120},[726,7244,7245,7247],{},[102,7246,124],{},[96,7248,7249,7251,7253],{},[111,7250,121],{"stretchy":120},[102,7252,736],{},[111,7254,127],{"stretchy":120},[111,7256,127],{"stretchy":120},[111,7258,4643],{},[726,7260,7261,7263],{},[102,7262,683],{},[96,7264,7265,7267,7269],{},[111,7266,121],{"stretchy":120},[102,7268,736],{},[111,7270,127],{"stretchy":120},[111,7272,127],{"fence":113},[1321,7274,1323],{},[145,7276,7277],{"encoding":147},"J(w,b) = \\frac{1}{2m}\\sum_{i=0}^{m-1}\\left(f_{w,b}(x^{(i)}) - y^{(i)}\\right)^2",[80,7279,7281,7317],{"className":7280,"ariaHidden":113},[152],[80,7282,7284,7287,7290,7293,7296,7299,7302,7305,7308,7311,7314],{"className":7283},[156],[80,7285],{"className":7286,"style":2316},[160],[80,7288,5606],{"className":7289,"style":5632},[165,169],[80,7291,121],{"className":7292},[235],[80,7294,109],{"className":7295,"style":210},[165,169],[80,7297,114],{"className":7298},[214],[80,7300],{"className":7301,"style":268},[246],[80,7303,117],{"className":7304},[165,169],[80,7306,127],{"className":7307},[242],[80,7309],{"className":7310,"style":247},[246],[80,7312,130],{"className":7313},[251],[80,7315],{"className":7316,"style":247},[246],[80,7318,7320,7324,7396,7399,7476,7479],{"className":7319},[156],[80,7321],{"className":7322,"style":7323},[160],"height:1.442em;vertical-align:-0.35em;",[80,7325,7327,7330,7393],{"className":7326},[165],[80,7328],{"className":7329},[235,4746],[80,7331,7333],{"className":7332},[4625],[80,7334,7336,7384],{"className":7335},[178,179],[80,7337,7339,7381],{"className":7338},[183],[80,7340,7342,7359,7367],{"className":7341,"style":5010},[187],[80,7343,7344,7347],{"style":5013},[80,7345],{"className":7346,"style":4766},[195],[80,7348,7350],{"className":7349},[200,201,202,203],[80,7351,7353,7356],{"className":7352},[165,203],[80,7354,1323],{"className":7355},[165,203],[80,7357,322],{"className":7358},[165,169,203],[80,7360,7361,7364],{"style":4859},[80,7362],{"className":7363,"style":4766},[195],[80,7365],{"className":7366,"style":4867},[4866],[80,7368,7369,7372],{"style":5042},[80,7370],{"className":7371,"style":4766},[195],[80,7373,7375],{"className":7374},[200,201,202,203],[80,7376,7378],{"className":7377},[165,203],[80,7379,1583],{"className":7380},[165,203],[80,7382,222],{"className":7383},[221],[80,7385,7387],{"className":7386},[183],[80,7388,7391],{"className":7389,"style":7390},[187],"height:0.345em;",[80,7392],{},[80,7394],{"className":7395},[242,4746],[80,7397],{"className":7398,"style":268},[246],[80,7400,7403,7409],{"className":7401},[7402],"mop",[80,7404,7206],{"className":7405,"style":7408},[7402,7406,7407],"op-symbol","small-op","position:relative;top:0em;",[80,7410,7412],{"className":7411},[174],[80,7413,7415,7467],{"className":7414},[178,179],[80,7416,7418,7464],{"className":7417},[183],[80,7419,7422,7443],{"className":7420,"style":7421},[187],"height:0.954em;",[80,7423,7425,7428],{"style":7424},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[80,7426],{"className":7427,"style":196},[195],[80,7429,7431],{"className":7430},[200,201,202,203],[80,7432,7434,7437,7440],{"className":7433},[165,203],[80,7435,736],{"className":7436},[165,169,203],[80,7438,130],{"className":7439},[251,203],[80,7441,2071],{"className":7442},[165,203],[80,7444,7446,7449],{"style":7445},"top:-3.2029em;margin-right:0.05em;",[80,7447],{"className":7448,"style":196},[195],[80,7450,7452],{"className":7451},[200,201,202,203],[80,7453,7455,7458,7461],{"className":7454},[165,203],[80,7456,322],{"className":7457},[165,169,203],[80,7459,4643],{"className":7460},[279,203],[80,7462,1583],{"className":7463},[165,203],[80,7465,222],{"className":7466},[221],[80,7468,7470],{"className":7469},[183],[80,7471,7474],{"className":7472,"style":7473},[187],"height:0.2997em;",[80,7475],{},[80,7477],{"className":7478,"style":268},[246],[80,7480,7483,7641],{"className":7481},[7482],"minner",[80,7484,7486,7495,7544,7547,7585,7588,7591,7594,7597,7635],{"className":7485},[7482],[80,7487,7491],{"className":7488,"style":7490},[235,7489],"delimcenter","top:0em;",[80,7492,121],{"className":7493},[7494,4803],"delimsizing",[80,7496,7498,7501],{"className":7497},[165],[80,7499,104],{"className":7500,"style":170},[165,169],[80,7502,7504],{"className":7503},[174],[80,7505,7507,7536],{"className":7506},[178,179],[80,7508,7510,7533],{"className":7509},[183],[80,7511,7513],{"className":7512,"style":188},[187],[80,7514,7515,7518],{"style":191},[80,7516],{"className":7517,"style":196},[195],[80,7519,7521],{"className":7520},[200,201,202,203],[80,7522,7524,7527,7530],{"className":7523},[165,203],[80,7525,109],{"className":7526,"style":210},[165,169,203],[80,7528,114],{"className":7529},[214,203],[80,7531,117],{"className":7532},[165,169,203],[80,7534,222],{"className":7535},[221],[80,7537,7539],{"className":7538},[183],[80,7540,7542],{"className":7541,"style":229},[187],[80,7543],{},[80,7545,121],{"className":7546},[235],[80,7548,7550,7553],{"className":7549},[165],[80,7551,124],{"className":7552},[165,169],[80,7554,7556],{"className":7555},[174],[80,7557,7559],{"className":7558},[178],[80,7560,7562],{"className":7561},[183],[80,7563,7565],{"className":7564,"style":751},[187],[80,7566,7567,7570],{"style":772},[80,7568],{"className":7569,"style":196},[195],[80,7571,7573],{"className":7572},[200,201,202,203],[80,7574,7576,7579,7582],{"className":7575},[165,203],[80,7577,121],{"className":7578},[235,203],[80,7580,736],{"className":7581},[165,169,203],[80,7583,127],{"className":7584},[242,203],[80,7586,127],{"className":7587},[242],[80,7589],{"className":7590,"style":275},[246],[80,7592,4643],{"className":7593},[279],[80,7595],{"className":7596,"style":275},[246],[80,7598,7600,7603],{"className":7599},[165],[80,7601,683],{"className":7602,"style":834},[165,169],[80,7604,7606],{"className":7605},[174],[80,7607,7609],{"className":7608},[178],[80,7610,7612],{"className":7611},[183],[80,7613,7615],{"className":7614,"style":751},[187],[80,7616,7617,7620],{"style":772},[80,7618],{"className":7619,"style":196},[195],[80,7621,7623],{"className":7622},[200,201,202,203],[80,7624,7626,7629,7632],{"className":7625},[165,203],[80,7627,121],{"className":7628},[235,203],[80,7630,736],{"className":7631},[165,169,203],[80,7633,127],{"className":7634},[242,203],[80,7636,7638],{"className":7637,"style":7490},[242,7489],[80,7639,127],{"className":7640},[7494,4803],[80,7642,7644],{"className":7643},[174],[80,7645,7647],{"className":7646},[178],[80,7648,7650],{"className":7649},[183],[80,7651,7654],{"className":7652,"style":7653},[187],"height:1.092em;",[80,7655,7657,7660],{"style":7656},"top:-3.3409em;margin-right:0.05em;",[80,7658],{"className":7659,"style":196},[195],[80,7661,7663],{"className":7662},[200,201,202,203],[80,7664,1323],{"className":7665},[165,203],[11,7667,7668],{},"I know, it looks scary. Let's take that formula apart piece by piece, like disassembling a Lego set.",[294,7670,7672],{"id":7671},"watch-out-for-the-word-cost","Watch out for the word \"cost\"",[11,7674,7675,7676,7679],{},"Before anything else: here ",[73,7677,7678],{},"cost"," has nothing to do with a house's price. It's the measure of how wrong the model is. For the house's value we keep using the word price. This vocabulary mix-up trips a lot of people up early on, because \"cost\" in everyday speech always means money.",[294,7681,7683],{"id":7682},"taking-the-formula-apart","Taking the formula apart",[11,7685,7686,7687,7760,7761,7905,7906,1618],{},"Think of a target-shooting contest. Each house in your training set is an attempt to hit the bullseye (the real price, ",[80,7688,7690,7713],{"className":7689},[83],[80,7691,7693],{"className":7692},[87],[89,7694,7695],{"xmlns":91},[93,7696,7697,7711],{},[96,7698,7699],{},[726,7700,7701,7703],{},[102,7702,683],{},[96,7704,7705,7707,7709],{},[111,7706,121],{"stretchy":120},[102,7708,736],{},[111,7710,127],{"stretchy":120},[145,7712,817],{"encoding":147},[80,7714,7716],{"className":7715,"ariaHidden":113},[152],[80,7717,7719,7722],{"className":7718},[156],[80,7720],{"className":7721,"style":827},[160],[80,7723,7725,7728],{"className":7724},[165],[80,7726,683],{"className":7727,"style":834},[165,169],[80,7729,7731],{"className":7730},[174],[80,7732,7734],{"className":7733},[178],[80,7735,7737],{"className":7736},[183],[80,7738,7740],{"className":7739,"style":751},[187],[80,7741,7742,7745],{"style":772},[80,7743],{"className":7744,"style":196},[195],[80,7746,7748],{"className":7747},[200,201,202,203],[80,7749,7751,7754,7757],{"className":7750},[165,203],[80,7752,121],{"className":7753},[235,203],[80,7755,736],{"className":7756},[165,169,203],[80,7758,127],{"className":7759},[242,203],"). ",[80,7762,7764,7803],{"className":7763},[83],[80,7765,7767],{"className":7766},[87],[89,7768,7769],{"xmlns":91},[93,7770,7771,7801],{},[96,7772,7773,7785,7787,7799],{},[99,7774,7775,7777],{},[102,7776,104],{},[96,7778,7779,7781,7783],{},[102,7780,109],{},[111,7782,114],{"separator":113},[102,7784,117],{},[111,7786,121],{"stretchy":120},[726,7788,7789,7791],{},[102,7790,124],{},[96,7792,7793,7795,7797],{},[111,7794,121],{"stretchy":120},[102,7796,736],{},[111,7798,127],{"stretchy":120},[111,7800,127],{"stretchy":120},[145,7802,1070],{"encoding":147},[80,7804,7806],{"className":7805,"ariaHidden":113},[152],[80,7807,7809,7812,7861,7864,7902],{"className":7808},[156],[80,7810],{"className":7811,"style":1080},[160],[80,7813,7815,7818],{"className":7814},[165],[80,7816,104],{"className":7817,"style":170},[165,169],[80,7819,7821],{"className":7820},[174],[80,7822,7824,7853],{"className":7823},[178,179],[80,7825,7827,7850],{"className":7826},[183],[80,7828,7830],{"className":7829,"style":188},[187],[80,7831,7832,7835],{"style":191},[80,7833],{"className":7834,"style":196},[195],[80,7836,7838],{"className":7837},[200,201,202,203],[80,7839,7841,7844,7847],{"className":7840},[165,203],[80,7842,109],{"className":7843,"style":210},[165,169,203],[80,7845,114],{"className":7846},[214,203],[80,7848,117],{"className":7849},[165,169,203],[80,7851,222],{"className":7852},[221],[80,7854,7856],{"className":7855},[183],[80,7857,7859],{"className":7858,"style":229},[187],[80,7860],{},[80,7862,121],{"className":7863},[235],[80,7865,7867,7870],{"className":7866},[165],[80,7868,124],{"className":7869},[165,169],[80,7871,7873],{"className":7872},[174],[80,7874,7876],{"className":7875},[178],[80,7877,7879],{"className":7878},[183],[80,7880,7882],{"className":7881,"style":751},[187],[80,7883,7884,7887],{"style":772},[80,7885],{"className":7886,"style":196},[195],[80,7888,7890],{"className":7889},[200,201,202,203],[80,7891,7893,7896,7899],{"className":7892},[165,203],[80,7894,121],{"className":7895},[235,203],[80,7897,736],{"className":7898},[165,169,203],[80,7900,127],{"className":7901},[242,203],[80,7903,127],{"className":7904},[242]," is where your arrow actually landed (the model's prediction). The distance between where it landed and the bullseye is that shot's ",[7103,7907,4263],{"definition":7908},"the difference between what the model predicted and the real value for that example, also called a residual",[521,7910,7911,7924],{},[524,7912,7913],{},[527,7914,7915,7918,7921],{},[530,7916,7917],{"align":532},"Piece of the formula",[530,7919,7920],{"align":532},"Name",[530,7922,7923],{"align":532},"What it does",[541,7925,7926,8148,8229,8353],{},[527,7927,7928,8142,8145],{},[546,7929,7930],{"align":532},[80,7931,7933,7987],{"className":7932},[83],[80,7934,7936],{"className":7935},[87],[89,7937,7938],{"xmlns":91},[93,7939,7940,7984],{},[96,7941,7942,7954,7956,7968,7970,7972],{},[99,7943,7944,7946],{},[102,7945,104],{},[96,7947,7948,7950,7952],{},[102,7949,109],{},[111,7951,114],{"separator":113},[102,7953,117],{},[111,7955,121],{"stretchy":120},[726,7957,7958,7960],{},[102,7959,124],{},[96,7961,7962,7964,7966],{},[111,7963,121],{"stretchy":120},[102,7965,736],{},[111,7967,127],{"stretchy":120},[111,7969,127],{"stretchy":120},[111,7971,4643],{},[726,7973,7974,7976],{},[102,7975,683],{},[96,7977,7978,7980,7982],{},[111,7979,121],{"stretchy":120},[102,7981,736],{},[111,7983,127],{"stretchy":120},[145,7985,7986],{"encoding":147},"f_{w,b}(x^{(i)}) - y^{(i)}",[80,7988,7990,8098],{"className":7989,"ariaHidden":113},[152],[80,7991,7993,7996,8045,8048,8086,8089,8092,8095],{"className":7992},[156],[80,7994],{"className":7995,"style":1080},[160],[80,7997,7999,8002],{"className":7998},[165],[80,8000,104],{"className":8001,"style":170},[165,169],[80,8003,8005],{"className":8004},[174],[80,8006,8008,8037],{"className":8007},[178,179],[80,8009,8011,8034],{"className":8010},[183],[80,8012,8014],{"className":8013,"style":188},[187],[80,8015,8016,8019],{"style":191},[80,8017],{"className":8018,"style":196},[195],[80,8020,8022],{"className":8021},[200,201,202,203],[80,8023,8025,8028,8031],{"className":8024},[165,203],[80,8026,109],{"className":8027,"style":210},[165,169,203],[80,8029,114],{"className":8030},[214,203],[80,8032,117],{"className":8033},[165,169,203],[80,8035,222],{"className":8036},[221],[80,8038,8040],{"className":8039},[183],[80,8041,8043],{"className":8042,"style":229},[187],[80,8044],{},[80,8046,121],{"className":8047},[235],[80,8049,8051,8054],{"className":8050},[165],[80,8052,124],{"className":8053},[165,169],[80,8055,8057],{"className":8056},[174],[80,8058,8060],{"className":8059},[178],[80,8061,8063],{"className":8062},[183],[80,8064,8066],{"className":8065,"style":751},[187],[80,8067,8068,8071],{"style":772},[80,8069],{"className":8070,"style":196},[195],[80,8072,8074],{"className":8073},[200,201,202,203],[80,8075,8077,8080,8083],{"className":8076},[165,203],[80,8078,121],{"className":8079},[235,203],[80,8081,736],{"className":8082},[165,169,203],[80,8084,127],{"className":8085},[242,203],[80,8087,127],{"className":8088},[242],[80,8090],{"className":8091,"style":275},[246],[80,8093,4643],{"className":8094},[279],[80,8096],{"className":8097,"style":275},[246],[80,8099,8101,8104],{"className":8100},[156],[80,8102],{"className":8103,"style":827},[160],[80,8105,8107,8110],{"className":8106},[165],[80,8108,683],{"className":8109,"style":834},[165,169],[80,8111,8113],{"className":8112},[174],[80,8114,8116],{"className":8115},[178],[80,8117,8119],{"className":8118},[183],[80,8120,8122],{"className":8121,"style":751},[187],[80,8123,8124,8127],{"style":772},[80,8125],{"className":8126,"style":196},[195],[80,8128,8130],{"className":8129},[200,201,202,203],[80,8131,8133,8136,8139],{"className":8132},[165,203],[80,8134,121],{"className":8135},[235,203],[80,8137,736],{"className":8138},[165,169,203],[80,8140,127],{"className":8141},[242,203],[546,8143,8144],{"align":532},"error (residual)",[546,8146,8147],{"align":532},"the distance between the prediction and the real value",[527,8149,8150,8223,8226],{},[546,8151,8152],{"align":532},[80,8153,8155,8178],{"className":8154},[83],[80,8156,8158],{"className":8157},[87],[89,8159,8160],{"xmlns":91},[93,8161,8162,8175],{},[96,8163,8164,8166,8169],{},[111,8165,121],{"stretchy":120},[111,8167,8168],{},"…",[726,8170,8171,8173],{},[111,8172,127],{"stretchy":120},[1321,8174,1323],{},[145,8176,8177],{"encoding":147},"(\\ldots)^2",[80,8179,8181],{"className":8180,"ariaHidden":113},[152],[80,8182,8184,8188,8191,8194],{"className":8183},[156],[80,8185],{"className":8186,"style":8187},[160],"height:1.0641em;vertical-align:-0.25em;",[80,8189,121],{"className":8190},[235],[80,8192,8168],{"className":8193},[7482],[80,8195,8197,8200],{"className":8196},[242],[80,8198,127],{"className":8199},[242],[80,8201,8203],{"className":8202},[174],[80,8204,8206],{"className":8205},[178],[80,8207,8209],{"className":8208},[183],[80,8210,8212],{"className":8211,"style":1407},[187],[80,8213,8214,8217],{"style":772},[80,8215],{"className":8216,"style":196},[195],[80,8218,8220],{"className":8219},[200,201,202,203],[80,8221,1323],{"className":8222},[165,203],[546,8224,8225],{"align":532},"squared error",[546,8227,8228],{"align":532},"removes the sign (missing high or low weighs the same) and punishes big misses much harder",[527,8230,8231,8347,8350],{},[546,8232,8233],{"align":532},[80,8234,8236,8268],{"className":8235},[83],[80,8237,8239],{"className":8238},[87],[89,8240,8241],{"xmlns":91},[93,8242,8243,8265],{},[96,8244,8245],{},[7202,8246,8247,8249,8257],{},[111,8248,7206],{},[96,8250,8251,8253,8255],{},[102,8252,736],{},[111,8254,130],{},[1321,8256,2071],{},[96,8258,8259,8261,8263],{},[102,8260,322],{},[111,8262,4643],{},[1321,8264,1583],{},[145,8266,8267],{"encoding":147},"\\sum_{i=0}^{m-1}",[80,8269,8271],{"className":8270,"ariaHidden":113},[152],[80,8272,8274,8278],{"className":8273},[156],[80,8275],{"className":8276,"style":8277},[160],"height:1.2537em;vertical-align:-0.2997em;",[80,8279,8281,8284],{"className":8280},[7402],[80,8282,7206],{"className":8283,"style":7408},[7402,7406,7407],[80,8285,8287],{"className":8286},[174],[80,8288,8290,8339],{"className":8289},[178,179],[80,8291,8293,8336],{"className":8292},[183],[80,8294,8296,8316],{"className":8295,"style":7421},[187],[80,8297,8298,8301],{"style":7424},[80,8299],{"className":8300,"style":196},[195],[80,8302,8304],{"className":8303},[200,201,202,203],[80,8305,8307,8310,8313],{"className":8306},[165,203],[80,8308,736],{"className":8309},[165,169,203],[80,8311,130],{"className":8312},[251,203],[80,8314,2071],{"className":8315},[165,203],[80,8317,8318,8321],{"style":7445},[80,8319],{"className":8320,"style":196},[195],[80,8322,8324],{"className":8323},[200,201,202,203],[80,8325,8327,8330,8333],{"className":8326},[165,203],[80,8328,322],{"className":8329},[165,169,203],[80,8331,4643],{"className":8332},[279,203],[80,8334,1583],{"className":8335},[165,203],[80,8337,222],{"className":8338},[221],[80,8340,8342],{"className":8341},[183],[80,8343,8345],{"className":8344,"style":7473},[187],[80,8346],{},[546,8348,8349],{"align":532},"sum",[546,8351,8352],{"align":532},"adds up everyone's error",[527,8354,8355,8463,8466],{},[546,8356,8357],{"align":532},[80,8358,8360,8382],{"className":8359},[83],[80,8361,8363],{"className":8362},[87],[89,8364,8365],{"xmlns":91},[93,8366,8367,8379],{},[96,8368,8369],{},[4625,8370,8371,8373],{},[1321,8372,1583],{},[96,8374,8375,8377],{},[1321,8376,1323],{},[102,8378,322],{},[145,8380,8381],{"encoding":147},"\\frac{1}{2m}",[80,8383,8385],{"className":8384,"ariaHidden":113},[152],[80,8386,8388,8392],{"className":8387},[156],[80,8389],{"className":8390,"style":8391},[160],"height:1.1901em;vertical-align:-0.345em;",[80,8393,8395,8398,8460],{"className":8394},[165],[80,8396],{"className":8397},[235,4746],[80,8399,8401],{"className":8400},[4625],[80,8402,8404,8452],{"className":8403},[178,179],[80,8405,8407,8449],{"className":8406},[183],[80,8408,8410,8427,8435],{"className":8409,"style":5010},[187],[80,8411,8412,8415],{"style":5013},[80,8413],{"className":8414,"style":4766},[195],[80,8416,8418],{"className":8417},[200,201,202,203],[80,8419,8421,8424],{"className":8420},[165,203],[80,8422,1323],{"className":8423},[165,203],[80,8425,322],{"className":8426},[165,169,203],[80,8428,8429,8432],{"style":4859},[80,8430],{"className":8431,"style":4766},[195],[80,8433],{"className":8434,"style":4867},[4866],[80,8436,8437,8440],{"style":5042},[80,8438],{"className":8439,"style":4766},[195],[80,8441,8443],{"className":8442},[200,201,202,203],[80,8444,8446],{"className":8445},[165,203],[80,8447,1583],{"className":8448},[165,203],[80,8450,222],{"className":8451},[221],[80,8453,8455],{"className":8454},[183],[80,8456,8458],{"className":8457,"style":7390},[187],[80,8459],{},[80,8461],{"className":8462},[242,4746],[546,8464,8465],{"align":532},"average (with an extra 2)",[546,8467,8468,8469,8497],{"align":532},"dividing by ",[80,8470,8472,8485],{"className":8471},[83],[80,8473,8475],{"className":8474},[87],[89,8476,8477],{"xmlns":91},[93,8478,8479,8483],{},[96,8480,8481],{},[102,8482,322],{},[145,8484,322],{"encoding":147},[80,8486,8488],{"className":8487,"ariaHidden":113},[152],[80,8489,8491,8494],{"className":8490},[156],[80,8492],{"className":8493,"style":334},[160],[80,8495,322],{"className":8496},[165,169]," turns the sum into an average (otherwise the cost would only grow from having more data, even for an equally good fit), and the 2 is just a bit of bookkeeping that makes the next post's math cleaner, more on that there",[11,8499,8500],{},"Two things fall right out of this formula, and I didn't have to memorize any of it:",[299,8502,8503,8562],{},[302,8504,8505,8561],{},[80,8506,8508,8531],{"className":8507},[83],[80,8509,8511],{"className":8510},[87],[89,8512,8513],{"xmlns":91},[93,8514,8515,8529],{},[96,8516,8517,8519,8521,8523,8525,8527],{},[102,8518,5606],{},[111,8520,121],{"stretchy":120},[102,8522,109],{},[111,8524,114],{"separator":113},[102,8526,117],{},[111,8528,127],{"stretchy":120},[145,8530,5619],{"encoding":147},[80,8532,8534],{"className":8533,"ariaHidden":113},[152],[80,8535,8537,8540,8543,8546,8549,8552,8555,8558],{"className":8536},[156],[80,8538],{"className":8539,"style":2316},[160],[80,8541,5606],{"className":8542,"style":5632},[165,169],[80,8544,121],{"className":8545},[235],[80,8547,109],{"className":8548,"style":210},[165,169],[80,8550,114],{"className":8551},[214],[80,8553],{"className":8554,"style":268},[246],[80,8556,117],{"className":8557},[165,169],[80,8559,127],{"className":8560},[242]," is never negative. It's a sum of squared terms, so the smallest possible value is zero.",[302,8563,8564,8643],{},[80,8565,8567,8595],{"className":8566},[83],[80,8568,8570],{"className":8569},[87],[89,8571,8572],{"xmlns":91},[93,8573,8574,8592],{},[96,8575,8576,8578,8580,8582,8584,8586,8588,8590],{},[102,8577,5606],{},[111,8579,121],{"stretchy":120},[102,8581,109],{},[111,8583,114],{"separator":113},[102,8585,117],{},[111,8587,127],{"stretchy":120},[111,8589,130],{},[1321,8591,2071],{},[145,8593,8594],{"encoding":147},"J(w,b) = 0",[80,8596,8598,8634],{"className":8597,"ariaHidden":113},[152],[80,8599,8601,8604,8607,8610,8613,8616,8619,8622,8625,8628,8631],{"className":8600},[156],[80,8602],{"className":8603,"style":2316},[160],[80,8605,5606],{"className":8606,"style":5632},[165,169],[80,8608,121],{"className":8609},[235],[80,8611,109],{"className":8612,"style":210},[165,169],[80,8614,114],{"className":8615},[214],[80,8617],{"className":8618,"style":268},[246],[80,8620,117],{"className":8621},[165,169],[80,8623,127],{"className":8624},[242],[80,8626],{"className":8627,"style":247},[246],[80,8629,130],{"className":8630},[251],[80,8632],{"className":8633,"style":247},[246],[80,8635,8637,8640],{"className":8636},[156],[80,8638],{"className":8639,"style":1614},[160],[80,8641,2071],{"className":8642},[165]," means a perfect fit. The line passes exactly through every point.",[11,8645,8646,8647,8676],{},"And why square the error instead of, say, taking its absolute value? Because squaring disproportionately punishes big misses. Missing by twice as much costs four times as much in ",[80,8648,8650,8663],{"className":8649},[83],[80,8651,8653],{"className":8652},[87],[89,8654,8655],{"xmlns":91},[93,8656,8657,8661],{},[96,8658,8659],{},[102,8660,5606],{},[145,8662,5606],{"encoding":147},[80,8664,8666],{"className":8665,"ariaHidden":113},[152],[80,8667,8669,8673],{"className":8668},[156],[80,8670],{"className":8671,"style":8672},[160],"height:0.6833em;",[80,8674,5606],{"className":8675,"style":5632},[165,169],", not twice as much. That makes the model hate really bad predictions in a way absolute value wouldn't, and it's exactly this behavior that gives the cost its U shape, which we'll see in a moment. Back to the target-shooting contest: it's as if the judge scored exponentially worse the farther the arrow lands from the bullseye. Missing by a little barely stings, missing badly stings way more than twice as much.",[294,8678,8680],{"id":8679},"putting-this-into-code","Putting this into code",[2611,8682,8684],{"className":2613,"code":8683,"language":2615,"meta":26,"style":26},"def compute_cost(x, y, w, b):\n    \"\"\"\n    Computes the cost function for linear regression.\n\n    Args:\n      x (ndarray (m,)) : input data, m examples\n      y (ndarray (m,)) : target values\n      w, b (scalar)    : model parameters\n\n    Returns:\n      total_cost (float): the cost of using w and b as parameters to fit\n                          the points (x, y)\n    \"\"\"\n    m = x.shape[0]\n\n    cost_sum = 0\n\n    for i in range(m):\n        f_wb = w * x[i] + b           # model's prediction for example i\n        cost = (f_wb - y[i]) ** 2     # example i's error, squared\n        cost_sum = cost_sum + cost    # add it to the total\n\n    total_cost = (1 \u002F (2 * m)) * cost_sum\n\n    return total_cost\n",[65,8685,8686,8691,8695,8700,8704,8708,8713,8718,8723,8727,8731,8736,8741,8745,8750,8754,8759,8763,8768,8774,8780,8786,8791,8797,8802],{"__ignoreMap":26},[80,8687,8688],{"class":2620,"line":33},[80,8689,8690],{},"def compute_cost(x, y, w, b):\n",[80,8692,8693],{"class":2620,"line":27},[80,8694,4105],{},[80,8696,8697],{"class":2620,"line":2631},[80,8698,8699],{},"    Computes the cost function for linear regression.\n",[80,8701,8702],{"class":2620,"line":2636},[80,8703,2657],{"emptyLinePlaceholder":32},[80,8705,8706],{"class":2620,"line":2642},[80,8707,4119],{},[80,8709,8710],{"class":2620,"line":2648},[80,8711,8712],{},"      x (ndarray (m,)) : input data, m examples\n",[80,8714,8715],{"class":2620,"line":2654},[80,8716,8717],{},"      y (ndarray (m,)) : target values\n",[80,8719,8720],{"class":2620,"line":2660},[80,8721,8722],{},"      w, b (scalar)    : model parameters\n",[80,8724,8725],{"class":2620,"line":2666},[80,8726,2657],{"emptyLinePlaceholder":32},[80,8728,8729],{"class":2620,"line":2672},[80,8730,4138],{},[80,8732,8733],{"class":2620,"line":2677},[80,8734,8735],{},"      total_cost (float): the cost of using w and b as parameters to fit\n",[80,8737,8738],{"class":2620,"line":2683},[80,8739,8740],{},"                          the points (x, y)\n",[80,8742,8743],{"class":2620,"line":4155},[80,8744,4105],{},[80,8746,8747],{"class":2620,"line":4161},[80,8748,8749],{},"    m = x.shape[0]\n",[80,8751,8752],{"class":2620,"line":4166},[80,8753,2657],{"emptyLinePlaceholder":32},[80,8755,8756],{"class":2620,"line":4172},[80,8757,8758],{},"    cost_sum = 0\n",[80,8760,8761],{"class":2620,"line":4178},[80,8762,2657],{"emptyLinePlaceholder":32},[80,8764,8765],{"class":2620,"line":4183},[80,8766,8767],{},"    for i in range(m):\n",[80,8769,8771],{"class":2620,"line":8770},19,[80,8772,8773],{},"        f_wb = w * x[i] + b           # model's prediction for example i\n",[80,8775,8777],{"class":2620,"line":8776},20,[80,8778,8779],{},"        cost = (f_wb - y[i]) ** 2     # example i's error, squared\n",[80,8781,8783],{"class":2620,"line":8782},21,[80,8784,8785],{},"        cost_sum = cost_sum + cost    # add it to the total\n",[80,8787,8789],{"class":2620,"line":8788},22,[80,8790,2657],{"emptyLinePlaceholder":32},[80,8792,8794],{"class":2620,"line":8793},23,[80,8795,8796],{},"    total_cost = (1 \u002F (2 * m)) * cost_sum\n",[80,8798,8800],{"class":2620,"line":8799},24,[80,8801,2657],{"emptyLinePlaceholder":32},[80,8803,8805],{"class":2620,"line":8804},25,[80,8806,8807],{},"    return total_cost\n",[11,8809,8810,8811,8814,8815,3255,8818,8821],{},"Notice it's the same ",[65,8812,8813],{},"for"," loop from ",[562,8816,8817],{"href":6979},"the last post's",[65,8819,8820],{},"compute_model_output",", except instead of storing the predictions, we accumulate each one's squared error. Same structure, different purpose.",[294,8823,8825],{"id":8824},"freezing-one-parameter-to-see-the-other","Freezing one parameter to see the other",[11,8827,8828,8829,2338,8857,8885,8886,8936,8937,1618],{},"With two parameters to move (",[80,8830,8832,8845],{"className":8831},[83],[80,8833,8835],{"className":8834},[87],[89,8836,8837],{"xmlns":91},[93,8838,8839,8843],{},[96,8840,8841],{},[102,8842,109],{},[145,8844,109],{"encoding":147},[80,8846,8848],{"className":8847,"ariaHidden":113},[152],[80,8849,8851,8854],{"className":8850},[156],[80,8852],{"className":8853,"style":334},[160],[80,8855,109],{"className":8856,"style":210},[165,169],[80,8858,8860,8873],{"className":8859},[83],[80,8861,8863],{"className":8862},[87],[89,8864,8865],{"xmlns":91},[93,8866,8867,8871],{},[96,8868,8869],{},[102,8870,117],{},[145,8872,117],{"encoding":147},[80,8874,8876],{"className":8875,"ariaHidden":113},[152],[80,8877,8879,8882],{"className":8878},[156],[80,8880],{"className":8881,"style":289},[160],[80,8883,117],{"className":8884},[165,169],"), I had a hard time visualizing the cost directly. Classic trick: freeze one and look only at the other. Fix ",[80,8887,8889,8906],{"className":8888},[83],[80,8890,8892],{"className":8891},[87],[89,8893,8894],{"xmlns":91},[93,8895,8896,8904],{},[96,8897,8898,8900,8902],{},[102,8899,117],{},[111,8901,130],{},[1321,8903,4286],{},[145,8905,5434],{"encoding":147},[80,8907,8909,8927],{"className":8908,"ariaHidden":113},[152],[80,8910,8912,8915,8918,8921,8924],{"className":8911},[156],[80,8913],{"className":8914,"style":289},[160],[80,8916,117],{"className":8917},[165,169],[80,8919],{"className":8920,"style":247},[246],[80,8922,130],{"className":8923},[251],[80,8925],{"className":8926,"style":247},[246],[80,8928,8930,8933],{"className":8929},[156],[80,8931],{"className":8932,"style":1614},[160],[80,8934,4286],{"className":8935},[165]," and move only ",[80,8938,8940,8953],{"className":8939},[83],[80,8941,8943],{"className":8942},[87],[89,8944,8945],{"xmlns":91},[93,8946,8947,8951],{},[96,8948,8949],{},[102,8950,109],{},[145,8952,109],{"encoding":147},[80,8954,8956],{"className":8955,"ariaHidden":113},[152],[80,8957,8959,8962],{"className":8958},[156],[80,8960],{"className":8961,"style":334},[160],[80,8963,109],{"className":8964,"style":210},[165,169],[11,8966,8967,8968,9012],{},"Drag the slider below and watch two things at once: how the line (left panel) gets closer to or farther from the points, and where the red dot sits on the ",[80,8969,8971,8991],{"className":8970},[83],[80,8972,8974],{"className":8973},[87],[89,8975,8976],{"xmlns":91},[93,8977,8978,8988],{},[96,8979,8980,8982,8984,8986],{},[102,8981,5606],{},[111,8983,121],{"stretchy":120},[102,8985,109],{},[111,8987,127],{"stretchy":120},[145,8989,8990],{"encoding":147},"J(w)",[80,8992,8994],{"className":8993,"ariaHidden":113},[152],[80,8995,8997,9000,9003,9006,9009],{"className":8996},[156],[80,8998],{"className":8999,"style":2316},[160],[80,9001,5606],{"className":9002,"style":5632},[165,169],[80,9004,121],{"className":9005},[235],[80,9007,109],{"className":9008,"style":210},[165,169],[80,9010,127],{"className":9011},[242]," curve (right panel). As the fit improves, the dot moves down the curve.",[9014,9015],"cost-intuition-explorer",{":b-fixed":4286,":initial-w":9016,":w-max":9017,":w-min":2071,":x-train":3350,":y-train":3351,"dataLabel":4589,"prediction-label":9018,"w-label":109,"x-label":3353,"y-label":3354},"150","400","Our prediction",[11,9020,9021],{},"Notice three things:",[1214,9023,9024,9135,9170],{},[302,9025,9026,9079,9080,9083,9084,9134],{},[15,9027,9028,9029],{},"The cost is at its lowest exactly at ",[80,9030,9032,9049],{"className":9031},[83],[80,9033,9035],{"className":9034},[87],[89,9036,9037],{"xmlns":91},[93,9038,9039,9047],{},[96,9040,9041,9043,9045],{},[102,9042,109],{},[111,9044,130],{},[1321,9046,4708],{},[145,9048,5383],{"encoding":147},[80,9050,9052,9070],{"className":9051,"ariaHidden":113},[152],[80,9053,9055,9058,9061,9064,9067],{"className":9054},[156],[80,9056],{"className":9057,"style":334},[160],[80,9059,109],{"className":9060,"style":210},[165,169],[80,9062],{"className":9063,"style":247},[246],[80,9065,130],{"className":9066},[251],[80,9068],{"className":9069,"style":247},[246],[80,9071,9073,9076],{"className":9072},[156],[80,9074],{"className":9075,"style":1614},[160],[80,9077,4708],{"className":9078},[165],", the same value you found by eye in ",[562,9081,9082],{"href":6979},"the last post",". With ",[80,9085,9087,9104],{"className":9086},[83],[80,9088,9090],{"className":9089},[87],[89,9091,9092],{"xmlns":91},[93,9093,9094,9102],{},[96,9095,9096,9098,9100],{},[102,9097,117],{},[111,9099,130],{},[1321,9101,4286],{},[145,9103,5434],{"encoding":147},[80,9105,9107,9125],{"className":9106,"ariaHidden":113},[152],[80,9108,9110,9113,9116,9119,9122],{"className":9109},[156],[80,9111],{"className":9112,"style":289},[160],[80,9114,117],{"className":9115},[165,169],[80,9117],{"className":9118,"style":247},[246],[80,9120,130],{"className":9121},[251],[80,9123],{"className":9124,"style":247},[246],[80,9126,9128,9131],{"className":9127},[156],[80,9129],{"className":9130,"style":1614},[160],[80,9132,4286],{"className":9133},[165],", the cost there hits zero, because the line passes through both points.",[302,9136,9137,9140,9141,9169],{},[15,9138,9139],{},"The cost shoots up fast"," when ",[80,9142,9144,9157],{"className":9143},[83],[80,9145,9147],{"className":9146},[87],[89,9148,9149],{"xmlns":91},[93,9150,9151,9155],{},[96,9152,9153],{},[102,9154,109],{},[145,9156,109],{"encoding":147},[80,9158,9160],{"className":9159,"ariaHidden":113},[152],[80,9161,9163,9166],{"className":9162},[156],[80,9164],{"className":9165,"style":334},[160],[80,9167,109],{"className":9168,"style":210},[165,169]," gets too big or too small. That's the square in the formula doing its job.",[302,9171,9172,9175,9176,2338,9204,9232,9233,9261],{},[15,9173,9174],{},"That's when it clicked: minimizing the cost is the same thing as finding the best fit."," It's not a coincidence, it's the definition. Choosing ",[80,9177,9179,9192],{"className":9178},[83],[80,9180,9182],{"className":9181},[87],[89,9183,9184],{"xmlns":91},[93,9185,9186,9190],{},[96,9187,9188],{},[102,9189,109],{},[145,9191,109],{"encoding":147},[80,9193,9195],{"className":9194,"ariaHidden":113},[152],[80,9196,9198,9201],{"className":9197},[156],[80,9199],{"className":9200,"style":334},[160],[80,9202,109],{"className":9203,"style":210},[165,169],[80,9205,9207,9220],{"className":9206},[83],[80,9208,9210],{"className":9209},[87],[89,9211,9212],{"xmlns":91},[93,9213,9214,9218],{},[96,9215,9216],{},[102,9217,117],{},[145,9219,117],{"encoding":147},[80,9221,9223],{"className":9222,"ariaHidden":113},[152],[80,9224,9226,9229],{"className":9225},[156],[80,9227],{"className":9228,"style":289},[160],[80,9230,117],{"className":9231},[165,169]," that minimize ",[80,9234,9236,9249],{"className":9235},[83],[80,9237,9239],{"className":9238},[87],[89,9240,9241],{"xmlns":91},[93,9242,9243,9247],{},[96,9244,9245],{},[102,9246,5606],{},[145,9248,5606],{"encoding":147},[80,9250,9252],{"className":9251,"ariaHidden":113},[152],[80,9253,9255,9258],{"className":9254},[156],[80,9256],{"className":9257,"style":8672},[160],[80,9259,5606],{"className":9260,"style":5632},[165,169]," is literally what we mean by training a model.",[11,9263,9264,9265,9316],{},"Try dragging the slider to ",[80,9266,9268,9286],{"className":9267},[83],[80,9269,9271],{"className":9270},[87],[89,9272,9273],{"xmlns":91},[93,9274,9275,9283],{},[96,9276,9277,9279,9281],{},[102,9278,109],{},[111,9280,130],{},[1321,9282,2071],{},[145,9284,9285],{"encoding":147},"w = 0",[80,9287,9289,9307],{"className":9288,"ariaHidden":113},[152],[80,9290,9292,9295,9298,9301,9304],{"className":9291},[156],[80,9293],{"className":9294,"style":334},[160],[80,9296,109],{"className":9297,"style":210},[165,169],[80,9299],{"className":9300,"style":247},[246],[80,9302,130],{"className":9303},[251],[80,9305],{"className":9306,"style":247},[246],[80,9308,9310,9313],{"className":9309},[156],[80,9311],{"className":9312,"style":1614},[160],[80,9314,2071],{"className":9315},[165],": the line goes flat (predicts the same price for any house size) and the cost spikes.",[294,9318,9320],{"id":9319},"the-real-world-doesnt-hand-you-2-perfect-points","The real world doesn't hand you 2 perfect points",[11,9322,9323],{},"So far, with 2 points and 2 parameters, I could get the cost to hit zero. That's rare. Let's swap the training set for a more realistic one, 6 houses, where the prices don't fall exactly on any single line (real noise, like similar houses selling for slightly different prices):",[521,9325,9326,9334],{},[524,9327,9328],{},[527,9329,9330,9332],{},[530,9331,3353],{"align":1884},[530,9333,3354],{"align":1884},[541,9335,9336,9343,9350,9357,9365,9373],{},[527,9337,9338,9340],{},[546,9339,2074],{"align":1884},[546,9341,9342],{"align":1884},"250",[527,9344,9345,9348],{},[546,9346,9347],{"align":1884},"1.7",[546,9349,1646],{"align":1884},[527,9351,9352,9354],{},[546,9353,2083],{"align":1884},[546,9355,9356],{"align":1884},"480",[527,9358,9359,9362],{},[546,9360,9361],{"align":1884},"2.5",[546,9363,9364],{"align":1884},"430",[527,9366,9367,9370],{},[546,9368,9369],{"align":1884},"3.0",[546,9371,9372],{"align":1884},"630",[527,9374,9375,9378],{},[546,9376,9377],{"align":1884},"3.2",[546,9379,9380],{"align":1884},"730",[11,9382,9383,9384,9435],{},"The open question: is there still some ",[80,9385,9387,9408],{"className":9386},[83],[80,9388,9390],{"className":9389},[87],[89,9391,9392],{"xmlns":91},[93,9393,9394,9406],{},[96,9395,9396,9398,9400,9402,9404],{},[111,9397,121],{"stretchy":120},[102,9399,109],{},[111,9401,114],{"separator":113},[102,9403,117],{},[111,9405,127],{"stretchy":120},[145,9407,4030],{"encoding":147},[80,9409,9411],{"className":9410,"ariaHidden":113},[152],[80,9412,9414,9417,9420,9423,9426,9429,9432],{"className":9413},[156],[80,9415],{"className":9416,"style":2316},[160],[80,9418,121],{"className":9419},[235],[80,9421,109],{"className":9422,"style":210},[165,169],[80,9424,114],{"className":9425},[214],[80,9427],{"className":9428,"style":268},[246],[80,9430,117],{"className":9431},[165,169],[80,9433,127],{"className":9434},[242]," that zeroes out the cost here?",[11,9437,9438,9439,3255,9467,3255,9469,9497],{},"Move both sliders below (now ",[80,9440,9442,9455],{"className":9441},[83],[80,9443,9445],{"className":9444},[87],[89,9446,9447],{"xmlns":91},[93,9448,9449,9453],{},[96,9450,9451],{},[102,9452,109],{},[145,9454,109],{"encoding":147},[80,9456,9458],{"className":9457,"ariaHidden":113},[152],[80,9459,9461,9464],{"className":9460},[156],[80,9462],{"className":9463,"style":334},[160],[80,9465,109],{"className":9466,"style":210},[165,169],[15,9468,2876],{},[80,9470,9472,9485],{"className":9471},[83],[80,9473,9475],{"className":9474},[87],[89,9476,9477],{"xmlns":91},[93,9478,9479,9483],{},[96,9480,9481],{},[102,9482,117],{},[145,9484,117],{"encoding":147},[80,9486,9488],{"className":9487,"ariaHidden":113},[152],[80,9489,9491,9494],{"className":9490},[156],[80,9492],{"className":9493,"style":289},[160],[80,9495,117],{"className":9496},[165,169]," at once) and try to reach the center of the heatmap, where the cost is lowest. Switch to the 3D view and rotate the surface with your finger or mouse to feel the shape of the \"valley\".",[9499,9500],"cost-explorer",{":b-max":4708,":b-min":9501,":initial-b":2071,":initial-w":9016,":w-max":9017,":w-min":2071,":x-train":9502,":y-train":9503,"b-label":117,"dataLabel":4589,"prediction-label":4591,"w-label":109,"x-label":3353,"y-label":3354},"-200","[1.0, 1.7, 2.0, 2.5, 3.0, 3.2]","[250, 300, 480, 430, 630, 730]",[11,9505,9506,9507,9510,9511,2338,9565,9617],{},"You probably noticed: this time the cost ",[15,9508,9509],{},"doesn't"," hit zero. The best you can get lands around ",[80,9512,9514,9534],{"className":9513},[83],[80,9515,9517],{"className":9516},[87],[89,9518,9519],{"xmlns":91},[93,9520,9521,9531],{},[96,9522,9523,9525,9528],{},[102,9524,109],{},[111,9526,9527],{},"≈",[1321,9529,9530],{},"209",[145,9532,9533],{"encoding":147},"w \\approx 209",[80,9535,9537,9556],{"className":9536,"ariaHidden":113},[152],[80,9538,9540,9544,9547,9550,9553],{"className":9539},[156],[80,9541],{"className":9542,"style":9543},[160],"height:0.4831em;",[80,9545,109],{"className":9546,"style":210},[165,169],[80,9548],{"className":9549,"style":247},[246],[80,9551,9527],{"className":9552},[251],[80,9554],{"className":9555,"style":247},[246],[80,9557,9559,9562],{"className":9558},[156],[80,9560],{"className":9561,"style":1614},[160],[80,9563,9530],{"className":9564},[165],[80,9566,9568,9587],{"className":9567},[83],[80,9569,9571],{"className":9570},[87],[89,9572,9573],{"xmlns":91},[93,9574,9575,9584],{},[96,9576,9577,9579,9581],{},[102,9578,117],{},[111,9580,9527],{},[1321,9582,9583],{},"2.4",[145,9585,9586],{"encoding":147},"b \\approx 2.4",[80,9588,9590,9608],{"className":9589,"ariaHidden":113},[152],[80,9591,9593,9596,9599,9602,9605],{"className":9592},[156],[80,9594],{"className":9595,"style":289},[160],[80,9597,117],{"className":9598},[165,169],[80,9600],{"className":9601,"style":247},[246],[80,9603,9527],{"className":9604},[251],[80,9606],{"className":9607,"style":247},[246],[80,9609,9611,9614],{"className":9610},[156],[80,9612],{"className":9613,"style":1614},[160],[80,9615,9583],{"className":9616},[165],", with a cost around 1736. Why? Because these 6 houses aren't aligned. No single line passes exactly through all of them. The best fit is the one that leaves the smallest possible total squared error, and that's exactly what the cost function defines as \"the best\".",[11,9619,9620,9621,9624,9625,9629],{},"Hold on to this: ",[15,9622,9623],{},"a nonzero minimum cost is the normal case",", not a bug. Zero cost tends to happen when you have too little data (like our 2-point case) or when the model memorized the data instead of learning its pattern (that has a name, ",[7103,9626,9628],{"definition":9627},"when a model memorizes the training data instead of learning its general pattern, and so performs poorly on new data","overfitting",", but that's a topic for further down the course).",[294,9631,9633],{"id":9632},"why-the-surface-is-always-a-bowl","Why the surface is always a bowl",[11,9635,9636,9637,9641],{},"Rotating the 3D surface above, you probably noticed it always looks like a soup bowl, a single valley, no fake peaks along the way. That's not a coincidence of our specific example, it's a direct consequence of squaring the error in the formula. Any time you square something, the result is a ",[7103,9638,9640],{"definition":9639},"a bowl-shaped surface with a single valley, no fake minima where a search algorithm could get stuck","convex"," surface.",[11,9643,9644,9645,9673,9674,9702,9703,9815],{},"To see this more cleanly, without the scale distortion the real data brings (notice ",[80,9646,9648,9661],{"className":9647},[83],[80,9649,9651],{"className":9650},[87],[89,9652,9653],{"xmlns":91},[93,9654,9655,9659],{},[96,9656,9657],{},[102,9658,109],{},[145,9660,109],{"encoding":147},[80,9662,9664],{"className":9663,"ariaHidden":113},[152],[80,9665,9667,9670],{"className":9666},[156],[80,9668],{"className":9669,"style":334},[160],[80,9671,109],{"className":9672,"style":210},[165,169]," ranges from 0 to 400 and ",[80,9675,9677,9690],{"className":9676},[83],[80,9678,9680],{"className":9679},[87],[89,9681,9682],{"xmlns":91},[93,9683,9684,9688],{},[96,9685,9686],{},[102,9687,117],{},[145,9689,117],{"encoding":147},[80,9691,9693],{"className":9692,"ariaHidden":113},[152],[80,9694,9696,9699],{"className":9695},[156],[80,9697],{"className":9698,"style":289},[160],[80,9700,117],{"className":9701},[165,169]," from -200 to 200 in the chart above, which stretches the valley), here's the idealized version of the same shape, just ",[80,9704,9706,9732],{"className":9705},[83],[80,9707,9709],{"className":9708},[87],[89,9710,9711],{"xmlns":91},[93,9712,9713,9729],{},[96,9714,9715,9721,9723],{},[726,9716,9717,9719],{},[102,9718,109],{},[1321,9720,1323],{},[111,9722,141],{},[726,9724,9725,9727],{},[102,9726,117],{},[1321,9728,1323],{},[145,9730,9731],{"encoding":147},"w^2 + b^2",[80,9733,9735,9780],{"className":9734,"ariaHidden":113},[152],[80,9736,9738,9742,9771,9774,9777],{"className":9737},[156],[80,9739],{"className":9740,"style":9741},[160],"height:0.8974em;vertical-align:-0.0833em;",[80,9743,9745,9748],{"className":9744},[165],[80,9746,109],{"className":9747,"style":210},[165,169],[80,9749,9751],{"className":9750},[174],[80,9752,9754],{"className":9753},[178],[80,9755,9757],{"className":9756},[183],[80,9758,9760],{"className":9759,"style":1407},[187],[80,9761,9762,9765],{"style":772},[80,9763],{"className":9764,"style":196},[195],[80,9766,9768],{"className":9767},[200,201,202,203],[80,9769,1323],{"className":9770},[165,203],[80,9772],{"className":9773,"style":275},[246],[80,9775,141],{"className":9776},[279],[80,9778],{"className":9779,"style":275},[246],[80,9781,9783,9786],{"className":9782},[156],[80,9784],{"className":9785,"style":1407},[160],[80,9787,9789,9792],{"className":9788},[165],[80,9790,117],{"className":9791},[165,169],[80,9793,9795],{"className":9794},[174],[80,9796,9798],{"className":9797},[178],[80,9799,9801],{"className":9800},[183],[80,9802,9804],{"className":9803,"style":1407},[187],[80,9805,9806,9809],{"style":772},[80,9807],{"className":9808,"style":196},[195],[80,9810,9812],{"className":9811},[200,201,202,203],[80,9813,1323],{"className":9814},[165,203],", with both axes on the same scale:",[9817,9818],"cost-surface3d",{":b-max":9819,":b-min":9820,":w-max":9819,":w-min":9820},"20","-20",[11,9822,9823,9824,9827],{},"Why does this shape matter so much? Because a convex surface guarantees there's only ",[15,9825,9826],{},"one"," minimum, the global minimum. There's no hidden valley elsewhere for a search algorithm to fall into and get stuck, mistakenly thinking it already reached the bottom when it hasn't. That guarantee is what makes the next step of the course reliable: an automatic way to walk down to the bottom of that bowl, without you having to drag a slider for the rest of your life.",[294,9829,6192],{"id":6191},[521,9831,9832,9842],{},[524,9833,9834],{},[527,9835,9836,9839],{},[530,9837,9838],{"align":532},"Concept",[530,9840,9841],{"align":532},"What we established",[541,9843,9844,9908,9916,9986,9994,10110],{},[527,9845,9846,9849],{},[546,9847,9848],{"align":532},"Cost function",[546,9850,9851,9907],{"align":532},[80,9852,9854,9877],{"className":9853},[83],[80,9855,9857],{"className":9856},[87],[89,9858,9859],{"xmlns":91},[93,9860,9861,9875],{},[96,9862,9863,9865,9867,9869,9871,9873],{},[102,9864,5606],{},[111,9866,121],{"stretchy":120},[102,9868,109],{},[111,9870,114],{"separator":113},[102,9872,117],{},[111,9874,127],{"stretchy":120},[145,9876,5619],{"encoding":147},[80,9878,9880],{"className":9879,"ariaHidden":113},[152],[80,9881,9883,9886,9889,9892,9895,9898,9901,9904],{"className":9882},[156],[80,9884],{"className":9885,"style":2316},[160],[80,9887,5606],{"className":9888,"style":5632},[165,169],[80,9890,121],{"className":9891},[235],[80,9893,109],{"className":9894,"style":210},[165,169],[80,9896,114],{"className":9897},[214],[80,9899],{"className":9900,"style":268},[246],[80,9902,117],{"className":9903},[165,169],[80,9905,127],{"className":9906},[242],", a number that measures how wrong the predictions are on the training data",[527,9909,9910,9913],{},[546,9911,9912],{"align":532},"Why squared",[546,9914,9915],{"align":532},"removes the sign of the error and heavily punishes big misses, plus guarantees a convex surface",[527,9917,9918,9955],{},[546,9919,9920,9921],{"align":532},"Why ",[80,9922,9924,9940],{"className":9923},[83],[80,9925,9927],{"className":9926},[87],[89,9928,9929],{"xmlns":91},[93,9930,9931,9937],{},[96,9932,9933,9935],{},[1321,9934,1323],{},[102,9936,322],{},[145,9938,9939],{"encoding":147},"2m",[80,9941,9943],{"className":9942,"ariaHidden":113},[152],[80,9944,9946,9949,9952],{"className":9945},[156],[80,9947],{"className":9948,"style":1614},[160],[80,9950,1323],{"className":9951},[165],[80,9953,322],{"className":9954},[165,169],[546,9956,9957,9985],{"align":532},[80,9958,9960,9973],{"className":9959},[83],[80,9961,9963],{"className":9962},[87],[89,9964,9965],{"xmlns":91},[93,9966,9967,9971],{},[96,9968,9969],{},[102,9970,322],{},[145,9972,322],{"encoding":147},[80,9974,9976],{"className":9975,"ariaHidden":113},[152],[80,9977,9979,9982],{"className":9978},[156],[80,9980],{"className":9981,"style":334},[160],[80,9983,322],{"className":9984},[165,169]," turns the sum into an average, and the 2 simplifies the math coming in the next post",[527,9987,9988,9991],{},[546,9989,9990],{"align":532},"Shape",[546,9992,9993],{"align":532},"a U curve (one parameter) and a soup bowl (two parameters)",[527,9995,9996,9999],{},[546,9997,9998],{"align":532},"Training the model",[546,10000,10001,10002,10053,10054],{"align":532},"means finding the pair ",[80,10003,10005,10026],{"className":10004},[83],[80,10006,10008],{"className":10007},[87],[89,10009,10010],{"xmlns":91},[93,10011,10012,10024],{},[96,10013,10014,10016,10018,10020,10022],{},[111,10015,121],{"stretchy":120},[102,10017,109],{},[111,10019,114],{"separator":113},[102,10021,117],{},[111,10023,127],{"stretchy":120},[145,10025,5561],{"encoding":147},[80,10027,10029],{"className":10028,"ariaHidden":113},[152],[80,10030,10032,10035,10038,10041,10044,10047,10050],{"className":10031},[156],[80,10033],{"className":10034,"style":2316},[160],[80,10036,121],{"className":10037},[235],[80,10039,109],{"className":10040,"style":210},[165,169],[80,10042,114],{"className":10043},[214],[80,10045],{"className":10046,"style":268},[246],[80,10048,117],{"className":10049},[165,169],[80,10051,127],{"className":10052},[242]," that minimizes ",[80,10055,10057,10080],{"className":10056},[83],[80,10058,10060],{"className":10059},[87],[89,10061,10062],{"xmlns":91},[93,10063,10064,10078],{},[96,10065,10066,10068,10070,10072,10074,10076],{},[102,10067,5606],{},[111,10069,121],{"stretchy":120},[102,10071,109],{},[111,10073,114],{"separator":113},[102,10075,117],{},[111,10077,127],{"stretchy":120},[145,10079,5619],{"encoding":147},[80,10081,10083],{"className":10082,"ariaHidden":113},[152],[80,10084,10086,10089,10092,10095,10098,10101,10104,10107],{"className":10085},[156],[80,10087],{"className":10088,"style":2316},[160],[80,10090,5606],{"className":10091,"style":5632},[165,169],[80,10093,121],{"className":10094},[235],[80,10096,109],{"className":10097,"style":210},[165,169],[80,10099,114],{"className":10100},[214],[80,10102],{"className":10103,"style":268},[246],[80,10105,117],{"className":10106},[165,169],[80,10108,127],{"className":10109},[242],[527,10111,10112,10115],{},[546,10113,10114],{"align":532},"Nonzero minimum cost",[546,10116,10117],{"align":532},"normal when the data has noise, not a sign something's wrong",[11,10119,10120,10124],{},[15,10121,10122,3774],{},[562,10123,6599],{"href":4070}," you've now felt firsthand what it's like to hunt for the bottom of the bowl by dragging sliders. It doesn't scale. The next post introduces gradient descent, an algorithm that uses the slope of the cost surface to walk down to the bottom on its own, with no guessing required from you.",[294,10126,6716],{"id":6715},[11,10128,10129,10130,10132,10133,10136,10137,1618],{},"Same real housing dataset from ",[562,10131,9082],{"href":6979}," (500 houses, ",[562,10134,6725],{"href":6722,"rel":10135},[6724],"). Now with a real ",[65,10138,10139],{},"compute_cost",[2611,10141,10143],{"className":2613,"code":10142,"language":2615,"meta":26,"style":26},"def compute_cost(x, y, w, b):\n    m = x.shape[0]\n    cost_sum = 0\n    for i in range(m):\n        f_wb = w * x[i] + b\n        cost_sum += (f_wb - y[i]) ** 2\n    return (1 \u002F (2 * m)) * cost_sum\n\n# x_sqft already in \"hundreds of sqft\", y_price already in \"thousands of dollars\"\n# (same 50-house sample from the last post)\n\nprint(\"cost with a bad guess, w=50, b=100:\", compute_cost(x_sqft, y_price, 50, 100))\nprint(\"cost near the optimal fit, w=116, b=399:\", compute_cost(x_sqft, y_price, 116, 399))\n",[65,10144,10145,10149,10153,10157,10161,10166,10171,10176,10180,10185,10190,10194,10199],{"__ignoreMap":26},[80,10146,10147],{"class":2620,"line":33},[80,10148,8690],{},[80,10150,10151],{"class":2620,"line":27},[80,10152,8749],{},[80,10154,10155],{"class":2620,"line":2631},[80,10156,8758],{},[80,10158,10159],{"class":2620,"line":2636},[80,10160,8767],{},[80,10162,10163],{"class":2620,"line":2642},[80,10164,10165],{},"        f_wb = w * x[i] + b\n",[80,10167,10168],{"class":2620,"line":2648},[80,10169,10170],{},"        cost_sum += (f_wb - y[i]) ** 2\n",[80,10172,10173],{"class":2620,"line":2654},[80,10174,10175],{},"    return (1 \u002F (2 * m)) * cost_sum\n",[80,10177,10178],{"class":2620,"line":2660},[80,10179,2657],{"emptyLinePlaceholder":32},[80,10181,10182],{"class":2620,"line":2666},[80,10183,10184],{},"# x_sqft already in \"hundreds of sqft\", y_price already in \"thousands of dollars\"\n",[80,10186,10187],{"class":2620,"line":2672},[80,10188,10189],{},"# (same 50-house sample from the last post)\n",[80,10191,10192],{"class":2620,"line":2677},[80,10193,2657],{"emptyLinePlaceholder":32},[80,10195,10196],{"class":2620,"line":2683},[80,10197,10198],{},"print(\"cost with a bad guess, w=50, b=100:\", compute_cost(x_sqft, y_price, 50, 100))\n",[80,10200,10201],{"class":2620,"line":4155},[80,10202,10203],{},"print(\"cost near the optimal fit, w=116, b=399:\", compute_cost(x_sqft, y_price, 116, 399))\n",[46,10205,10206],{},[11,10207,10208,3255,10210,10213,10214],{},[15,10209,2693],{},[65,10211,10212],{},"cost with a bad guess, w=50, b=100: 90317.6"," \u002F ",[65,10215,10216],{},"cost near the optimal fit, w=116, b=399: 5189.6",[11,10218,10219],{},"Almost 20 times less cost just from picking better parameters. Move the sliders yourself and try to get close to that ~5189 (notice: you can't hit zero, that really is the real minimum, real data has noise):",[10221,10222],"housing-cost-explorer",{"dataLabel":6876,"prediction-label":4591,"x-label":6877,"y-label":3354},[6949,10224,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":10226},[10227,10228,10229,10230,10231,10232,10233,10234],{"id":7671,"depth":27,"text":7672},{"id":7682,"depth":27,"text":7683},{"id":8679,"depth":27,"text":8680},{"id":8824,"depth":27,"text":8825},{"id":9319,"depth":27,"text":9320},{"id":9632,"depth":27,"text":9633},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716},"How to put a single number on how wrong a model is: J(w,b), why squared error, why divide by 2m, and why the cost surface is always a bowl.",{},{"title":6990,"description":10235},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab03-cost-function",[10240,10241,6986],"cost-function","squared-error","Qjmvwr-oCFxG6bDlDBH6R9_fNvaKZWiy6I6PwELBaBg",{"id":10244,"title":10245,"body":10246,"cover":3,"date":6976,"description":14665,"extension":30,"meta":14666,"navigation":32,"order":2631,"path":4070,"playlist":6980,"seo":14667,"status":36,"stem":14668,"tags":14669,"__hash__":14672},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab04-gradient-descent.md","Optional Lab: Gradient Descent",{"type":8,"value":10247,"toc":14651},[10248,10254,10521,10527,10893,10897,10903,11056,11060,11071,11427,11459,11463,11466,12048,12576,12798,12838,12959,12961,13099,13102,13242,13246,13249,13354,13359,13363,13423,13460,13492,13554,13557,13561,13568,13572,13575,13837,13893,13899,13902,13905,13911,13965,13969,14132,14135,14201,14233,14236,14240,14486,14488,14508,14522,14524,14535,14574,14583,14586,14589,14649],[11,10249,10250],{},[57,10251],{"alt":10252,"src":10253},"An Olympic podium meme: in the first five panels, the athlete celebrates the gold medal next to a graph of a nicely-behaved, single-bottomed bowl. In the last panel, the real podium shows up with a graph full of bumpy valleys instead, and who takes gold, silver, or bronze depends on which valley each one fell into","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Flab04-gradient-descent\u002Fmeme-gradient-descent.jpg",[11,10255,10256,10257,3255,10260,10408,10409,504,10412,10468,10469,10520],{},"Quick recap of the last two posts: you ",[562,10258,10259],{"href":6979},"built the model",[80,10261,10263,10302],{"className":10262},[83],[80,10264,10266],{"className":10265},[87],[89,10267,10268],{"xmlns":91},[93,10269,10270,10300],{},[96,10271,10272,10284,10286,10288,10290,10292,10294,10296,10298],{},[99,10273,10274,10276],{},[102,10275,104],{},[96,10277,10278,10280,10282],{},[102,10279,109],{},[111,10281,114],{"separator":113},[102,10283,117],{},[111,10285,121],{"stretchy":120},[102,10287,124],{},[111,10289,127],{"stretchy":120},[111,10291,130],{},[102,10293,109],{},[102,10295,124],{},[111,10297,141],{},[102,10299,117],{},[145,10301,387],{"encoding":147},[80,10303,10305,10378,10399],{"className":10304,"ariaHidden":113},[152],[80,10306,10308,10311,10360,10363,10366,10369,10372,10375],{"className":10307},[156],[80,10309],{"className":10310,"style":161},[160],[80,10312,10314,10317],{"className":10313},[165],[80,10315,104],{"className":10316,"style":170},[165,169],[80,10318,10320],{"className":10319},[174],[80,10321,10323,10352],{"className":10322},[178,179],[80,10324,10326,10349],{"className":10325},[183],[80,10327,10329],{"className":10328,"style":188},[187],[80,10330,10331,10334],{"style":191},[80,10332],{"className":10333,"style":196},[195],[80,10335,10337],{"className":10336},[200,201,202,203],[80,10338,10340,10343,10346],{"className":10339},[165,203],[80,10341,109],{"className":10342,"style":210},[165,169,203],[80,10344,114],{"className":10345},[214,203],[80,10347,117],{"className":10348},[165,169,203],[80,10350,222],{"className":10351},[221],[80,10353,10355],{"className":10354},[183],[80,10356,10358],{"className":10357,"style":229},[187],[80,10359],{},[80,10361,121],{"className":10362},[235],[80,10364,124],{"className":10365},[165,169],[80,10367,127],{"className":10368},[242],[80,10370],{"className":10371,"style":247},[246],[80,10373,130],{"className":10374},[251],[80,10376],{"className":10377,"style":247},[246],[80,10379,10381,10384,10387,10390,10393,10396],{"className":10380},[156],[80,10382],{"className":10383,"style":261},[160],[80,10385,109],{"className":10386,"style":210},[165,169],[80,10388,124],{"className":10389},[165,169],[80,10391],{"className":10392,"style":275},[246],[80,10394,141],{"className":10395},[279],[80,10397],{"className":10398,"style":275},[246],[80,10400,10402,10405],{"className":10401},[156],[80,10403],{"className":10404,"style":289},[160],[80,10406,117],{"className":10407},[165,169],", then ",[562,10410,10411],{"href":4066},"built a way to measure how wrong it is",[80,10413,10415,10438],{"className":10414},[83],[80,10416,10418],{"className":10417},[87],[89,10419,10420],{"xmlns":91},[93,10421,10422,10436],{},[96,10423,10424,10426,10428,10430,10432,10434],{},[102,10425,5606],{},[111,10427,121],{"stretchy":120},[102,10429,109],{},[111,10431,114],{"separator":113},[102,10433,117],{},[111,10435,127],{"stretchy":120},[145,10437,5619],{"encoding":147},[80,10439,10441],{"className":10440,"ariaHidden":113},[152],[80,10442,10444,10447,10450,10453,10456,10459,10462,10465],{"className":10443},[156],[80,10445],{"className":10446,"style":2316},[160],[80,10448,5606],{"className":10449,"style":5632},[165,169],[80,10451,121],{"className":10452},[235],[80,10454,109],{"className":10455,"style":210},[165,169],[80,10457,114],{"className":10458},[214],[80,10460],{"className":10461,"style":268},[246],[80,10463,117],{"className":10464},[165,169],[80,10466,127],{"className":10467},[242],". But to find the best ",[80,10470,10472,10493],{"className":10471},[83],[80,10473,10475],{"className":10474},[87],[89,10476,10477],{"xmlns":91},[93,10478,10479,10491],{},[96,10480,10481,10483,10485,10487,10489],{},[111,10482,121],{"stretchy":120},[102,10484,109],{},[111,10486,114],{"separator":113},[102,10488,117],{},[111,10490,127],{"stretchy":120},[145,10492,5561],{"encoding":147},[80,10494,10496],{"className":10495,"ariaHidden":113},[152],[80,10497,10499,10502,10505,10508,10511,10514,10517],{"className":10498},[156],[80,10500],{"className":10501,"style":2316},[160],[80,10503,121],{"className":10504},[235],[80,10506,109],{"className":10507,"style":210},[165,169],[80,10509,114],{"className":10510},[214],[80,10512],{"className":10513,"style":268},[246],[80,10515,117],{"className":10516},[165,169],[80,10518,127],{"className":10519},[242]," you were still doing the most primitive thing possible: dragging a slider and watching the number drop. That works for 2 points. For a real dataset, with thousands of examples and dozens of parameters, it's impossible.",[11,10522,10523,10524,1618],{},"This post closes the loop with the algorithm that does that search on its own: ",[7103,10525,4071],{"definition":10526},"an algorithm that adjusts a model's parameters step by step, always in the direction that most reduces the cost, until it stops near the minimum",[11,10528,10529],{},[80,10530,10532,10618],{"className":10531},[83],[80,10533,10535],{"className":10534},[87],[89,10536,10537],{"xmlns":91},[93,10538,10539,10615],{},[96,10540,10541,10543,10545,10547,10549,10552,10578,10581,10583,10585,10587,10589,10591],{},[102,10542,109],{},[111,10544,130],{},[102,10546,109],{},[111,10548,4643],{},[102,10550,10551],{},"α",[4625,10553,10554,10572],{},[96,10555,10556,10560,10562,10564,10566,10568,10570],{},[102,10557,10559],{"mathvariant":10558},"normal","∂",[102,10561,5606],{},[111,10563,121],{"stretchy":120},[102,10565,109],{},[111,10567,114],{"separator":113},[102,10569,117],{},[111,10571,127],{"stretchy":120},[96,10573,10574,10576],{},[102,10575,10559],{"mathvariant":10558},[102,10577,109],{},[246,10579],{"width":10580},"2em",[102,10582,117],{},[111,10584,130],{},[102,10586,117],{},[111,10588,4643],{},[102,10590,10551],{},[4625,10592,10593,10609],{},[96,10594,10595,10597,10599,10601,10603,10605,10607],{},[102,10596,10559],{"mathvariant":10558},[102,10598,5606],{},[111,10600,121],{"stretchy":120},[102,10602,109],{},[111,10604,114],{"separator":113},[102,10606,117],{},[111,10608,127],{"stretchy":120},[96,10610,10611,10613],{},[102,10612,10559],{"mathvariant":10558},[102,10614,117],{},[145,10616,10617],{"encoding":147},"w = w - \\alpha \\frac{\\partial J(w,b)}{\\partial w} \\qquad b = b - \\alpha \\frac{\\partial J(w,b)}{\\partial b}",[80,10619,10621,10639,10657,10776,10795],{"className":10620,"ariaHidden":113},[152],[80,10622,10624,10627,10630,10633,10636],{"className":10623},[156],[80,10625],{"className":10626,"style":334},[160],[80,10628,109],{"className":10629,"style":210},[165,169],[80,10631],{"className":10632,"style":247},[246],[80,10634,130],{"className":10635},[251],[80,10637],{"className":10638,"style":247},[246],[80,10640,10642,10645,10648,10651,10654],{"className":10641},[156],[80,10643],{"className":10644,"style":261},[160],[80,10646,109],{"className":10647,"style":210},[165,169],[80,10649],{"className":10650,"style":275},[246],[80,10652,4643],{"className":10653},[279],[80,10655],{"className":10656,"style":275},[246],[80,10658,10660,10664,10668,10760,10764,10767,10770,10773],{"className":10659},[156],[80,10661],{"className":10662,"style":10663},[160],"height:1.355em;vertical-align:-0.345em;",[80,10665,10551],{"className":10666,"style":10667},[165,169],"margin-right:0.0037em;",[80,10669,10671,10674,10757],{"className":10670},[165],[80,10672],{"className":10673},[235,4746],[80,10675,10677],{"className":10676},[4625],[80,10678,10680,10749],{"className":10679},[178,179],[80,10681,10683,10746],{"className":10682},[183],[80,10684,10687,10705,10713],{"className":10685,"style":10686},[187],"height:1.01em;",[80,10688,10689,10692],{"style":5013},[80,10690],{"className":10691,"style":4766},[195],[80,10693,10695],{"className":10694},[200,201,202,203],[80,10696,10698,10702],{"className":10697},[165,203],[80,10699,10559],{"className":10700,"style":10701},[165,203],"margin-right:0.0556em;",[80,10703,109],{"className":10704,"style":210},[165,169,203],[80,10706,10707,10710],{"style":4859},[80,10708],{"className":10709,"style":4766},[195],[80,10711],{"className":10712,"style":4867},[4866],[80,10714,10716,10719],{"style":10715},"top:-3.485em;",[80,10717],{"className":10718,"style":4766},[195],[80,10720,10722],{"className":10721},[200,201,202,203],[80,10723,10725,10728,10731,10734,10737,10740,10743],{"className":10724},[165,203],[80,10726,10559],{"className":10727,"style":10701},[165,203],[80,10729,5606],{"className":10730,"style":5632},[165,169,203],[80,10732,121],{"className":10733},[235,203],[80,10735,109],{"className":10736,"style":210},[165,169,203],[80,10738,114],{"className":10739},[214,203],[80,10741,117],{"className":10742},[165,169,203],[80,10744,127],{"className":10745},[242,203],[80,10747,222],{"className":10748},[221],[80,10750,10752],{"className":10751},[183],[80,10753,10755],{"className":10754,"style":7390},[187],[80,10756],{},[80,10758],{"className":10759},[242,4746],[80,10761],{"className":10762,"style":10763},[246],"margin-right:2em;",[80,10765,117],{"className":10766},[165,169],[80,10768],{"className":10769,"style":247},[246],[80,10771,130],{"className":10772},[251],[80,10774],{"className":10775,"style":247},[246],[80,10777,10779,10783,10786,10789,10792],{"className":10778},[156],[80,10780],{"className":10781,"style":10782},[160],"height:0.7778em;vertical-align:-0.0833em;",[80,10784,117],{"className":10785},[165,169],[80,10787],{"className":10788,"style":275},[246],[80,10790,4643],{"className":10791},[279],[80,10793],{"className":10794,"style":275},[246],[80,10796,10798,10801,10804],{"className":10797},[156],[80,10799],{"className":10800,"style":10663},[160],[80,10802,10551],{"className":10803,"style":10667},[165,169],[80,10805,10807,10810,10890],{"className":10806},[165],[80,10808],{"className":10809},[235,4746],[80,10811,10813],{"className":10812},[4625],[80,10814,10816,10882],{"className":10815},[178,179],[80,10817,10819,10879],{"className":10818},[183],[80,10820,10822,10839,10847],{"className":10821,"style":10686},[187],[80,10823,10824,10827],{"style":5013},[80,10825],{"className":10826,"style":4766},[195],[80,10828,10830],{"className":10829},[200,201,202,203],[80,10831,10833,10836],{"className":10832},[165,203],[80,10834,10559],{"className":10835,"style":10701},[165,203],[80,10837,117],{"className":10838},[165,169,203],[80,10840,10841,10844],{"style":4859},[80,10842],{"className":10843,"style":4766},[195],[80,10845],{"className":10846,"style":4867},[4866],[80,10848,10849,10852],{"style":10715},[80,10850],{"className":10851,"style":4766},[195],[80,10853,10855],{"className":10854},[200,201,202,203],[80,10856,10858,10861,10864,10867,10870,10873,10876],{"className":10857},[165,203],[80,10859,10559],{"className":10860,"style":10701},[165,203],[80,10862,5606],{"className":10863,"style":5632},[165,169,203],[80,10865,121],{"className":10866},[235,203],[80,10868,109],{"className":10869,"style":210},[165,169,203],[80,10871,114],{"className":10872},[214,203],[80,10874,117],{"className":10875},[165,169,203],[80,10877,127],{"className":10878},[242,203],[80,10880,222],{"className":10881},[221],[80,10883,10885],{"className":10884},[183],[80,10886,10888],{"className":10887,"style":7390},[187],[80,10889],{},[80,10891],{"className":10892},[242,4746],[294,10894,10896],{"id":10895},"the-idea-in-one-sentence","The idea in one sentence",[11,10898,10899,10900,10902],{},"Remember the soup bowl from ",[562,10901,9082],{"href":4066},"? Gradient descent is literally that: you start at some point on the bowl and take steps downhill, always in the direction that descends fastest, until you land near the bottom.",[11,10904,10905,10906,11020,11021,11050,11051,11055],{},"The \"feeling which direction descends fastest\" part is the derivative's job, that ",[80,10907,10909,10935],{"className":10908},[83],[80,10910,10912],{"className":10911},[87],[89,10913,10914],{"xmlns":91},[93,10915,10916,10932],{},[96,10917,10918],{},[4625,10919,10920,10926],{},[96,10921,10922,10924],{},[102,10923,10559],{"mathvariant":10558},[102,10925,5606],{},[96,10927,10928,10930],{},[102,10929,10559],{"mathvariant":10558},[102,10931,109],{},[145,10933,10934],{"encoding":147},"\\frac{\\partial J}{\\partial w}",[80,10936,10938],{"className":10937,"ariaHidden":113},[152],[80,10939,10941,10945],{"className":10940},[156],[80,10942],{"className":10943,"style":10944},[160],"height:1.2251em;vertical-align:-0.345em;",[80,10946,10948,10951,11017],{"className":10947},[165],[80,10949],{"className":10950},[235,4746],[80,10952,10954],{"className":10953},[4625],[80,10955,10957,11009],{"className":10956},[178,179],[80,10958,10960,11006],{"className":10959},[183],[80,10961,10964,10981,10989],{"className":10962,"style":10963},[187],"height:0.8801em;",[80,10965,10966,10969],{"style":5013},[80,10967],{"className":10968,"style":4766},[195],[80,10970,10972],{"className":10971},[200,201,202,203],[80,10973,10975,10978],{"className":10974},[165,203],[80,10976,10559],{"className":10977,"style":10701},[165,203],[80,10979,109],{"className":10980,"style":210},[165,169,203],[80,10982,10983,10986],{"style":4859},[80,10984],{"className":10985,"style":4766},[195],[80,10987],{"className":10988,"style":4867},[4866],[80,10990,10991,10994],{"style":5042},[80,10992],{"className":10993,"style":4766},[195],[80,10995,10997],{"className":10996},[200,201,202,203],[80,10998,11000,11003],{"className":10999},[165,203],[80,11001,10559],{"className":11002,"style":10701},[165,203],[80,11004,5606],{"className":11005,"style":5632},[165,169,203],[80,11007,222],{"className":11008},[221],[80,11010,11012],{"className":11011},[183],[80,11013,11015],{"className":11014,"style":7390},[187],[80,11016],{},[80,11018],{"className":11019},[242,4746]," in the formula. It's the cost surface's slope at that specific point. And ",[80,11022,11024,11038],{"className":11023},[83],[80,11025,11027],{"className":11026},[87],[89,11028,11029],{"xmlns":91},[93,11030,11031,11035],{},[96,11032,11033],{},[102,11034,10551],{},[145,11036,11037],{"encoding":147},"\\alpha",[80,11039,11041],{"className":11040,"ariaHidden":113},[152],[80,11042,11044,11047],{"className":11043},[156],[80,11045],{"className":11046,"style":334},[160],[80,11048,10551],{"className":11049,"style":10667},[165,169]," (",[7103,11052,11054],{"definition":11053},"the greek letter used for the learning rate, the size of the step the algorithm takes on each iteration","alpha",") is how big a step you take each time.",[294,11057,11059],{"id":11058},"why-subtract-the-derivative","Why subtract the derivative",[11,11061,11062,11063,11066,11067,11070],{},"The derivative points toward where the cost ",[15,11064,11065],{},"increases",". Since you want a smaller cost, you walk in the ",[15,11068,11069],{},"opposite"," direction. Hence the minus sign in the formula.",[521,11072,11073,11225],{},[524,11074,11075],{},[527,11076,11077,11080,11083],{},[530,11078,11079],{"align":532},"Situation",[530,11081,11082],{"align":1884},"Sign of the derivative",[530,11084,11085,11086],{"align":532},"What happens to ",[80,11087,11089,11121],{"className":11088},[83],[80,11090,11092],{"className":11091},[87],[89,11093,11094],{"xmlns":91},[93,11095,11096,11118],{},[96,11097,11098,11100,11102,11104],{},[102,11099,109],{},[111,11101,4643],{},[102,11103,10551],{},[4625,11105,11106,11112],{},[96,11107,11108,11110],{},[102,11109,10559],{"mathvariant":10558},[102,11111,5606],{},[96,11113,11114,11116],{},[102,11115,10559],{"mathvariant":10558},[102,11117,109],{},[145,11119,11120],{"encoding":147},"w - \\alpha \\frac{\\partial J}{\\partial w}",[80,11122,11124,11142],{"className":11123,"ariaHidden":113},[152],[80,11125,11127,11130,11133,11136,11139],{"className":11126},[156],[80,11128],{"className":11129,"style":261},[160],[80,11131,109],{"className":11132,"style":210},[165,169],[80,11134],{"className":11135,"style":275},[246],[80,11137,4643],{"className":11138},[279],[80,11140],{"className":11141,"style":275},[246],[80,11143,11145,11148,11151],{"className":11144},[156],[80,11146],{"className":11147,"style":10944},[160],[80,11149,10551],{"className":11150,"style":10667},[165,169],[80,11152,11154,11157,11222],{"className":11153},[165],[80,11155],{"className":11156},[235,4746],[80,11158,11160],{"className":11159},[4625],[80,11161,11163,11214],{"className":11162},[178,179],[80,11164,11166,11211],{"className":11165},[183],[80,11167,11169,11186,11194],{"className":11168,"style":10963},[187],[80,11170,11171,11174],{"style":5013},[80,11172],{"className":11173,"style":4766},[195],[80,11175,11177],{"className":11176},[200,201,202,203],[80,11178,11180,11183],{"className":11179},[165,203],[80,11181,10559],{"className":11182,"style":10701},[165,203],[80,11184,109],{"className":11185,"style":210},[165,169,203],[80,11187,11188,11191],{"style":4859},[80,11189],{"className":11190,"style":4766},[195],[80,11192],{"className":11193,"style":4867},[4866],[80,11195,11196,11199],{"style":5042},[80,11197],{"className":11198,"style":4766},[195],[80,11200,11202],{"className":11201},[200,201,202,203],[80,11203,11205,11208],{"className":11204},[165,203],[80,11206,10559],{"className":11207,"style":10701},[165,203],[80,11209,5606],{"className":11210,"style":5632},[165,169,203],[80,11212,222],{"className":11213},[221],[80,11215,11217],{"className":11216},[183],[80,11218,11220],{"className":11219,"style":7390},[187],[80,11221],{},[80,11223],{"className":11224},[242,4746],[541,11226,11227,11294,11361],{},[527,11228,11229,11260,11263],{},[546,11230,11231,11259],{"align":532},[80,11232,11234,11247],{"className":11233},[83],[80,11235,11237],{"className":11236},[87],[89,11238,11239],{"xmlns":91},[93,11240,11241,11245],{},[96,11242,11243],{},[102,11244,109],{},[145,11246,109],{"encoding":147},[80,11248,11250],{"className":11249,"ariaHidden":113},[152],[80,11251,11253,11256],{"className":11252},[156],[80,11254],{"className":11255,"style":334},[160],[80,11257,109],{"className":11258,"style":210},[165,169]," is to the right of the minimum",[546,11261,11262],{"align":1884},"positive",[546,11264,11265,11293],{"align":532},[80,11266,11268,11281],{"className":11267},[83],[80,11269,11271],{"className":11270},[87],[89,11272,11273],{"xmlns":91},[93,11274,11275,11279],{},[96,11276,11277],{},[102,11278,109],{},[145,11280,109],{"encoding":147},[80,11282,11284],{"className":11283,"ariaHidden":113},[152],[80,11285,11287,11290],{"className":11286},[156],[80,11288],{"className":11289,"style":334},[160],[80,11291,109],{"className":11292,"style":210},[165,169]," decreases, moves left",[527,11295,11296,11327,11330],{},[546,11297,11298,11326],{"align":532},[80,11299,11301,11314],{"className":11300},[83],[80,11302,11304],{"className":11303},[87],[89,11305,11306],{"xmlns":91},[93,11307,11308,11312],{},[96,11309,11310],{},[102,11311,109],{},[145,11313,109],{"encoding":147},[80,11315,11317],{"className":11316,"ariaHidden":113},[152],[80,11318,11320,11323],{"className":11319},[156],[80,11321],{"className":11322,"style":334},[160],[80,11324,109],{"className":11325,"style":210},[165,169]," is to the left of the minimum",[546,11328,11329],{"align":1884},"negative",[546,11331,11332,11360],{"align":532},[80,11333,11335,11348],{"className":11334},[83],[80,11336,11338],{"className":11337},[87],[89,11339,11340],{"xmlns":91},[93,11341,11342,11346],{},[96,11343,11344],{},[102,11345,109],{},[145,11347,109],{"encoding":147},[80,11349,11351],{"className":11350,"ariaHidden":113},[152],[80,11352,11354,11357],{"className":11353},[156],[80,11355],{"className":11356,"style":334},[160],[80,11358,109],{"className":11359,"style":210},[165,169]," increases, moves right",[527,11362,11363,11394,11396],{},[546,11364,11365,11393],{"align":532},[80,11366,11368,11381],{"className":11367},[83],[80,11369,11371],{"className":11370},[87],[89,11372,11373],{"xmlns":91},[93,11374,11375,11379],{},[96,11376,11377],{},[102,11378,109],{},[145,11380,109],{"encoding":147},[80,11382,11384],{"className":11383,"ariaHidden":113},[152],[80,11385,11387,11390],{"className":11386},[156],[80,11388],{"className":11389,"style":334},[160],[80,11391,109],{"className":11392,"style":210},[165,169]," is exactly at the minimum",[546,11395,2891],{"align":1884},[546,11397,11398,11426],{"align":532},[80,11399,11401,11414],{"className":11400},[83],[80,11402,11404],{"className":11403},[87],[89,11405,11406],{"xmlns":91},[93,11407,11408,11412],{},[96,11409,11410],{},[102,11411,109],{},[145,11413,109],{"encoding":147},[80,11415,11417],{"className":11416,"ariaHidden":113},[152],[80,11418,11420,11423],{"className":11419},[156],[80,11421],{"className":11422,"style":334},[160],[80,11424,109],{"className":11425,"style":210},[165,169]," stops changing, the algorithm halted on its own",[11,11428,11429,11430,11458],{},"Notice the last row: the algorithm doesn't need an \"if you've reached the minimum, stop\" check. It simply stops moving on its own, because the derivative hits zero. And since the derivative shrinks as you get closer to the bottom, the steps also get smaller on their own, even with a fixed ",[80,11431,11433,11446],{"className":11432},[83],[80,11434,11436],{"className":11435},[87],[89,11437,11438],{"xmlns":91},[93,11439,11440,11444],{},[96,11441,11442],{},[102,11443,10551],{},[145,11445,11037],{"encoding":147},[80,11447,11449],{"className":11448,"ariaHidden":113},[152],[80,11450,11452,11455],{"className":11451},[156],[80,11453],{"className":11454,"style":334},[160],[80,11456,10551],{"className":11457,"style":10667},[165,169],". That's a free property, not something you code separately.",[294,11460,11462],{"id":11461},"the-two-partial-derivatives","The two partial derivatives",[11,11464,11465],{},"For single-variable linear regression, the math works out to these two formulas (I didn't need to memorize the derivation, just get the pattern):",[11,11467,11468],{},[80,11469,11471,11595],{"className":11470},[83],[80,11472,11474],{"className":11473},[87],[89,11475,11476],{"xmlns":91},[93,11477,11478,11592],{},[96,11479,11480,11504,11506,11512,11532,11580],{},[4625,11481,11482,11498],{},[96,11483,11484,11486,11488,11490,11492,11494,11496],{},[102,11485,10559],{"mathvariant":10558},[102,11487,5606],{},[111,11489,121],{"stretchy":120},[102,11491,109],{},[111,11493,114],{"separator":113},[102,11495,117],{},[111,11497,127],{"stretchy":120},[96,11499,11500,11502],{},[102,11501,10559],{"mathvariant":10558},[102,11503,109],{},[111,11505,130],{},[4625,11507,11508,11510],{},[1321,11509,1583],{},[102,11511,322],{},[7202,11513,11514,11516,11524],{},[111,11515,7206],{},[96,11517,11518,11520,11522],{},[102,11519,736],{},[111,11521,130],{},[1321,11523,2071],{},[96,11525,11526,11528,11530],{},[102,11527,322],{},[111,11529,4643],{},[1321,11531,1583],{},[96,11533,11534,11536,11548,11550,11562,11564,11566,11578],{},[111,11535,121],{"fence":113},[99,11537,11538,11540],{},[102,11539,104],{},[96,11541,11542,11544,11546],{},[102,11543,109],{},[111,11545,114],{"separator":113},[102,11547,117],{},[111,11549,121],{"stretchy":120},[726,11551,11552,11554],{},[102,11553,124],{},[96,11555,11556,11558,11560],{},[111,11557,121],{"stretchy":120},[102,11559,736],{},[111,11561,127],{"stretchy":120},[111,11563,127],{"stretchy":120},[111,11565,4643],{},[726,11567,11568,11570],{},[102,11569,683],{},[96,11571,11572,11574,11576],{},[111,11573,121],{"stretchy":120},[102,11575,736],{},[111,11577,127],{"stretchy":120},[111,11579,127],{"fence":113},[726,11581,11582,11584],{},[102,11583,124],{},[96,11585,11586,11588,11590],{},[111,11587,121],{"stretchy":120},[102,11589,736],{},[111,11591,127],{"stretchy":120},[145,11593,11594],{"encoding":147},"\\frac{\\partial J(w,b)}{\\partial w} = \\frac{1}{m} \\sum_{i=0}^{m-1} \\left(f_{w,b}(x^{(i)}) - y^{(i)}\\right) x^{(i)}",[80,11596,11598,11702],{"className":11597,"ariaHidden":113},[152],[80,11599,11601,11604,11693,11696,11699],{"className":11600},[156],[80,11602],{"className":11603,"style":10663},[160],[80,11605,11607,11610,11690],{"className":11606},[165],[80,11608],{"className":11609},[235,4746],[80,11611,11613],{"className":11612},[4625],[80,11614,11616,11682],{"className":11615},[178,179],[80,11617,11619,11679],{"className":11618},[183],[80,11620,11622,11639,11647],{"className":11621,"style":10686},[187],[80,11623,11624,11627],{"style":5013},[80,11625],{"className":11626,"style":4766},[195],[80,11628,11630],{"className":11629},[200,201,202,203],[80,11631,11633,11636],{"className":11632},[165,203],[80,11634,10559],{"className":11635,"style":10701},[165,203],[80,11637,109],{"className":11638,"style":210},[165,169,203],[80,11640,11641,11644],{"style":4859},[80,11642],{"className":11643,"style":4766},[195],[80,11645],{"className":11646,"style":4867},[4866],[80,11648,11649,11652],{"style":10715},[80,11650],{"className":11651,"style":4766},[195],[80,11653,11655],{"className":11654},[200,201,202,203],[80,11656,11658,11661,11664,11667,11670,11673,11676],{"className":11657},[165,203],[80,11659,10559],{"className":11660,"style":10701},[165,203],[80,11662,5606],{"className":11663,"style":5632},[165,169,203],[80,11665,121],{"className":11666},[235,203],[80,11668,109],{"className":11669,"style":210},[165,169,203],[80,11671,114],{"className":11672},[214,203],[80,11674,117],{"className":11675},[165,169,203],[80,11677,127],{"className":11678},[242,203],[80,11680,222],{"className":11681},[221],[80,11683,11685],{"className":11684},[183],[80,11686,11688],{"className":11687,"style":7390},[187],[80,11689],{},[80,11691],{"className":11692},[242,4746],[80,11694],{"className":11695,"style":247},[246],[80,11697,130],{"className":11698},[251],[80,11700],{"className":11701,"style":247},[246],[80,11703,11705,11709,11777,11780,11849,11852,12007,12010],{"className":11704},[156],[80,11706],{"className":11707,"style":11708},[160],"height:1.304em;vertical-align:-0.35em;",[80,11710,11712,11715,11774],{"className":11711},[165],[80,11713],{"className":11714},[235,4746],[80,11716,11718],{"className":11717},[4625],[80,11719,11721,11766],{"className":11720},[178,179],[80,11722,11724,11763],{"className":11723},[183],[80,11725,11727,11741,11749],{"className":11726,"style":5010},[187],[80,11728,11729,11732],{"style":5013},[80,11730],{"className":11731,"style":4766},[195],[80,11733,11735],{"className":11734},[200,201,202,203],[80,11736,11738],{"className":11737},[165,203],[80,11739,322],{"className":11740},[165,169,203],[80,11742,11743,11746],{"style":4859},[80,11744],{"className":11745,"style":4766},[195],[80,11747],{"className":11748,"style":4867},[4866],[80,11750,11751,11754],{"style":5042},[80,11752],{"className":11753,"style":4766},[195],[80,11755,11757],{"className":11756},[200,201,202,203],[80,11758,11760],{"className":11759},[165,203],[80,11761,1583],{"className":11762},[165,203],[80,11764,222],{"className":11765},[221],[80,11767,11769],{"className":11768},[183],[80,11770,11772],{"className":11771,"style":7390},[187],[80,11773],{},[80,11775],{"className":11776},[242,4746],[80,11778],{"className":11779,"style":268},[246],[80,11781,11783,11786],{"className":11782},[7402],[80,11784,7206],{"className":11785,"style":7408},[7402,7406,7407],[80,11787,11789],{"className":11788},[174],[80,11790,11792,11841],{"className":11791},[178,179],[80,11793,11795,11838],{"className":11794},[183],[80,11796,11798,11818],{"className":11797,"style":7421},[187],[80,11799,11800,11803],{"style":7424},[80,11801],{"className":11802,"style":196},[195],[80,11804,11806],{"className":11805},[200,201,202,203],[80,11807,11809,11812,11815],{"className":11808},[165,203],[80,11810,736],{"className":11811},[165,169,203],[80,11813,130],{"className":11814},[251,203],[80,11816,2071],{"className":11817},[165,203],[80,11819,11820,11823],{"style":7445},[80,11821],{"className":11822,"style":196},[195],[80,11824,11826],{"className":11825},[200,201,202,203],[80,11827,11829,11832,11835],{"className":11828},[165,203],[80,11830,322],{"className":11831},[165,169,203],[80,11833,4643],{"className":11834},[279,203],[80,11836,1583],{"className":11837},[165,203],[80,11839,222],{"className":11840},[221],[80,11842,11844],{"className":11843},[183],[80,11845,11847],{"className":11846,"style":7473},[187],[80,11848],{},[80,11850],{"className":11851,"style":268},[246],[80,11853,11855,11861,11910,11913,11951,11954,11957,11960,11963,12001],{"className":11854},[7482],[80,11856,11858],{"className":11857,"style":7490},[235,7489],[80,11859,121],{"className":11860},[7494,4803],[80,11862,11864,11867],{"className":11863},[165],[80,11865,104],{"className":11866,"style":170},[165,169],[80,11868,11870],{"className":11869},[174],[80,11871,11873,11902],{"className":11872},[178,179],[80,11874,11876,11899],{"className":11875},[183],[80,11877,11879],{"className":11878,"style":188},[187],[80,11880,11881,11884],{"style":191},[80,11882],{"className":11883,"style":196},[195],[80,11885,11887],{"className":11886},[200,201,202,203],[80,11888,11890,11893,11896],{"className":11889},[165,203],[80,11891,109],{"className":11892,"style":210},[165,169,203],[80,11894,114],{"className":11895},[214,203],[80,11897,117],{"className":11898},[165,169,203],[80,11900,222],{"className":11901},[221],[80,11903,11905],{"className":11904},[183],[80,11906,11908],{"className":11907,"style":229},[187],[80,11909],{},[80,11911,121],{"className":11912},[235],[80,11914,11916,11919],{"className":11915},[165],[80,11917,124],{"className":11918},[165,169],[80,11920,11922],{"className":11921},[174],[80,11923,11925],{"className":11924},[178],[80,11926,11928],{"className":11927},[183],[80,11929,11931],{"className":11930,"style":751},[187],[80,11932,11933,11936],{"style":772},[80,11934],{"className":11935,"style":196},[195],[80,11937,11939],{"className":11938},[200,201,202,203],[80,11940,11942,11945,11948],{"className":11941},[165,203],[80,11943,121],{"className":11944},[235,203],[80,11946,736],{"className":11947},[165,169,203],[80,11949,127],{"className":11950},[242,203],[80,11952,127],{"className":11953},[242],[80,11955],{"className":11956,"style":275},[246],[80,11958,4643],{"className":11959},[279],[80,11961],{"className":11962,"style":275},[246],[80,11964,11966,11969],{"className":11965},[165],[80,11967,683],{"className":11968,"style":834},[165,169],[80,11970,11972],{"className":11971},[174],[80,11973,11975],{"className":11974},[178],[80,11976,11978],{"className":11977},[183],[80,11979,11981],{"className":11980,"style":751},[187],[80,11982,11983,11986],{"style":772},[80,11984],{"className":11985,"style":196},[195],[80,11987,11989],{"className":11988},[200,201,202,203],[80,11990,11992,11995,11998],{"className":11991},[165,203],[80,11993,121],{"className":11994},[235,203],[80,11996,736],{"className":11997},[165,169,203],[80,11999,127],{"className":12000},[242,203],[80,12002,12004],{"className":12003,"style":7490},[242,7489],[80,12005,127],{"className":12006},[7494,4803],[80,12008],{"className":12009,"style":268},[246],[80,12011,12013,12016],{"className":12012},[165],[80,12014,124],{"className":12015},[165,169],[80,12017,12019],{"className":12018},[174],[80,12020,12022],{"className":12021},[178],[80,12023,12025],{"className":12024},[183],[80,12026,12028],{"className":12027,"style":751},[187],[80,12029,12030,12033],{"style":772},[80,12031],{"className":12032,"style":196},[195],[80,12034,12036],{"className":12035},[200,201,202,203],[80,12037,12039,12042,12045],{"className":12038},[165,203],[80,12040,121],{"className":12041},[235,203],[80,12043,736],{"className":12044},[165,169,203],[80,12046,127],{"className":12047},[242,203],[11,12049,12050],{},[80,12051,12053,12165],{"className":12052},[83],[80,12054,12056],{"className":12055},[87],[89,12057,12058],{"xmlns":91},[93,12059,12060,12162],{},[96,12061,12062,12086,12088,12094,12114],{},[4625,12063,12064,12080],{},[96,12065,12066,12068,12070,12072,12074,12076,12078],{},[102,12067,10559],{"mathvariant":10558},[102,12069,5606],{},[111,12071,121],{"stretchy":120},[102,12073,109],{},[111,12075,114],{"separator":113},[102,12077,117],{},[111,12079,127],{"stretchy":120},[96,12081,12082,12084],{},[102,12083,10559],{"mathvariant":10558},[102,12085,117],{},[111,12087,130],{},[4625,12089,12090,12092],{},[1321,12091,1583],{},[102,12093,322],{},[7202,12095,12096,12098,12106],{},[111,12097,7206],{},[96,12099,12100,12102,12104],{},[102,12101,736],{},[111,12103,130],{},[1321,12105,2071],{},[96,12107,12108,12110,12112],{},[102,12109,322],{},[111,12111,4643],{},[1321,12113,1583],{},[96,12115,12116,12118,12130,12132,12144,12146,12148,12160],{},[111,12117,121],{"fence":113},[99,12119,12120,12122],{},[102,12121,104],{},[96,12123,12124,12126,12128],{},[102,12125,109],{},[111,12127,114],{"separator":113},[102,12129,117],{},[111,12131,121],{"stretchy":120},[726,12133,12134,12136],{},[102,12135,124],{},[96,12137,12138,12140,12142],{},[111,12139,121],{"stretchy":120},[102,12141,736],{},[111,12143,127],{"stretchy":120},[111,12145,127],{"stretchy":120},[111,12147,4643],{},[726,12149,12150,12152],{},[102,12151,683],{},[96,12153,12154,12156,12158],{},[111,12155,121],{"stretchy":120},[102,12157,736],{},[111,12159,127],{"stretchy":120},[111,12161,127],{"fence":113},[145,12163,12164],{"encoding":147},"\\frac{\\partial J(w,b)}{\\partial b} = \\frac{1}{m} \\sum_{i=0}^{m-1} \\left(f_{w,b}(x^{(i)}) - y^{(i)}\\right)",[80,12166,12168,12272],{"className":12167,"ariaHidden":113},[152],[80,12169,12171,12174,12263,12266,12269],{"className":12170},[156],[80,12172],{"className":12173,"style":10663},[160],[80,12175,12177,12180,12260],{"className":12176},[165],[80,12178],{"className":12179},[235,4746],[80,12181,12183],{"className":12182},[4625],[80,12184,12186,12252],{"className":12185},[178,179],[80,12187,12189,12249],{"className":12188},[183],[80,12190,12192,12209,12217],{"className":12191,"style":10686},[187],[80,12193,12194,12197],{"style":5013},[80,12195],{"className":12196,"style":4766},[195],[80,12198,12200],{"className":12199},[200,201,202,203],[80,12201,12203,12206],{"className":12202},[165,203],[80,12204,10559],{"className":12205,"style":10701},[165,203],[80,12207,117],{"className":12208},[165,169,203],[80,12210,12211,12214],{"style":4859},[80,12212],{"className":12213,"style":4766},[195],[80,12215],{"className":12216,"style":4867},[4866],[80,12218,12219,12222],{"style":10715},[80,12220],{"className":12221,"style":4766},[195],[80,12223,12225],{"className":12224},[200,201,202,203],[80,12226,12228,12231,12234,12237,12240,12243,12246],{"className":12227},[165,203],[80,12229,10559],{"className":12230,"style":10701},[165,203],[80,12232,5606],{"className":12233,"style":5632},[165,169,203],[80,12235,121],{"className":12236},[235,203],[80,12238,109],{"className":12239,"style":210},[165,169,203],[80,12241,114],{"className":12242},[214,203],[80,12244,117],{"className":12245},[165,169,203],[80,12247,127],{"className":12248},[242,203],[80,12250,222],{"className":12251},[221],[80,12253,12255],{"className":12254},[183],[80,12256,12258],{"className":12257,"style":7390},[187],[80,12259],{},[80,12261],{"className":12262},[242,4746],[80,12264],{"className":12265,"style":247},[246],[80,12267,130],{"className":12268},[251],[80,12270],{"className":12271,"style":247},[246],[80,12273,12275,12278,12346,12349,12418,12421],{"className":12274},[156],[80,12276],{"className":12277,"style":11708},[160],[80,12279,12281,12284,12343],{"className":12280},[165],[80,12282],{"className":12283},[235,4746],[80,12285,12287],{"className":12286},[4625],[80,12288,12290,12335],{"className":12289},[178,179],[80,12291,12293,12332],{"className":12292},[183],[80,12294,12296,12310,12318],{"className":12295,"style":5010},[187],[80,12297,12298,12301],{"style":5013},[80,12299],{"className":12300,"style":4766},[195],[80,12302,12304],{"className":12303},[200,201,202,203],[80,12305,12307],{"className":12306},[165,203],[80,12308,322],{"className":12309},[165,169,203],[80,12311,12312,12315],{"style":4859},[80,12313],{"className":12314,"style":4766},[195],[80,12316],{"className":12317,"style":4867},[4866],[80,12319,12320,12323],{"style":5042},[80,12321],{"className":12322,"style":4766},[195],[80,12324,12326],{"className":12325},[200,201,202,203],[80,12327,12329],{"className":12328},[165,203],[80,12330,1583],{"className":12331},[165,203],[80,12333,222],{"className":12334},[221],[80,12336,12338],{"className":12337},[183],[80,12339,12341],{"className":12340,"style":7390},[187],[80,12342],{},[80,12344],{"className":12345},[242,4746],[80,12347],{"className":12348,"style":268},[246],[80,12350,12352,12355],{"className":12351},[7402],[80,12353,7206],{"className":12354,"style":7408},[7402,7406,7407],[80,12356,12358],{"className":12357},[174],[80,12359,12361,12410],{"className":12360},[178,179],[80,12362,12364,12407],{"className":12363},[183],[80,12365,12367,12387],{"className":12366,"style":7421},[187],[80,12368,12369,12372],{"style":7424},[80,12370],{"className":12371,"style":196},[195],[80,12373,12375],{"className":12374},[200,201,202,203],[80,12376,12378,12381,12384],{"className":12377},[165,203],[80,12379,736],{"className":12380},[165,169,203],[80,12382,130],{"className":12383},[251,203],[80,12385,2071],{"className":12386},[165,203],[80,12388,12389,12392],{"style":7445},[80,12390],{"className":12391,"style":196},[195],[80,12393,12395],{"className":12394},[200,201,202,203],[80,12396,12398,12401,12404],{"className":12397},[165,203],[80,12399,322],{"className":12400},[165,169,203],[80,12402,4643],{"className":12403},[279,203],[80,12405,1583],{"className":12406},[165,203],[80,12408,222],{"className":12409},[221],[80,12411,12413],{"className":12412},[183],[80,12414,12416],{"className":12415,"style":7473},[187],[80,12417],{},[80,12419],{"className":12420,"style":268},[246],[80,12422,12424,12430,12479,12482,12520,12523,12526,12529,12532,12570],{"className":12423},[7482],[80,12425,12427],{"className":12426,"style":7490},[235,7489],[80,12428,121],{"className":12429},[7494,4803],[80,12431,12433,12436],{"className":12432},[165],[80,12434,104],{"className":12435,"style":170},[165,169],[80,12437,12439],{"className":12438},[174],[80,12440,12442,12471],{"className":12441},[178,179],[80,12443,12445,12468],{"className":12444},[183],[80,12446,12448],{"className":12447,"style":188},[187],[80,12449,12450,12453],{"style":191},[80,12451],{"className":12452,"style":196},[195],[80,12454,12456],{"className":12455},[200,201,202,203],[80,12457,12459,12462,12465],{"className":12458},[165,203],[80,12460,109],{"className":12461,"style":210},[165,169,203],[80,12463,114],{"className":12464},[214,203],[80,12466,117],{"className":12467},[165,169,203],[80,12469,222],{"className":12470},[221],[80,12472,12474],{"className":12473},[183],[80,12475,12477],{"className":12476,"style":229},[187],[80,12478],{},[80,12480,121],{"className":12481},[235],[80,12483,12485,12488],{"className":12484},[165],[80,12486,124],{"className":12487},[165,169],[80,12489,12491],{"className":12490},[174],[80,12492,12494],{"className":12493},[178],[80,12495,12497],{"className":12496},[183],[80,12498,12500],{"className":12499,"style":751},[187],[80,12501,12502,12505],{"style":772},[80,12503],{"className":12504,"style":196},[195],[80,12506,12508],{"className":12507},[200,201,202,203],[80,12509,12511,12514,12517],{"className":12510},[165,203],[80,12512,121],{"className":12513},[235,203],[80,12515,736],{"className":12516},[165,169,203],[80,12518,127],{"className":12519},[242,203],[80,12521,127],{"className":12522},[242],[80,12524],{"className":12525,"style":275},[246],[80,12527,4643],{"className":12528},[279],[80,12530],{"className":12531,"style":275},[246],[80,12533,12535,12538],{"className":12534},[165],[80,12536,683],{"className":12537,"style":834},[165,169],[80,12539,12541],{"className":12540},[174],[80,12542,12544],{"className":12543},[178],[80,12545,12547],{"className":12546},[183],[80,12548,12550],{"className":12549,"style":751},[187],[80,12551,12552,12555],{"style":772},[80,12553],{"className":12554,"style":196},[195],[80,12556,12558],{"className":12557},[200,201,202,203],[80,12559,12561,12564,12567],{"className":12560},[165,203],[80,12562,121],{"className":12563},[235,203],[80,12565,736],{"className":12566},[165,169,203],[80,12568,127],{"className":12569},[242,203],[80,12571,12573],{"className":12572,"style":7490},[242,7489],[80,12574,127],{"className":12575},[7494,4803],[11,12577,12578,12579,12607,12608,12681,12682,12710,12711,12739,12740,12768,12769,12797],{},"Notice the two are nearly twins: the one for ",[80,12580,12582,12595],{"className":12581},[83],[80,12583,12585],{"className":12584},[87],[89,12586,12587],{"xmlns":91},[93,12588,12589,12593],{},[96,12590,12591],{},[102,12592,109],{},[145,12594,109],{"encoding":147},[80,12596,12598],{"className":12597,"ariaHidden":113},[152],[80,12599,12601,12604],{"className":12600},[156],[80,12602],{"className":12603,"style":334},[160],[80,12605,109],{"className":12606,"style":210},[165,169]," has an ",[80,12609,12611,12634],{"className":12610},[83],[80,12612,12614],{"className":12613},[87],[89,12615,12616],{"xmlns":91},[93,12617,12618,12632],{},[96,12619,12620],{},[726,12621,12622,12624],{},[102,12623,124],{},[96,12625,12626,12628,12630],{},[111,12627,121],{"stretchy":120},[102,12629,736],{},[111,12631,127],{"stretchy":120},[145,12633,741],{"encoding":147},[80,12635,12637],{"className":12636,"ariaHidden":113},[152],[80,12638,12640,12643],{"className":12639},[156],[80,12641],{"className":12642,"style":751},[160],[80,12644,12646,12649],{"className":12645},[165],[80,12647,124],{"className":12648},[165,169],[80,12650,12652],{"className":12651},[174],[80,12653,12655],{"className":12654},[178],[80,12656,12658],{"className":12657},[183],[80,12659,12661],{"className":12660,"style":751},[187],[80,12662,12663,12666],{"style":772},[80,12664],{"className":12665,"style":196},[195],[80,12667,12669],{"className":12668},[200,201,202,203],[80,12670,12672,12675,12678],{"className":12671},[165,203],[80,12673,121],{"className":12674},[235,203],[80,12676,736],{"className":12677},[165,169,203],[80,12679,127],{"className":12680},[242,203]," multiplying the error, the one for ",[80,12683,12685,12698],{"className":12684},[83],[80,12686,12688],{"className":12687},[87],[89,12689,12690],{"xmlns":91},[93,12691,12692,12696],{},[96,12693,12694],{},[102,12695,117],{},[145,12697,117],{"encoding":147},[80,12699,12701],{"className":12700,"ariaHidden":113},[152],[80,12702,12704,12707],{"className":12703},[156],[80,12705],{"className":12706,"style":289},[160],[80,12708,117],{"className":12709},[165,169]," doesn't. That makes sense geometrically: changing ",[80,12712,12714,12727],{"className":12713},[83],[80,12715,12717],{"className":12716},[87],[89,12718,12719],{"xmlns":91},[93,12720,12721,12725],{},[96,12722,12723],{},[102,12724,109],{},[145,12726,109],{"encoding":147},[80,12728,12730],{"className":12729,"ariaHidden":113},[152],[80,12731,12733,12736],{"className":12732},[156],[80,12734],{"className":12735,"style":334},[160],[80,12737,109],{"className":12738,"style":210},[165,169]," affects examples with large ",[80,12741,12743,12756],{"className":12742},[83],[80,12744,12746],{"className":12745},[87],[89,12747,12748],{"xmlns":91},[93,12749,12750,12754],{},[96,12751,12752],{},[102,12753,124],{},[145,12755,124],{"encoding":147},[80,12757,12759],{"className":12758,"ariaHidden":113},[152],[80,12760,12762,12765],{"className":12761},[156],[80,12763],{"className":12764,"style":334},[160],[80,12766,124],{"className":12767},[165,169]," more strongly (the slope carries more weight farther from the origin), while ",[80,12770,12772,12785],{"className":12771},[83],[80,12773,12775],{"className":12774},[87],[89,12776,12777],{"xmlns":91},[93,12778,12779,12783],{},[96,12780,12781],{},[102,12782,117],{},[145,12784,117],{"encoding":147},[80,12786,12788],{"className":12787,"ariaHidden":113},[152],[80,12789,12791,12794],{"className":12790},[156],[80,12792],{"className":12793,"style":289},[160],[80,12795,117],{"className":12796},[165,169]," shifts the entire line the same amount for everyone.",[11,12799,12800,12801,12834,12835,12837],{},"And remember that \"2\" we put in ",[80,12802,12804,12819],{"className":12803},[83],[80,12805,12807],{"className":12806},[87],[89,12808,12809],{"xmlns":91},[93,12810,12811,12817],{},[96,12812,12813,12815],{},[1321,12814,1323],{},[102,12816,322],{},[145,12818,9939],{"encoding":147},[80,12820,12822],{"className":12821,"ariaHidden":113},[152],[80,12823,12825,12828,12831],{"className":12824},[156],[80,12826],{"className":12827,"style":1614},[160],[80,12829,1323],{"className":12830},[165],[80,12832,322],{"className":12833},[165,169]," in ",[562,12836,9082],{"href":4066},", where I said it was just there to simplify the math later? Here's where it pays off: differentiating the squared term leaves behind a factor of 2 that cancels exactly with the 2 in the denominator. Without that 2 back there, these formulas here would carry a leftover 2.",[11,12839,12840,12843,12844,2338,12872,12900,12901,12929,12930,12958],{},[15,12841,12842],{},"Simultaneous updates matter."," You compute both derivatives first, using the current values of ",[80,12845,12847,12860],{"className":12846},[83],[80,12848,12850],{"className":12849},[87],[89,12851,12852],{"xmlns":91},[93,12853,12854,12858],{},[96,12855,12856],{},[102,12857,109],{},[145,12859,109],{"encoding":147},[80,12861,12863],{"className":12862,"ariaHidden":113},[152],[80,12864,12866,12869],{"className":12865},[156],[80,12867],{"className":12868,"style":334},[160],[80,12870,109],{"className":12871,"style":210},[165,169],[80,12873,12875,12888],{"className":12874},[83],[80,12876,12878],{"className":12877},[87],[89,12879,12880],{"xmlns":91},[93,12881,12882,12886],{},[96,12883,12884],{},[102,12885,117],{},[145,12887,117],{"encoding":147},[80,12889,12891],{"className":12890,"ariaHidden":113},[152],[80,12892,12894,12897],{"className":12893},[156],[80,12895],{"className":12896,"style":289},[160],[80,12898,117],{"className":12899},[165,169],", and only then swap both parameters at once. Using the new ",[80,12902,12904,12917],{"className":12903},[83],[80,12905,12907],{"className":12906},[87],[89,12908,12909],{"xmlns":91},[93,12910,12911,12915],{},[96,12912,12913],{},[102,12914,109],{},[145,12916,109],{"encoding":147},[80,12918,12920],{"className":12919,"ariaHidden":113},[152],[80,12921,12923,12926],{"className":12922},[156],[80,12924],{"className":12925,"style":334},[160],[80,12927,109],{"className":12928,"style":210},[165,169]," to compute ",[80,12931,12933,12946],{"className":12932},[83],[80,12934,12936],{"className":12935},[87],[89,12937,12938],{"xmlns":91},[93,12939,12940,12944],{},[96,12941,12942],{},[102,12943,117],{},[145,12945,117],{"encoding":147},[80,12947,12949],{"className":12948,"ariaHidden":113},[152],[80,12950,12952,12955],{"className":12951},[156],[80,12953],{"className":12954,"style":289},[160],[80,12956,117],{"className":12957},[165,169],"'s derivative is a classic mistake that changes the algorithm's behavior.",[294,12960,8680],{"id":8679},[2611,12962,12964],{"className":2613,"code":12963,"language":2615,"meta":26,"style":26},"def compute_gradient(x, y, w, b):\n    \"\"\"\n    Computes the gradient of the cost function for linear regression.\n\n    Args:\n      x (ndarray (m,)) : input data, m examples\n      y (ndarray (m,)) : target values\n      w, b (scalar)    : model parameters\n\n    Returns:\n      dj_dw (scalar): partial derivative of the cost with respect to w\n      dj_db (scalar): partial derivative of the cost with respect to b\n    \"\"\"\n    m = x.shape[0]\n\n    dj_dw = 0\n    dj_db = 0\n\n    for i in range(m):\n        f_wb = w * x[i] + b\n        dj_dw_i = (f_wb - y[i]) * x[i]   # example i's contribution to dj_dw\n        dj_db_i = f_wb - y[i]            # example i's contribution to dj_db\n        dj_db += dj_db_i\n        dj_dw += dj_dw_i\n\n    dj_dw = dj_dw \u002F m\n    dj_db = dj_db \u002F m\n\n    return dj_dw, dj_db\n",[65,12965,12966,12971,12975,12980,12984,12988,12992,12996,13000,13004,13008,13013,13018,13022,13026,13030,13035,13040,13044,13048,13052,13057,13062,13067,13072,13076,13082,13088,13093],{"__ignoreMap":26},[80,12967,12968],{"class":2620,"line":33},[80,12969,12970],{},"def compute_gradient(x, y, w, b):\n",[80,12972,12973],{"class":2620,"line":27},[80,12974,4105],{},[80,12976,12977],{"class":2620,"line":2631},[80,12978,12979],{},"    Computes the gradient of the cost function for linear regression.\n",[80,12981,12982],{"class":2620,"line":2636},[80,12983,2657],{"emptyLinePlaceholder":32},[80,12985,12986],{"class":2620,"line":2642},[80,12987,4119],{},[80,12989,12990],{"class":2620,"line":2648},[80,12991,8712],{},[80,12993,12994],{"class":2620,"line":2654},[80,12995,8717],{},[80,12997,12998],{"class":2620,"line":2660},[80,12999,8722],{},[80,13001,13002],{"class":2620,"line":2666},[80,13003,2657],{"emptyLinePlaceholder":32},[80,13005,13006],{"class":2620,"line":2672},[80,13007,4138],{},[80,13009,13010],{"class":2620,"line":2677},[80,13011,13012],{},"      dj_dw (scalar): partial derivative of the cost with respect to w\n",[80,13014,13015],{"class":2620,"line":2683},[80,13016,13017],{},"      dj_db (scalar): partial derivative of the cost with respect to b\n",[80,13019,13020],{"class":2620,"line":4155},[80,13021,4105],{},[80,13023,13024],{"class":2620,"line":4161},[80,13025,8749],{},[80,13027,13028],{"class":2620,"line":4166},[80,13029,2657],{"emptyLinePlaceholder":32},[80,13031,13032],{"class":2620,"line":4172},[80,13033,13034],{},"    dj_dw = 0\n",[80,13036,13037],{"class":2620,"line":4178},[80,13038,13039],{},"    dj_db = 0\n",[80,13041,13042],{"class":2620,"line":4183},[80,13043,2657],{"emptyLinePlaceholder":32},[80,13045,13046],{"class":2620,"line":8770},[80,13047,8767],{},[80,13049,13050],{"class":2620,"line":8776},[80,13051,10165],{},[80,13053,13054],{"class":2620,"line":8782},[80,13055,13056],{},"        dj_dw_i = (f_wb - y[i]) * x[i]   # example i's contribution to dj_dw\n",[80,13058,13059],{"class":2620,"line":8788},[80,13060,13061],{},"        dj_db_i = f_wb - y[i]            # example i's contribution to dj_db\n",[80,13063,13064],{"class":2620,"line":8793},[80,13065,13066],{},"        dj_db += dj_db_i\n",[80,13068,13069],{"class":2620,"line":8799},[80,13070,13071],{},"        dj_dw += dj_dw_i\n",[80,13073,13074],{"class":2620,"line":8804},[80,13075,2657],{"emptyLinePlaceholder":32},[80,13077,13079],{"class":2620,"line":13078},26,[80,13080,13081],{},"    dj_dw = dj_dw \u002F m\n",[80,13083,13085],{"class":2620,"line":13084},27,[80,13086,13087],{},"    dj_db = dj_db \u002F m\n",[80,13089,13091],{"class":2620,"line":13090},28,[80,13092,2657],{"emptyLinePlaceholder":32},[80,13094,13096],{"class":2620,"line":13095},29,[80,13097,13098],{},"    return dj_dw, dj_db\n",[11,13100,13101],{},"And the main loop, which repeats the update until the iterations run out:",[2611,13103,13105],{"className":2613,"code":13104,"language":2615,"meta":26,"style":26},"def gradient_descent(x, y, w_in, b_in, alpha, num_iters, cost_function, gradient_function):\n    \"\"\"\n    Runs gradient descent to fit w and b.\n\n    Args:\n      x, y                : training data\n      w_in, b_in (scalar) : INITIAL parameter values\n      alpha (float)       : learning rate\n      num_iters (int)     : how many iterations to run\n      cost_function       : function to compute the cost\n      gradient_function   : function to compute the gradient\n\n    Returns:\n      w, b (scalar)    : parameters after training\n      J_history (list) : cost at every iteration\n    \"\"\"\n    J_history = []\n    w = w_in\n    b = b_in\n\n    for i in range(num_iters):\n        dj_dw, dj_db = gradient_function(x, y, w, b)\n\n        b = b - alpha * dj_db   # simultaneous update: both derivatives were\n        w = w - alpha * dj_dw   # already computed from the old values\n\n        J_history.append(cost_function(x, y, w, b))\n\n    return w, b, J_history\n",[65,13106,13107,13112,13116,13121,13125,13129,13134,13139,13144,13149,13154,13159,13163,13167,13172,13177,13181,13186,13191,13196,13200,13205,13210,13214,13219,13224,13228,13233,13237],{"__ignoreMap":26},[80,13108,13109],{"class":2620,"line":33},[80,13110,13111],{},"def gradient_descent(x, y, w_in, b_in, alpha, num_iters, cost_function, gradient_function):\n",[80,13113,13114],{"class":2620,"line":27},[80,13115,4105],{},[80,13117,13118],{"class":2620,"line":2631},[80,13119,13120],{},"    Runs gradient descent to fit w and b.\n",[80,13122,13123],{"class":2620,"line":2636},[80,13124,2657],{"emptyLinePlaceholder":32},[80,13126,13127],{"class":2620,"line":2642},[80,13128,4119],{},[80,13130,13131],{"class":2620,"line":2648},[80,13132,13133],{},"      x, y                : training data\n",[80,13135,13136],{"class":2620,"line":2654},[80,13137,13138],{},"      w_in, b_in (scalar) : INITIAL parameter values\n",[80,13140,13141],{"class":2620,"line":2660},[80,13142,13143],{},"      alpha (float)       : learning rate\n",[80,13145,13146],{"class":2620,"line":2666},[80,13147,13148],{},"      num_iters (int)     : how many iterations to run\n",[80,13150,13151],{"class":2620,"line":2672},[80,13152,13153],{},"      cost_function       : function to compute the cost\n",[80,13155,13156],{"class":2620,"line":2677},[80,13157,13158],{},"      gradient_function   : function to compute the gradient\n",[80,13160,13161],{"class":2620,"line":2683},[80,13162,2657],{"emptyLinePlaceholder":32},[80,13164,13165],{"class":2620,"line":4155},[80,13166,4138],{},[80,13168,13169],{"class":2620,"line":4161},[80,13170,13171],{},"      w, b (scalar)    : parameters after training\n",[80,13173,13174],{"class":2620,"line":4166},[80,13175,13176],{},"      J_history (list) : cost at every iteration\n",[80,13178,13179],{"class":2620,"line":4172},[80,13180,4105],{},[80,13182,13183],{"class":2620,"line":4178},[80,13184,13185],{},"    J_history = []\n",[80,13187,13188],{"class":2620,"line":4183},[80,13189,13190],{},"    w = w_in\n",[80,13192,13193],{"class":2620,"line":8770},[80,13194,13195],{},"    b = b_in\n",[80,13197,13198],{"class":2620,"line":8776},[80,13199,2657],{"emptyLinePlaceholder":32},[80,13201,13202],{"class":2620,"line":8782},[80,13203,13204],{},"    for i in range(num_iters):\n",[80,13206,13207],{"class":2620,"line":8788},[80,13208,13209],{},"        dj_dw, dj_db = gradient_function(x, y, w, b)\n",[80,13211,13212],{"class":2620,"line":8793},[80,13213,2657],{"emptyLinePlaceholder":32},[80,13215,13216],{"class":2620,"line":8799},[80,13217,13218],{},"        b = b - alpha * dj_db   # simultaneous update: both derivatives were\n",[80,13220,13221],{"class":2620,"line":8804},[80,13222,13223],{},"        w = w - alpha * dj_dw   # already computed from the old values\n",[80,13225,13226],{"class":2620,"line":13078},[80,13227,2657],{"emptyLinePlaceholder":32},[80,13229,13230],{"class":2620,"line":13084},[80,13231,13232],{},"        J_history.append(cost_function(x, y, w, b))\n",[80,13234,13235],{"class":2620,"line":13090},[80,13236,2657],{"emptyLinePlaceholder":32},[80,13238,13239],{"class":2620,"line":13095},[80,13240,13241],{},"    return w, b, J_history\n",[294,13243,13245],{"id":13244},"now-its-your-turn-but-with-the-algorithm-doing-the-work","Now it's your turn, but with the algorithm doing the work",[11,13247,13248],{},"No more dragging a slider until you land on the right value by hand. Below is the real algorithm running, with full control: pick a learning rate, take one step at a time or run a batch at once, and watch the little red dot walk downhill on its own, on the heatmap, on the 3D surface, or on the parabola.",[11,13250,13251,13252,504,13302,13353],{},"Start at ",[80,13253,13255,13272],{"className":13254},[83],[80,13256,13258],{"className":13257},[87],[89,13259,13260],{"xmlns":91},[93,13261,13262,13270],{},[96,13263,13264,13266,13268],{},[102,13265,109],{},[111,13267,130],{},[1321,13269,2071],{},[145,13271,9285],{"encoding":147},[80,13273,13275,13293],{"className":13274,"ariaHidden":113},[152],[80,13276,13278,13281,13284,13287,13290],{"className":13277},[156],[80,13279],{"className":13280,"style":334},[160],[80,13282,109],{"className":13283,"style":210},[165,169],[80,13285],{"className":13286,"style":247},[246],[80,13288,130],{"className":13289},[251],[80,13291],{"className":13292,"style":247},[246],[80,13294,13296,13299],{"className":13295},[156],[80,13297],{"className":13298,"style":1614},[160],[80,13300,2071],{"className":13301},[165],[80,13303,13305,13323],{"className":13304},[83],[80,13306,13308],{"className":13307},[87],[89,13309,13310],{"xmlns":91},[93,13311,13312,13320],{},[96,13313,13314,13316,13318],{},[102,13315,117],{},[111,13317,130],{},[1321,13319,2071],{},[145,13321,13322],{"encoding":147},"b = 0",[80,13324,13326,13344],{"className":13325,"ariaHidden":113},[152],[80,13327,13329,13332,13335,13338,13341],{"className":13328},[156],[80,13330],{"className":13331,"style":289},[160],[80,13333,117],{"className":13334},[165,169],[80,13336],{"className":13337,"style":247},[246],[80,13339,130],{"className":13340},[251],[80,13342],{"className":13343,"style":247},[246],[80,13345,13347,13350],{"className":13346},[156],[80,13348],{"className":13349,"style":1614},[160],[80,13351,2071],{"className":13352},[165]," (far from the answer) and click \"Rodar 2000\" a few times with the default alpha of 0.01. Notice how the cost drops fast at first and then slows down on its own, with no input from you.",[13355,13356],"gradient-descent-simulator",{":b-range":13357,":initial-b":2071,":initial-w":2071,":w-range":13358,"b-label":117,"w-label":109,":x-train":3350,":y-train":3351},"[-200, 400]","[-100, 500]",[294,13360,13362],{"id":13361},"when-the-learning-rate-is-too-big","When the learning rate is too big",[11,13364,13365,13366,2338,13394,13422],{},"Now click the alpha 0.8 preset (way bigger than the 0.01 that worked) and run a few steps. You'll watch ",[80,13367,13369,13382],{"className":13368},[83],[80,13370,13372],{"className":13371},[87],[89,13373,13374],{"xmlns":91},[93,13375,13376,13380],{},[96,13377,13378],{},[102,13379,109],{},[145,13381,109],{"encoding":147},[80,13383,13385],{"className":13384,"ariaHidden":113},[152],[80,13386,13388,13391],{"className":13387},[156],[80,13389],{"className":13390,"style":334},[160],[80,13392,109],{"className":13393,"style":210},[165,169],[80,13395,13397,13410],{"className":13396},[83],[80,13398,13400],{"className":13399},[87],[89,13401,13402],{"xmlns":91},[93,13403,13404,13408],{},[96,13405,13406],{},[102,13407,117],{},[145,13409,117],{"encoding":147},[80,13411,13413],{"className":13412,"ariaHidden":113},[152],[80,13414,13416,13419],{"className":13415},[156],[80,13417],{"className":13418,"style":289},[160],[80,13420,117],{"className":13421},[165,169]," become increasingly absurd, and instead of dropping, the cost climbs.",[11,13424,13425,13426,13430,13431,13459],{},"That's ",[7103,13427,13429],{"definition":13428},"when gradient descent, instead of getting closer to the minimum, moves farther and farther away from it because the step is too big","divergence",", and the reason is easy to picture: the step size is proportional to the derivative. If ",[80,13432,13434,13447],{"className":13433},[83],[80,13435,13437],{"className":13436},[87],[89,13438,13439],{"xmlns":91},[93,13440,13441,13445],{},[96,13442,13443],{},[102,13444,10551],{},[145,13446,11037],{"encoding":147},[80,13448,13450],{"className":13449,"ariaHidden":113},[152],[80,13451,13453,13456],{"className":13452},[156],[80,13454],{"className":13455,"style":334},[160],[80,13457,10551],{"className":13458,"style":10667},[165,169]," is large, the step overshoots the bottom of the bowl and lands on the other side, higher up than it started. From that new spot, the derivative is even bigger (in magnitude) and flipped in sign, so the next step is even bigger in the opposite direction. It turns into a self-feeding loop that explodes, like pushing a swing harder and harder until it flips upside down.",[11,13461,13462,13463,13491],{},"In practice, if you're training a real model and see the cost climbing or bouncing back and forth without settling, the first thing I try is lowering ",[80,13464,13466,13479],{"className":13465},[83],[80,13467,13469],{"className":13468},[87],[89,13470,13471],{"xmlns":91},[93,13472,13473,13477],{},[96,13474,13475],{},[102,13476,10551],{},[145,13478,11037],{"encoding":147},[80,13480,13482],{"className":13481,"ariaHidden":113},[152],[80,13483,13485,13488],{"className":13484},[156],[80,13486],{"className":13487,"style":334},[160],[80,13489,10551],{"className":13490,"style":10667},[165,169]," (divide by 3 or by 10, for instance). The course suggests trying a sequence like 0.001, 0.003, 0.01, 0.03, 0.1 and comparing the cost curves until you find one that drops smoothly.",[521,13493,13494,13504],{},[524,13495,13496],{},[527,13497,13498,13501],{},[530,13499,13500],{"align":1884},"Alpha",[530,13502,13503],{"align":532},"What happens (after 1000 steps, starting at w=0, b=0)",[541,13505,13506,13514,13522,13530,13538,13546],{},[527,13507,13508,13511],{},[546,13509,13510],{"align":1884},"0.0001",[546,13512,13513],{"align":532},"way too slow, barely left the starting point",[527,13515,13516,13519],{},[546,13517,13518],{"align":1884},"0.001",[546,13520,13521],{"align":532},"still far from the target",[527,13523,13524,13527],{},[546,13525,13526],{"align":1884},"0.01",[546,13528,13529],{"align":532},"good balance, this is what we used above",[527,13531,13532,13535],{},[546,13533,13534],{"align":1884},"0.1",[546,13536,13537],{"align":532},"converges fast",[527,13539,13540,13543],{},[546,13541,13542],{"align":1884},"0.3",[546,13544,13545],{"align":532},"still converges, but already close to the edge",[527,13547,13548,13551],{},[546,13549,13550],{"align":1884},"0.8",[546,13552,13553],{"align":532},"diverges",[11,13555,13556],{},"Try these values yourself in the simulator above and compare against the table.",[294,13558,13560],{"id":13559},"bonus-not-every-bowl-is-this-well-behaved","Bonus: not every bowl is this well-behaved",[11,13562,13563,13564,13567],{},"Every cost surface we've drawn so far looks like the same soup bowl, because it comes from squared error, and that guarantees convexity (",[562,13565,13566],{"href":4066},"last post","). But gradient descent doesn't only live on nicely-behaved bowls. Here are two classics that anyone studying optimization runs into sooner or later, just so you see that the trouble we hit with a big alpha is only the tip of the iceberg.",[2606,13569,13571],{"id":13570},"rosenbrocks-banana-valley","Rosenbrock's banana valley",[11,13573,13574],{},"This one's practically the standard stress test of the field, it even has its own name: the Rosenbrock function.",[11,13576,13577],{},[80,13578,13580,13642],{"className":13579},[83],[80,13581,13583],{"className":13582},[87],[89,13584,13585],{"xmlns":91},[93,13586,13587,13639],{},[96,13588,13589,13591,13593,13595,13597,13599,13601,13603,13605,13607,13609,13611,13617,13619,13621,13623,13625,13627,13633],{},[102,13590,104],{},[111,13592,121],{"stretchy":120},[102,13594,109],{},[111,13596,114],{"separator":113},[102,13598,117],{},[111,13600,127],{"stretchy":120},[111,13602,130],{},[111,13604,121],{"stretchy":120},[1321,13606,1583],{},[111,13608,4643],{},[102,13610,109],{},[726,13612,13613,13615],{},[111,13614,127],{"stretchy":120},[1321,13616,1323],{},[111,13618,141],{},[1321,13620,4286],{},[111,13622,121],{"stretchy":120},[102,13624,117],{},[111,13626,4643],{},[726,13628,13629,13631],{},[102,13630,109],{},[1321,13632,1323],{},[726,13634,13635,13637],{},[111,13636,127],{"stretchy":120},[1321,13638,1323],{},[145,13640,13641],{"encoding":147},"f(w,b) = (1-w)^2 + 100(b - w^2)^2",[80,13643,13645,13681,13702,13749,13773],{"className":13644,"ariaHidden":113},[152],[80,13646,13648,13651,13654,13657,13660,13663,13666,13669,13672,13675,13678],{"className":13647},[156],[80,13649],{"className":13650,"style":2316},[160],[80,13652,104],{"className":13653,"style":170},[165,169],[80,13655,121],{"className":13656},[235],[80,13658,109],{"className":13659,"style":210},[165,169],[80,13661,114],{"className":13662},[214],[80,13664],{"className":13665,"style":268},[246],[80,13667,117],{"className":13668},[165,169],[80,13670,127],{"className":13671},[242],[80,13673],{"className":13674,"style":247},[246],[80,13676,130],{"className":13677},[251],[80,13679],{"className":13680,"style":247},[246],[80,13682,13684,13687,13690,13693,13696,13699],{"className":13683},[156],[80,13685],{"className":13686,"style":2316},[160],[80,13688,121],{"className":13689},[235],[80,13691,1583],{"className":13692},[165],[80,13694],{"className":13695,"style":275},[246],[80,13697,4643],{"className":13698},[279],[80,13700],{"className":13701,"style":275},[246],[80,13703,13705,13708,13711,13740,13743,13746],{"className":13704},[156],[80,13706],{"className":13707,"style":8187},[160],[80,13709,109],{"className":13710,"style":210},[165,169],[80,13712,13714,13717],{"className":13713},[242],[80,13715,127],{"className":13716},[242],[80,13718,13720],{"className":13719},[174],[80,13721,13723],{"className":13722},[178],[80,13724,13726],{"className":13725},[183],[80,13727,13729],{"className":13728,"style":1407},[187],[80,13730,13731,13734],{"style":772},[80,13732],{"className":13733,"style":196},[195],[80,13735,13737],{"className":13736},[200,201,202,203],[80,13738,1323],{"className":13739},[165,203],[80,13741],{"className":13742,"style":275},[246],[80,13744,141],{"className":13745},[279],[80,13747],{"className":13748,"style":275},[246],[80,13750,13752,13755,13758,13761,13764,13767,13770],{"className":13751},[156],[80,13753],{"className":13754,"style":2316},[160],[80,13756,4286],{"className":13757},[165],[80,13759,121],{"className":13760},[235],[80,13762,117],{"className":13763},[165,169],[80,13765],{"className":13766,"style":275},[246],[80,13768,4643],{"className":13769},[279],[80,13771],{"className":13772,"style":275},[246],[80,13774,13776,13779,13808],{"className":13775},[156],[80,13777],{"className":13778,"style":8187},[160],[80,13780,13782,13785],{"className":13781},[165],[80,13783,109],{"className":13784,"style":210},[165,169],[80,13786,13788],{"className":13787},[174],[80,13789,13791],{"className":13790},[178],[80,13792,13794],{"className":13793},[183],[80,13795,13797],{"className":13796,"style":1407},[187],[80,13798,13799,13802],{"style":772},[80,13800],{"className":13801,"style":196},[195],[80,13803,13805],{"className":13804},[200,201,202,203],[80,13806,1323],{"className":13807},[165,203],[80,13809,13811,13814],{"className":13810},[242],[80,13812,127],{"className":13813},[242],[80,13815,13817],{"className":13816},[174],[80,13818,13820],{"className":13819},[178],[80,13821,13823],{"className":13822},[183],[80,13824,13826],{"className":13825,"style":1407},[187],[80,13827,13828,13831],{"style":772},[80,13829],{"className":13830,"style":196},[195],[80,13832,13834],{"className":13833},[200,201,202,203],[80,13835,1323],{"className":13836},[165,203],[11,13838,13839,13840,13892],{},"The global minimum sits at ",[80,13841,13843,13865],{"className":13842},[83],[80,13844,13846],{"className":13845},[87],[89,13847,13848],{"xmlns":91},[93,13849,13850,13862],{},[96,13851,13852,13854,13856,13858,13860],{},[111,13853,121],{"stretchy":120},[1321,13855,1583],{},[111,13857,114],{"separator":113},[1321,13859,1583],{},[111,13861,127],{"stretchy":120},[145,13863,13864],{"encoding":147},"(1,1)",[80,13866,13868],{"className":13867,"ariaHidden":113},[152],[80,13869,13871,13874,13877,13880,13883,13886,13889],{"className":13870},[156],[80,13872],{"className":13873,"style":2316},[160],[80,13875,121],{"className":13876},[235],[80,13878,1583],{"className":13879},[165],[80,13881,114],{"className":13882},[214],[80,13884],{"className":13885,"style":268},[246],[80,13887,1583],{"className":13888},[165],[80,13890,127],{"className":13891},[242],", cost zero, but look at the shape:",[9817,13894],{":b-max":13895,":b-min":13896,":w-max":1323,":w-min":13897,"fn":13898},"3","-1","-2","rosenbrock",[11,13900,13901],{},"It's not a round bowl, it's a curved valley, shaped like a banana. That's a real problem for gradient descent: the direction that descends fastest almost never points toward the bottom of the valley, it points toward the nearest wall. The algorithm ends up bouncing from one side of the banana to the other, barely making progress with each zig-zag, even near the bottom.",[11,13903,13904],{},"Try alpha 0.01 here (the same value that worked smoothly in our housing example) and watch what happens:",[13355,13906],{":b-range":13907,":initial-b":1583,":initial-w":13896,":w-range":13908,"b-label":117,"w-label":109,":alpha-presets":13909,":initial-alpha":13910,"fn":13898},"[-1, 3]","[-2, 2]","[0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01]","0.002",[11,13912,13913,13914,1618],{},"Alpha 0.01 diverges almost instantly here, the same value that was the \"good balance\" above. There's no universal alpha, it depends entirely on the shape of the surface you're descending. Drop it to 0.002 and click \"Rodar 2000\" a few times: now the red dot snakes slowly through the valley until it lands near ",[80,13915,13917,13938],{"className":13916},[83],[80,13918,13920],{"className":13919},[87],[89,13921,13922],{"xmlns":91},[93,13923,13924,13936],{},[96,13925,13926,13928,13930,13932,13934],{},[111,13927,121],{"stretchy":120},[1321,13929,1583],{},[111,13931,114],{"separator":113},[1321,13933,1583],{},[111,13935,127],{"stretchy":120},[145,13937,13864],{"encoding":147},[80,13939,13941],{"className":13940,"ariaHidden":113},[152],[80,13942,13944,13947,13950,13953,13956,13959,13962],{"className":13943},[156],[80,13945],{"className":13946,"style":2316},[160],[80,13948,121],{"className":13949},[235],[80,13951,1583],{"className":13952},[165],[80,13954,114],{"className":13955},[214],[80,13957],{"className":13958,"style":268},[246],[80,13960,1583],{"className":13961},[165],[80,13963,127],{"className":13964},[242],[2606,13966,13968],{"id":13967},"the-saddle-point","The saddle point",[11,13970,13971],{},[80,13972,13974,14014],{"className":13973},[83],[80,13975,13977],{"className":13976},[87],[89,13978,13979],{"xmlns":91},[93,13980,13981,14011],{},[96,13982,13983,13985,13987,13989,13991,13993,13995,13997,14003,14005],{},[102,13984,104],{},[111,13986,121],{"stretchy":120},[102,13988,109],{},[111,13990,114],{"separator":113},[102,13992,117],{},[111,13994,127],{"stretchy":120},[111,13996,130],{},[726,13998,13999,14001],{},[102,14000,109],{},[1321,14002,1323],{},[111,14004,4643],{},[726,14006,14007,14009],{},[102,14008,117],{},[1321,14010,1323],{},[145,14012,14013],{"encoding":147},"f(w,b) = w^2 - b^2",[80,14015,14017,14053,14097],{"className":14016,"ariaHidden":113},[152],[80,14018,14020,14023,14026,14029,14032,14035,14038,14041,14044,14047,14050],{"className":14019},[156],[80,14021],{"className":14022,"style":2316},[160],[80,14024,104],{"className":14025,"style":170},[165,169],[80,14027,121],{"className":14028},[235],[80,14030,109],{"className":14031,"style":210},[165,169],[80,14033,114],{"className":14034},[214],[80,14036],{"className":14037,"style":268},[246],[80,14039,117],{"className":14040},[165,169],[80,14042,127],{"className":14043},[242],[80,14045],{"className":14046,"style":247},[246],[80,14048,130],{"className":14049},[251],[80,14051],{"className":14052,"style":247},[246],[80,14054,14056,14059,14088,14091,14094],{"className":14055},[156],[80,14057],{"className":14058,"style":9741},[160],[80,14060,14062,14065],{"className":14061},[165],[80,14063,109],{"className":14064,"style":210},[165,169],[80,14066,14068],{"className":14067},[174],[80,14069,14071],{"className":14070},[178],[80,14072,14074],{"className":14073},[183],[80,14075,14077],{"className":14076,"style":1407},[187],[80,14078,14079,14082],{"style":772},[80,14080],{"className":14081,"style":196},[195],[80,14083,14085],{"className":14084},[200,201,202,203],[80,14086,1323],{"className":14087},[165,203],[80,14089],{"className":14090,"style":275},[246],[80,14092,4643],{"className":14093},[279],[80,14095],{"className":14096,"style":275},[246],[80,14098,14100,14103],{"className":14099},[156],[80,14101],{"className":14102,"style":1407},[160],[80,14104,14106,14109],{"className":14105},[165],[80,14107,117],{"className":14108},[165,169],[80,14110,14112],{"className":14111},[174],[80,14113,14115],{"className":14114},[178],[80,14116,14118],{"className":14117},[183],[80,14119,14121],{"className":14120,"style":1407},[187],[80,14122,14123,14126],{"style":772},[80,14124],{"className":14125,"style":196},[195],[80,14127,14129],{"className":14128},[200,201,202,203],[80,14130,1323],{"className":14131},[165,203],[9817,14133],{":b-max":1323,":b-min":13897,":w-max":1323,":w-min":13897,"fn":14134,":marker-b":2071,":marker-w":2071},"saddle",[11,14136,14137,14138,14166,14167,14195,14196,14200],{},"Rotate this one slowly. It's a minimum if you only look along the ",[80,14139,14141,14154],{"className":14140},[83],[80,14142,14144],{"className":14143},[87],[89,14145,14146],{"xmlns":91},[93,14147,14148,14152],{},[96,14149,14150],{},[102,14151,109],{},[145,14153,109],{"encoding":147},[80,14155,14157],{"className":14156,"ariaHidden":113},[152],[80,14158,14160,14163],{"className":14159},[156],[80,14161],{"className":14162,"style":334},[160],[80,14164,109],{"className":14165,"style":210},[165,169]," axis, and a maximum if you only look along the ",[80,14168,14170,14183],{"className":14169},[83],[80,14171,14173],{"className":14172},[87],[89,14174,14175],{"xmlns":91},[93,14176,14177,14181],{},[96,14178,14179],{},[102,14180,117],{},[145,14182,117],{"encoding":147},[80,14184,14186],{"className":14185,"ariaHidden":113},[152],[80,14187,14189,14192],{"className":14188},[156],[80,14190],{"className":14191,"style":289},[160],[80,14193,117],{"className":14194},[165,169]," axis, at the same time, at the same point. That's called a ",[7103,14197,14199],{"definition":14198},"a point where the gradient is zero, but that's neither a minimum nor a maximum, it's a minimum in one direction and a maximum in another at the same time, like the middle of a horse saddle","saddle point",", the red dot in the center marks exactly that.",[11,14202,14203,14204,14232],{},"Remember the rule \"when the derivative hits zero, the algorithm stops on its own\"? Well, right at this point it hits zero just the same. If gradient descent landed exactly there, it would think it was done, without being done at all, just balanced on top of a mountain pass. Any tiny nudge away from that exact center, and it slides downhill in whichever direction actually descends (the ",[80,14205,14207,14220],{"className":14206},[83],[80,14208,14210],{"className":14209},[87],[89,14211,14212],{"xmlns":91},[93,14213,14214,14218],{},[96,14215,14216],{},[102,14217,117],{},[145,14219,117],{"encoding":147},[80,14221,14223],{"className":14222,"ariaHidden":113},[152],[80,14224,14226,14229],{"className":14225},[156],[80,14227],{"className":14228,"style":289},[160],[80,14230,117],{"className":14231},[165,169]," axis), away from any real minimum.",[11,14234,14235],{},"This never happens on our own regression cost (it's always convex, no hidden saddle anywhere), but this exact kind of surface shows up constantly in more complex models, like neural networks. Keep that name in your back pocket, it comes back.",[294,14237,14239],{"id":14238},"wrapping-up-the-trilogy","Wrapping up the trilogy",[521,14241,14242,14252],{},[524,14243,14244],{},[527,14245,14246,14249],{},[530,14247,14248],{"align":532},"Post",[530,14250,14251],{"align":532},"What's done",[541,14253,14254,14412,14478],{},[527,14255,14256,14261],{},[546,14257,14258],{"align":532},[562,14259,14260],{"href":6979},"Lab 02",[546,14262,14263,14264],{"align":532},"the model, ",[80,14265,14267,14306],{"className":14266},[83],[80,14268,14270],{"className":14269},[87],[89,14271,14272],{"xmlns":91},[93,14273,14274,14304],{},[96,14275,14276,14288,14290,14292,14294,14296,14298,14300,14302],{},[99,14277,14278,14280],{},[102,14279,104],{},[96,14281,14282,14284,14286],{},[102,14283,109],{},[111,14285,114],{"separator":113},[102,14287,117],{},[111,14289,121],{"stretchy":120},[102,14291,124],{},[111,14293,127],{"stretchy":120},[111,14295,130],{},[102,14297,109],{},[102,14299,124],{},[111,14301,141],{},[102,14303,117],{},[145,14305,387],{"encoding":147},[80,14307,14309,14382,14403],{"className":14308,"ariaHidden":113},[152],[80,14310,14312,14315,14364,14367,14370,14373,14376,14379],{"className":14311},[156],[80,14313],{"className":14314,"style":161},[160],[80,14316,14318,14321],{"className":14317},[165],[80,14319,104],{"className":14320,"style":170},[165,169],[80,14322,14324],{"className":14323},[174],[80,14325,14327,14356],{"className":14326},[178,179],[80,14328,14330,14353],{"className":14329},[183],[80,14331,14333],{"className":14332,"style":188},[187],[80,14334,14335,14338],{"style":191},[80,14336],{"className":14337,"style":196},[195],[80,14339,14341],{"className":14340},[200,201,202,203],[80,14342,14344,14347,14350],{"className":14343},[165,203],[80,14345,109],{"className":14346,"style":210},[165,169,203],[80,14348,114],{"className":14349},[214,203],[80,14351,117],{"className":14352},[165,169,203],[80,14354,222],{"className":14355},[221],[80,14357,14359],{"className":14358},[183],[80,14360,14362],{"className":14361,"style":229},[187],[80,14363],{},[80,14365,121],{"className":14366},[235],[80,14368,124],{"className":14369},[165,169],[80,14371,127],{"className":14372},[242],[80,14374],{"className":14375,"style":247},[246],[80,14377,130],{"className":14378},[251],[80,14380],{"className":14381,"style":247},[246],[80,14383,14385,14388,14391,14394,14397,14400],{"className":14384},[156],[80,14386],{"className":14387,"style":261},[160],[80,14389,109],{"className":14390,"style":210},[165,169],[80,14392,124],{"className":14393},[165,169],[80,14395],{"className":14396,"style":275},[246],[80,14398,141],{"className":14399},[279],[80,14401],{"className":14402,"style":275},[246],[80,14404,14406,14409],{"className":14405},[156],[80,14407],{"className":14408,"style":289},[160],[80,14410,117],{"className":14411},[165,169],[527,14413,14414,14419],{},[546,14415,14416],{"align":532},[562,14417,14418],{"href":4066},"Lab 03",[546,14420,14421,14422],{"align":532},"the error measurement, ",[80,14423,14425,14448],{"className":14424},[83],[80,14426,14428],{"className":14427},[87],[89,14429,14430],{"xmlns":91},[93,14431,14432,14446],{},[96,14433,14434,14436,14438,14440,14442,14444],{},[102,14435,5606],{},[111,14437,121],{"stretchy":120},[102,14439,109],{},[111,14441,114],{"separator":113},[102,14443,117],{},[111,14445,127],{"stretchy":120},[145,14447,5619],{"encoding":147},[80,14449,14451],{"className":14450,"ariaHidden":113},[152],[80,14452,14454,14457,14460,14463,14466,14469,14472,14475],{"className":14453},[156],[80,14455],{"className":14456,"style":2316},[160],[80,14458,5606],{"className":14459,"style":5632},[165,169],[80,14461,121],{"className":14462},[235],[80,14464,109],{"className":14465,"style":210},[165,169],[80,14467,114],{"className":14468},[214],[80,14470],{"className":14471,"style":268},[246],[80,14473,117],{"className":14474},[165,169],[80,14476,127],{"className":14477},[242],[527,14479,14480,14483],{},[546,14481,14482],{"align":532},"Lab 04 (this one)",[546,14484,14485],{"align":532},"the algorithm that minimizes that error on its own, gradient descent",[11,14487,6507],{},[299,14489,14490,14496,14502],{},[302,14491,14492,14495],{},[15,14493,14494],{},"The gradient points toward where the cost increases",", so the algorithm always walks the opposite way.",[302,14497,14498,14501],{},[15,14499,14500],{},"The update is simultaneous",", compute both derivatives first, swap the parameters after.",[302,14503,14504,14507],{},[15,14505,14506],{},"The learning rate is the single most sensitive knob you'll touch",": too small is slow, too big diverges.",[11,14509,14510,14513,14514,14516,14517,14521],{},[15,14511,14512],{},"Coming up next:"," everything you've seen so far used one feature (the house's size). In Week 2 of the course, the model gains several features at once, and computing with a ",[65,14515,8813],{}," loop stops cutting it. Before touching a model with multiple features, ",[562,14518,14520],{"href":14519},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization","the next post"," covers the tool that makes it viable: NumPy and vectorization.",[294,14523,6716],{"id":6715},[11,14525,14526,14527,14530,14531,2338,14533,1618],{},"Same real housing dataset from the last two posts (",[562,14528,6725],{"href":6722,"rel":14529},[6724],"), same scale (size in \"hundreds of sqft\", price in \"thousands of dollars\"). Let's watch the algorithm find, on its own, in real data, the fit you went hunting for by hand in posts ",[562,14532,1583],{"href":6979},[562,14534,1323],{"href":4066},[2611,14536,14538],{"className":2613,"code":14537,"language":2615,"meta":26,"style":26},"w, b, J_hist = gradient_descent(\n    x_sqft, y_price,          # the 50 real houses\n    w_in=0, b_in=0,           # starting from zero, same as the toy example\n    alpha=0.01, num_iters=4000,\n    cost_function=compute_cost, gradient_function=compute_gradient)\n\nprint(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[65,14539,14540,14545,14550,14555,14560,14565,14569],{"__ignoreMap":26},[80,14541,14542],{"class":2620,"line":33},[80,14543,14544],{},"w, b, J_hist = gradient_descent(\n",[80,14546,14547],{"class":2620,"line":27},[80,14548,14549],{},"    x_sqft, y_price,          # the 50 real houses\n",[80,14551,14552],{"class":2620,"line":2631},[80,14553,14554],{},"    w_in=0, b_in=0,           # starting from zero, same as the toy example\n",[80,14556,14557],{"class":2620,"line":2636},[80,14558,14559],{},"    alpha=0.01, num_iters=4000,\n",[80,14561,14562],{"class":2620,"line":2642},[80,14563,14564],{},"    cost_function=compute_cost, gradient_function=compute_gradient)\n",[80,14566,14567],{"class":2620,"line":2648},[80,14568,2657],{"emptyLinePlaceholder":32},[80,14570,14571],{"class":2620,"line":2654},[80,14572,14573],{},"print(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[46,14575,14576],{},[11,14577,14578,3255,14580],{},[15,14579,2693],{},[65,14581,14582],{},"(w, b) found: (116.5, 398.3)",[11,14584,14585],{},"Matches almost exactly the fit cited in the last two posts. Click \"Rodar 2000\" twice on the simulator below and watch it live:",[14587,14588],"housing-gradient-descent-simulator",{},[11,14590,14591,14592,2338,14620,14648],{},"Notice convergence here is quite a bit slower than in the 2-point toy example, even already using the same small scale as before. That happens because ",[80,14593,14595,14608],{"className":14594},[83],[80,14596,14598],{"className":14597},[87],[89,14599,14600],{"xmlns":91},[93,14601,14602,14606],{},[96,14603,14604],{},[102,14605,109],{},[145,14607,109],{"encoding":147},[80,14609,14611],{"className":14610,"ariaHidden":113},[152],[80,14612,14614,14617],{"className":14613},[156],[80,14615],{"className":14616,"style":334},[160],[80,14618,109],{"className":14619,"style":210},[165,169],[80,14621,14623,14636],{"className":14622},[83],[80,14624,14626],{"className":14625},[87],[89,14627,14628],{"xmlns":91},[93,14629,14630,14634],{},[96,14631,14632],{},[102,14633,117],{},[145,14635,117],{"encoding":147},[80,14637,14639],{"className":14638,"ariaHidden":113},[152],[80,14640,14642,14645],{"className":14641},[156],[80,14643],{"className":14644,"style":289},[160],[80,14646,117],{"className":14647},[165,169]," still live on pretty different ranges from each other (one goes up to 300, the other up to 600), which stretches the cost valley. This is exactly the kind of situation feature scaling, later in the course, fixes for good.",[6949,14650,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":14652},[14653,14654,14655,14656,14657,14658,14659,14663,14664],{"id":10895,"depth":27,"text":10896},{"id":11058,"depth":27,"text":11059},{"id":11461,"depth":27,"text":11462},{"id":8679,"depth":27,"text":8680},{"id":13244,"depth":27,"text":13245},{"id":13361,"depth":27,"text":13362},{"id":13559,"depth":27,"text":13560,"children":14660},[14661,14662],{"id":13570,"depth":2631,"text":13571},{"id":13967,"depth":2631,"text":13968},{"id":14238,"depth":27,"text":14239},{"id":6715,"depth":27,"text":6716},"The algorithm that walks itself down to the bottom of the cost bowl: the math behind gradient descent, what happens when the learning rate is too big, and why it stops on its own near the minimum.",{},{"title":10245,"description":14665},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab04-gradient-descent",[14670,14671,6986],"gradient-descent","optimization","hxYODkFSVzZOMg12Zsd-Rt3eVggi00IEWJ5kpaCr-B4",{"id":14674,"title":14675,"body":14676,"cover":3,"date":6976,"description":16414,"extension":30,"meta":16415,"navigation":32,"order":2636,"path":14519,"playlist":6980,"seo":16416,"status":36,"stem":16417,"tags":16418,"__hash__":16420},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization.md","NumPy and Vectorization",{"type":8,"value":14677,"toc":16400},[14678,14684,14690,14694,14700,14708,14711,14715,14928,14976,14983,14987,15008,15047,15051,15057,15091,15103,15109,15113,15129,15388,15391,15448,15452,15462,15468,15481,15492,15495,15498,15502,15594,15628,15653,15900,15902,16114,16131,16133,16145,16194,16209,16243,16258,16300,16304,16395,16398],[11,14679,14680],{},[57,14681],{"alt":14682,"src":14683},"The \"silent protector\" meme: a soldier labeled Linear Algebra, Statistics, Calculus, and Probability Theory takes hit after hit in place of an \"ML Newbie\" sleeping peacefully in bed, completely unaware any of it is happening","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization\u002Fmeme-vectorization.jpg",[11,14685,14686,14687,14689],{},"Let's take a break from the trilogy. This post has no new model to train, no cost to compute, no hill to walk down. It's a tool post, like pulling into a gas station to swap a bald tire before it blows out on the highway. It exists because starting in Week 2 of the course the model gains several features at once (size, bedroom count, house age, all together), and at that point computing with a ",[65,14688,8813],{}," loop stops being viable. Let's understand the tool that solves this before we actually need it.",[294,14691,14693],{"id":14692},"python-lists-work-so-why-switch","Python lists work, so why switch",[11,14695,14696,14697,1618],{},"Python already ships with lists, which store numbers just fine. So why does the entire ML community use a separate library for this? Short answer: ",[15,14698,14699],{},"memory",[11,14701,14702,14703,14707],{},"A Python list stores each number as a separate object, scattered around memory, with a bunch of extra bookkeeping stuck to it (reference count, and so on). Doing math on top of that means the processor keeps jumping from address to address, unwrapping each object to get the value inside. A ",[7103,14704,14706],{"definition":14705},"a Python library specialized in numeric arrays, with every element of the same type stored side by side in memory","NumPy"," array, on the other hand, stores raw numbers of the same type, packed right next to each other in one contiguous block of memory. The processor can crunch a whole bunch of them at once, using special instructions that exist exactly for this.",[11,14709,14710],{},"That difference in memory layout is the real reason behind NumPy's speed. It's not magic, it's low-level engineering.",[294,14712,14714],{"id":14713},"vectors-the-basic-building-block","Vectors: the basic building block",[11,14716,14717,14718,14746,14747,7106,14751,14798,14799,14827,14828,14874,14875,14927],{},"A vector is a bunch of numbers arranged in order, all of the same type. In the course's notation, a vector is a bold lowercase letter, like ",[80,14719,14721,14734],{"className":14720},[83],[80,14722,14724],{"className":14723},[87],[89,14725,14726],{"xmlns":91},[93,14727,14728,14732],{},[96,14729,14730],{},[102,14731,124],{"mathvariant":601},[145,14733,644],{"encoding":147},[80,14735,14737],{"className":14736,"ariaHidden":113},[152],[80,14738,14740,14743],{"className":14739},[156],[80,14741],{"className":14742,"style":614},[160],[80,14744,124],{"className":14745},[165,618],". A vector's size is called its ",[7103,14748,14750],{"definition":14749},"a tuple describing an array's layout, how many elements exist along each dimension","shape",[80,14752,14754,14774],{"className":14753},[83],[80,14755,14757],{"className":14756},[87],[89,14758,14759],{"xmlns":91},[93,14760,14761,14771],{},[96,14762,14763,14765,14767,14769],{},[111,14764,121],{"stretchy":120},[102,14766,1487],{},[111,14768,114],{"separator":113},[111,14770,127],{"stretchy":120},[145,14772,14773],{"encoding":147},"(n,)",[80,14775,14777],{"className":14776,"ariaHidden":113},[152],[80,14778,14780,14783,14786,14789,14792,14795],{"className":14779},[156],[80,14781],{"className":14782,"style":2316},[160],[80,14784,121],{"className":14785},[235],[80,14787,1487],{"className":14788},[165,169],[80,14790,114],{"className":14791},[214],[80,14793],{"className":14794,"style":268},[246],[80,14796,127],{"className":14797},[242]," for a vector with ",[80,14800,14802,14815],{"className":14801},[83],[80,14803,14805],{"className":14804},[87],[89,14806,14807],{"xmlns":91},[93,14808,14809,14813],{},[96,14810,14811],{},[102,14812,1487],{},[145,14814,1487],{"encoding":147},[80,14816,14818],{"className":14817,"ariaHidden":113},[152],[80,14819,14821,14824],{"className":14820},[156],[80,14822],{"className":14823,"style":334},[160],[80,14825,1487],{"className":14826},[165,169]," elements. Notice the lone comma inside the parentheses: that's Python's way of saying \"this is a one-element tuple\", and it's easy to confuse ",[80,14829,14831,14850],{"className":14830},[83],[80,14832,14834],{"className":14833},[87],[89,14835,14836],{"xmlns":91},[93,14837,14838,14848],{},[96,14839,14840,14842,14844,14846],{},[111,14841,121],{"stretchy":120},[102,14843,1487],{},[111,14845,114],{"separator":113},[111,14847,127],{"stretchy":120},[145,14849,14773],{"encoding":147},[80,14851,14853],{"className":14852,"ariaHidden":113},[152],[80,14854,14856,14859,14862,14865,14868,14871],{"className":14855},[156],[80,14857],{"className":14858,"style":2316},[160],[80,14860,121],{"className":14861},[235],[80,14863,1487],{"className":14864},[165,169],[80,14866,114],{"className":14867},[214],[80,14869],{"className":14870,"style":268},[246],[80,14872,127],{"className":14873},[242]," with ",[80,14876,14878,14900],{"className":14877},[83],[80,14879,14881],{"className":14880},[87],[89,14882,14883],{"xmlns":91},[93,14884,14885,14897],{},[96,14886,14887,14889,14891,14893,14895],{},[111,14888,121],{"stretchy":120},[102,14890,1487],{},[111,14892,114],{"separator":113},[1321,14894,1583],{},[111,14896,127],{"stretchy":120},[145,14898,14899],{"encoding":147},"(n,1)",[80,14901,14903],{"className":14902,"ariaHidden":113},[152],[80,14904,14906,14909,14912,14915,14918,14921,14924],{"className":14905},[156],[80,14907],{"className":14908,"style":2316},[160],[80,14910,121],{"className":14911},[235],[80,14913,1487],{"className":14914},[165,169],[80,14916,114],{"className":14917},[214],[80,14919],{"className":14920,"style":268},[246],[80,14922,1583],{"className":14923},[165],[80,14925,127],{"className":14926},[242]," later on, so keep an eye on that.",[2611,14929,14931],{"className":2613,"code":14930,"language":2615,"meta":26,"style":26},"import numpy as np\n\na = np.zeros(4)\nprint(f\"np.zeros(4): a = {a}, shape = {a.shape}, dtype = {a.dtype}\")\n# a = [0. 0. 0. 0.], shape = (4,), dtype = float64\n\na = np.array([5, 4, 3, 2])\nprint(f\"np.array: a = {a}, shape = {a.shape}, dtype = {a.dtype}\")\n# a = [5 4 3 2], shape = (4,), dtype = int64\n",[65,14932,14933,14938,14942,14947,14952,14957,14961,14966,14971],{"__ignoreMap":26},[80,14934,14935],{"class":2620,"line":33},[80,14936,14937],{},"import numpy as np\n",[80,14939,14940],{"class":2620,"line":27},[80,14941,2657],{"emptyLinePlaceholder":32},[80,14943,14944],{"class":2620,"line":2631},[80,14945,14946],{},"a = np.zeros(4)\n",[80,14948,14949],{"class":2620,"line":2636},[80,14950,14951],{},"print(f\"np.zeros(4): a = {a}, shape = {a.shape}, dtype = {a.dtype}\")\n",[80,14953,14954],{"class":2620,"line":2642},[80,14955,14956],{},"# a = [0. 0. 0. 0.], shape = (4,), dtype = float64\n",[80,14958,14959],{"class":2620,"line":2648},[80,14960,2657],{"emptyLinePlaceholder":32},[80,14962,14963],{"class":2620,"line":2654},[80,14964,14965],{},"a = np.array([5, 4, 3, 2])\n",[80,14967,14968],{"class":2620,"line":2660},[80,14969,14970],{},"print(f\"np.array: a = {a}, shape = {a.shape}, dtype = {a.dtype}\")\n",[80,14972,14973],{"class":2620,"line":2666},[80,14974,14975],{},"# a = [5 4 3 2], shape = (4,), dtype = int64\n",[11,14977,14978,14979,14982],{},"A detail that tripped me up as a newcomer: a single decimal-point value in the list is enough for the whole array to become ",[65,14980,14981],{},"float64",". Makes sense, since every element of an array has to share the same type.",[2606,14984,14986],{"id":14985},"indexing-and-slicing","Indexing and slicing",[11,14988,14989,14990,14993,14994,14997,14998,15001,15002,15004,15005,15007],{},"Indexing (",[65,14991,14992],{},"a[2]",") grabs one element, while slicing (",[65,14995,14996],{},"a[2:7]",") grabs a chunk. The rules are the same as Python lists: counting starts at zero, negative indices count from the end (",[65,14999,15000],{},"a[-1]"," is the last element), and the end of a slice is ",[15,15003,1294],{}," included (",[65,15006,14996],{}," grabs indices 2, 3, 4, 5, and 6, five elements, not six).",[2611,15009,15011],{"className":2613,"code":15010,"language":2615,"meta":26,"style":26},"a = np.arange(10)          # [0 1 2 3 4 5 6 7 8 9]\n\nprint(a[2])                 # 2, grabbing one element = a scalar\nprint(a[-1])                 # 9, the last element\nprint(a[2:7:1])              # [2 3 4 5 6], from index 2 through 6\nprint(a[3:])                  # [3 4 5 6 7 8 9], from index 3 to the end\nprint(a[:3])                   # [0 1 2], from the start through index 2\n",[65,15012,15013,15018,15022,15027,15032,15037,15042],{"__ignoreMap":26},[80,15014,15015],{"class":2620,"line":33},[80,15016,15017],{},"a = np.arange(10)          # [0 1 2 3 4 5 6 7 8 9]\n",[80,15019,15020],{"class":2620,"line":27},[80,15021,2657],{"emptyLinePlaceholder":32},[80,15023,15024],{"class":2620,"line":2631},[80,15025,15026],{},"print(a[2])                 # 2, grabbing one element = a scalar\n",[80,15028,15029],{"class":2620,"line":2636},[80,15030,15031],{},"print(a[-1])                 # 9, the last element\n",[80,15033,15034],{"class":2620,"line":2642},[80,15035,15036],{},"print(a[2:7:1])              # [2 3 4 5 6], from index 2 through 6\n",[80,15038,15039],{"class":2620,"line":2648},[80,15040,15041],{},"print(a[3:])                  # [3 4 5 6 7 8 9], from index 3 to the end\n",[80,15043,15044],{"class":2620,"line":2654},[80,15045,15046],{},"print(a[:3])                   # [0 1 2], from the start through index 2\n",[2606,15048,15050],{"id":15049},"operations-with-no-loop-at-all","Operations with no loop at all",[11,15052,15053,15054,15056],{},"This is where the good part lives. Vector operations run over the entire array at once, with no ",[65,15055,8813],{}," written by you:",[2611,15058,15060],{"className":2613,"code":15059,"language":2615,"meta":26,"style":26},"a = np.array([1, 2, 3, 4])\n\nprint(-a)          # [-1 -2 -3 -4], flips the sign of everyone at once\nprint(np.sum(a))   # 10, sums it all up\nprint(a ** 2)       # [1 4 9 16], squares every element\nprint(5 * a)          # [5 10 15 20], multiplies every element by 5\n",[65,15061,15062,15067,15071,15076,15081,15086],{"__ignoreMap":26},[80,15063,15064],{"class":2620,"line":33},[80,15065,15066],{},"a = np.array([1, 2, 3, 4])\n",[80,15068,15069],{"class":2620,"line":27},[80,15070,2657],{"emptyLinePlaceholder":32},[80,15072,15073],{"class":2620,"line":2631},[80,15074,15075],{},"print(-a)          # [-1 -2 -3 -4], flips the sign of everyone at once\n",[80,15077,15078],{"class":2620,"line":2636},[80,15079,15080],{},"print(np.sum(a))   # 10, sums it all up\n",[80,15082,15083],{"class":2620,"line":2642},[80,15084,15085],{},"print(a ** 2)       # [1 4 9 16], squares every element\n",[80,15087,15088],{"class":2620,"line":2648},[80,15089,15090],{},"print(5 * a)          # [5 10 15 20], multiplies every element by 5\n",[11,15092,15093,15094,15097,15098,15102],{},"Notice that last example: the ",[65,15095,15096],{},"5"," was just a loose number, but NumPy \"stretched\" it on its own to match the array's 4 elements. That has a name, ",[7103,15099,15101],{"definition":15100},"when NumPy stretches a number (or a smaller array) to match the shape of a bigger array, with no loop written by you","broadcasting",", and it'll show up a lot from here on.",[11,15104,15105,15106,15108],{},"That's the whole point of this lab: you describe the operation over the entire array, and let NumPy figure out how to apply it to each element under the hood. If you catch yourself writing a ",[65,15107,8813],{}," to walk through an array, there's probably a vectorized equivalent waiting to be used instead.",[294,15110,15112],{"id":15111},"the-dot-product-the-operation-linear-regression-runs-on","The dot product: the operation linear regression runs on",[11,15114,15115,15116,15120,15121,15124,15125,15128],{},"The ",[7103,15117,15119],{"definition":15118},"multiplying two vectors element by element and summing everything up, ending in a single number","dot product"," is this lab's most important operation, because it's literally what's going to replace that ",[65,15122,15123],{},"w * x[i] + b"," we wrote with a loop ",[562,15126,15127],{"href":6979},"in the previous posts",", once the model gains several features.",[11,15130,15131],{},[80,15132,15134,15187],{"className":15133},[83],[80,15135,15137],{"className":15136},[87],[89,15138,15139],{"xmlns":91},[93,15140,15141,15184],{},[96,15142,15143,15145,15148,15150,15152,15172,15178],{},[102,15144,562],{"mathvariant":601},[111,15146,15147],{},"⋅",[102,15149,117],{"mathvariant":601},[111,15151,130],{},[7202,15153,15154,15156,15164],{},[111,15155,7206],{},[96,15157,15158,15160,15162],{},[102,15159,736],{},[111,15161,130],{},[1321,15163,2071],{},[96,15165,15166,15168,15170],{},[102,15167,1487],{},[111,15169,4643],{},[1321,15171,1583],{},[99,15173,15174,15176],{},[102,15175,562],{},[102,15177,736],{},[99,15179,15180,15182],{},[102,15181,117],{},[102,15183,736],{},[145,15185,15186],{"encoding":147},"\\mathbf{a} \\cdot \\mathbf{b} = \\sum_{i=0}^{n-1} a_i b_i",[80,15188,15190,15209,15227],{"className":15189,"ariaHidden":113},[152],[80,15191,15193,15197,15200,15203,15206],{"className":15192},[156],[80,15194],{"className":15195,"style":15196},[160],"height:0.4445em;",[80,15198,562],{"className":15199},[165,618],[80,15201],{"className":15202,"style":275},[246],[80,15204,15147],{"className":15205},[279],[80,15207],{"className":15208,"style":275},[246],[80,15210,15212,15215,15218,15221,15224],{"className":15211},[156],[80,15213],{"className":15214,"style":289},[160],[80,15216,117],{"className":15217},[165,618],[80,15219],{"className":15220,"style":247},[246],[80,15222,130],{"className":15223},[251],[80,15225],{"className":15226,"style":247},[246],[80,15228,15230,15233,15302,15305,15348],{"className":15229},[156],[80,15231],{"className":15232,"style":8277},[160],[80,15234,15236,15239],{"className":15235},[7402],[80,15237,7206],{"className":15238,"style":7408},[7402,7406,7407],[80,15240,15242],{"className":15241},[174],[80,15243,15245,15294],{"className":15244},[178,179],[80,15246,15248,15291],{"className":15247},[183],[80,15249,15251,15271],{"className":15250,"style":7421},[187],[80,15252,15253,15256],{"style":7424},[80,15254],{"className":15255,"style":196},[195],[80,15257,15259],{"className":15258},[200,201,202,203],[80,15260,15262,15265,15268],{"className":15261},[165,203],[80,15263,736],{"className":15264},[165,169,203],[80,15266,130],{"className":15267},[251,203],[80,15269,2071],{"className":15270},[165,203],[80,15272,15273,15276],{"style":7445},[80,15274],{"className":15275,"style":196},[195],[80,15277,15279],{"className":15278},[200,201,202,203],[80,15280,15282,15285,15288],{"className":15281},[165,203],[80,15283,1487],{"className":15284},[165,169,203],[80,15286,4643],{"className":15287},[279,203],[80,15289,1583],{"className":15290},[165,203],[80,15292,222],{"className":15293},[221],[80,15295,15297],{"className":15296},[183],[80,15298,15300],{"className":15299,"style":7473},[187],[80,15301],{},[80,15303],{"className":15304,"style":268},[246],[80,15306,15308,15311],{"className":15307},[165],[80,15309,562],{"className":15310},[165,169],[80,15312,15314],{"className":15313},[174],[80,15315,15317,15339],{"className":15316},[178,179],[80,15318,15320,15336],{"className":15319},[183],[80,15321,15324],{"className":15322,"style":15323},[187],"height:0.3117em;",[80,15325,15327,15330],{"style":15326},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[80,15328],{"className":15329,"style":196},[195],[80,15331,15333],{"className":15332},[200,201,202,203],[80,15334,736],{"className":15335},[165,169,203],[80,15337,222],{"className":15338},[221],[80,15340,15342],{"className":15341},[183],[80,15343,15346],{"className":15344,"style":15345},[187],"height:0.15em;",[80,15347],{},[80,15349,15351,15354],{"className":15350},[165],[80,15352,117],{"className":15353},[165,169],[80,15355,15357],{"className":15356},[174],[80,15358,15360,15380],{"className":15359},[178,179],[80,15361,15363,15377],{"className":15362},[183],[80,15364,15366],{"className":15365,"style":15323},[187],[80,15367,15368,15371],{"style":15326},[80,15369],{"className":15370,"style":196},[195],[80,15372,15374],{"className":15373},[200,201,202,203],[80,15375,736],{"className":15376},[165,169,203],[80,15378,222],{"className":15379},[221],[80,15381,15383],{"className":15382},[183],[80,15384,15386],{"className":15385,"style":15345},[187],[80,15387],{},[11,15389,15390],{},"Multiply pairwise, sum it all up. That simple. Before reaching for NumPy's built-in version, I wrote my own, just to make what's happening under the hood explicit:",[2611,15392,15394],{"className":2613,"code":15393,"language":2615,"meta":26,"style":26},"def my_dot(a, b):\n    x = 0\n    for i in range(a.shape[0]):\n        x = x + a[i] * b[i]\n    return x\n\na = np.array([1, 2, 3, 4])\nb = np.array([-1, 4, 3, 2])\n\nprint(my_dot(a, b))       # 24\nprint(np.dot(a, b))       # 24, same result, NumPy's built-in function\n",[65,15395,15396,15401,15406,15411,15416,15421,15425,15429,15434,15438,15443],{"__ignoreMap":26},[80,15397,15398],{"class":2620,"line":33},[80,15399,15400],{},"def my_dot(a, b):\n",[80,15402,15403],{"class":2620,"line":27},[80,15404,15405],{},"    x = 0\n",[80,15407,15408],{"class":2620,"line":2631},[80,15409,15410],{},"    for i in range(a.shape[0]):\n",[80,15412,15413],{"class":2620,"line":2636},[80,15414,15415],{},"        x = x + a[i] * b[i]\n",[80,15417,15418],{"class":2620,"line":2642},[80,15419,15420],{},"    return x\n",[80,15422,15423],{"class":2620,"line":2648},[80,15424,2657],{"emptyLinePlaceholder":32},[80,15426,15427],{"class":2620,"line":2654},[80,15428,15066],{},[80,15430,15431],{"class":2620,"line":2660},[80,15432,15433],{},"b = np.array([-1, 4, 3, 2])\n",[80,15435,15436],{"class":2620,"line":2666},[80,15437,2657],{"emptyLinePlaceholder":32},[80,15439,15440],{"class":2620,"line":2672},[80,15441,15442],{},"print(my_dot(a, b))       # 24\n",[80,15444,15445],{"class":2620,"line":2677},[80,15446,15447],{},"print(np.dot(a, b))       # 24, same result, NumPy's built-in function\n",[294,15449,15451],{"id":15450},"the-speed-gap-live-in-your-browser","The speed gap, live in your browser",[11,15453,15454,15455,15458,15459,15461],{},"The original notebook tests this with 10-million-element arrays, comparing ",[65,15456,15457],{},"np.dot"," (vectorized) against a hand-written ",[65,15460,8813],{}," loop, in Python. The gain there runs dozens to hundreds of times over.",[11,15463,15464,15465,15467],{},"Below, you can run a similar test, except in your own browser, right now, in JavaScript. I need to be honest with you about one thing first, though: the browser is ",[15,15466,1294],{}," going to show a 100x gain. And it's not because the test is broken.",[11,15469,15470,15471,15473,15474,15477,15478,15480],{},"Your browser's JavaScript engine (V8, if you're on Chrome or Edge) already optimizes a plain ",[65,15472,8813],{}," loop in a way pretty similar to what it does for built-in methods like ",[65,15475,15476],{},".reduce()",". When I compared \"loop\" against \"",[65,15479,15476],{},"\" on the same kind of array, it came out close to a tie, and that was expected, not a bug in the test.",[11,15482,15483,15484,15487,15488,15491],{},"The comparison that actually shows a real, honest gap is a different one: a plain JavaScript ",[65,15485,15486],{},"Array"," (which stores numbers somewhat \"boxed up\", similar to the Python list we talked about above) versus a ",[65,15489,15490],{},"Float64Array"," (contiguous memory, single type, no boxing). That's the same underlying reason NumPy is fast, just reproduced here in the browser with a smaller, honest gain (usually 2 to 5x, sometimes more), instead of me pretending I could recreate Python's giant number in an environment that wasn't built for that.",[15493,15494],"vectorization-benchmark",{},[11,15496,15497],{},"Run it a few times with different sizes. Notice that \"Array comum + laço\" (plain Array + loop) tends to be visibly slower than the other two, and that \"Float64Array + laço\" (typed array + loop) and \"Float64Array + .reduce()\" land close to each other. That's exactly the pattern the explanation above predicted.",[294,15499,15501],{"id":15500},"matrices-when-one-vector-isnt-enough","Matrices: when one vector isn't enough",[11,15503,15504,15505,15536,15537,15589,15590,15593],{},"A matrix is a two-dimensional array, written with a bold uppercase letter (",[80,15506,15508,15523],{"className":15507},[83],[80,15509,15511],{"className":15510},[87],[89,15512,15513],{"xmlns":91},[93,15514,15515,15520],{},[96,15516,15517],{},[102,15518,15519],{"mathvariant":601},"X",[145,15521,15522],{"encoding":147},"\\mathbf{X}",[80,15524,15526],{"className":15525,"ariaHidden":113},[152],[80,15527,15529,15533],{"className":15528},[156],[80,15530],{"className":15531,"style":15532},[160],"height:0.6861em;",[80,15534,15519],{"className":15535},[165,618],"), with shape ",[80,15538,15540,15562],{"className":15539},[83],[80,15541,15543],{"className":15542},[87],[89,15544,15545],{"xmlns":91},[93,15546,15547,15559],{},[96,15548,15549,15551,15553,15555,15557],{},[111,15550,121],{"stretchy":120},[102,15552,322],{},[111,15554,114],{"separator":113},[102,15556,1487],{},[111,15558,127],{"stretchy":120},[145,15560,15561],{"encoding":147},"(m, n)",[80,15563,15565],{"className":15564,"ariaHidden":113},[152],[80,15566,15568,15571,15574,15577,15580,15583,15586],{"className":15567},[156],[80,15569],{"className":15570,"style":2316},[160],[80,15572,121],{"className":15573},[235],[80,15575,322],{"className":15576},[165,169],[80,15578,114],{"className":15579},[214],[80,15581],{"className":15582,"style":268},[246],[80,15584,1487],{"className":15585},[165,169],[80,15587,127],{"className":15588},[242],". In the course's context the convention is always the same: ",[15,15591,15592],{},"row is a training example, column is a feature",". One house per row, one attribute (size, bedrooms, age) per column.",[2611,15595,15597],{"className":2613,"code":15596,"language":2615,"meta":26,"style":26},"X = np.array([[1, 5], [2, 3], [3, 1]])   # shape (3, 2): 3 examples, 2 features each\n\nprint(X.shape)      # (3, 2)\nprint(X[1])          # [2 3], the entire row 1, becomes a 1-D vector\nprint(X[1, 0])         # 2, one specific element, becomes a scalar\nprint(X[:, 0])           # [1 2 3], the entire column 0\n",[65,15598,15599,15604,15608,15613,15618,15623],{"__ignoreMap":26},[80,15600,15601],{"class":2620,"line":33},[80,15602,15603],{},"X = np.array([[1, 5], [2, 3], [3, 1]])   # shape (3, 2): 3 examples, 2 features each\n",[80,15605,15606],{"class":2620,"line":27},[80,15607,2657],{"emptyLinePlaceholder":32},[80,15609,15610],{"class":2620,"line":2631},[80,15611,15612],{},"print(X.shape)      # (3, 2)\n",[80,15614,15615],{"class":2620,"line":2636},[80,15616,15617],{},"print(X[1])          # [2 3], the entire row 1, becomes a 1-D vector\n",[80,15619,15620],{"class":2620,"line":2642},[80,15621,15622],{},"print(X[1, 0])         # 2, one specific element, becomes a scalar\n",[80,15624,15625],{"class":2620,"line":2648},[80,15626,15627],{},"print(X[:, 0])           # [1 2 3], the entire column 0\n",[11,15629,15630,15631,15633,15634,15637,15638,15641,15642,15644,15645,15648,15649,15652],{},"The detail that confuses newcomers the most here: indexing a matrix with just ",[15,15632,9826],{}," index (",[65,15635,15636],{},"X[1]",") returns an array with ",[15,15639,15640],{},"one fewer dimension",", not a single-row matrix. ",[65,15643,15636],{}," has shape ",[65,15646,15647],{},"(2,)",", not ",[65,15650,15651],{},"(1, 2)",". This specific gotcha is behind a good chunk of the dimension errors you'll run into later in the course, and I've learned to flag it mentally every time I touch shapes.",[11,15654,15655,15656,15684,15685,15713,15714,15760,15761,15790,15791,15837,15838,15866,15867,15870,15871,15899],{},"And this is exactly why the dot product comes back into play: once the model has ",[80,15657,15659,15672],{"className":15658},[83],[80,15660,15662],{"className":15661},[87],[89,15663,15664],{"xmlns":91},[93,15665,15666,15670],{},[96,15667,15668],{},[102,15669,1487],{},[145,15671,1487],{"encoding":147},[80,15673,15675],{"className":15674,"ariaHidden":113},[152],[80,15676,15678,15681],{"className":15677},[156],[80,15679],{"className":15680,"style":334},[160],[80,15682,1487],{"className":15683},[165,169]," features, each row of the matrix ",[80,15686,15688,15701],{"className":15687},[83],[80,15689,15691],{"className":15690},[87],[89,15692,15693],{"xmlns":91},[93,15694,15695,15699],{},[96,15696,15697],{},[102,15698,15519],{"mathvariant":601},[145,15700,15522],{"encoding":147},[80,15702,15704],{"className":15703,"ariaHidden":113},[152],[80,15705,15707,15710],{"className":15706},[156],[80,15708],{"className":15709,"style":15532},[160],[80,15711,15519],{"className":15712},[165,618]," becomes a vector of shape ",[80,15715,15717,15736],{"className":15716},[83],[80,15718,15720],{"className":15719},[87],[89,15721,15722],{"xmlns":91},[93,15723,15724,15734],{},[96,15725,15726,15728,15730,15732],{},[111,15727,121],{"stretchy":120},[102,15729,1487],{},[111,15731,114],{"separator":113},[111,15733,127],{"stretchy":120},[145,15735,14773],{"encoding":147},[80,15737,15739],{"className":15738,"ariaHidden":113},[152],[80,15740,15742,15745,15748,15751,15754,15757],{"className":15741},[156],[80,15743],{"className":15744,"style":2316},[160],[80,15746,121],{"className":15747},[235],[80,15749,1487],{"className":15750},[165,169],[80,15752,114],{"className":15753},[214],[80,15755],{"className":15756,"style":268},[246],[80,15758,127],{"className":15759},[242],", ready to take a dot product directly with the weight vector ",[80,15762,15764,15778],{"className":15763},[83],[80,15765,15767],{"className":15766},[87],[89,15768,15769],{"xmlns":91},[93,15770,15771,15775],{},[96,15772,15773],{},[102,15774,109],{"mathvariant":601},[145,15776,15777],{"encoding":147},"\\mathbf{w}",[80,15779,15781],{"className":15780,"ariaHidden":113},[152],[80,15782,15784,15787],{"className":15783},[156],[80,15785],{"className":15786,"style":614},[160],[80,15788,109],{"className":15789,"style":700},[165,618],", also of shape ",[80,15792,15794,15813],{"className":15793},[83],[80,15795,15797],{"className":15796},[87],[89,15798,15799],{"xmlns":91},[93,15800,15801,15811],{},[96,15802,15803,15805,15807,15809],{},[111,15804,121],{"stretchy":120},[102,15806,1487],{},[111,15808,114],{"separator":113},[111,15810,127],{"stretchy":120},[145,15812,14773],{"encoding":147},[80,15814,15816],{"className":15815,"ariaHidden":113},[152],[80,15817,15819,15822,15825,15828,15831,15834],{"className":15818},[156],[80,15820],{"className":15821,"style":2316},[160],[80,15823,121],{"className":15824},[235],[80,15826,1487],{"className":15827},[165,169],[80,15829,114],{"className":15830},[214],[80,15832],{"className":15833,"style":268},[246],[80,15835,127],{"className":15836},[242],". Example ",[80,15839,15841,15854],{"className":15840},[83],[80,15842,15844],{"className":15843},[87],[89,15845,15846],{"xmlns":91},[93,15847,15848,15852],{},[96,15849,15850],{},[102,15851,736],{},[145,15853,736],{"encoding":147},[80,15855,15857],{"className":15856,"ariaHidden":113},[152],[80,15858,15860,15863],{"className":15859},[156],[80,15861],{"className":15862,"style":895},[160],[80,15864,736],{"className":15865},[165,169],"'s prediction becomes ",[65,15868,15869],{},"np.dot(w, X[i]) + b",", a single line of code, no loop at all, whether ",[80,15872,15874,15887],{"className":15873},[83],[80,15875,15877],{"className":15876},[87],[89,15878,15879],{"xmlns":91},[93,15880,15881,15885],{},[96,15882,15883],{},[102,15884,1487],{},[145,15886,1487],{"encoding":147},[80,15888,15890],{"className":15889,"ariaHidden":113},[152],[80,15891,15893,15896],{"className":15892},[156],[80,15894],{"className":15895,"style":334},[160],[80,15897,1487],{"className":15898},[165,169]," is 1 or 100.",[294,15901,6192],{"id":6191},[521,15903,15904,15913],{},[524,15905,15906],{},[527,15907,15908,15911],{},[530,15909,15910],{"align":532},"Topic",[530,15912,9841],{"align":532},[541,15914,15915,15923,16076,16084,16095,16106],{},[527,15916,15917,15920],{},[546,15918,15919],{"align":532},"Why NumPy",[546,15921,15922],{"align":532},"contiguous, typed memory with no per-element boxing is where the speed comes from",[527,15924,15925,15927],{},[546,15926,9990],{"align":532},[546,15928,15929,15975,15976,16028,16029,16075],{"align":532},[80,15930,15932,15951],{"className":15931},[83],[80,15933,15935],{"className":15934},[87],[89,15936,15937],{"xmlns":91},[93,15938,15939,15949],{},[96,15940,15941,15943,15945,15947],{},[111,15942,121],{"stretchy":120},[102,15944,1487],{},[111,15946,114],{"separator":113},[111,15948,127],{"stretchy":120},[145,15950,14773],{"encoding":147},[80,15952,15954],{"className":15953,"ariaHidden":113},[152],[80,15955,15957,15960,15963,15966,15969,15972],{"className":15956},[156],[80,15958],{"className":15959,"style":2316},[160],[80,15961,121],{"className":15962},[235],[80,15964,1487],{"className":15965},[165,169],[80,15967,114],{"className":15968},[214],[80,15970],{"className":15971,"style":268},[246],[80,15973,127],{"className":15974},[242]," is a vector, ",[80,15977,15979,16001],{"className":15978},[83],[80,15980,15982],{"className":15981},[87],[89,15983,15984],{"xmlns":91},[93,15985,15986,15998],{},[96,15987,15988,15990,15992,15994,15996],{},[111,15989,121],{"stretchy":120},[102,15991,322],{},[111,15993,114],{"separator":113},[102,15995,1487],{},[111,15997,127],{"stretchy":120},[145,15999,16000],{"encoding":147},"(m,n)",[80,16002,16004],{"className":16003,"ariaHidden":113},[152],[80,16005,16007,16010,16013,16016,16019,16022,16025],{"className":16006},[156],[80,16008],{"className":16009,"style":2316},[160],[80,16011,121],{"className":16012},[235],[80,16014,322],{"className":16015},[165,169],[80,16017,114],{"className":16018},[214],[80,16020],{"className":16021,"style":268},[246],[80,16023,1487],{"className":16024},[165,169],[80,16026,127],{"className":16027},[242]," is a matrix, and the lone comma in ",[80,16030,16032,16051],{"className":16031},[83],[80,16033,16035],{"className":16034},[87],[89,16036,16037],{"xmlns":91},[93,16038,16039,16049],{},[96,16040,16041,16043,16045,16047],{},[111,16042,121],{"stretchy":120},[102,16044,1487],{},[111,16046,114],{"separator":113},[111,16048,127],{"stretchy":120},[145,16050,14773],{"encoding":147},[80,16052,16054],{"className":16053,"ariaHidden":113},[152],[80,16055,16057,16060,16063,16066,16069,16072],{"className":16056},[156],[80,16058],{"className":16059,"style":2316},[160],[80,16061,121],{"className":16062},[235],[80,16064,1487],{"className":16065},[165,169],[80,16067,114],{"className":16068},[214],[80,16070],{"className":16071,"style":268},[246],[80,16073,127],{"className":16074},[242]," matters",[527,16077,16078,16081],{},[546,16079,16080],{"align":532},"Indexing",[546,16082,16083],{"align":532},"starts at zero, and a single index into a matrix returns one fewer dimension",[527,16085,16086,16089],{},[546,16087,16088],{"align":532},"Slicing",[546,16090,16091,16092],{"align":532},"the end is always exclusive, same as Python's ",[65,16093,16094],{},"range()",[527,16096,16097,16100],{},[546,16098,16099],{"align":532},"Dot product",[546,16101,16102,16105],{"align":532},[65,16103,16104],{},"np.dot(a, b)",", multiplies pairwise and sums, returns a scalar",[527,16107,16108,16111],{},[546,16109,16110],{"align":532},"Vectorization",[546,16112,16113],{"align":532},"the real gain comes from contiguous, single-type memory, not \"library magic\"",[11,16115,16116,16119,16120,16122,16123,16126,16127,16130],{},[15,16117,16118],{},"How this connects to the rest of the playlist:"," with these tools in hand, the model you built in ",[562,16121,14260],{"href":6979}," with a hand-written ",[65,16124,16125],{},"w * x + b"," becomes ",[65,16128,16129],{},"np.dot(w, x) + b",", working for any number of features without rewriting anything. This exact foundation is what the course uses to extend everything you've already seen (model, cost, gradient descent) to linear regression with multiple variables.",[294,16132,6716],{"id":6715},[11,16134,16135,16136,16139,16140,2338,16142,16144],{},"Same real housing dataset from the previous posts (",[562,16137,6725],{"href":6722,"rel":16138},[6724],"). This post doesn't train anything, so here the idea is just applying the matrix\u002Fvector mechanics on top of real column names, instead of the toy ",[65,16141,562],{},[65,16143,117],{}," examples.",[2611,16146,16148],{"className":2613,"code":16147,"language":2615,"meta":26,"style":26},"import pandas as pd\nimport numpy as np\n\ndf = pd.read_csv(\"real_estate_dataset.csv\")\n\nX = df[[\"Square_Feet\", \"Num_Bedrooms\"]].to_numpy()[:3]   # first 3 houses, 2 features\n\nprint(\"X.shape:\", X.shape)        # (3, 2): 3 examples, 2 features\nprint(\"X[1]:\", X[1])              # the entire second house, becomes a 1-D vector\nprint(\"X[:, 0]:\", X[:, 0])        # the entire Square_Feet column, all houses\n",[65,16149,16150,16154,16158,16162,16166,16170,16175,16179,16184,16189],{"__ignoreMap":26},[80,16151,16152],{"class":2620,"line":33},[80,16153,6740],{},[80,16155,16156],{"class":2620,"line":27},[80,16157,14937],{},[80,16159,16160],{"class":2620,"line":2631},[80,16161,2657],{"emptyLinePlaceholder":32},[80,16163,16164],{"class":2620,"line":2636},[80,16165,6749],{},[80,16167,16168],{"class":2620,"line":2642},[80,16169,2657],{"emptyLinePlaceholder":32},[80,16171,16172],{"class":2620,"line":2648},[80,16173,16174],{},"X = df[[\"Square_Feet\", \"Num_Bedrooms\"]].to_numpy()[:3]   # first 3 houses, 2 features\n",[80,16176,16177],{"class":2620,"line":2654},[80,16178,2657],{"emptyLinePlaceholder":32},[80,16180,16181],{"class":2620,"line":2660},[80,16182,16183],{},"print(\"X.shape:\", X.shape)        # (3, 2): 3 examples, 2 features\n",[80,16185,16186],{"class":2620,"line":2666},[80,16187,16188],{},"print(\"X[1]:\", X[1])              # the entire second house, becomes a 1-D vector\n",[80,16190,16191],{"class":2620,"line":2672},[80,16192,16193],{},"print(\"X[:, 0]:\", X[:, 0])        # the entire Square_Feet column, all houses\n",[46,16195,16196],{},[11,16197,16198,3255,16200,10213,16203,10213,16206],{},[15,16199,2693],{},[65,16201,16202],{},"X.shape: (3, 2)",[65,16204,16205],{},"X[1]: [55.15  5.]",[65,16207,16208],{},"X[:, 0]: [143.64  55.15 202.96]",[2611,16210,16212],{"className":2613,"code":16211,"language":2615,"meta":26,"style":26},"# A real dot product, with arbitrary weights just to illustrate the mechanics\n# (we're not training anything in this post, this isn't the \"right\" fit):\nw = np.array([2000, 10000])\n\nfor i in range(X.shape[0]):\n    print(f\"np.dot(w, X[{i}]) = {np.dot(w, X[i]):,.0f}\")\n",[65,16213,16214,16219,16224,16229,16233,16238],{"__ignoreMap":26},[80,16215,16216],{"class":2620,"line":33},[80,16217,16218],{},"# A real dot product, with arbitrary weights just to illustrate the mechanics\n",[80,16220,16221],{"class":2620,"line":27},[80,16222,16223],{},"# (we're not training anything in this post, this isn't the \"right\" fit):\n",[80,16225,16226],{"class":2620,"line":2631},[80,16227,16228],{},"w = np.array([2000, 10000])\n",[80,16230,16231],{"class":2620,"line":2636},[80,16232,2657],{"emptyLinePlaceholder":32},[80,16234,16235],{"class":2620,"line":2642},[80,16236,16237],{},"for i in range(X.shape[0]):\n",[80,16239,16240],{"class":2620,"line":2648},[80,16241,16242],{},"    print(f\"np.dot(w, X[{i}]) = {np.dot(w, X[i]):,.0f}\")\n",[46,16244,16245],{},[11,16246,16247,3255,16249,10213,16252,10213,16255],{},[15,16248,2693],{},[65,16250,16251],{},"np.dot(w, X[0]) = 297,280",[65,16253,16254],{},"np.dot(w, X[1]) = 160,300",[65,16256,16257],{},"np.dot(w, X[2]) = 455,920",[11,16259,16260,16261,2338,16263,16266,16267,16270,16271,16299],{},"Same math, same mechanics from the rest of the post, just running on top of real ",[65,16262,6791],{},[65,16264,16265],{},"Num_Bedrooms"," instead of ",[65,16268,16269],{},"[1, 2, 3, 4]",". Actually training these weights for real (finding the ",[80,16272,16274,16287],{"className":16273},[83],[80,16275,16277],{"className":16276},[87],[89,16278,16279],{"xmlns":91},[93,16280,16281,16285],{},[96,16282,16283],{},[102,16284,109],{"mathvariant":601},[145,16286,15777],{"encoding":147},[80,16288,16290],{"className":16289,"ariaHidden":113},[152],[80,16291,16293,16296],{"className":16292},[156],[80,16294],{"className":16295,"style":614},[160],[80,16297,109],{"className":16298,"style":700},[165,618]," that makes sense for the data) is content further ahead in the course, with multiple features at once.",[2606,16301,16303],{"id":16302},"picking-the-matrix-columns-live","Picking the matrix columns, live",[11,16305,16306,16307,16335,16336,2338,16364,16394],{},"Remember ",[80,16308,16310,16323],{"className":16309},[83],[80,16311,16313],{"className":16312},[87],[89,16314,16315],{"xmlns":91},[93,16316,16317,16321],{},[96,16318,16319],{},[102,16320,15519],{"mathvariant":601},[145,16322,15522],{"encoding":147},[80,16324,16326],{"className":16325,"ariaHidden":113},[152],[80,16327,16329,16332],{"className":16328},[156],[80,16330],{"className":16331,"style":15532},[160],[80,16333,15519],{"className":16334},[165,618]," is just a matrix, one row per example, one column per feature? Pick which two columns become the ",[80,16337,16339,16352],{"className":16338},[83],[80,16340,16342],{"className":16341},[87],[89,16343,16344],{"xmlns":91},[93,16345,16346,16350],{},[96,16347,16348],{},[102,16349,124],{},[145,16351,124],{"encoding":147},[80,16353,16355],{"className":16354,"ariaHidden":113},[152],[80,16356,16358,16361],{"className":16357},[156],[80,16359],{"className":16360,"style":334},[160],[80,16362,124],{"className":16363},[165,169],[80,16365,16367,16381],{"className":16366},[83],[80,16368,16370],{"className":16369},[87],[89,16371,16372],{"xmlns":91},[93,16373,16374,16379],{},[96,16375,16376],{},[102,16377,16378],{},"z",[145,16380,16378],{"encoding":147},[80,16382,16384],{"className":16383,"ariaHidden":113},[152],[80,16385,16387,16390],{"className":16386},[156],[80,16388],{"className":16389,"style":334},[160],[80,16391,16378],{"className":16392,"style":16393},[165,169],"margin-right:0.044em;"," axes below (height always stays price) and rotate to feel how each column pairing relates to the price of the 50 houses:",[16396,16397],"housing-feature-explorer3d",{"y-label":6804},[6949,16399,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":16401},[16402,16403,16407,16408,16409,16410,16411],{"id":14692,"depth":27,"text":14693},{"id":14713,"depth":27,"text":14714,"children":16404},[16405,16406],{"id":14985,"depth":2631,"text":14986},{"id":15049,"depth":2631,"text":15050},{"id":15111,"depth":27,"text":15112},{"id":15450,"depth":27,"text":15451},{"id":15500,"depth":27,"text":15501},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716,"children":16412},[16413],{"id":16302,"depth":2631,"text":16303},"Why every serious ML codebase uses NumPy instead of a Python list: vectors, matrices, the dot product, and the real (not the legendary) gap between a loop and a vectorized operation.",{},{"title":14675,"description":16414},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab01-numpy-vectorization",[6985,2715,16419],"tools","qbhEIwMXkIpf4m8_kfHWuHUpB1psJgLRD8EH-d6eFko",{"id":16422,"title":16423,"body":16424,"cover":3,"date":6976,"description":18468,"extension":30,"meta":18469,"navigation":32,"order":2642,"path":18470,"playlist":6980,"seo":18471,"status":36,"stem":18472,"tags":18473,"__hash__":18479},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions.md","Just a Little Extra: Loss Functions (MSE, MAE, Huber)",{"type":8,"value":16425,"toc":18459},[16426,16432,16442,16446,16458,16911,16917,16921,16929,17428,17434,17438,17441,17448,17451,17455,17495,18075,18107,18111,18125,18130,18133,18135,18194,18196,18216,18218,18224,18286,18301,18451,18454,18457],[11,16427,16428],{},[57,16429],{"alt":16430,"src":16431},"A \"look what they need to mimic a fraction of our power\" meme: a GPU and a 3D plot of a bumpy cost surface, next to a giant brain, with the caption underneath","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Floss-functions\u002Fmeme-loss-function.jpeg",[11,16433,16434,16435,3255,16438,16441],{},"This post doesn't come from any specific course lab, it's a bonus we earned from hammering on the cost function so much in ",[562,16436,16437],{"href":4066},"posts 2",[562,16439,16440],{"href":4070},"and 3",". Back there we squared the error without questioning the choice much, but \"squaring it\" is a choice, not the only option. Let's open that up.",[294,16443,16445],{"id":16444},"naming-what-you-already-built-mse","Naming what you already built: MSE",[11,16447,16448,16449,16452,16453,16457],{},"The cost function you've been using since ",[562,16450,16451],{"href":4066},"post 2"," has an official name: ",[7103,16454,16456],{"definition":16455},"Mean Squared Error, the same cost function you've already been using since post 2","MSE"," (Mean Squared Error).",[11,16459,16460],{},[80,16461,16463,16557],{"className":16462},[83],[80,16464,16466],{"className":16465},[87],[89,16467,16468],{"xmlns":91},[93,16469,16470,16554],{},[96,16471,16472,16474,16476,16482,16502],{},[134,16473,16456],{},[111,16475,130],{},[4625,16477,16478,16480],{},[1321,16479,1583],{},[102,16481,322],{},[7202,16483,16484,16486,16494],{},[111,16485,7206],{},[96,16487,16488,16490,16492],{},[102,16489,736],{},[111,16491,130],{},[1321,16493,2071],{},[96,16495,16496,16498,16500],{},[102,16497,322],{},[111,16499,4643],{},[1321,16501,1583],{},[726,16503,16504,16552],{},[96,16505,16506,16508,16520,16522,16534,16536,16538,16550],{},[111,16507,121],{"fence":113},[99,16509,16510,16512],{},[102,16511,104],{},[96,16513,16514,16516,16518],{},[102,16515,109],{},[111,16517,114],{"separator":113},[102,16519,117],{},[111,16521,121],{"stretchy":120},[726,16523,16524,16526],{},[102,16525,124],{},[96,16527,16528,16530,16532],{},[111,16529,121],{"stretchy":120},[102,16531,736],{},[111,16533,127],{"stretchy":120},[111,16535,127],{"stretchy":120},[111,16537,4643],{},[726,16539,16540,16542],{},[102,16541,683],{},[96,16543,16544,16546,16548],{},[111,16545,121],{"stretchy":120},[102,16547,736],{},[111,16549,127],{"stretchy":120},[111,16551,127],{"fence":113},[1321,16553,1323],{},[145,16555,16556],{"encoding":147},"\\text{MSE} = \\frac{1}{m}\\sum_{i=0}^{m-1}\\left(f_{w,b}(x^{(i)}) - y^{(i)}\\right)^2",[80,16558,16560,16581],{"className":16559,"ariaHidden":113},[152],[80,16561,16563,16566,16572,16575,16578],{"className":16562},[156],[80,16564],{"className":16565,"style":8672},[160],[80,16567,16569],{"className":16568},[165,2699],[80,16570,16456],{"className":16571},[165],[80,16573],{"className":16574,"style":247},[246],[80,16576,130],{"className":16577},[251],[80,16579],{"className":16580,"style":247},[246],[80,16582,16584,16587,16655,16658,16727,16730],{"className":16583},[156],[80,16585],{"className":16586,"style":7323},[160],[80,16588,16590,16593,16652],{"className":16589},[165],[80,16591],{"className":16592},[235,4746],[80,16594,16596],{"className":16595},[4625],[80,16597,16599,16644],{"className":16598},[178,179],[80,16600,16602,16641],{"className":16601},[183],[80,16603,16605,16619,16627],{"className":16604,"style":5010},[187],[80,16606,16607,16610],{"style":5013},[80,16608],{"className":16609,"style":4766},[195],[80,16611,16613],{"className":16612},[200,201,202,203],[80,16614,16616],{"className":16615},[165,203],[80,16617,322],{"className":16618},[165,169,203],[80,16620,16621,16624],{"style":4859},[80,16622],{"className":16623,"style":4766},[195],[80,16625],{"className":16626,"style":4867},[4866],[80,16628,16629,16632],{"style":5042},[80,16630],{"className":16631,"style":4766},[195],[80,16633,16635],{"className":16634},[200,201,202,203],[80,16636,16638],{"className":16637},[165,203],[80,16639,1583],{"className":16640},[165,203],[80,16642,222],{"className":16643},[221],[80,16645,16647],{"className":16646},[183],[80,16648,16650],{"className":16649,"style":7390},[187],[80,16651],{},[80,16653],{"className":16654},[242,4746],[80,16656],{"className":16657,"style":268},[246],[80,16659,16661,16664],{"className":16660},[7402],[80,16662,7206],{"className":16663,"style":7408},[7402,7406,7407],[80,16665,16667],{"className":16666},[174],[80,16668,16670,16719],{"className":16669},[178,179],[80,16671,16673,16716],{"className":16672},[183],[80,16674,16676,16696],{"className":16675,"style":7421},[187],[80,16677,16678,16681],{"style":7424},[80,16679],{"className":16680,"style":196},[195],[80,16682,16684],{"className":16683},[200,201,202,203],[80,16685,16687,16690,16693],{"className":16686},[165,203],[80,16688,736],{"className":16689},[165,169,203],[80,16691,130],{"className":16692},[251,203],[80,16694,2071],{"className":16695},[165,203],[80,16697,16698,16701],{"style":7445},[80,16699],{"className":16700,"style":196},[195],[80,16702,16704],{"className":16703},[200,201,202,203],[80,16705,16707,16710,16713],{"className":16706},[165,203],[80,16708,322],{"className":16709},[165,169,203],[80,16711,4643],{"className":16712},[279,203],[80,16714,1583],{"className":16715},[165,203],[80,16717,222],{"className":16718},[221],[80,16720,16722],{"className":16721},[183],[80,16723,16725],{"className":16724,"style":7473},[187],[80,16726],{},[80,16728],{"className":16729,"style":268},[246],[80,16731,16733,16888],{"className":16732},[7482],[80,16734,16736,16742,16791,16794,16832,16835,16838,16841,16844,16882],{"className":16735},[7482],[80,16737,16739],{"className":16738,"style":7490},[235,7489],[80,16740,121],{"className":16741},[7494,4803],[80,16743,16745,16748],{"className":16744},[165],[80,16746,104],{"className":16747,"style":170},[165,169],[80,16749,16751],{"className":16750},[174],[80,16752,16754,16783],{"className":16753},[178,179],[80,16755,16757,16780],{"className":16756},[183],[80,16758,16760],{"className":16759,"style":188},[187],[80,16761,16762,16765],{"style":191},[80,16763],{"className":16764,"style":196},[195],[80,16766,16768],{"className":16767},[200,201,202,203],[80,16769,16771,16774,16777],{"className":16770},[165,203],[80,16772,109],{"className":16773,"style":210},[165,169,203],[80,16775,114],{"className":16776},[214,203],[80,16778,117],{"className":16779},[165,169,203],[80,16781,222],{"className":16782},[221],[80,16784,16786],{"className":16785},[183],[80,16787,16789],{"className":16788,"style":229},[187],[80,16790],{},[80,16792,121],{"className":16793},[235],[80,16795,16797,16800],{"className":16796},[165],[80,16798,124],{"className":16799},[165,169],[80,16801,16803],{"className":16802},[174],[80,16804,16806],{"className":16805},[178],[80,16807,16809],{"className":16808},[183],[80,16810,16812],{"className":16811,"style":751},[187],[80,16813,16814,16817],{"style":772},[80,16815],{"className":16816,"style":196},[195],[80,16818,16820],{"className":16819},[200,201,202,203],[80,16821,16823,16826,16829],{"className":16822},[165,203],[80,16824,121],{"className":16825},[235,203],[80,16827,736],{"className":16828},[165,169,203],[80,16830,127],{"className":16831},[242,203],[80,16833,127],{"className":16834},[242],[80,16836],{"className":16837,"style":275},[246],[80,16839,4643],{"className":16840},[279],[80,16842],{"className":16843,"style":275},[246],[80,16845,16847,16850],{"className":16846},[165],[80,16848,683],{"className":16849,"style":834},[165,169],[80,16851,16853],{"className":16852},[174],[80,16854,16856],{"className":16855},[178],[80,16857,16859],{"className":16858},[183],[80,16860,16862],{"className":16861,"style":751},[187],[80,16863,16864,16867],{"style":772},[80,16865],{"className":16866,"style":196},[195],[80,16868,16870],{"className":16869},[200,201,202,203],[80,16871,16873,16876,16879],{"className":16872},[165,203],[80,16874,121],{"className":16875},[235,203],[80,16877,736],{"className":16878},[165,169,203],[80,16880,127],{"className":16881},[242,203],[80,16883,16885],{"className":16884,"style":7490},[242,7489],[80,16886,127],{"className":16887},[7494,4803],[80,16889,16891],{"className":16890},[174],[80,16892,16894],{"className":16893},[178],[80,16895,16897],{"className":16896},[183],[80,16898,16900],{"className":16899,"style":7653},[187],[80,16901,16902,16905],{"style":7656},[80,16903],{"className":16904,"style":196},[195],[80,16906,16908],{"className":16907},[200,201,202,203],[80,16909,1323],{"className":16910},[165,203],[11,16912,16913,16914,16916],{},"Notice it's basically the same formula from ",[562,16915,16451],{"href":4066},", just without the extra \"2\" in the denominator (that 2 only existed to make gradient descent's derivative cleaner, remember?). The name changes, the substance doesn't.",[294,16918,16920],{"id":16919},"the-most-direct-alternative-mae","The most direct alternative: MAE",[11,16922,16923,16924,16928],{},"What if, instead of squaring the error, we just took its absolute value? That's ",[7103,16925,16927],{"definition":16926},"Mean Absolute Error, uses the absolute value of the error instead of squaring it","MAE"," (Mean Absolute Error):",[11,16930,16931],{},[80,16932,16934,17025],{"className":16933},[83],[80,16935,16937],{"className":16936},[87],[89,16938,16939],{"xmlns":91},[93,16940,16941,17022],{},[96,16942,16943,16945,16947,16953,16973],{},[134,16944,16927],{},[111,16946,130],{},[4625,16948,16949,16951],{},[1321,16950,1583],{},[102,16952,322],{},[7202,16954,16955,16957,16965],{},[111,16956,7206],{},[96,16958,16959,16961,16963],{},[102,16960,736],{},[111,16962,130],{},[1321,16964,2071],{},[96,16966,16967,16969,16971],{},[102,16968,322],{},[111,16970,4643],{},[1321,16972,1583],{},[96,16974,16975,16978,16990,16992,17004,17006,17008,17020],{},[111,16976,16977],{"fence":113},"∣",[99,16979,16980,16982],{},[102,16981,104],{},[96,16983,16984,16986,16988],{},[102,16985,109],{},[111,16987,114],{"separator":113},[102,16989,117],{},[111,16991,121],{"stretchy":120},[726,16993,16994,16996],{},[102,16995,124],{},[96,16997,16998,17000,17002],{},[111,16999,121],{"stretchy":120},[102,17001,736],{},[111,17003,127],{"stretchy":120},[111,17005,127],{"stretchy":120},[111,17007,4643],{},[726,17009,17010,17012],{},[102,17011,683],{},[96,17013,17014,17016,17018],{},[111,17015,121],{"stretchy":120},[102,17017,736],{},[111,17019,127],{"stretchy":120},[111,17021,16977],{"fence":113},[145,17023,17024],{"encoding":147},"\\text{MAE} = \\frac{1}{m}\\sum_{i=0}^{m-1}\\left|f_{w,b}(x^{(i)}) - y^{(i)}\\right|",[80,17026,17028,17049],{"className":17027,"ariaHidden":113},[152],[80,17029,17031,17034,17040,17043,17046],{"className":17030},[156],[80,17032],{"className":17033,"style":8672},[160],[80,17035,17037],{"className":17036},[165,2699],[80,17038,16927],{"className":17039},[165],[80,17041],{"className":17042,"style":247},[246],[80,17044,130],{"className":17045},[251],[80,17047],{"className":17048,"style":247},[246],[80,17050,17052,17055,17123,17126,17195,17198],{"className":17051},[156],[80,17053],{"className":17054,"style":11708},[160],[80,17056,17058,17061,17120],{"className":17057},[165],[80,17059],{"className":17060},[235,4746],[80,17062,17064],{"className":17063},[4625],[80,17065,17067,17112],{"className":17066},[178,179],[80,17068,17070,17109],{"className":17069},[183],[80,17071,17073,17087,17095],{"className":17072,"style":5010},[187],[80,17074,17075,17078],{"style":5013},[80,17076],{"className":17077,"style":4766},[195],[80,17079,17081],{"className":17080},[200,201,202,203],[80,17082,17084],{"className":17083},[165,203],[80,17085,322],{"className":17086},[165,169,203],[80,17088,17089,17092],{"style":4859},[80,17090],{"className":17091,"style":4766},[195],[80,17093],{"className":17094,"style":4867},[4866],[80,17096,17097,17100],{"style":5042},[80,17098],{"className":17099,"style":4766},[195],[80,17101,17103],{"className":17102},[200,201,202,203],[80,17104,17106],{"className":17105},[165,203],[80,17107,1583],{"className":17108},[165,203],[80,17110,222],{"className":17111},[221],[80,17113,17115],{"className":17114},[183],[80,17116,17118],{"className":17117,"style":7390},[187],[80,17119],{},[80,17121],{"className":17122},[242,4746],[80,17124],{"className":17125,"style":268},[246],[80,17127,17129,17132],{"className":17128},[7402],[80,17130,7206],{"className":17131,"style":7408},[7402,7406,7407],[80,17133,17135],{"className":17134},[174],[80,17136,17138,17187],{"className":17137},[178,179],[80,17139,17141,17184],{"className":17140},[183],[80,17142,17144,17164],{"className":17143,"style":7421},[187],[80,17145,17146,17149],{"style":7424},[80,17147],{"className":17148,"style":196},[195],[80,17150,17152],{"className":17151},[200,201,202,203],[80,17153,17155,17158,17161],{"className":17154},[165,203],[80,17156,736],{"className":17157},[165,169,203],[80,17159,130],{"className":17160},[251,203],[80,17162,2071],{"className":17163},[165,203],[80,17165,17166,17169],{"style":7445},[80,17167],{"className":17168,"style":196},[195],[80,17170,17172],{"className":17171},[200,201,202,203],[80,17173,17175,17178,17181],{"className":17174},[165,203],[80,17176,322],{"className":17177},[165,169,203],[80,17179,4643],{"className":17180},[279,203],[80,17182,1583],{"className":17183},[165,203],[80,17185,222],{"className":17186},[221],[80,17188,17190],{"className":17189},[183],[80,17191,17193],{"className":17192,"style":7473},[187],[80,17194],{},[80,17196],{"className":17197,"style":268},[246],[80,17199,17201,17251,17300,17303,17341,17344,17347,17350,17353,17391],{"className":17200},[7482],[80,17202,17204],{"className":17203},[235],[80,17205,17208],{"className":17206},[7494,17207],"mult",[80,17209,17211,17242],{"className":17210},[178,179],[80,17212,17214,17239],{"className":17213},[183],[80,17215,17218],{"className":17216,"style":17217},[187],"height:0.85em;",[80,17219,17221,17225],{"style":17220},"top:-2.85em;",[80,17222],{"className":17223,"style":17224},[195],"height:3.2em;",[80,17226,17228],{"style":17227},"width:0.333em;height:1.2em;",[17229,17230,17235],"svg",{"xmlns":17231,"width":17232,"height":17233,"viewBox":17234},"http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","0.333em","1.2em","0 0 333 1200",[17236,17237],"path",{"d":17238},"M145 15 v585 v0 v585 c2.667,10,9.667,15,21,15\nc10,0,16.667,-5,20,-15 v-585 v0 v-585 c-2.667,-10,-9.667,-15,-21,-15\nc-10,0,-16.667,5,-20,15z M188 15 H145 v585 v0 v585 h43z",[80,17240,222],{"className":17241},[221],[80,17243,17245],{"className":17244},[183],[80,17246,17249],{"className":17247,"style":17248},[187],"height:0.35em;",[80,17250],{},[80,17252,17254,17257],{"className":17253},[165],[80,17255,104],{"className":17256,"style":170},[165,169],[80,17258,17260],{"className":17259},[174],[80,17261,17263,17292],{"className":17262},[178,179],[80,17264,17266,17289],{"className":17265},[183],[80,17267,17269],{"className":17268,"style":188},[187],[80,17270,17271,17274],{"style":191},[80,17272],{"className":17273,"style":196},[195],[80,17275,17277],{"className":17276},[200,201,202,203],[80,17278,17280,17283,17286],{"className":17279},[165,203],[80,17281,109],{"className":17282,"style":210},[165,169,203],[80,17284,114],{"className":17285},[214,203],[80,17287,117],{"className":17288},[165,169,203],[80,17290,222],{"className":17291},[221],[80,17293,17295],{"className":17294},[183],[80,17296,17298],{"className":17297,"style":229},[187],[80,17299],{},[80,17301,121],{"className":17302},[235],[80,17304,17306,17309],{"className":17305},[165],[80,17307,124],{"className":17308},[165,169],[80,17310,17312],{"className":17311},[174],[80,17313,17315],{"className":17314},[178],[80,17316,17318],{"className":17317},[183],[80,17319,17321],{"className":17320,"style":751},[187],[80,17322,17323,17326],{"style":772},[80,17324],{"className":17325,"style":196},[195],[80,17327,17329],{"className":17328},[200,201,202,203],[80,17330,17332,17335,17338],{"className":17331},[165,203],[80,17333,121],{"className":17334},[235,203],[80,17336,736],{"className":17337},[165,169,203],[80,17339,127],{"className":17340},[242,203],[80,17342,127],{"className":17343},[242],[80,17345],{"className":17346,"style":275},[246],[80,17348,4643],{"className":17349},[279],[80,17351],{"className":17352,"style":275},[246],[80,17354,17356,17359],{"className":17355},[165],[80,17357,683],{"className":17358,"style":834},[165,169],[80,17360,17362],{"className":17361},[174],[80,17363,17365],{"className":17364},[178],[80,17366,17368],{"className":17367},[183],[80,17369,17371],{"className":17370,"style":751},[187],[80,17372,17373,17376],{"style":772},[80,17374],{"className":17375,"style":196},[195],[80,17377,17379],{"className":17378},[200,201,202,203],[80,17380,17382,17385,17388],{"className":17381},[165,203],[80,17383,121],{"className":17384},[235,203],[80,17386,736],{"className":17387},[165,169,203],[80,17389,127],{"className":17390},[242,203],[80,17392,17394],{"className":17393},[242],[80,17395,17397],{"className":17396},[7494,17207],[80,17398,17400,17420],{"className":17399},[178,179],[80,17401,17403,17417],{"className":17402},[183],[80,17404,17406],{"className":17405,"style":17217},[187],[80,17407,17408,17411],{"style":17220},[80,17409],{"className":17410,"style":17224},[195],[80,17412,17413],{"style":17227},[17229,17414,17415],{"xmlns":17231,"width":17232,"height":17233,"viewBox":17234},[17236,17416],{"d":17238},[80,17418,222],{"className":17419},[221],[80,17421,17423],{"className":17422},[183],[80,17424,17426],{"className":17425,"style":17248},[187],[80,17427],{},[11,17429,17430,17431,17433],{},"The difference looks small on paper, but it changes everything about the behavior. With MSE, missing by twice as much costs four times as much (remember ",[562,17432,16451],{"href":4066},"?). With MAE, missing by twice as much costs exactly twice as much, no more, no less. It's the difference between a judge who loses their mind when you miss badly and a judge who just counts points proportionally, no extra drama.",[294,17435,17437],{"id":17436},"both-shapes-side-by-side","Both shapes side by side",[11,17439,17440],{},"No need to imagine the shape, here it is:",[17442,17443],"loss-shape-chart",{":error-max":17444,":initial-delta":17445,"x-label":17446,"y-label":17447},"10","4","error (prediction - actual)","loss",[11,17449,17450],{},"Notice: MSE is a parabola (grows faster and faster), MAE is a V (grows at a constant rate). Hold that image in your head, it explains everything that follows.",[294,17452,17454],{"id":17453},"the-best-of-both-worlds-huber","The best of both worlds: Huber",[11,17456,17457,17458,17462,17463,17494],{},"MSE is too sensitive to big errors. MAE is too harsh, even on small ones (look at the V: even tiny errors cost proportionally to their size, without the \"discount\" squaring gives near zero). The ",[7103,17459,17461],{"definition":17460},"a hybrid loss function, a parabola for small errors and a straight line for big ones, with a threshold delta deciding where the switch happens","Huber loss"," tries to get the best of both: a smooth parabola right near zero, a straight line past a threshold ",[80,17464,17466,17481],{"className":17465},[83],[80,17467,17469],{"className":17468},[87],[89,17470,17471],{"xmlns":91},[93,17472,17473,17478],{},[96,17474,17475],{},[102,17476,17477],{},"δ",[145,17479,17480],{"encoding":147},"\\delta",[80,17482,17484],{"className":17483,"ariaHidden":113},[152],[80,17485,17487,17490],{"className":17486},[156],[80,17488],{"className":17489,"style":289},[160],[80,17491,17477],{"className":17492,"style":17493},[165,169],"margin-right:0.0379em;"," (delta).",[11,17496,17497],{},[80,17498,17500,17632],{"className":17499},[83],[80,17501,17503],{"className":17502},[87],[89,17504,17505],{"xmlns":91},[93,17506,17507,17629],{},[96,17508,17509,17516,17518,17520,17522,17524],{},[99,17510,17511,17514],{},[102,17512,17513],{},"L",[102,17515,17477],{},[111,17517,121],{"stretchy":120},[102,17519,1671],{},[111,17521,127],{"stretchy":120},[111,17523,130],{},[96,17525,17526,17529],{},[111,17527,17528],{"fence":113},"{",[17530,17531,17535,17578],"mtable",{"rowspacing":17532,"columnalign":17533,"columnspacing":17534},"0.36em","left left","1em",[17536,17537,17538,17558],"mtr",{},[17539,17540,17541],"mtd",{},[17542,17543,17544],"mstyle",{"scriptlevel":2071,"displaystyle":120},[96,17545,17546,17552],{},[4625,17547,17548,17550],{},[1321,17549,1583],{},[1321,17551,1323],{},[726,17553,17554,17556],{},[102,17555,1671],{},[1321,17557,1323],{},[17539,17559,17560],{},[17542,17561,17562],{"scriptlevel":2071,"displaystyle":120},[96,17563,17564,17567,17569,17571,17573,17576],{},[134,17565,17566],{},"if ",[102,17568,16977],{"mathvariant":10558},[102,17570,1671],{},[102,17572,16977],{"mathvariant":10558},[111,17574,17575],{},"≤",[102,17577,17477],{},[17536,17579,17580,17610],{},[17539,17581,17582],{},[17542,17583,17584],{"scriptlevel":2071,"displaystyle":120},[96,17585,17586,17588],{},[102,17587,17477],{},[96,17589,17590,17592,17594,17596,17598,17600,17606,17608],{},[111,17591,121],{"fence":113},[102,17593,16977],{"mathvariant":10558},[102,17595,1671],{},[102,17597,16977],{"mathvariant":10558},[111,17599,4643],{},[4625,17601,17602,17604],{},[1321,17603,1583],{},[1321,17605,1323],{},[102,17607,17477],{},[111,17609,127],{"fence":113},[17539,17611,17612],{},[17542,17613,17614],{"scriptlevel":2071,"displaystyle":120},[96,17615,17616,17618,17620,17622,17624,17627],{},[134,17617,17566],{},[102,17619,16977],{"mathvariant":10558},[102,17621,1671],{},[102,17623,16977],{"mathvariant":10558},[111,17625,17626],{},">",[102,17628,17477],{},[145,17630,17631],{"encoding":147},"L_\\delta(e) = \\begin{cases} \\frac{1}{2}e^2 & \\text{if } |e| \\le \\delta \\\\ \\delta\\left(|e| - \\frac{1}{2}\\delta\\right) & \\text{if } |e| > \\delta \\end{cases}",[80,17633,17635,17699],{"className":17634,"ariaHidden":113},[152],[80,17636,17638,17641,17681,17684,17687,17690,17693,17696],{"className":17637},[156],[80,17639],{"className":17640,"style":2316},[160],[80,17642,17644,17647],{"className":17643},[165],[80,17645,17513],{"className":17646},[165,169],[80,17648,17650],{"className":17649},[174],[80,17651,17653,17673],{"className":17652},[178,179],[80,17654,17656,17670],{"className":17655},[183],[80,17657,17659],{"className":17658,"style":188},[187],[80,17660,17661,17664],{"style":15326},[80,17662],{"className":17663,"style":196},[195],[80,17665,17667],{"className":17666},[200,201,202,203],[80,17668,17477],{"className":17669,"style":17493},[165,169,203],[80,17671,222],{"className":17672},[221],[80,17674,17676],{"className":17675},[183],[80,17677,17679],{"className":17678,"style":15345},[187],[80,17680],{},[80,17682,121],{"className":17683},[235],[80,17685,1671],{"className":17686},[165,169],[80,17688,127],{"className":17689},[242],[80,17691],{"className":17692,"style":247},[246],[80,17694,130],{"className":17695},[251],[80,17697],{"className":17698,"style":247},[246],[80,17700,17702,17706],{"className":17701},[156],[80,17703],{"className":17704,"style":17705},[160],"height:3em;vertical-align:-1.25em;",[80,17707,17709,17716,18072],{"className":17708},[7482],[80,17710,17712],{"className":17711,"style":7490},[235,7489],[80,17713,17528],{"className":17714},[7494,17715],"size4",[80,17717,17719],{"className":17718},[165],[80,17720,17722,17974,17979],{"className":17721},[17530],[80,17723,17726],{"className":17724},[17725],"col-align-l",[80,17727,17729,17965],{"className":17728},[178,179],[80,17730,17732,17962],{"className":17731},[183],[80,17733,17736,17843],{"className":17734,"style":17735},[187],"height:1.69em;",[80,17737,17739,17743],{"style":17738},"top:-3.69em;",[80,17740],{"className":17741,"style":17742},[195],"height:3.008em;",[80,17744,17746,17814],{"className":17745},[165],[80,17747,17749,17752,17811],{"className":17748},[165],[80,17750],{"className":17751},[235,4746],[80,17753,17755],{"className":17754},[4625],[80,17756,17758,17803],{"className":17757},[178,179],[80,17759,17761,17800],{"className":17760},[183],[80,17762,17764,17778,17786],{"className":17763,"style":5010},[187],[80,17765,17766,17769],{"style":5013},[80,17767],{"className":17768,"style":4766},[195],[80,17770,17772],{"className":17771},[200,201,202,203],[80,17773,17775],{"className":17774},[165,203],[80,17776,1323],{"className":17777},[165,203],[80,17779,17780,17783],{"style":4859},[80,17781],{"className":17782,"style":4766},[195],[80,17784],{"className":17785,"style":4867},[4866],[80,17787,17788,17791],{"style":5042},[80,17789],{"className":17790,"style":4766},[195],[80,17792,17794],{"className":17793},[200,201,202,203],[80,17795,17797],{"className":17796},[165,203],[80,17798,1583],{"className":17799},[165,203],[80,17801,222],{"className":17802},[221],[80,17804,17806],{"className":17805},[183],[80,17807,17809],{"className":17808,"style":7390},[187],[80,17810],{},[80,17812],{"className":17813},[242,4746],[80,17815,17817,17820],{"className":17816},[165],[80,17818,1671],{"className":17819},[165,169],[80,17821,17823],{"className":17822},[174],[80,17824,17826],{"className":17825},[178],[80,17827,17829],{"className":17828},[183],[80,17830,17832],{"className":17831,"style":1407},[187],[80,17833,17834,17837],{"style":772},[80,17835],{"className":17836,"style":196},[195],[80,17838,17840],{"className":17839},[200,201,202,203],[80,17841,1323],{"className":17842},[165,203],[80,17844,17846,17849],{"style":17845},"top:-2.25em;",[80,17847],{"className":17848,"style":17742},[195],[80,17850,17852,17855,17858],{"className":17851},[165],[80,17853,17477],{"className":17854,"style":17493},[165,169],[80,17856],{"className":17857,"style":268},[246],[80,17859,17861,17867,17870,17873,17876,17879,17882,17885,17953,17956],{"className":17860},[7482],[80,17862,17864],{"className":17863,"style":7490},[235,7489],[80,17865,121],{"className":17866},[7494,4803],[80,17868,16977],{"className":17869},[165],[80,17871,1671],{"className":17872},[165,169],[80,17874,16977],{"className":17875},[165],[80,17877],{"className":17878,"style":275},[246],[80,17880,4643],{"className":17881},[279],[80,17883],{"className":17884,"style":275},[246],[80,17886,17888,17891,17950],{"className":17887},[165],[80,17889],{"className":17890},[235,4746],[80,17892,17894],{"className":17893},[4625],[80,17895,17897,17942],{"className":17896},[178,179],[80,17898,17900,17939],{"className":17899},[183],[80,17901,17903,17917,17925],{"className":17902,"style":5010},[187],[80,17904,17905,17908],{"style":5013},[80,17906],{"className":17907,"style":4766},[195],[80,17909,17911],{"className":17910},[200,201,202,203],[80,17912,17914],{"className":17913},[165,203],[80,17915,1323],{"className":17916},[165,203],[80,17918,17919,17922],{"style":4859},[80,17920],{"className":17921,"style":4766},[195],[80,17923],{"className":17924,"style":4867},[4866],[80,17926,17927,17930],{"style":5042},[80,17928],{"className":17929,"style":4766},[195],[80,17931,17933],{"className":17932},[200,201,202,203],[80,17934,17936],{"className":17935},[165,203],[80,17937,1583],{"className":17938},[165,203],[80,17940,222],{"className":17941},[221],[80,17943,17945],{"className":17944},[183],[80,17946,17948],{"className":17947,"style":7390},[187],[80,17949],{},[80,17951],{"className":17952},[242,4746],[80,17954,17477],{"className":17955,"style":17493},[165,169],[80,17957,17959],{"className":17958,"style":7490},[242,7489],[80,17960,127],{"className":17961},[7494,4803],[80,17963,222],{"className":17964},[221],[80,17966,17968],{"className":17967},[183],[80,17969,17972],{"className":17970,"style":17971},[187],"height:1.19em;",[80,17973],{},[80,17975],{"className":17976,"style":17978},[17977],"arraycolsep","width:1em;",[80,17980,17982],{"className":17981},[17725],[80,17983,17985,18064],{"className":17984},[178,179],[80,17986,17988,18061],{"className":17987},[183],[80,17989,17991,18026],{"className":17990,"style":17735},[187],[80,17992,17993,17996],{"style":17738},[80,17994],{"className":17995,"style":17742},[195],[80,17997,17999,18005,18008,18011,18014,18017,18020,18023],{"className":17998},[165],[80,18000,18002],{"className":18001},[165,2699],[80,18003,17566],{"className":18004},[165],[80,18006,16977],{"className":18007},[165],[80,18009,1671],{"className":18010},[165,169],[80,18012,16977],{"className":18013},[165],[80,18015],{"className":18016,"style":247},[246],[80,18018,17575],{"className":18019},[251],[80,18021],{"className":18022,"style":247},[246],[80,18024,17477],{"className":18025,"style":17493},[165,169],[80,18027,18028,18031],{"style":17845},[80,18029],{"className":18030,"style":17742},[195],[80,18032,18034,18040,18043,18046,18049,18052,18055,18058],{"className":18033},[165],[80,18035,18037],{"className":18036},[165,2699],[80,18038,17566],{"className":18039},[165],[80,18041,16977],{"className":18042},[165],[80,18044,1671],{"className":18045},[165,169],[80,18047,16977],{"className":18048},[165],[80,18050],{"className":18051,"style":247},[246],[80,18053,17626],{"className":18054},[251],[80,18056],{"className":18057,"style":247},[246],[80,18059,17477],{"className":18060,"style":17493},[165,169],[80,18062,222],{"className":18063},[221],[80,18065,18067],{"className":18066},[183],[80,18068,18070],{"className":18069,"style":17971},[187],[80,18071],{},[80,18073],{"className":18074},[242,4746],[11,18076,18077,18078,18106],{},"Where ",[80,18079,18081,18094],{"className":18080},[83],[80,18082,18084],{"className":18083},[87],[89,18085,18086],{"xmlns":91},[93,18087,18088,18092],{},[96,18089,18090],{},[102,18091,1671],{},[145,18093,1671],{"encoding":147},[80,18095,18097],{"className":18096,"ariaHidden":113},[152],[80,18098,18100,18103],{"className":18099},[156],[80,18101],{"className":18102,"style":334},[160],[80,18104,1671],{"className":18105},[165,169]," is one example's error. Drag the delta slider above again and watch the orange curve: with a small delta, Huber turns nearly into MAE, and with a big delta, it turns nearly into MSE. Delta is literally the control for \"past what error size do I stop giving a discount\".",[294,18108,18110],{"id":18109},"the-impact-of-an-outlier","The impact of an outlier",[11,18112,18113,18114,18116,18117,18119,18120,18124],{},"This is where picking a loss function stops being theory and becomes a real decision. Take the 6-house dataset from ",[562,18115,16451],{"href":4066}," and add ",[15,18118,9826],{}," more house, at a fixed position. You control its price with the slider, dragging from a reasonable value to one way off the trend (an ",[7103,18121,18123],{"definition":18122},"a data point way outside the pattern of the others, whether from a typo, a weird measurement, or a genuinely rare case","outlier","), and watch three fitted lines live, one per loss function:",[18126,18127],"outlier-impact-explorer",{":base-x":9502,":base-y":9503,":huber-delta":4286,":initial-outlier-y":9380,":outlier-x":18128,":outlier-y-max":18129,":outlier-y-min":4708,"x-label":3353,"y-label":3354},"2.2","1800",[11,18131,18132],{},"Drag it all the way to the max and watch: the blue line (MSE) runs after the red dot, twisting itself to try to \"please\" the outlier. The green line (MAE) barely budges. The orange one (Huber) sits in between. That's not a coincidence of this one example, it's the direct consequence of the two shapes you saw above: squaring punishes big errors without limit, absolute value doesn't.",[294,18134,6192],{"id":6191},[521,18136,18137,18153],{},[524,18138,18139],{},[527,18140,18141,18144,18147,18150],{},[530,18142,18143],{"align":532},"Loss function",[530,18145,18146],{"align":532},"Formula",[530,18148,18149],{"align":532},"Reaction to an outlier",[530,18151,18152],{"align":532},"When to use",[541,18154,18155,18167,18180],{},[527,18156,18157,18159,18161,18164],{},[546,18158,16456],{"align":532},[546,18160,8225],{"align":532},[546,18162,18163],{"align":532},"sensitive, gets pulled",[546,18165,18166],{"align":532},"clean data, no meaningful outliers",[527,18168,18169,18171,18174,18177],{},[546,18170,16927],{"align":532},[546,18172,18173],{"align":532},"absolute error",[546,18175,18176],{"align":532},"robust, nearly immune",[546,18178,18179],{"align":532},"data with outliers you want to ignore",[527,18181,18182,18185,18188,18191],{},[546,18183,18184],{"align":532},"Huber",[546,18186,18187],{"align":532},"hybrid (squared near zero, linear far)",[546,18189,18190],{"align":532},"adjustable middle ground",[546,18192,18193],{"align":532},"when you want a bit of both, tuned via delta",[11,18195,6507],{},[299,18197,18198,18204,18210],{},[302,18199,18200,18203],{},[15,18201,18202],{},"The loss function isn't a hidden technical detail, it's a design choice"," that changes the entire model's behavior, especially in the presence of real, messy data.",[302,18205,18206,18209],{},[15,18207,18208],{},"Squaring punishes big errors disproportionately",", great when you trust your dataset, dangerous when you don't.",[302,18211,18212,18215],{},[15,18213,18214],{},"Huber exists exactly so you don't have to choose blind",": delta is a dial that slides between the two extremes.",[294,18217,6716],{"id":6715},[11,18219,16135,18220,18223],{},[562,18221,6725],{"href":6722,"rel":18222},[6724],"). This time, a very realistic scenario: a typo. A 60 sqft house (tiny) got mistakenly logged as costing almost $2 million.",[2611,18225,18227],{"className":2613,"code":18226,"language":2615,"meta":26,"style":26},"# 50 real houses + 1 typo: 60 sqft, $1.9 million\nx_with_typo = [*x_sqft, 0.6]\ny_with_typo = [*y_price, 1900]\n\nols   = fit_mse(x_with_typo, y_with_typo)                 # ordinary least squares fit\nmae   = fit_irls(x_with_typo, y_with_typo, \"mae\")         # robust to the outlier\nhuber = fit_irls(x_with_typo, y_with_typo, \"huber\", delta=50)\n\nnew_house = 1.5   # 150 sqft, same scale as the previous posts\n\nfor name, (w, b) in [(\"MSE\", ols), (\"MAE\", mae), (\"Huber\", huber)]:\n    print(f\"{name}: ${(w * new_house + b) * 1000:,.0f}\")\n",[65,18228,18229,18234,18239,18244,18248,18253,18258,18263,18267,18272,18276,18281],{"__ignoreMap":26},[80,18230,18231],{"class":2620,"line":33},[80,18232,18233],{},"# 50 real houses + 1 typo: 60 sqft, $1.9 million\n",[80,18235,18236],{"class":2620,"line":27},[80,18237,18238],{},"x_with_typo = [*x_sqft, 0.6]\n",[80,18240,18241],{"class":2620,"line":2631},[80,18242,18243],{},"y_with_typo = [*y_price, 1900]\n",[80,18245,18246],{"class":2620,"line":2636},[80,18247,2657],{"emptyLinePlaceholder":32},[80,18249,18250],{"class":2620,"line":2642},[80,18251,18252],{},"ols   = fit_mse(x_with_typo, y_with_typo)                 # ordinary least squares fit\n",[80,18254,18255],{"class":2620,"line":2648},[80,18256,18257],{},"mae   = fit_irls(x_with_typo, y_with_typo, \"mae\")         # robust to the outlier\n",[80,18259,18260],{"class":2620,"line":2654},[80,18261,18262],{},"huber = fit_irls(x_with_typo, y_with_typo, \"huber\", delta=50)\n",[80,18264,18265],{"class":2620,"line":2660},[80,18266,2657],{"emptyLinePlaceholder":32},[80,18268,18269],{"class":2620,"line":2666},[80,18270,18271],{},"new_house = 1.5   # 150 sqft, same scale as the previous posts\n",[80,18273,18274],{"class":2620,"line":2672},[80,18275,2657],{"emptyLinePlaceholder":32},[80,18277,18278],{"class":2620,"line":2677},[80,18279,18280],{},"for name, (w, b) in [(\"MSE\", ols), (\"MAE\", mae), (\"Huber\", huber)]:\n",[80,18282,18283],{"class":2620,"line":2683},[80,18284,18285],{},"    print(f\"{name}: ${(w * new_house + b) * 1000:,.0f}\")\n",[46,18287,18288],{},[11,18289,18290,3255,18292,10213,18295,10213,18298],{},[15,18291,2693],{},[65,18293,18294],{},"MSE: $608,900",[65,18296,18297],{},"MAE: $570,700",[65,18299,18300],{},"Huber: $569,800",[11,18302,18303,18304,18349,18350,18447,18448,18450],{},"Without the typo, all three give roughly the same guess for this house (around ",[80,18305,18307,18328],{"className":18306},[83],[80,18308,18310],{"className":18309},[87],[89,18311,18312],{"xmlns":91},[93,18313,18314,18325],{},[96,18315,18316,18319,18321,18323],{},[1321,18317,18318],{},"567",[102,18320,1649],{},[102,18322,1676],{},[102,18324,1666],{},[145,18326,18327],{"encoding":147},"567k to ",[80,18329,18331],{"className":18330,"ariaHidden":113},[152],[80,18332,18334,18337,18340,18343,18346],{"className":18333},[156],[80,18335],{"className":18336,"style":289},[160],[80,18338,18318],{"className":18339},[165],[80,18341,1649],{"className":18342,"style":1736},[165,169],[80,18344,1676],{"className":18345},[165,169],[80,18347,1666],{"className":18348},[165,169],"573k). With the typo, MSE jumps to almost ",[80,18351,18353,18396],{"className":18352},[83],[80,18354,18356],{"className":18355},[87],[89,18357,18358],{"xmlns":91},[93,18359,18360,18393],{},[96,18361,18362,18365,18367,18369,18371,18374,18376,18378,18380,18382,18384,18386,18389,18391],{},[1321,18363,18364],{},"609",[102,18366,1649],{},[111,18368,114],{"separator":113},[102,18370,562],{},[102,18372,18373],{},"j",[102,18375,1666],{},[102,18377,1704],{},[102,18379,1676],{},[102,18381,1666],{},[102,18383,104],{},[102,18385,1666],{},[102,18387,18388],{},"v",[102,18390,1671],{},[102,18392,1713],{},[145,18394,18395],{"encoding":147},"609k, a jolt of over ",[80,18397,18399],{"className":18398,"ariaHidden":113},[152],[80,18400,18402,18405,18408,18411,18414,18417,18422,18425,18428,18431,18434,18437,18440,18443],{"className":18401},[156],[80,18403],{"className":18404,"style":1752},[160],[80,18406,18364],{"className":18407},[165],[80,18409,1649],{"className":18410,"style":1736},[165,169],[80,18412,114],{"className":18413},[214],[80,18415],{"className":18416,"style":268},[246],[80,18418,18421],{"className":18419,"style":18420},[165,169],"margin-right:0.0572em;","aj",[80,18423,1666],{"className":18424},[165,169],[80,18426,1704],{"className":18427,"style":1842},[165,169],[80,18429,1676],{"className":18430},[165,169],[80,18432,1666],{"className":18433},[165,169],[80,18435,104],{"className":18436,"style":170},[165,169],[80,18438,1666],{"className":18439},[165,169],[80,18441,18388],{"className":18442,"style":834},[165,169],[80,18444,18446],{"className":18445,"style":1850},[165,169],"er","35k just from ",[15,18449,9826],{}," wrong row in the spreadsheet, while MAE and Huber barely move (under $3k of difference). Try it yourself, dragging the outlier:",[18452,18453],"housing-outlier-explorer",{"x-label":6877,"y-label":3354},[11,18455,18456],{},"In a real database, with thousands of rows, typos happen. The choice of loss function decides whether one of those turns into your problem or not.",[6949,18458,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":18460},[18461,18462,18463,18464,18465,18466,18467],{"id":16444,"depth":27,"text":16445},{"id":16919,"depth":27,"text":16920},{"id":17436,"depth":27,"text":17437},{"id":17453,"depth":27,"text":17454},{"id":18109,"depth":27,"text":18110},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716},"Why we squared the error back in post 2, and what happens if you choose differently: MAE, Huber, and the real impact a single outlier has on each choice.",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions",{"title":16423,"description":18468},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Floss-functions",[18474,18475,18476,18477,18478],"loss-function","mse","mae","huber","robustness","arTSdKZgqx98Fje6zNQdsIV1u51dSyyxETRISLcdEtQ",{"id":18481,"title":18482,"body":18483,"cover":3,"date":26913,"description":26914,"extension":30,"meta":26915,"navigation":32,"order":2648,"path":26916,"playlist":6980,"seo":26917,"status":36,"stem":26918,"tags":26919,"__hash__":26923},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling.md","Feature Scaling and Learning Rate",{"type":8,"value":18484,"toc":26898},[18485,18491,18534,18538,18545,18645,18652,18655,18659,18662,18665,18763,18766,18770,19319,19624,19656,19660,19663,19678,19804,20689,20692,20701,20894,20937,20940,21202,21355,21883,22085,22298,22330,22336,22433,22598,22775,22779,22811,24370,24549,24579,24582,24588,24592,24720,24725,24990,25057,25061,25064,25079,25085,25091,25166,26030,26195,26199,26209,26219,26224,26230,26235,26238,26241,26244,26249,26254,26259,26265,26268,26323,26325,26420,26422,26501,26503,26671,26716,26725,26888,26890,26893,26896],[11,18486,18487],{},[57,18488],{"alt":18489,"src":18490},"A confused kid meme with open hands and the caption \"what do you mean I need scaling?\"","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling\u002Fmeme-feature-scaling.jpeg",[11,18492,18493,18494,18497,18498,18500,18501,18504,18505,18533],{},"I picked up the optional lab from Course 1, Week 2 (",[65,18495,18496],{},"C1_W2_Lab03",") expecting just another follow-up post, and ended up finding the topic that annoyed me the most in the whole specialization so far: the exact gradient descent I built in ",[562,18499,9082],{"href":4070},", same algorithm, zero code changes, simply ",[15,18502,18503],{},"stops working"," depending on the unit I use to measure a feature. I swapped meters for centimeters and the algorithm diverged. That's not a bug, it's math, and after this post you'll never pick an ",[80,18506,18508,18521],{"className":18507},[83],[80,18509,18511],{"className":18510},[87],[89,18512,18513],{"xmlns":91},[93,18514,18515,18519],{},[96,18516,18517],{},[102,18518,10551],{},[145,18520,11037],{"encoding":147},[80,18522,18524],{"className":18523,"ariaHidden":113},[152],[80,18525,18527,18530],{"className":18526},[156],[80,18528],{"className":18529,"style":334},[160],[80,18531,10551],{"className":18532,"style":10667},[165,169]," in the dark again without understanding why.",[294,18535,18537],{"id":18536},"the-problem-4-features-wildly-different-scales","The problem: 4 features, wildly different scales",[11,18539,18540,18541,18544],{},"The original notebook uses a 100-house dataset (derived from the Ames Housing dataset, the same one used in the course) with 4 features: size in sqft, number of bedrooms, number of floors, and age. The goal is to predict the price of a ",[15,18542,18543],{},"1200 sqft, 3-bedroom, 1-floor, 40-year-old"," house.",[521,18546,18547,18570],{},[524,18548,18549],{},[527,18550,18551,18554,18558,18561,18564,18567],{},[530,18552,18553],{"align":532},"Feature",[530,18555,18557],{"align":18556},"right","Min",[530,18559,18560],{"align":18556},"Max",[530,18562,18563],{"align":18556},"Mean",[530,18565,18566],{"align":18556},"Std",[530,18568,18569],{"align":18556},"Range",[541,18571,18572,18591,18608,18625],{},[527,18573,18574,18576,18579,18582,18585,18588],{},[546,18575,6803],{"align":532},[546,18577,18578],{"align":18556},"788",[546,18580,18581],{"align":18556},"3194",[546,18583,18584],{"align":18556},"1413.7",[546,18586,18587],{"align":18556},"412.2",[546,18589,18590],{"align":18556},"2406",[527,18592,18593,18596,18598,18600,18603,18606],{},[546,18594,18595],{"align":532},"bedrooms",[546,18597,2071],{"align":18556},[546,18599,17445],{"align":18556},[546,18601,18602],{"align":18556},"2.7",[546,18604,18605],{"align":18556},"0.7",[546,18607,17445],{"align":18556},[527,18609,18610,18613,18615,18617,18620,18623],{},[546,18611,18612],{"align":532},"floors",[546,18614,1583],{"align":18556},[546,18616,1323],{"align":18556},[546,18618,18619],{"align":18556},"1.4",[546,18621,18622],{"align":18556},"0.5",[546,18624,1583],{"align":18556},[527,18626,18627,18630,18633,18636,18639,18642],{},[546,18628,18629],{"align":532},"age (years)",[546,18631,18632],{"align":18556},"12",[546,18634,18635],{"align":18556},"107",[546,18637,18638],{"align":18556},"38.6",[546,18640,18641],{"align":18556},"25.8",[546,18643,18644],{"align":18556},"95",[11,18646,18647,18648,18651],{},"I noticed this number the moment I ran the cell: the feature with the biggest range (size) is ",[15,18649,18650],{},"2406 times"," bigger than the one with the smallest range (floors). I kept that number in mind, because it's the root of everything that follows.",[11,18653,18654],{},"Before jumping into modeling, I looked at each feature against price, one at a time:",[18656,18657],"feature-scaling-overview",{"age-label":18629,"bedrooms-label":18595,"floors-label":18612,"price-label":18658,"size-label":6803},"price (thousand US$)",[11,18660,18661],{},"Size carries real signal (the cloud of points clearly climbs from left to right). Bedrooms and floors are discrete and noisy, you can see 2-bedroom houses costing more than 3-bedroom ones. Age pulls price down, but with a lot of scatter. That doesn't mean bedrooms and floors are useless, just that alone they explain little, their value shows up once they join the model together with the others (the negative bedrooms coefficient I compute further down is exactly that kind of hidden effect).",[11,18663,18664],{},"I also computed the correlation between every pair of features (and between each feature and price):",[521,18666,18667,18686],{},[524,18668,18669],{},[527,18670,18671,18673,18676,18678,18680,18683],{},[530,18672],{"align":532},[530,18674,18675],{"align":18556},"size",[530,18677,18595],{"align":18556},[530,18679,18612],{"align":18556},[530,18681,18682],{"align":18556},"age",[530,18684,18685],{"align":18556},"price",[541,18687,18688,18709,18728,18746],{},[527,18689,18690,18694,18697,18700,18703,18706],{},[546,18691,18692],{"align":532},[15,18693,18675],{},[546,18695,18696],{"align":18556},"1.00",[546,18698,18699],{"align":18556},"0.56",[546,18701,18702],{"align":18556},"0.60",[546,18704,18705],{"align":18556},"-0.27",[546,18707,18708],{"align":18556},"0.86",[527,18710,18711,18715,18717,18719,18722,18725],{},[546,18712,18713],{"align":532},[15,18714,18595],{},[546,18716,18699],{"align":18556},[546,18718,18696],{"align":18556},[546,18720,18721],{"align":18556},"0.38",[546,18723,18724],{"align":18556},"-0.06",[546,18726,18727],{"align":18556},"0.29",[527,18729,18730,18734,18736,18738,18740,18743],{},[546,18731,18732],{"align":532},[15,18733,18612],{},[546,18735,18702],{"align":18556},[546,18737,18721],{"align":18556},[546,18739,18696],{"align":18556},[546,18741,18742],{"align":18556},"-0.20",[546,18744,18745],{"align":18556},"0.32",[527,18747,18748,18752,18754,18756,18758,18760],{},[546,18749,18750],{"align":532},[15,18751,18682],{},[546,18753,18705],{"align":18556},[546,18755,18724],{"align":18556},[546,18757,18742],{"align":18556},[546,18759,18696],{"align":18556},[546,18761,18762],{"align":18556},"-0.58",[11,18764,18765],{},"Size is by far the feature most correlated with price (0.86), which matches the chart above. And notice size and bedrooms aren't independent (0.56), bigger houses tend to have more bedrooms, which is kind of obvious once you stop to think about it, but it's always good to confirm with a number instead of a guess.",[294,18767,18769],{"id":18768},"quick-recap-with-a-single-feature","Quick recap, with a single feature",[11,18771,18772,18773,18921,18922,19314,19315,19318],{},"To keep the foundation I already built, I'll simplify to 1 feature (size) for this first part, exactly like the previous posts: ",[80,18774,18776,18815],{"className":18775},[83],[80,18777,18779],{"className":18778},[87],[89,18780,18781],{"xmlns":91},[93,18782,18783,18813],{},[96,18784,18785,18797,18799,18801,18803,18805,18807,18809,18811],{},[99,18786,18787,18789],{},[102,18788,104],{},[96,18790,18791,18793,18795],{},[102,18792,109],{},[111,18794,114],{"separator":113},[102,18796,117],{},[111,18798,121],{"stretchy":120},[102,18800,124],{},[111,18802,127],{"stretchy":120},[111,18804,130],{},[102,18806,109],{},[102,18808,124],{},[111,18810,141],{},[102,18812,117],{},[145,18814,387],{"encoding":147},[80,18816,18818,18891,18912],{"className":18817,"ariaHidden":113},[152],[80,18819,18821,18824,18873,18876,18879,18882,18885,18888],{"className":18820},[156],[80,18822],{"className":18823,"style":161},[160],[80,18825,18827,18830],{"className":18826},[165],[80,18828,104],{"className":18829,"style":170},[165,169],[80,18831,18833],{"className":18832},[174],[80,18834,18836,18865],{"className":18835},[178,179],[80,18837,18839,18862],{"className":18838},[183],[80,18840,18842],{"className":18841,"style":188},[187],[80,18843,18844,18847],{"style":191},[80,18845],{"className":18846,"style":196},[195],[80,18848,18850],{"className":18849},[200,201,202,203],[80,18851,18853,18856,18859],{"className":18852},[165,203],[80,18854,109],{"className":18855,"style":210},[165,169,203],[80,18857,114],{"className":18858},[214,203],[80,18860,117],{"className":18861},[165,169,203],[80,18863,222],{"className":18864},[221],[80,18866,18868],{"className":18867},[183],[80,18869,18871],{"className":18870,"style":229},[187],[80,18872],{},[80,18874,121],{"className":18875},[235],[80,18877,124],{"className":18878},[165,169],[80,18880,127],{"className":18881},[242],[80,18883],{"className":18884,"style":247},[246],[80,18886,130],{"className":18887},[251],[80,18889],{"className":18890,"style":247},[246],[80,18892,18894,18897,18900,18903,18906,18909],{"className":18893},[156],[80,18895],{"className":18896,"style":261},[160],[80,18898,109],{"className":18899,"style":210},[165,169],[80,18901,124],{"className":18902},[165,169],[80,18904],{"className":18905,"style":275},[246],[80,18907,141],{"className":18908},[279],[80,18910],{"className":18911,"style":275},[246],[80,18913,18915,18918],{"className":18914},[156],[80,18916],{"className":18917,"style":289},[160],[80,18919,117],{"className":18920},[165,169],", cost ",[80,18923,18925,19013],{"className":18924},[83],[80,18926,18928],{"className":18927},[87],[89,18929,18930],{"xmlns":91},[93,18931,18932,19010],{},[96,18933,18934,18936,18938,18940,18942,18944,18946,18948,18958,18960,18962,18974,18976,18988,18990,18992,19004],{},[102,18935,5606],{},[111,18937,121],{"stretchy":120},[102,18939,109],{},[111,18941,114],{"separator":113},[102,18943,117],{},[111,18945,127],{"stretchy":120},[111,18947,130],{},[4625,18949,18950,18952],{},[1321,18951,1583],{},[96,18953,18954,18956],{},[1321,18955,1323],{},[102,18957,322],{},[111,18959,7206],{},[111,18961,121],{"stretchy":120},[99,18963,18964,18966],{},[102,18965,104],{},[96,18967,18968,18970,18972],{},[102,18969,109],{},[111,18971,114],{"separator":113},[102,18973,117],{},[111,18975,121],{"stretchy":120},[726,18977,18978,18980],{},[102,18979,124],{},[96,18981,18982,18984,18986],{},[111,18983,121],{"stretchy":120},[102,18985,736],{},[111,18987,127],{"stretchy":120},[111,18989,127],{"stretchy":120},[111,18991,4643],{},[726,18993,18994,18996],{},[102,18995,683],{},[96,18997,18998,19000,19002],{},[111,18999,121],{"stretchy":120},[102,19001,736],{},[111,19003,127],{"stretchy":120},[726,19005,19006,19008],{},[111,19007,127],{"stretchy":120},[1321,19009,1323],{},[145,19011,19012],{"encoding":147},"J(w,b) = \\frac{1}{2m}\\sum(f_{w,b}(x^{(i)}) - y^{(i)})^2",[80,19014,19016,19052,19241],{"className":19015,"ariaHidden":113},[152],[80,19017,19019,19022,19025,19028,19031,19034,19037,19040,19043,19046,19049],{"className":19018},[156],[80,19020],{"className":19021,"style":2316},[160],[80,19023,5606],{"className":19024,"style":5632},[165,169],[80,19026,121],{"className":19027},[235],[80,19029,109],{"className":19030,"style":210},[165,169],[80,19032,114],{"className":19033},[214],[80,19035],{"className":19036,"style":268},[246],[80,19038,117],{"className":19039},[165,169],[80,19041,127],{"className":19042},[242],[80,19044],{"className":19045,"style":247},[246],[80,19047,130],{"className":19048},[251],[80,19050],{"className":19051,"style":247},[246],[80,19053,19055,19059,19130,19133,19136,19139,19188,19191,19229,19232,19235,19238],{"className":19054},[156],[80,19056],{"className":19057,"style":19058},[160],"height:1.233em;vertical-align:-0.345em;",[80,19060,19062,19065,19127],{"className":19061},[165],[80,19063],{"className":19064},[235,4746],[80,19066,19068],{"className":19067},[4625],[80,19069,19071,19119],{"className":19070},[178,179],[80,19072,19074,19116],{"className":19073},[183],[80,19075,19077,19094,19102],{"className":19076,"style":5010},[187],[80,19078,19079,19082],{"style":5013},[80,19080],{"className":19081,"style":4766},[195],[80,19083,19085],{"className":19084},[200,201,202,203],[80,19086,19088,19091],{"className":19087},[165,203],[80,19089,1323],{"className":19090},[165,203],[80,19092,322],{"className":19093},[165,169,203],[80,19095,19096,19099],{"style":4859},[80,19097],{"className":19098,"style":4766},[195],[80,19100],{"className":19101,"style":4867},[4866],[80,19103,19104,19107],{"style":5042},[80,19105],{"className":19106,"style":4766},[195],[80,19108,19110],{"className":19109},[200,201,202,203],[80,19111,19113],{"className":19112},[165,203],[80,19114,1583],{"className":19115},[165,203],[80,19117,222],{"className":19118},[221],[80,19120,19122],{"className":19121},[183],[80,19123,19125],{"className":19124,"style":7390},[187],[80,19126],{},[80,19128],{"className":19129},[242,4746],[80,19131],{"className":19132,"style":268},[246],[80,19134,7206],{"className":19135,"style":7408},[7402,7406,7407],[80,19137,121],{"className":19138},[235],[80,19140,19142,19145],{"className":19141},[165],[80,19143,104],{"className":19144,"style":170},[165,169],[80,19146,19148],{"className":19147},[174],[80,19149,19151,19180],{"className":19150},[178,179],[80,19152,19154,19177],{"className":19153},[183],[80,19155,19157],{"className":19156,"style":188},[187],[80,19158,19159,19162],{"style":191},[80,19160],{"className":19161,"style":196},[195],[80,19163,19165],{"className":19164},[200,201,202,203],[80,19166,19168,19171,19174],{"className":19167},[165,203],[80,19169,109],{"className":19170,"style":210},[165,169,203],[80,19172,114],{"className":19173},[214,203],[80,19175,117],{"className":19176},[165,169,203],[80,19178,222],{"className":19179},[221],[80,19181,19183],{"className":19182},[183],[80,19184,19186],{"className":19185,"style":229},[187],[80,19187],{},[80,19189,121],{"className":19190},[235],[80,19192,19194,19197],{"className":19193},[165],[80,19195,124],{"className":19196},[165,169],[80,19198,19200],{"className":19199},[174],[80,19201,19203],{"className":19202},[178],[80,19204,19206],{"className":19205},[183],[80,19207,19209],{"className":19208,"style":751},[187],[80,19210,19211,19214],{"style":772},[80,19212],{"className":19213,"style":196},[195],[80,19215,19217],{"className":19216},[200,201,202,203],[80,19218,19220,19223,19226],{"className":19219},[165,203],[80,19221,121],{"className":19222},[235,203],[80,19224,736],{"className":19225},[165,169,203],[80,19227,127],{"className":19228},[242,203],[80,19230,127],{"className":19231},[242],[80,19233],{"className":19234,"style":275},[246],[80,19236,4643],{"className":19237},[279],[80,19239],{"className":19240,"style":275},[246],[80,19242,19244,19247,19285],{"className":19243},[156],[80,19245],{"className":19246,"style":2212},[160],[80,19248,19250,19253],{"className":19249},[165],[80,19251,683],{"className":19252,"style":834},[165,169],[80,19254,19256],{"className":19255},[174],[80,19257,19259],{"className":19258},[178],[80,19260,19262],{"className":19261},[183],[80,19263,19265],{"className":19264,"style":751},[187],[80,19266,19267,19270],{"style":772},[80,19268],{"className":19269,"style":196},[195],[80,19271,19273],{"className":19272},[200,201,202,203],[80,19274,19276,19279,19282],{"className":19275},[165,203],[80,19277,121],{"className":19278},[235,203],[80,19280,736],{"className":19281},[165,169,203],[80,19283,127],{"className":19284},[242,203],[80,19286,19288,19291],{"className":19287},[242],[80,19289,127],{"className":19290},[242],[80,19292,19294],{"className":19293},[174],[80,19295,19297],{"className":19296},[178],[80,19298,19300],{"className":19299},[183],[80,19301,19303],{"className":19302,"style":1407},[187],[80,19304,19305,19308],{"style":772},[80,19306],{"className":19307,"style":196},[195],[80,19309,19311],{"className":19310},[200,201,202,203],[80,19312,1323],{"className":19313},[165,203],". The gradient descent I built ",[562,19316,19317],{"href":4070},"two posts ago"," doesn't change a single line:",[11,19320,19321],{},[80,19322,19324,19386],{"className":19323},[83],[80,19325,19327],{"className":19326},[87],[89,19328,19329],{"xmlns":91},[93,19330,19331,19383],{},[96,19332,19333,19335,19337,19339,19341,19343,19357,19359,19361,19363,19365,19367,19369],{},[102,19334,109],{},[111,19336,130],{},[102,19338,109],{},[111,19340,4643],{},[102,19342,10551],{},[4625,19344,19345,19351],{},[96,19346,19347,19349],{},[102,19348,10559],{"mathvariant":10558},[102,19350,5606],{},[96,19352,19353,19355],{},[102,19354,10559],{"mathvariant":10558},[102,19356,109],{},[246,19358],{"width":10580},[102,19360,117],{},[111,19362,130],{},[102,19364,117],{},[111,19366,4643],{},[102,19368,10551],{},[4625,19370,19371,19377],{},[96,19372,19373,19375],{},[102,19374,10559],{"mathvariant":10558},[102,19376,5606],{},[96,19378,19379,19381],{},[102,19380,10559],{"mathvariant":10558},[102,19382,117],{},[145,19384,19385],{"encoding":147},"w = w - \\alpha \\frac{\\partial J}{\\partial w} \\qquad b = b - \\alpha \\frac{\\partial J}{\\partial b}",[80,19387,19389,19407,19425,19523,19541],{"className":19388,"ariaHidden":113},[152],[80,19390,19392,19395,19398,19401,19404],{"className":19391},[156],[80,19393],{"className":19394,"style":334},[160],[80,19396,109],{"className":19397,"style":210},[165,169],[80,19399],{"className":19400,"style":247},[246],[80,19402,130],{"className":19403},[251],[80,19405],{"className":19406,"style":247},[246],[80,19408,19410,19413,19416,19419,19422],{"className":19409},[156],[80,19411],{"className":19412,"style":261},[160],[80,19414,109],{"className":19415,"style":210},[165,169],[80,19417],{"className":19418,"style":275},[246],[80,19420,4643],{"className":19421},[279],[80,19423],{"className":19424,"style":275},[246],[80,19426,19428,19431,19434,19508,19511,19514,19517,19520],{"className":19427},[156],[80,19429],{"className":19430,"style":10944},[160],[80,19432,10551],{"className":19433,"style":10667},[165,169],[80,19435,19437,19440,19505],{"className":19436},[165],[80,19438],{"className":19439},[235,4746],[80,19441,19443],{"className":19442},[4625],[80,19444,19446,19497],{"className":19445},[178,179],[80,19447,19449,19494],{"className":19448},[183],[80,19450,19452,19469,19477],{"className":19451,"style":10963},[187],[80,19453,19454,19457],{"style":5013},[80,19455],{"className":19456,"style":4766},[195],[80,19458,19460],{"className":19459},[200,201,202,203],[80,19461,19463,19466],{"className":19462},[165,203],[80,19464,10559],{"className":19465,"style":10701},[165,203],[80,19467,109],{"className":19468,"style":210},[165,169,203],[80,19470,19471,19474],{"style":4859},[80,19472],{"className":19473,"style":4766},[195],[80,19475],{"className":19476,"style":4867},[4866],[80,19478,19479,19482],{"style":5042},[80,19480],{"className":19481,"style":4766},[195],[80,19483,19485],{"className":19484},[200,201,202,203],[80,19486,19488,19491],{"className":19487},[165,203],[80,19489,10559],{"className":19490,"style":10701},[165,203],[80,19492,5606],{"className":19493,"style":5632},[165,169,203],[80,19495,222],{"className":19496},[221],[80,19498,19500],{"className":19499},[183],[80,19501,19503],{"className":19502,"style":7390},[187],[80,19504],{},[80,19506],{"className":19507},[242,4746],[80,19509],{"className":19510,"style":10763},[246],[80,19512,117],{"className":19513},[165,169],[80,19515],{"className":19516,"style":247},[246],[80,19518,130],{"className":19519},[251],[80,19521],{"className":19522,"style":247},[246],[80,19524,19526,19529,19532,19535,19538],{"className":19525},[156],[80,19527],{"className":19528,"style":10782},[160],[80,19530,117],{"className":19531},[165,169],[80,19533],{"className":19534,"style":275},[246],[80,19536,4643],{"className":19537},[279],[80,19539],{"className":19540,"style":275},[246],[80,19542,19544,19547,19550],{"className":19543},[156],[80,19545],{"className":19546,"style":10944},[160],[80,19548,10551],{"className":19549,"style":10667},[165,169],[80,19551,19553,19556,19621],{"className":19552},[165],[80,19554],{"className":19555},[235,4746],[80,19557,19559],{"className":19558},[4625],[80,19560,19562,19613],{"className":19561},[178,179],[80,19563,19565,19610],{"className":19564},[183],[80,19566,19568,19585,19593],{"className":19567,"style":10963},[187],[80,19569,19570,19573],{"style":5013},[80,19571],{"className":19572,"style":4766},[195],[80,19574,19576],{"className":19575},[200,201,202,203],[80,19577,19579,19582],{"className":19578},[165,203],[80,19580,10559],{"className":19581,"style":10701},[165,203],[80,19583,117],{"className":19584},[165,169,203],[80,19586,19587,19590],{"style":4859},[80,19588],{"className":19589,"style":4766},[195],[80,19591],{"className":19592,"style":4867},[4866],[80,19594,19595,19598],{"style":5042},[80,19596],{"className":19597,"style":4766},[195],[80,19599,19601],{"className":19600},[200,201,202,203],[80,19602,19604,19607],{"className":19603},[165,203],[80,19605,10559],{"className":19606,"style":10701},[165,203],[80,19608,5606],{"className":19609,"style":5632},[165,169,203],[80,19611,222],{"className":19612},[221],[80,19614,19616],{"className":19615},[183],[80,19617,19619],{"className":19618,"style":7390},[187],[80,19620],{},[80,19622],{"className":19623},[242,4746],[11,19625,19626,19627,19655],{},"The real notebook runs this with all 4 features at once (one ",[80,19628,19630,19643],{"className":19629},[83],[80,19631,19633],{"className":19632},[87],[89,19634,19635],{"xmlns":91},[93,19636,19637,19641],{},[96,19638,19639],{},[102,19640,109],{},[145,19642,109],{"encoding":147},[80,19644,19646],{"className":19645,"ariaHidden":113},[152],[80,19647,19649,19652],{"className":19648},[156],[80,19650],{"className":19651,"style":334},[160],[80,19653,109],{"className":19654,"style":210},[165,169]," per feature), but the core lesson shows up completely already with a single feature, and it's much easier to visualize on a 2D chart. Keep this simplification in mind, it comes back later when I show the result with the real 4 features.",[294,19657,19659],{"id":19658},"gradient-descent-diverges-on-its-own","Gradient descent diverges on its own",[11,19661,19662],{},"I picked 8 real houses from the notebook's dataset (size in sqft, price in thousands of dollars) to test this by hand:",[2611,19664,19666],{"className":2613,"code":19665,"language":2615,"meta":26,"style":26},"x_train = [952, 1244, 1947, 1725, 1959, 1314, 864, 1836]\ny_train = [271.5, 300, 509.8, 394, 540, 415, 230, 560]\n",[65,19667,19668,19673],{"__ignoreMap":26},[80,19669,19670],{"class":2620,"line":33},[80,19671,19672],{},"x_train = [952, 1244, 1947, 1725, 1959, 1314, 864, 1836]\n",[80,19674,19675],{"class":2620,"line":27},[80,19676,19677],{},"y_train = [271.5, 300, 509.8, 394, 540, 415, 230, 560]\n",[11,19679,19680,19681,19684,19685,19688,19689,19717,19718,3774],{},"I ran the same ",[65,19682,19683],{},"gradient_descent"," from ",[562,19686,19687],{"href":4070},"the previous post"," with three different ",[80,19690,19692,19705],{"className":19691},[83],[80,19693,19695],{"className":19694},[87],[89,19696,19697],{"xmlns":91},[93,19698,19699,19703],{},[96,19700,19701],{},[102,19702,10551],{},[145,19704,11037],{"encoding":147},[80,19706,19708],{"className":19707,"ariaHidden":113},[152],[80,19709,19711,19714],{"className":19710},[156],[80,19712],{"className":19713,"style":334},[160],[80,19715,10551],{"className":19716,"style":10667},[165,169]," values, just 10 iterations, starting at ",[80,19719,19721,19747],{"className":19720},[83],[80,19722,19724],{"className":19723},[87],[89,19725,19726],{"xmlns":91},[93,19727,19728,19744],{},[96,19729,19730,19732,19734,19736,19738,19740,19742],{},[102,19731,109],{},[111,19733,130],{},[1321,19735,2071],{},[111,19737,114],{"separator":113},[102,19739,117],{},[111,19741,130],{},[1321,19743,2071],{},[145,19745,19746],{"encoding":147},"w=0, b=0",[80,19748,19750,19768,19795],{"className":19749,"ariaHidden":113},[152],[80,19751,19753,19756,19759,19762,19765],{"className":19752},[156],[80,19754],{"className":19755,"style":334},[160],[80,19757,109],{"className":19758,"style":210},[165,169],[80,19760],{"className":19761,"style":247},[246],[80,19763,130],{"className":19764},[251],[80,19766],{"className":19767,"style":247},[246],[80,19769,19771,19774,19777,19780,19783,19786,19789,19792],{"className":19770},[156],[80,19772],{"className":19773,"style":1752},[160],[80,19775,2071],{"className":19776},[165],[80,19778,114],{"className":19779},[214],[80,19781],{"className":19782,"style":268},[246],[80,19784,117],{"className":19785},[165,169],[80,19787],{"className":19788,"style":247},[246],[80,19790,130],{"className":19791},[251],[80,19793],{"className":19794,"style":247},[246],[80,19796,19798,19801],{"className":19797},[156],[80,19799],{"className":19800,"style":1614},[160],[80,19802,2071],{"className":19803},[165],[46,19805,19806,20105,20398],{},[11,19807,19808,19929,19930,20015,20016,20101,20102,20104],{},[15,19809,19810,19811,3774],{},"Output with ",[80,19812,19814,19846],{"className":19813},[83],[80,19815,19817],{"className":19816},[87],[89,19818,19819],{"xmlns":91},[93,19820,19821,19843],{},[96,19822,19823,19825,19827,19830,19832],{},[102,19824,10551],{},[111,19826,130],{},[1321,19828,19829],{},"9",[111,19831,2834],{},[726,19833,19834,19836],{},[1321,19835,17444],{},[96,19837,19838,19840],{},[111,19839,4643],{},[1321,19841,19842],{},"7",[145,19844,19845],{"encoding":147},"\\alpha = 9\\times10^{-7}",[80,19847,19849,19867,19885],{"className":19848,"ariaHidden":113},[152],[80,19850,19852,19855,19858,19861,19864],{"className":19851},[156],[80,19853],{"className":19854,"style":334},[160],[80,19856,10551],{"className":19857,"style":10667},[165,169],[80,19859],{"className":19860,"style":247},[246],[80,19862,130],{"className":19863},[251],[80,19865],{"className":19866,"style":247},[246],[80,19868,19870,19873,19876,19879,19882],{"className":19869},[156],[80,19871],{"className":19872,"style":5303},[160],[80,19874,19829],{"className":19875},[165],[80,19877],{"className":19878,"style":275},[246],[80,19880,2834],{"className":19881},[279],[80,19883],{"className":19884,"style":275},[246],[80,19886,19888,19891,19894],{"className":19887},[156],[80,19889],{"className":19890,"style":1407},[160],[80,19892,1583],{"className":19893},[165],[80,19895,19897,19900],{"className":19896},[165],[80,19898,2071],{"className":19899},[165],[80,19901,19903],{"className":19902},[174],[80,19904,19906],{"className":19905},[178],[80,19907,19909],{"className":19908},[183],[80,19910,19912],{"className":19911,"style":1407},[187],[80,19913,19914,19917],{"style":772},[80,19915],{"className":19916,"style":196},[195],[80,19918,19920],{"className":19919},[200,201,202,203],[80,19921,19923,19926],{"className":19922},[165,203],[80,19924,4643],{"className":19925},[165,203],[80,19927,19842],{"className":19928},[165,203]," cost climbs every iteration, from ",[80,19931,19933,19956],{"className":19932},[83],[80,19934,19936],{"className":19935},[87],[89,19937,19938],{"xmlns":91},[93,19939,19940,19953],{},[96,19941,19942,19945,19947],{},[1321,19943,19944],{},"8.8",[111,19946,2834],{},[726,19948,19949,19951],{},[1321,19950,17444],{},[1321,19952,17445],{},[145,19954,19955],{"encoding":147},"8.8\\times10^4",[80,19957,19959,19977],{"className":19958,"ariaHidden":113},[152],[80,19960,19962,19965,19968,19971,19974],{"className":19961},[156],[80,19963],{"className":19964,"style":5303},[160],[80,19966,19944],{"className":19967},[165],[80,19969],{"className":19970,"style":275},[246],[80,19972,2834],{"className":19973},[279],[80,19975],{"className":19976,"style":275},[246],[80,19978,19980,19983,19986],{"className":19979},[156],[80,19981],{"className":19982,"style":1407},[160],[80,19984,1583],{"className":19985},[165],[80,19987,19989,19992],{"className":19988},[165],[80,19990,2071],{"className":19991},[165],[80,19993,19995],{"className":19994},[174],[80,19996,19998],{"className":19997},[178],[80,19999,20001],{"className":20000},[183],[80,20002,20004],{"className":20003,"style":1407},[187],[80,20005,20006,20009],{"style":772},[80,20007],{"className":20008,"style":196},[195],[80,20010,20012],{"className":20011},[200,201,202,203],[80,20013,17445],{"className":20014},[165,203]," to ",[80,20017,20019,20042],{"className":20018},[83],[80,20020,20022],{"className":20021},[87],[89,20023,20024],{"xmlns":91},[93,20025,20026,20039],{},[96,20027,20028,20031,20033],{},[1321,20029,20030],{},"9.4",[111,20032,2834],{},[726,20034,20035,20037],{},[1321,20036,17444],{},[1321,20038,15096],{},[145,20040,20041],{"encoding":147},"9.4\\times10^5",[80,20043,20045,20063],{"className":20044,"ariaHidden":113},[152],[80,20046,20048,20051,20054,20057,20060],{"className":20047},[156],[80,20049],{"className":20050,"style":5303},[160],[80,20052,20030],{"className":20053},[165],[80,20055],{"className":20056,"style":275},[246],[80,20058,2834],{"className":20059},[279],[80,20061],{"className":20062,"style":275},[246],[80,20064,20066,20069,20072],{"className":20065},[156],[80,20067],{"className":20068,"style":1407},[160],[80,20070,1583],{"className":20071},[165],[80,20073,20075,20078],{"className":20074},[165],[80,20076,2071],{"className":20077},[165],[80,20079,20081],{"className":20080},[174],[80,20082,20084],{"className":20083},[178],[80,20085,20087],{"className":20086},[183],[80,20088,20090],{"className":20089,"style":1407},[187],[80,20091,20092,20095],{"style":772},[80,20093],{"className":20094,"style":196},[195],[80,20096,20098],{"className":20097},[200,201,202,203],[80,20099,15096],{"className":20100},[165,203],". ",[65,20103,109],{}," flips sign constantly.",[11,20106,20107,20226,20227,20015,20310,20394,20395,20397],{},[15,20108,19810,20109,3774],{},[80,20110,20112,20143],{"className":20111},[83],[80,20113,20115],{"className":20114},[87],[89,20116,20117],{"xmlns":91},[93,20118,20119,20140],{},[96,20120,20121,20123,20125,20128,20130],{},[102,20122,10551],{},[111,20124,130],{},[1321,20126,20127],{},"8",[111,20129,2834],{},[726,20131,20132,20134],{},[1321,20133,17444],{},[96,20135,20136,20138],{},[111,20137,4643],{},[1321,20139,19842],{},[145,20141,20142],{"encoding":147},"\\alpha = 8\\times10^{-7}",[80,20144,20146,20164,20182],{"className":20145,"ariaHidden":113},[152],[80,20147,20149,20152,20155,20158,20161],{"className":20148},[156],[80,20150],{"className":20151,"style":334},[160],[80,20153,10551],{"className":20154,"style":10667},[165,169],[80,20156],{"className":20157,"style":247},[246],[80,20159,130],{"className":20160},[251],[80,20162],{"className":20163,"style":247},[246],[80,20165,20167,20170,20173,20176,20179],{"className":20166},[156],[80,20168],{"className":20169,"style":5303},[160],[80,20171,20127],{"className":20172},[165],[80,20174],{"className":20175,"style":275},[246],[80,20177,2834],{"className":20178},[279],[80,20180],{"className":20181,"style":275},[246],[80,20183,20185,20188,20191],{"className":20184},[156],[80,20186],{"className":20187,"style":1407},[160],[80,20189,1583],{"className":20190},[165],[80,20192,20194,20197],{"className":20193},[165],[80,20195,2071],{"className":20196},[165],[80,20198,20200],{"className":20199},[174],[80,20201,20203],{"className":20202},[178],[80,20204,20206],{"className":20205},[183],[80,20207,20209],{"className":20208,"style":1407},[187],[80,20210,20211,20214],{"style":772},[80,20212],{"className":20213,"style":196},[195],[80,20215,20217],{"className":20216},[200,201,202,203],[80,20218,20220,20223],{"className":20219},[165,203],[80,20221,4643],{"className":20222},[165,203],[80,20224,19842],{"className":20225},[165,203]," cost drops from ",[80,20228,20230,20251],{"className":20229},[83],[80,20231,20233],{"className":20232},[87],[89,20234,20235],{"xmlns":91},[93,20236,20237,20249],{},[96,20238,20239,20241,20243],{},[1321,20240,19944],{},[111,20242,2834],{},[726,20244,20245,20247],{},[1321,20246,17444],{},[1321,20248,17445],{},[145,20250,19955],{"encoding":147},[80,20252,20254,20272],{"className":20253,"ariaHidden":113},[152],[80,20255,20257,20260,20263,20266,20269],{"className":20256},[156],[80,20258],{"className":20259,"style":5303},[160],[80,20261,19944],{"className":20262},[165],[80,20264],{"className":20265,"style":275},[246],[80,20267,2834],{"className":20268},[279],[80,20270],{"className":20271,"style":275},[246],[80,20273,20275,20278,20281],{"className":20274},[156],[80,20276],{"className":20277,"style":1407},[160],[80,20279,1583],{"className":20280},[165],[80,20282,20284,20287],{"className":20283},[165],[80,20285,2071],{"className":20286},[165],[80,20288,20290],{"className":20289},[174],[80,20291,20293],{"className":20292},[178],[80,20294,20296],{"className":20295},[183],[80,20297,20299],{"className":20298,"style":1407},[187],[80,20300,20301,20304],{"style":772},[80,20302],{"className":20303,"style":196},[195],[80,20305,20307],{"className":20306},[200,201,202,203],[80,20308,17445],{"className":20309},[165,203],[80,20311,20313,20335],{"className":20312},[83],[80,20314,20316],{"className":20315},[87],[89,20317,20318],{"xmlns":91},[93,20319,20320,20332],{},[96,20321,20322,20324,20326],{},[1321,20323,20030],{},[111,20325,2834],{},[726,20327,20328,20330],{},[1321,20329,17444],{},[1321,20331,13895],{},[145,20333,20334],{"encoding":147},"9.4\\times10^3",[80,20336,20338,20356],{"className":20337,"ariaHidden":113},[152],[80,20339,20341,20344,20347,20350,20353],{"className":20340},[156],[80,20342],{"className":20343,"style":5303},[160],[80,20345,20030],{"className":20346},[165],[80,20348],{"className":20349,"style":275},[246],[80,20351,2834],{"className":20352},[279],[80,20354],{"className":20355,"style":275},[246],[80,20357,20359,20362,20365],{"className":20358},[156],[80,20360],{"className":20361,"style":1407},[160],[80,20363,1583],{"className":20364},[165],[80,20366,20368,20371],{"className":20367},[165],[80,20369,2071],{"className":20370},[165],[80,20372,20374],{"className":20373},[174],[80,20375,20377],{"className":20376},[178],[80,20378,20380],{"className":20379},[183],[80,20381,20383],{"className":20382,"style":1407},[187],[80,20384,20385,20388],{"style":772},[80,20386],{"className":20387,"style":196},[195],[80,20389,20391],{"className":20390},[200,201,202,203],[80,20392,13895],{"className":20393},[165,203],", but ",[65,20396,109],{}," still oscillates (0 → 0.51 → 0.06 → 0.46 → 0.10...).",[11,20399,20400,20226,20518,20015,20601,504,20686,20688],{},[15,20401,19810,20402,3774],{},[80,20403,20405,20435],{"className":20404},[83],[80,20406,20408],{"className":20407},[87],[89,20409,20410],{"xmlns":91},[93,20411,20412,20432],{},[96,20413,20414,20416,20418,20420,20422],{},[102,20415,10551],{},[111,20417,130],{},[1321,20419,1583],{},[111,20421,2834],{},[726,20423,20424,20426],{},[1321,20425,17444],{},[96,20427,20428,20430],{},[111,20429,4643],{},[1321,20431,19842],{},[145,20433,20434],{"encoding":147},"\\alpha = 1\\times10^{-7}",[80,20436,20438,20456,20474],{"className":20437,"ariaHidden":113},[152],[80,20439,20441,20444,20447,20450,20453],{"className":20440},[156],[80,20442],{"className":20443,"style":334},[160],[80,20445,10551],{"className":20446,"style":10667},[165,169],[80,20448],{"className":20449,"style":247},[246],[80,20451,130],{"className":20452},[251],[80,20454],{"className":20455,"style":247},[246],[80,20457,20459,20462,20465,20468,20471],{"className":20458},[156],[80,20460],{"className":20461,"style":5303},[160],[80,20463,1583],{"className":20464},[165],[80,20466],{"className":20467,"style":275},[246],[80,20469,2834],{"className":20470},[279],[80,20472],{"className":20473,"style":275},[246],[80,20475,20477,20480,20483],{"className":20476},[156],[80,20478],{"className":20479,"style":1407},[160],[80,20481,1583],{"className":20482},[165],[80,20484,20486,20489],{"className":20485},[165],[80,20487,2071],{"className":20488},[165],[80,20490,20492],{"className":20491},[174],[80,20493,20495],{"className":20494},[178],[80,20496,20498],{"className":20497},[183],[80,20499,20501],{"className":20500,"style":1407},[187],[80,20502,20503,20506],{"style":772},[80,20504],{"className":20505,"style":196},[195],[80,20507,20509],{"className":20508},[200,201,202,203],[80,20510,20512,20515],{"className":20511},[165,203],[80,20513,4643],{"className":20514},[165,203],[80,20516,19842],{"className":20517},[165,203],[80,20519,20521,20542],{"className":20520},[83],[80,20522,20524],{"className":20523},[87],[89,20525,20526],{"xmlns":91},[93,20527,20528,20540],{},[96,20529,20530,20532,20534],{},[1321,20531,19944],{},[111,20533,2834],{},[726,20535,20536,20538],{},[1321,20537,17444],{},[1321,20539,17445],{},[145,20541,19955],{"encoding":147},[80,20543,20545,20563],{"className":20544,"ariaHidden":113},[152],[80,20546,20548,20551,20554,20557,20560],{"className":20547},[156],[80,20549],{"className":20550,"style":5303},[160],[80,20552,19944],{"className":20553},[165],[80,20555],{"className":20556,"style":275},[246],[80,20558,2834],{"className":20559},[279],[80,20561],{"className":20562,"style":275},[246],[80,20564,20566,20569,20572],{"className":20565},[156],[80,20567],{"className":20568,"style":1407},[160],[80,20570,1583],{"className":20571},[165],[80,20573,20575,20578],{"className":20574},[165],[80,20576,2071],{"className":20577},[165],[80,20579,20581],{"className":20580},[174],[80,20582,20584],{"className":20583},[178],[80,20585,20587],{"className":20586},[183],[80,20588,20590],{"className":20589,"style":1407},[187],[80,20591,20592,20595],{"style":772},[80,20593],{"className":20594,"style":196},[195],[80,20596,20598],{"className":20597},[200,201,202,203],[80,20599,17445],{"className":20600},[165,203],[80,20602,20604,20627],{"className":20603},[83],[80,20605,20607],{"className":20606},[87],[89,20608,20609],{"xmlns":91},[93,20610,20611,20624],{},[96,20612,20613,20616,20618],{},[1321,20614,20615],{},"1.3",[111,20617,2834],{},[726,20619,20620,20622],{},[1321,20621,17444],{},[1321,20623,13895],{},[145,20625,20626],{"encoding":147},"1.3\\times10^3",[80,20628,20630,20648],{"className":20629,"ariaHidden":113},[152],[80,20631,20633,20636,20639,20642,20645],{"className":20632},[156],[80,20634],{"className":20635,"style":5303},[160],[80,20637,20615],{"className":20638},[165],[80,20640],{"className":20641,"style":275},[246],[80,20643,2834],{"className":20644},[279],[80,20646],{"className":20647,"style":275},[246],[80,20649,20651,20654,20657],{"className":20650},[156],[80,20652],{"className":20653,"style":1407},[160],[80,20655,1583],{"className":20656},[165],[80,20658,20660,20663],{"className":20659},[165],[80,20661,2071],{"className":20662},[165],[80,20664,20666],{"className":20665},[174],[80,20667,20669],{"className":20668},[178],[80,20670,20672],{"className":20671},[183],[80,20673,20675],{"className":20674,"style":1407},[187],[80,20676,20677,20680],{"style":772},[80,20678],{"className":20679,"style":196},[195],[80,20681,20683],{"className":20682},[200,201,202,203],[80,20684,13895],{"className":20685},[165,203],[65,20687,109],{}," climbs straight up without oscillating (0 → 0.06 → 0.11 → 0.15...).",[11,20690,20691],{},"Try it yourself in the simulator below. I started with a fairly small alpha on purpose, drag the slider up slowly and notice exactly where the cost flips from \"dropping\" to \"climbing\":",[13355,20693],{":b-range":20694,":initial-b":2071,":initial-w":2071,":w-range":20695,":x-train":20696,":y-train":20697,"b-label":117,"w-label":109,":alpha-presets":20698,":alpha-slider-max":20699,":alpha-slider-min":20700,":alpha-slider-step":20700},"[-250, 250]","[0, 0.6]","[952, 1244, 1947, 1725, 1959, 1314, 864, 1836]","[271.5, 300, 509.8, 394, 540, 415, 230, 560]","[1e-7, 3e-7, 8e-7, 9e-7, 9.9e-7]","2e-6","1e-8",[11,20702,20703,20704,2338,20798,20893],{},"The pattern I saw matches exactly what the notebook describes for the full dataset (with all 100 houses and all 4 features, the threshold sits between ",[80,20705,20707,20733],{"className":20706},[83],[80,20708,20710],{"className":20709},[87],[89,20711,20712],{"xmlns":91},[93,20713,20714,20730],{},[96,20715,20716,20718,20720],{},[1321,20717,19829],{},[111,20719,2834],{},[726,20721,20722,20724],{},[1321,20723,17444],{},[96,20725,20726,20728],{},[111,20727,4643],{},[1321,20729,19842],{},[145,20731,20732],{"encoding":147},"9\\times10^{-7}",[80,20734,20736,20754],{"className":20735,"ariaHidden":113},[152],[80,20737,20739,20742,20745,20748,20751],{"className":20738},[156],[80,20740],{"className":20741,"style":5303},[160],[80,20743,19829],{"className":20744},[165],[80,20746],{"className":20747,"style":275},[246],[80,20749,2834],{"className":20750},[279],[80,20752],{"className":20753,"style":275},[246],[80,20755,20757,20760,20763],{"className":20756},[156],[80,20758],{"className":20759,"style":1407},[160],[80,20761,1583],{"className":20762},[165],[80,20764,20766,20769],{"className":20765},[165],[80,20767,2071],{"className":20768},[165],[80,20770,20772],{"className":20771},[174],[80,20773,20775],{"className":20774},[178],[80,20776,20778],{"className":20777},[183],[80,20779,20781],{"className":20780,"style":1407},[187],[80,20782,20783,20786],{"style":772},[80,20784],{"className":20785,"style":196},[195],[80,20787,20789],{"className":20788},[200,201,202,203],[80,20790,20792,20795],{"className":20791},[165,203],[80,20793,4643],{"className":20794},[165,203],[80,20796,19842],{"className":20797},[165,203],[80,20799,20801,20828],{"className":20800},[83],[80,20802,20804],{"className":20803},[87],[89,20805,20806],{"xmlns":91},[93,20807,20808,20825],{},[96,20809,20810,20813,20815],{},[1321,20811,20812],{},"9.9",[111,20814,2834],{},[726,20816,20817,20819],{},[1321,20818,17444],{},[96,20820,20821,20823],{},[111,20822,4643],{},[1321,20824,19842],{},[145,20826,20827],{"encoding":147},"9.9\\times10^{-7}",[80,20829,20831,20849],{"className":20830,"ariaHidden":113},[152],[80,20832,20834,20837,20840,20843,20846],{"className":20833},[156],[80,20835],{"className":20836,"style":5303},[160],[80,20838,20812],{"className":20839},[165],[80,20841],{"className":20842,"style":275},[246],[80,20844,2834],{"className":20845},[279],[80,20847],{"className":20848,"style":275},[246],[80,20850,20852,20855,20858],{"className":20851},[156],[80,20853],{"className":20854,"style":1407},[160],[80,20856,1583],{"className":20857},[165],[80,20859,20861,20864],{"className":20860},[165],[80,20862,2071],{"className":20863},[165],[80,20865,20867],{"className":20866},[174],[80,20868,20870],{"className":20869},[178],[80,20871,20873],{"className":20872},[183],[80,20874,20876],{"className":20875,"style":1407},[187],[80,20877,20878,20881],{"style":772},[80,20879],{"className":20880,"style":196},[195],[80,20882,20884],{"className":20883},[200,201,202,203],[80,20885,20887,20890],{"className":20886},[165,203],[80,20888,4643],{"className":20889},[165,203],[80,20891,19842],{"className":20892},[165,203],", very close to what I found here with just size and a smaller subset). That's not a coincidence, and the next section explains exactly why.",[294,20895,20897,20898,20936],{"id":20896},"why-exactly-there-the-2l2l2l-bound-and-the-condition-number","Why exactly there: the ",[80,20899,20901,20920],{"className":20900},[83],[80,20902,20904],{"className":20903},[87],[89,20905,20906],{"xmlns":91},[93,20907,20908,20917],{},[96,20909,20910,20912,20915],{},[1321,20911,1323],{},[102,20913,20914],{"mathvariant":10558},"\u002F",[102,20916,17513],{},[145,20918,20919],{"encoding":147},"2\u002FL",[80,20921,20923],{"className":20922,"ariaHidden":113},[152],[80,20924,20926,20929,20933],{"className":20925},[156],[80,20927],{"className":20928,"style":2316},[160],[80,20930,20932],{"className":20931},[165],"2\u002F",[80,20934,17513],{"className":20935},[165,169]," bound and the condition number",[11,20938,20939],{},"This isn't trial and error, it can be calculated. For a quadratic cost function like ours, gradient descent converges if and only if",[11,20941,20942],{},[80,20943,20945,20998],{"className":20944},[83],[80,20946,20948],{"className":20947},[87],[89,20949,20950],{"xmlns":91},[93,20951,20952,20995],{},[96,20953,20954,20956,20959,20961,20963,20969,20971,20973,20975,20977,20988,20990,20993],{},[1321,20955,2071],{},[111,20957,20958],{},"\u003C",[102,20960,10551],{},[111,20962,20958],{},[4625,20964,20965,20967],{},[1321,20966,1323],{},[102,20968,17513],{},[111,20970,114],{"separator":113},[246,20972],{"width":10580},[102,20974,17513],{},[111,20976,130],{},[99,20978,20979,20982,20985],{},[102,20980,20981],{},"λ",[102,20983,20984],{},"max",[111,20986,20987],{},"⁡",[111,20989,121],{"stretchy":120},[102,20991,20992],{"mathvariant":601},"H",[111,20994,127],{"stretchy":120},[145,20996,20997],{"encoding":147},"0 \u003C \\alpha \u003C \\frac{2}{L}, \\qquad L = \\lambda_{\\max}(\\mathbf{H})",[80,20999,21001,21020,21039,21134],{"className":21000,"ariaHidden":113},[152],[80,21002,21004,21008,21011,21014,21017],{"className":21003},[156],[80,21005],{"className":21006,"style":21007},[160],"height:0.6835em;vertical-align:-0.0391em;",[80,21009,2071],{"className":21010},[165],[80,21012],{"className":21013,"style":247},[246],[80,21015,20958],{"className":21016},[251],[80,21018],{"className":21019,"style":247},[246],[80,21021,21023,21027,21030,21033,21036],{"className":21022},[156],[80,21024],{"className":21025,"style":21026},[160],"height:0.5782em;vertical-align:-0.0391em;",[80,21028,10551],{"className":21029,"style":10667},[165,169],[80,21031],{"className":21032,"style":247},[246],[80,21034,20958],{"className":21035},[251],[80,21037],{"className":21038,"style":247},[246],[80,21040,21042,21045,21113,21116,21119,21122,21125,21128,21131],{"className":21041},[156],[80,21043],{"className":21044,"style":8391},[160],[80,21046,21048,21051,21110],{"className":21047},[165],[80,21049],{"className":21050},[235,4746],[80,21052,21054],{"className":21053},[4625],[80,21055,21057,21102],{"className":21056},[178,179],[80,21058,21060,21099],{"className":21059},[183],[80,21061,21063,21077,21085],{"className":21062,"style":5010},[187],[80,21064,21065,21068],{"style":5013},[80,21066],{"className":21067,"style":4766},[195],[80,21069,21071],{"className":21070},[200,201,202,203],[80,21072,21074],{"className":21073},[165,203],[80,21075,17513],{"className":21076},[165,169,203],[80,21078,21079,21082],{"style":4859},[80,21080],{"className":21081,"style":4766},[195],[80,21083],{"className":21084,"style":4867},[4866],[80,21086,21087,21090],{"style":5042},[80,21088],{"className":21089,"style":4766},[195],[80,21091,21093],{"className":21092},[200,201,202,203],[80,21094,21096],{"className":21095},[165,203],[80,21097,1323],{"className":21098},[165,203],[80,21100,222],{"className":21101},[221],[80,21103,21105],{"className":21104},[183],[80,21106,21108],{"className":21107,"style":7390},[187],[80,21109],{},[80,21111],{"className":21112},[242,4746],[80,21114,114],{"className":21115},[214],[80,21117],{"className":21118,"style":10763},[246],[80,21120],{"className":21121,"style":268},[246],[80,21123,17513],{"className":21124},[165,169],[80,21126],{"className":21127,"style":247},[246],[80,21129,130],{"className":21130},[251],[80,21132],{"className":21133,"style":247},[246],[80,21135,21137,21140,21193,21196,21199],{"className":21136},[156],[80,21138],{"className":21139,"style":2316},[160],[80,21141,21143,21146],{"className":21142},[165],[80,21144,20981],{"className":21145},[165,169],[80,21147,21149],{"className":21148},[174],[80,21150,21152,21185],{"className":21151},[178,179],[80,21153,21155,21182],{"className":21154},[183],[80,21156,21159],{"className":21157,"style":21158},[187],"height:0.1514em;",[80,21160,21161,21164],{"style":15326},[80,21162],{"className":21163,"style":196},[195],[80,21165,21167],{"className":21166},[200,201,202,203],[80,21168,21170],{"className":21169},[165,203],[80,21171,21173,21176,21179],{"className":21172},[7402,203],[80,21174,322],{"className":21175},[203],[80,21177,562],{"className":21178},[203],[80,21180,124],{"className":21181},[203],[80,21183,222],{"className":21184},[221],[80,21186,21188],{"className":21187},[183],[80,21189,21191],{"className":21190,"style":15345},[187],[80,21192],{},[80,21194,121],{"className":21195},[235],[80,21197,20992],{"className":21198},[165,618],[80,21200,127],{"className":21201},[242],[11,21203,21204,21205,21234,21235,21239,21240,21325,21326,21354],{},"where ",[80,21206,21208,21222],{"className":21207},[83],[80,21209,21211],{"className":21210},[87],[89,21212,21213],{"xmlns":91},[93,21214,21215,21219],{},[96,21216,21217],{},[102,21218,20992],{"mathvariant":601},[145,21220,21221],{"encoding":147},"\\mathbf{H}",[80,21223,21225],{"className":21224,"ariaHidden":113},[152],[80,21226,21228,21231],{"className":21227},[156],[80,21229],{"className":21230,"style":15532},[160],[80,21232,20992],{"className":21233},[165,618]," is the ",[7103,21236,21238],{"definition":21237},"the matrix of second derivatives of the cost function, it describes the bowl's curvature in every direction","Hessian"," of the cost and ",[80,21241,21243,21263],{"className":21242},[83],[80,21244,21246],{"className":21245},[87],[89,21247,21248],{"xmlns":91},[93,21249,21250,21260],{},[96,21251,21252],{},[99,21253,21254,21256,21258],{},[102,21255,20981],{},[102,21257,20984],{},[111,21259,20987],{},[145,21261,21262],{"encoding":147},"\\lambda_{\\max}",[80,21264,21266],{"className":21265,"ariaHidden":113},[152],[80,21267,21269,21273],{"className":21268},[156],[80,21270],{"className":21271,"style":21272},[160],"height:0.8444em;vertical-align:-0.15em;",[80,21274,21276,21279],{"className":21275},[165],[80,21277,20981],{"className":21278},[165,169],[80,21280,21282],{"className":21281},[174],[80,21283,21285,21317],{"className":21284},[178,179],[80,21286,21288,21314],{"className":21287},[183],[80,21289,21291],{"className":21290,"style":21158},[187],[80,21292,21293,21296],{"style":15326},[80,21294],{"className":21295,"style":196},[195],[80,21297,21299],{"className":21298},[200,201,202,203],[80,21300,21302],{"className":21301},[165,203],[80,21303,21305,21308,21311],{"className":21304},[7402,203],[80,21306,322],{"className":21307},[203],[80,21309,562],{"className":21310},[203],[80,21312,124],{"className":21313},[203],[80,21315,222],{"className":21316},[221],[80,21318,21320],{"className":21319},[183],[80,21321,21323],{"className":21322,"style":15345},[187],[80,21324],{}," is its largest eigenvalue. ",[80,21327,21329,21342],{"className":21328},[83],[80,21330,21332],{"className":21331},[87],[89,21333,21334],{"xmlns":91},[93,21335,21336,21340],{},[96,21337,21338],{},[102,21339,17513],{},[145,21341,17513],{"encoding":147},[80,21343,21345],{"className":21344,"ariaHidden":113},[152],[80,21346,21348,21351],{"className":21347},[156],[80,21349],{"className":21350,"style":8672},[160],[80,21352,17513],{"className":21353},[165,169]," is the bowl's maximum curvature: the sharper the steepest direction, the smaller the step allowed before it overshoots to the other side of the valley.",[11,21356,21357,21358,21409,21410,2338,21438,21466,21467,21575,21576,21693,21694,21787,21788,21882],{},"I computed the Hessian for our 8-house example (a ",[80,21359,21361,21379],{"className":21360},[83],[80,21362,21364],{"className":21363},[87],[89,21365,21366],{"xmlns":91},[93,21367,21368,21376],{},[96,21369,21370,21372,21374],{},[1321,21371,1323],{},[111,21373,2834],{},[1321,21375,1323],{},[145,21377,21378],{"encoding":147},"2\\times2",[80,21380,21382,21400],{"className":21381,"ariaHidden":113},[152],[80,21383,21385,21388,21391,21394,21397],{"className":21384},[156],[80,21386],{"className":21387,"style":5303},[160],[80,21389,1323],{"className":21390},[165],[80,21392],{"className":21393,"style":275},[246],[80,21395,2834],{"className":21396},[279],[80,21398],{"className":21399,"style":275},[246],[80,21401,21403,21406],{"className":21402},[156],[80,21404],{"className":21405,"style":1614},[160],[80,21407,1323],{"className":21408},[165]," calculation, since it's just ",[80,21411,21413,21426],{"className":21412},[83],[80,21414,21416],{"className":21415},[87],[89,21417,21418],{"xmlns":91},[93,21419,21420,21424],{},[96,21421,21422],{},[102,21423,109],{},[145,21425,109],{"encoding":147},[80,21427,21429],{"className":21428,"ariaHidden":113},[152],[80,21430,21432,21435],{"className":21431},[156],[80,21433],{"className":21434,"style":334},[160],[80,21436,109],{"className":21437,"style":210},[165,169],[80,21439,21441,21454],{"className":21440},[83],[80,21442,21444],{"className":21443},[87],[89,21445,21446],{"xmlns":91},[93,21447,21448,21452],{},[96,21449,21450],{},[102,21451,117],{},[145,21453,117],{"encoding":147},[80,21455,21457],{"className":21456,"ariaHidden":113},[152],[80,21458,21460,21463],{"className":21459},[156],[80,21461],{"className":21462,"style":289},[160],[80,21464,117],{"className":21465},[165,169],") and got ",[80,21468,21470,21498],{"className":21469},[83],[80,21471,21473],{"className":21472},[87],[89,21474,21475],{"xmlns":91},[93,21476,21477,21495],{},[96,21478,21479,21481,21483,21486,21488],{},[102,21480,17513],{},[111,21482,9527],{},[1321,21484,21485],{},"2.36",[111,21487,2834],{},[726,21489,21490,21492],{},[1321,21491,17444],{},[1321,21493,21494],{},"6",[145,21496,21497],{"encoding":147},"L \\approx 2.36\\times10^6",[80,21499,21501,21519,21537],{"className":21500,"ariaHidden":113},[152],[80,21502,21504,21507,21510,21513,21516],{"className":21503},[156],[80,21505],{"className":21506,"style":8672},[160],[80,21508,17513],{"className":21509},[165,169],[80,21511],{"className":21512,"style":247},[246],[80,21514,9527],{"className":21515},[251],[80,21517],{"className":21518,"style":247},[246],[80,21520,21522,21525,21528,21531,21534],{"className":21521},[156],[80,21523],{"className":21524,"style":5303},[160],[80,21526,21485],{"className":21527},[165],[80,21529],{"className":21530,"style":275},[246],[80,21532,2834],{"className":21533},[279],[80,21535],{"className":21536,"style":275},[246],[80,21538,21540,21543,21546],{"className":21539},[156],[80,21541],{"className":21542,"style":1407},[160],[80,21544,1583],{"className":21545},[165],[80,21547,21549,21552],{"className":21548},[165],[80,21550,2071],{"className":21551},[165],[80,21553,21555],{"className":21554},[174],[80,21556,21558],{"className":21557},[178],[80,21559,21561],{"className":21560},[183],[80,21562,21564],{"className":21563,"style":1407},[187],[80,21565,21566,21569],{"style":772},[80,21567],{"className":21568,"style":196},[195],[80,21570,21572],{"className":21571},[200,201,202,203],[80,21573,21494],{"className":21574},[165,203],", giving a critical ",[80,21577,21579,21610],{"className":21578},[83],[80,21580,21582],{"className":21581},[87],[89,21583,21584],{"xmlns":91},[93,21585,21586,21607],{},[96,21587,21588,21590,21592,21595,21597],{},[102,21589,10551],{},[111,21591,9527],{},[1321,21593,21594],{},"8.46",[111,21596,2834],{},[726,21598,21599,21601],{},[1321,21600,17444],{},[96,21602,21603,21605],{},[111,21604,4643],{},[1321,21606,19842],{},[145,21608,21609],{"encoding":147},"\\alpha \\approx 8.46\\times10^{-7}",[80,21611,21613,21631,21649],{"className":21612,"ariaHidden":113},[152],[80,21614,21616,21619,21622,21625,21628],{"className":21615},[156],[80,21617],{"className":21618,"style":9543},[160],[80,21620,10551],{"className":21621,"style":10667},[165,169],[80,21623],{"className":21624,"style":247},[246],[80,21626,9527],{"className":21627},[251],[80,21629],{"className":21630,"style":247},[246],[80,21632,21634,21637,21640,21643,21646],{"className":21633},[156],[80,21635],{"className":21636,"style":5303},[160],[80,21638,21594],{"className":21639},[165],[80,21641],{"className":21642,"style":275},[246],[80,21644,2834],{"className":21645},[279],[80,21647],{"className":21648,"style":275},[246],[80,21650,21652,21655,21658],{"className":21651},[156],[80,21653],{"className":21654,"style":1407},[160],[80,21656,1583],{"className":21657},[165],[80,21659,21661,21664],{"className":21660},[165],[80,21662,2071],{"className":21663},[165],[80,21665,21667],{"className":21666},[174],[80,21668,21670],{"className":21669},[178],[80,21671,21673],{"className":21672},[183],[80,21674,21676],{"className":21675,"style":1407},[187],[80,21677,21678,21681],{"style":772},[80,21679],{"className":21680,"style":196},[195],[80,21682,21684],{"className":21683},[200,201,202,203],[80,21685,21687,21690],{"className":21686},[165,203],[80,21688,4643],{"className":21689},[165,203],[80,21691,19842],{"className":21692},[165,203],". That matches what I saw in the simulator: ",[80,21695,21697,21722],{"className":21696},[83],[80,21698,21700],{"className":21699},[87],[89,21701,21702],{"xmlns":91},[93,21703,21704,21720],{},[96,21705,21706,21708,21710],{},[1321,21707,19829],{},[111,21709,2834],{},[726,21711,21712,21714],{},[1321,21713,17444],{},[96,21715,21716,21718],{},[111,21717,4643],{},[1321,21719,19842],{},[145,21721,20732],{"encoding":147},[80,21723,21725,21743],{"className":21724,"ariaHidden":113},[152],[80,21726,21728,21731,21734,21737,21740],{"className":21727},[156],[80,21729],{"className":21730,"style":5303},[160],[80,21732,19829],{"className":21733},[165],[80,21735],{"className":21736,"style":275},[246],[80,21738,2834],{"className":21739},[279],[80,21741],{"className":21742,"style":275},[246],[80,21744,21746,21749,21752],{"className":21745},[156],[80,21747],{"className":21748,"style":1407},[160],[80,21750,1583],{"className":21751},[165],[80,21753,21755,21758],{"className":21754},[165],[80,21756,2071],{"className":21757},[165],[80,21759,21761],{"className":21760},[174],[80,21762,21764],{"className":21763},[178],[80,21765,21767],{"className":21766},[183],[80,21768,21770],{"className":21769,"style":1407},[187],[80,21771,21772,21775],{"style":772},[80,21773],{"className":21774,"style":196},[195],[80,21776,21778],{"className":21777},[200,201,202,203],[80,21779,21781,21784],{"className":21780},[165,203],[80,21782,4643],{"className":21783},[165,203],[80,21785,19842],{"className":21786},[165,203]," crosses that threshold and diverges, ",[80,21789,21791,21817],{"className":21790},[83],[80,21792,21794],{"className":21793},[87],[89,21795,21796],{"xmlns":91},[93,21797,21798,21814],{},[96,21799,21800,21802,21804],{},[1321,21801,20127],{},[111,21803,2834],{},[726,21805,21806,21808],{},[1321,21807,17444],{},[96,21809,21810,21812],{},[111,21811,4643],{},[1321,21813,19842],{},[145,21815,21816],{"encoding":147},"8\\times10^{-7}",[80,21818,21820,21838],{"className":21819,"ariaHidden":113},[152],[80,21821,21823,21826,21829,21832,21835],{"className":21822},[156],[80,21824],{"className":21825,"style":5303},[160],[80,21827,20127],{"className":21828},[165],[80,21830],{"className":21831,"style":275},[246],[80,21833,2834],{"className":21834},[279],[80,21836],{"className":21837,"style":275},[246],[80,21839,21841,21844,21847],{"className":21840},[156],[80,21842],{"className":21843,"style":1407},[160],[80,21845,1583],{"className":21846},[165],[80,21848,21850,21853],{"className":21849},[165],[80,21851,2071],{"className":21852},[165],[80,21854,21856],{"className":21855},[174],[80,21857,21859],{"className":21858},[178],[80,21860,21862],{"className":21861},[183],[80,21863,21865],{"className":21864,"style":1407},[187],[80,21866,21867,21870],{"style":772},[80,21868],{"className":21869,"style":196},[195],[80,21871,21873],{"className":21872},[200,201,202,203],[80,21874,21876,21879],{"className":21875},[165,203],[80,21877,4643],{"className":21878},[165,203],[80,21880,19842],{"className":21881},[165,203]," stays under it and converges.",[11,21884,21885,21886,21937,21938,21966,21967,22084],{},"The original notebook runs this same calculation with the real 4 features (the Hessian becomes ",[80,21887,21889,21907],{"className":21888},[83],[80,21890,21892],{"className":21891},[87],[89,21893,21894],{"xmlns":91},[93,21895,21896,21904],{},[96,21897,21898,21900,21902],{},[1321,21899,15096],{},[111,21901,2834],{},[1321,21903,15096],{},[145,21905,21906],{"encoding":147},"5\\times5",[80,21908,21910,21928],{"className":21909,"ariaHidden":113},[152],[80,21911,21913,21916,21919,21922,21925],{"className":21912},[156],[80,21914],{"className":21915,"style":5303},[160],[80,21917,15096],{"className":21918},[165],[80,21920],{"className":21921,"style":275},[246],[80,21923,2834],{"className":21924},[279],[80,21926],{"className":21927,"style":275},[246],[80,21929,21931,21934],{"className":21930},[156],[80,21932],{"className":21933,"style":1614},[160],[80,21935,15096],{"className":21936},[165],", counting ",[80,21939,21941,21954],{"className":21940},[83],[80,21942,21944],{"className":21943},[87],[89,21945,21946],{"xmlns":91},[93,21947,21948,21952],{},[96,21949,21950],{},[102,21951,117],{},[145,21953,117],{"encoding":147},[80,21955,21957],{"className":21956,"ariaHidden":113},[152],[80,21958,21960,21963],{"className":21959},[156],[80,21961],{"className":21962,"style":289},[160],[80,21964,117],{"className":21965},[165,169],") and lands on a critical ",[80,21968,21970,22001],{"className":21969},[83],[80,21971,21973],{"className":21972},[87],[89,21974,21975],{"xmlns":91},[93,21976,21977,21998],{},[96,21978,21979,21981,21983,21986,21988],{},[102,21980,10551],{},[111,21982,9527],{},[1321,21984,21985],{},"9.22",[111,21987,2834],{},[726,21989,21990,21992],{},[1321,21991,17444],{},[96,21993,21994,21996],{},[111,21995,4643],{},[1321,21997,19842],{},[145,21999,22000],{"encoding":147},"\\alpha \\approx 9.22\\times10^{-7}",[80,22002,22004,22022,22040],{"className":22003,"ariaHidden":113},[152],[80,22005,22007,22010,22013,22016,22019],{"className":22006},[156],[80,22008],{"className":22009,"style":9543},[160],[80,22011,10551],{"className":22012,"style":10667},[165,169],[80,22014],{"className":22015,"style":247},[246],[80,22017,9527],{"className":22018},[251],[80,22020],{"className":22021,"style":247},[246],[80,22023,22025,22028,22031,22034,22037],{"className":22024},[156],[80,22026],{"className":22027,"style":5303},[160],[80,22029,21985],{"className":22030},[165],[80,22032],{"className":22033,"style":275},[246],[80,22035,2834],{"className":22036},[279],[80,22038],{"className":22039,"style":275},[246],[80,22041,22043,22046,22049],{"className":22042},[156],[80,22044],{"className":22045,"style":1407},[160],[80,22047,1583],{"className":22048},[165],[80,22050,22052,22055],{"className":22051},[165],[80,22053,2071],{"className":22054},[165],[80,22056,22058],{"className":22057},[174],[80,22059,22061],{"className":22060},[178],[80,22062,22064],{"className":22063},[183],[80,22065,22067],{"className":22066,"style":1407},[187],[80,22068,22069,22072],{"style":772},[80,22070],{"className":22071,"style":196},[195],[80,22073,22075],{"className":22074},[200,201,202,203],[80,22076,22078,22081],{"className":22077},[165,203],[80,22079,4643],{"className":22080},[165,203],[80,22082,19842],{"className":22083},[165,203],", practically the same number I found by simplifying to 1 feature. That makes sense: size is the feature with the most absurd scale, so it alone already dominates most of the bowl's curvature.",[11,22086,22087,22088,3255,22092,22155,22156,22185,22186,22293,22294,22297],{},"There's a second number that comes out of this same calculation, the ",[7103,22089,22091],{"definition":22090},"the ratio between the largest and smallest curvature of the cost bowl, kappa = L\u002Fmu, and the bigger it is, the more elongated the bowl and the slower gradient descent moves","condition number",[80,22093,22095,22119],{"className":22094},[83],[80,22096,22098],{"className":22097},[87],[89,22099,22100],{"xmlns":91},[93,22101,22102,22116],{},[96,22103,22104,22107,22109,22111,22113],{},[102,22105,22106],{},"κ",[111,22108,130],{},[102,22110,17513],{},[102,22112,20914],{"mathvariant":10558},[102,22114,22115],{},"μ",[145,22117,22118],{"encoding":147},"\\kappa = L\u002F\\mu",[80,22120,22122,22140],{"className":22121,"ariaHidden":113},[152],[80,22123,22125,22128,22131,22134,22137],{"className":22124},[156],[80,22126],{"className":22127,"style":334},[160],[80,22129,22106],{"className":22130},[165,169],[80,22132],{"className":22133,"style":247},[246],[80,22135,130],{"className":22136},[251],[80,22138],{"className":22139,"style":247},[246],[80,22141,22143,22146,22149,22152],{"className":22142},[156],[80,22144],{"className":22145,"style":2316},[160],[80,22147,17513],{"className":22148},[165,169],[80,22150,20914],{"className":22151},[165],[80,22153,22115],{"className":22154},[165,169]," (where ",[80,22157,22159,22173],{"className":22158},[83],[80,22160,22162],{"className":22161},[87],[89,22163,22164],{"xmlns":91},[93,22165,22166,22170],{},[96,22167,22168],{},[102,22169,22115],{},[145,22171,22172],{"encoding":147},"\\mu",[80,22174,22176],{"className":22175,"ariaHidden":113},[152],[80,22177,22179,22182],{"className":22178},[156],[80,22180],{"className":22181,"style":4514},[160],[80,22183,22115],{"className":22184},[165,169]," is the smallest curvature). It measures how \"stretched\" the bowl is. I found ",[80,22187,22189,22216],{"className":22188},[83],[80,22190,22192],{"className":22191},[87],[89,22193,22194],{"xmlns":91},[93,22195,22196,22213],{},[96,22197,22198,22200,22202,22205,22207],{},[102,22199,22106],{},[111,22201,9527],{},[1321,22203,22204],{},"5.8",[111,22206,2834],{},[726,22208,22209,22211],{},[1321,22210,17444],{},[1321,22212,19842],{},[145,22214,22215],{"encoding":147},"\\kappa \\approx 5.8\\times10^7",[80,22217,22219,22237,22255],{"className":22218,"ariaHidden":113},[152],[80,22220,22222,22225,22228,22231,22234],{"className":22221},[156],[80,22223],{"className":22224,"style":9543},[160],[80,22226,22106],{"className":22227},[165,169],[80,22229],{"className":22230,"style":247},[246],[80,22232,9527],{"className":22233},[251],[80,22235],{"className":22236,"style":247},[246],[80,22238,22240,22243,22246,22249,22252],{"className":22239},[156],[80,22241],{"className":22242,"style":5303},[160],[80,22244,22204],{"className":22245},[165],[80,22247],{"className":22248,"style":275},[246],[80,22250,2834],{"className":22251},[279],[80,22253],{"className":22254,"style":275},[246],[80,22256,22258,22261,22264],{"className":22257},[156],[80,22259],{"className":22260,"style":1407},[160],[80,22262,1583],{"className":22263},[165],[80,22265,22267,22270],{"className":22266},[165],[80,22268,2071],{"className":22269},[165],[80,22271,22273],{"className":22272},[174],[80,22274,22276],{"className":22275},[178],[80,22277,22279],{"className":22278},[183],[80,22280,22282],{"className":22281,"style":1407},[187],[80,22283,22284,22287],{"style":772},[80,22285],{"className":22286,"style":196},[195],[80,22288,22290],{"className":22289},[200,201,202,203],[80,22291,19842],{"className":22292},[165,203]," for the full 100-house dataset with all 4 features. I also computed how many iterations would be needed, in the best case, just to reduce the error by 10x: ",[15,22295,22296],{},"around 67 million",". That's when I fully understood why \"lower alpha a bit and wait longer\" isn't a real solution here.",[294,22299,22301,22302],{"id":22300},"a-practical-recipe-for-picking-αalphaα","A practical recipe for picking ",[80,22303,22305,22318],{"className":22304},[83],[80,22306,22308],{"className":22307},[87],[89,22309,22310],{"xmlns":91},[93,22311,22312,22316],{},[96,22313,22314],{},[102,22315,10551],{},[145,22317,11037],{"encoding":147},[80,22319,22321],{"className":22320,"ariaHidden":113},[152],[80,22322,22324,22327],{"className":22323},[156],[80,22325],{"className":22326,"style":334},[160],[80,22328,10551],{"className":22329,"style":10667},[165,169],[11,22331,22332,22333,3774],{},"I won't always want to compute an eigenvalue. The recipe Andrew Ng teaches, and that I've already been using since ",[562,22334,22335],{"href":4070},"the gradient descent post",[299,22337,22338,22392,22398,22401],{},[302,22339,22340,22341,1618],{},"Start small, like ",[80,22342,22344,22362],{"className":22343},[83],[80,22345,22347],{"className":22346},[87],[89,22348,22349],{"xmlns":91},[93,22350,22351,22359],{},[96,22352,22353,22355,22357],{},[102,22354,10551],{},[111,22356,130],{},[1321,22358,13518],{},[145,22360,22361],{"encoding":147},"\\alpha = 0.001",[80,22363,22365,22383],{"className":22364,"ariaHidden":113},[152],[80,22366,22368,22371,22374,22377,22380],{"className":22367},[156],[80,22369],{"className":22370,"style":334},[160],[80,22372,10551],{"className":22373,"style":10667},[165,169],[80,22375],{"className":22376,"style":247},[246],[80,22378,130],{"className":22379},[251],[80,22381],{"className":22382,"style":247},[246],[80,22384,22386,22389],{"className":22385},[156],[80,22387],{"className":22388,"style":1614},[160],[80,22390,13518],{"className":22391},[165],[302,22393,22394,22395],{},"Multiply by roughly 3 each attempt: ",[65,22396,22397],{},"0.001, 0.003, 0.01, 0.03, 0.1...",[302,22399,22400],{},"Run a few iterations and look at the cost-vs-iteration chart.",[302,22402,22403,22404,22432],{},"Pick the largest ",[80,22405,22407,22420],{"className":22406},[83],[80,22408,22410],{"className":22409},[87],[89,22411,22412],{"xmlns":91},[93,22413,22414,22418],{},[96,22415,22416],{},[102,22417,10551],{},[145,22419,11037],{"encoding":147},[80,22421,22423],{"className":22422,"ariaHidden":113},[152],[80,22424,22426,22429],{"className":22425},[156],[80,22427],{"className":22428,"style":334},[160],[80,22430,10551],{"className":22431,"style":10667},[165,169]," that still gives a smooth, decreasing curve.",[521,22434,22435,22445],{},[524,22436,22437],{},[527,22438,22439,22442],{},[530,22440,22441],{"align":532},"What shows up on the chart",[530,22443,22444],{"align":532},"Diagnosis",[541,22446,22447,22489,22525,22561],{},[527,22448,22449,22458],{},[546,22450,22451,22452,20914,22455],{"align":532},"Cost climbs, or turns into ",[65,22453,22454],{},"nan",[65,22456,22457],{},"inf",[546,22459,22460,22488],{"align":532},[80,22461,22463,22476],{"className":22462},[83],[80,22464,22466],{"className":22465},[87],[89,22467,22468],{"xmlns":91},[93,22469,22470,22474],{},[96,22471,22472],{},[102,22473,10551],{},[145,22475,11037],{"encoding":147},[80,22477,22479],{"className":22478,"ariaHidden":113},[152],[80,22480,22482,22485],{"className":22481},[156],[80,22483],{"className":22484,"style":334},[160],[80,22486,10551],{"className":22487,"style":10667},[165,169]," too big",[527,22490,22491,22494],{},[546,22492,22493],{"align":532},"Cost drops but jagged, with spikes",[546,22495,22496,22524],{"align":532},[80,22497,22499,22512],{"className":22498},[83],[80,22500,22502],{"className":22501},[87],[89,22503,22504],{"xmlns":91},[93,22505,22506,22510],{},[96,22507,22508],{},[102,22509,10551],{},[145,22511,11037],{"encoding":147},[80,22513,22515],{"className":22514,"ariaHidden":113},[152],[80,22516,22518,22521],{"className":22517},[156],[80,22519],{"className":22520,"style":334},[160],[80,22522,10551],{"className":22523,"style":10667},[165,169]," at the limit, lower it",[527,22526,22527,22530],{},[546,22528,22529],{"align":532},"Cost drops in an almost straight, slow line",[546,22531,22532,22560],{"align":532},[80,22533,22535,22548],{"className":22534},[83],[80,22536,22538],{"className":22537},[87],[89,22539,22540],{"xmlns":91},[93,22541,22542,22546],{},[96,22543,22544],{},[102,22545,10551],{},[145,22547,11037],{"encoding":147},[80,22549,22551],{"className":22550,"ariaHidden":113},[152],[80,22552,22554,22557],{"className":22553},[156],[80,22555],{"className":22556,"style":334},[160],[80,22558,10551],{"className":22559,"style":10667},[165,169]," too small",[527,22562,22563,22566],{},[546,22564,22565],{"align":532},"Cost drops fast and then flattens",[546,22567,22568,22569,22597],{"align":532},"good ",[80,22570,22572,22585],{"className":22571},[83],[80,22573,22575],{"className":22574},[87],[89,22576,22577],{"xmlns":91},[93,22578,22579,22583],{},[96,22580,22581],{},[102,22582,10551],{},[145,22584,11037],{"encoding":147},[80,22586,22588],{"className":22587,"ariaHidden":113},[152],[80,22589,22591,22594],{"className":22590},[156],[80,22592],{"className":22593,"style":334},[160],[80,22595,10551],{"className":22596,"style":10667},[165,169],", converged",[11,22599,22600,22601,22629,22630,2338,22702,22774],{},"I ran this sweep on the full dataset and found the usable range of ",[80,22602,22604,22617],{"className":22603},[83],[80,22605,22607],{"className":22606},[87],[89,22608,22609],{"xmlns":91},[93,22610,22611,22615],{},[96,22612,22613],{},[102,22614,10551],{},[145,22616,11037],{"encoding":147},[80,22618,22620],{"className":22619,"ariaHidden":113},[152],[80,22621,22623,22626],{"className":22622},[156],[80,22624],{"className":22625,"style":334},[160],[80,22627,10551],{"className":22628,"style":10667},[165,169]," (with no scaling at all) sits between ",[80,22631,22633,22655],{"className":22632},[83],[80,22634,22636],{"className":22635},[87],[89,22637,22638],{"xmlns":91},[93,22639,22640,22652],{},[96,22641,22642],{},[726,22643,22644,22646],{},[1321,22645,17444],{},[96,22647,22648,22650],{},[111,22649,4643],{},[1321,22651,20127],{},[145,22653,22654],{"encoding":147},"10^{-8}",[80,22656,22658],{"className":22657,"ariaHidden":113},[152],[80,22659,22661,22664,22667],{"className":22660},[156],[80,22662],{"className":22663,"style":1407},[160],[80,22665,1583],{"className":22666},[165],[80,22668,22670,22673],{"className":22669},[165],[80,22671,2071],{"className":22672},[165],[80,22674,22676],{"className":22675},[174],[80,22677,22679],{"className":22678},[178],[80,22680,22682],{"className":22681},[183],[80,22683,22685],{"className":22684,"style":1407},[187],[80,22686,22687,22690],{"style":772},[80,22688],{"className":22689,"style":196},[195],[80,22691,22693],{"className":22692},[200,201,202,203],[80,22694,22696,22699],{"className":22695},[165,203],[80,22697,4643],{"className":22698},[165,203],[80,22700,20127],{"className":22701},[165,203],[80,22703,22705,22727],{"className":22704},[83],[80,22706,22708],{"className":22707},[87],[89,22709,22710],{"xmlns":91},[93,22711,22712,22724],{},[96,22713,22714],{},[726,22715,22716,22718],{},[1321,22717,17444],{},[96,22719,22720,22722],{},[111,22721,4643],{},[1321,22723,21494],{},[145,22725,22726],{"encoding":147},"10^{-6}",[80,22728,22730],{"className":22729,"ariaHidden":113},[152],[80,22731,22733,22736,22739],{"className":22732},[156],[80,22734],{"className":22735,"style":1407},[160],[80,22737,1583],{"className":22738},[165],[80,22740,22742,22745],{"className":22741},[165],[80,22743,2071],{"className":22744},[165],[80,22746,22748],{"className":22747},[174],[80,22749,22751],{"className":22750},[178],[80,22752,22754],{"className":22753},[183],[80,22755,22757],{"className":22756,"style":1407},[187],[80,22758,22759,22762],{"style":772},[80,22760],{"className":22761,"style":196},[195],[80,22763,22765],{"className":22764},[200,201,202,203],[80,22766,22768,22771],{"className":22767},[165,203],[80,22769,4643],{"className":22770},[165,203],[80,22772,21494],{"className":22773},[165,203],". An extremely narrow range, and one that depends entirely on the unit I used to measure size. If I'd measured in square meters instead of sqft, every one of these numbers would shift. That's fragile, and it's not how I want to train any model.",[294,22776,22778],{"id":22777},"the-scaling-techniques","The scaling techniques",[11,22780,22781,22782,22810],{},"The fix isn't hunting for a better ",[80,22783,22785,22798],{"className":22784},[83],[80,22786,22788],{"className":22787},[87],[89,22789,22790],{"xmlns":91},[93,22791,22792,22796],{},[96,22793,22794],{},[102,22795,10551],{},[145,22797,11037],{"encoding":147},[80,22799,22801],{"className":22800,"ariaHidden":113},[152],[80,22802,22804,22807],{"className":22803},[156],[80,22805],{"className":22806,"style":334},[160],[80,22808,10551],{"className":22809,"style":10667},[165,169],", it's fixing the scale of the features before running anything. There are a few ways to do that:",[521,22812,22813,22825],{},[524,22814,22815],{},[527,22816,22817,22820,22822],{},[530,22818,22819],{"align":532},"Technique",[530,22821,18146],{"align":532},[530,22823,22824],{"align":532},"Resulting range",[541,22826,22827,23096,23592,24062],{},[527,22828,22829,22832,23040],{},[546,22830,22831],{"align":532},"Divide by max",[546,22833,22834],{"align":532},[80,22835,22837,22880],{"className":22836},[83],[80,22838,22840],{"className":22839},[87],[89,22841,22842],{"xmlns":91},[93,22843,22844,22877],{},[96,22845,22846,22852,22855,22861,22863,22865,22867,22869,22875],{},[99,22847,22848,22850],{},[102,22849,124],{},[102,22851,18373],{},[111,22853,22854],{},"←",[99,22856,22857,22859],{},[102,22858,124],{},[102,22860,18373],{},[102,22862,20914],{"mathvariant":10558},[102,22864,20984],{},[111,22866,20987],{},[111,22868,121],{"stretchy":120},[99,22870,22871,22873],{},[102,22872,124],{},[102,22874,18373],{},[111,22876,127],{"stretchy":120},[145,22878,22879],{"encoding":147},"x_j \\leftarrow x_j \u002F \\max(x_j)",[80,22881,22883,22939],{"className":22882,"ariaHidden":113},[152],[80,22884,22886,22890,22930,22933,22936],{"className":22885},[156],[80,22887],{"className":22888,"style":22889},[160],"height:0.7167em;vertical-align:-0.2861em;",[80,22891,22893,22896],{"className":22892},[165],[80,22894,124],{"className":22895},[165,169],[80,22897,22899],{"className":22898},[174],[80,22900,22902,22922],{"className":22901},[178,179],[80,22903,22905,22919],{"className":22904},[183],[80,22906,22908],{"className":22907,"style":15323},[187],[80,22909,22910,22913],{"style":15326},[80,22911],{"className":22912,"style":196},[195],[80,22914,22916],{"className":22915},[200,201,202,203],[80,22917,18373],{"className":22918,"style":18420},[165,169,203],[80,22920,222],{"className":22921},[221],[80,22923,22925],{"className":22924},[183],[80,22926,22928],{"className":22927,"style":229},[187],[80,22929],{},[80,22931],{"className":22932,"style":247},[246],[80,22934,22854],{"className":22935},[251],[80,22937],{"className":22938,"style":247},[246],[80,22940,22942,22945,22985,22988,22991,22994,22997,23037],{"className":22941},[156],[80,22943],{"className":22944,"style":161},[160],[80,22946,22948,22951],{"className":22947},[165],[80,22949,124],{"className":22950},[165,169],[80,22952,22954],{"className":22953},[174],[80,22955,22957,22977],{"className":22956},[178,179],[80,22958,22960,22974],{"className":22959},[183],[80,22961,22963],{"className":22962,"style":15323},[187],[80,22964,22965,22968],{"style":15326},[80,22966],{"className":22967,"style":196},[195],[80,22969,22971],{"className":22970},[200,201,202,203],[80,22972,18373],{"className":22973,"style":18420},[165,169,203],[80,22975,222],{"className":22976},[221],[80,22978,22980],{"className":22979},[183],[80,22981,22983],{"className":22982,"style":229},[187],[80,22984],{},[80,22986,20914],{"className":22987},[165],[80,22989],{"className":22990,"style":268},[246],[80,22992,20984],{"className":22993},[7402],[80,22995,121],{"className":22996},[235],[80,22998,23000,23003],{"className":22999},[165],[80,23001,124],{"className":23002},[165,169],[80,23004,23006],{"className":23005},[174],[80,23007,23009,23029],{"className":23008},[178,179],[80,23010,23012,23026],{"className":23011},[183],[80,23013,23015],{"className":23014,"style":15323},[187],[80,23016,23017,23020],{"style":15326},[80,23018],{"className":23019,"style":196},[195],[80,23021,23023],{"className":23022},[200,201,202,203],[80,23024,18373],{"className":23025,"style":18420},[165,169,203],[80,23027,222],{"className":23028},[221],[80,23030,23032],{"className":23031},[183],[80,23033,23035],{"className":23034,"style":229},[187],[80,23036],{},[80,23038,127],{"className":23039},[242],[546,23041,23042],{"align":532},[80,23043,23045,23069],{"className":23044},[83],[80,23046,23048],{"className":23047},[87],[89,23049,23050],{"xmlns":91},[93,23051,23052,23066],{},[96,23053,23054,23057,23059,23061,23063],{},[111,23055,23056],{"stretchy":120},"[",[1321,23058,2071],{},[111,23060,114],{"separator":113},[1321,23062,1583],{},[111,23064,23065],{"stretchy":120},"]",[145,23067,23068],{"encoding":147},"[0, 1]",[80,23070,23072],{"className":23071,"ariaHidden":113},[152],[80,23073,23075,23078,23081,23084,23087,23090,23093],{"className":23074},[156],[80,23076],{"className":23077,"style":2316},[160],[80,23079,23056],{"className":23080},[235],[80,23082,2071],{"className":23083},[165],[80,23085,114],{"className":23086},[214],[80,23088],{"className":23089,"style":268},[246],[80,23091,1583],{"className":23092},[165],[80,23094,23065],{"className":23095},[242],[527,23097,23098,23101,23539],{},[546,23099,23100],{"align":532},"Min-max",[546,23102,23103],{"align":532},[80,23104,23106,23185],{"className":23105},[83],[80,23107,23109],{"className":23108},[87],[89,23110,23111],{"xmlns":91},[93,23112,23113,23182],{},[96,23114,23115,23121,23123],{},[99,23116,23117,23119],{},[102,23118,124],{},[102,23120,18373],{},[111,23122,22854],{},[4625,23124,23125,23150],{},[96,23126,23127,23133,23135,23138,23140,23142,23148],{},[99,23128,23129,23131],{},[102,23130,124],{},[102,23132,18373],{},[111,23134,4643],{},[102,23136,23137],{},"min",[111,23139,20987],{},[111,23141,121],{"stretchy":120},[99,23143,23144,23146],{},[102,23145,124],{},[102,23147,18373],{},[111,23149,127],{"stretchy":120},[96,23151,23152,23154,23156,23158,23164,23166,23168,23170,23172,23174,23180],{},[102,23153,20984],{},[111,23155,20987],{},[111,23157,121],{"stretchy":120},[99,23159,23160,23162],{},[102,23161,124],{},[102,23163,18373],{},[111,23165,127],{"stretchy":120},[111,23167,4643],{},[102,23169,23137],{},[111,23171,20987],{},[111,23173,121],{"stretchy":120},[99,23175,23176,23178],{},[102,23177,124],{},[102,23179,18373],{},[111,23181,127],{"stretchy":120},[145,23183,23184],{"encoding":147},"x_j \\leftarrow \\frac{x_j - \\min(x_j)}{\\max(x_j) - \\min(x_j)}",[80,23186,23188,23243],{"className":23187,"ariaHidden":113},[152],[80,23189,23191,23194,23234,23237,23240],{"className":23190},[156],[80,23192],{"className":23193,"style":22889},[160],[80,23195,23197,23200],{"className":23196},[165],[80,23198,124],{"className":23199},[165,169],[80,23201,23203],{"className":23202},[174],[80,23204,23206,23226],{"className":23205},[178,179],[80,23207,23209,23223],{"className":23208},[183],[80,23210,23212],{"className":23211,"style":15323},[187],[80,23213,23214,23217],{"style":15326},[80,23215],{"className":23216,"style":196},[195],[80,23218,23220],{"className":23219},[200,201,202,203],[80,23221,18373],{"className":23222,"style":18420},[165,169,203],[80,23224,222],{"className":23225},[221],[80,23227,23229],{"className":23228},[183],[80,23230,23232],{"className":23231,"style":229},[187],[80,23233],{},[80,23235],{"className":23236,"style":247},[246],[80,23238,22854],{"className":23239},[251],[80,23241],{"className":23242,"style":247},[246],[80,23244,23246,23250],{"className":23245},[156],[80,23247],{"className":23248,"style":23249},[160],"height:1.5746em;vertical-align:-0.5423em;",[80,23251,23253,23256,23536],{"className":23252},[165],[80,23254],{"className":23255},[235,4746],[80,23257,23259],{"className":23258},[4625],[80,23260,23262,23527],{"className":23261},[178,179],[80,23263,23265,23524],{"className":23264},[183],[80,23266,23269,23403,23411],{"className":23267,"style":23268},[187],"height:1.0323em;",[80,23270,23271,23274],{"style":5013},[80,23272],{"className":23273,"style":4766},[195],[80,23275,23277],{"className":23276},[200,201,202,203],[80,23278,23280,23292,23295,23339,23342,23345,23357,23360,23400],{"className":23279},[165,203],[80,23281,23283,23286,23289],{"className":23282},[7402,203],[80,23284,322],{"className":23285},[203],[80,23287,562],{"className":23288},[203],[80,23290,124],{"className":23291},[203],[80,23293,121],{"className":23294},[235,203],[80,23296,23298,23301],{"className":23297},[165,203],[80,23299,124],{"className":23300},[165,169,203],[80,23302,23304],{"className":23303},[174],[80,23305,23307,23330],{"className":23306},[178,179],[80,23308,23310,23327],{"className":23309},[183],[80,23311,23314],{"className":23312,"style":23313},[187],"height:0.3281em;",[80,23315,23317,23321],{"style":23316},"top:-2.357em;margin-left:0em;margin-right:0.0714em;",[80,23318],{"className":23319,"style":23320},[195],"height:2.5em;",[80,23322,23324],{"className":23323},[200,4802,4803,203],[80,23325,18373],{"className":23326,"style":18420},[165,169,203],[80,23328,222],{"className":23329},[221],[80,23331,23333],{"className":23332},[183],[80,23334,23337],{"className":23335,"style":23336},[187],"height:0.2819em;",[80,23338],{},[80,23340,127],{"className":23341},[242,203],[80,23343,4643],{"className":23344},[279,203],[80,23346,23348,23351,23354],{"className":23347},[7402,203],[80,23349,322],{"className":23350},[203],[80,23352,736],{"className":23353},[203],[80,23355,1487],{"className":23356},[203],[80,23358,121],{"className":23359},[235,203],[80,23361,23363,23366],{"className":23362},[165,203],[80,23364,124],{"className":23365},[165,169,203],[80,23367,23369],{"className":23368},[174],[80,23370,23372,23392],{"className":23371},[178,179],[80,23373,23375,23389],{"className":23374},[183],[80,23376,23378],{"className":23377,"style":23313},[187],[80,23379,23380,23383],{"style":23316},[80,23381],{"className":23382,"style":23320},[195],[80,23384,23386],{"className":23385},[200,4802,4803,203],[80,23387,18373],{"className":23388,"style":18420},[165,169,203],[80,23390,222],{"className":23391},[221],[80,23393,23395],{"className":23394},[183],[80,23396,23398],{"className":23397,"style":23336},[187],[80,23399],{},[80,23401,127],{"className":23402},[242,203],[80,23404,23405,23408],{"style":4859},[80,23406],{"className":23407,"style":4766},[195],[80,23409],{"className":23410,"style":4867},[4866],[80,23412,23414,23417],{"style":23413},"top:-3.5073em;",[80,23415],{"className":23416,"style":4766},[195],[80,23418,23420],{"className":23419},[200,201,202,203],[80,23421,23423,23463,23466,23478,23481,23521],{"className":23422},[165,203],[80,23424,23426,23429],{"className":23425},[165,203],[80,23427,124],{"className":23428},[165,169,203],[80,23430,23432],{"className":23431},[174],[80,23433,23435,23455],{"className":23434},[178,179],[80,23436,23438,23452],{"className":23437},[183],[80,23439,23441],{"className":23440,"style":23313},[187],[80,23442,23443,23446],{"style":23316},[80,23444],{"className":23445,"style":23320},[195],[80,23447,23449],{"className":23448},[200,4802,4803,203],[80,23450,18373],{"className":23451,"style":18420},[165,169,203],[80,23453,222],{"className":23454},[221],[80,23456,23458],{"className":23457},[183],[80,23459,23461],{"className":23460,"style":23336},[187],[80,23462],{},[80,23464,4643],{"className":23465},[279,203],[80,23467,23469,23472,23475],{"className":23468},[7402,203],[80,23470,322],{"className":23471},[203],[80,23473,736],{"className":23474},[203],[80,23476,1487],{"className":23477},[203],[80,23479,121],{"className":23480},[235,203],[80,23482,23484,23487],{"className":23483},[165,203],[80,23485,124],{"className":23486},[165,169,203],[80,23488,23490],{"className":23489},[174],[80,23491,23493,23513],{"className":23492},[178,179],[80,23494,23496,23510],{"className":23495},[183],[80,23497,23499],{"className":23498,"style":23313},[187],[80,23500,23501,23504],{"style":23316},[80,23502],{"className":23503,"style":23320},[195],[80,23505,23507],{"className":23506},[200,4802,4803,203],[80,23508,18373],{"className":23509,"style":18420},[165,169,203],[80,23511,222],{"className":23512},[221],[80,23514,23516],{"className":23515},[183],[80,23517,23519],{"className":23518,"style":23336},[187],[80,23520],{},[80,23522,127],{"className":23523},[242,203],[80,23525,222],{"className":23526},[221],[80,23528,23530],{"className":23529},[183],[80,23531,23534],{"className":23532,"style":23533},[187],"height:0.5423em;",[80,23535],{},[80,23537],{"className":23538},[242,4746],[546,23540,23541],{"align":532},[80,23542,23544,23565],{"className":23543},[83],[80,23545,23547],{"className":23546},[87],[89,23548,23549],{"xmlns":91},[93,23550,23551,23563],{},[96,23552,23553,23555,23557,23559,23561],{},[111,23554,23056],{"stretchy":120},[1321,23556,2071],{},[111,23558,114],{"separator":113},[1321,23560,1583],{},[111,23562,23065],{"stretchy":120},[145,23564,23068],{"encoding":147},[80,23566,23568],{"className":23567,"ariaHidden":113},[152],[80,23569,23571,23574,23577,23580,23583,23586,23589],{"className":23570},[156],[80,23572],{"className":23573,"style":2316},[160],[80,23575,23056],{"className":23576},[235],[80,23578,2071],{"className":23579},[165],[80,23581,114],{"className":23582},[214],[80,23584],{"className":23585,"style":268},[246],[80,23587,1583],{"className":23588},[165],[80,23590,23065],{"className":23591},[242],[527,23593,23594,23597,24002],{},[546,23595,23596],{"align":532},"Mean normalization",[546,23598,23599],{"align":532},[80,23600,23602,23672],{"className":23601},[83],[80,23603,23605],{"className":23604},[87],[89,23606,23607],{"xmlns":91},[93,23608,23609,23669],{},[96,23610,23611,23617,23619],{},[99,23612,23613,23615],{},[102,23614,124],{},[102,23616,18373],{},[111,23618,22854],{},[4625,23620,23621,23637],{},[96,23622,23623,23629,23631],{},[99,23624,23625,23627],{},[102,23626,124],{},[102,23628,18373],{},[111,23630,4643],{},[99,23632,23633,23635],{},[102,23634,22115],{},[102,23636,18373],{},[96,23638,23639,23641,23643,23645,23651,23653,23655,23657,23659,23661,23667],{},[102,23640,20984],{},[111,23642,20987],{},[111,23644,121],{"stretchy":120},[99,23646,23647,23649],{},[102,23648,124],{},[102,23650,18373],{},[111,23652,127],{"stretchy":120},[111,23654,4643],{},[102,23656,23137],{},[111,23658,20987],{},[111,23660,121],{"stretchy":120},[99,23662,23663,23665],{},[102,23664,124],{},[102,23666,18373],{},[111,23668,127],{"stretchy":120},[145,23670,23671],{"encoding":147},"x_j \\leftarrow \\frac{x_j - \\mu_j}{\\max(x_j) - \\min(x_j)}",[80,23673,23675,23730],{"className":23674,"ariaHidden":113},[152],[80,23676,23678,23681,23721,23724,23727],{"className":23677},[156],[80,23679],{"className":23680,"style":22889},[160],[80,23682,23684,23687],{"className":23683},[165],[80,23685,124],{"className":23686},[165,169],[80,23688,23690],{"className":23689},[174],[80,23691,23693,23713],{"className":23692},[178,179],[80,23694,23696,23710],{"className":23695},[183],[80,23697,23699],{"className":23698,"style":15323},[187],[80,23700,23701,23704],{"style":15326},[80,23702],{"className":23703,"style":196},[195],[80,23705,23707],{"className":23706},[200,201,202,203],[80,23708,18373],{"className":23709,"style":18420},[165,169,203],[80,23711,222],{"className":23712},[221],[80,23714,23716],{"className":23715},[183],[80,23717,23719],{"className":23718,"style":229},[187],[80,23720],{},[80,23722],{"className":23723,"style":247},[246],[80,23725,22854],{"className":23726},[251],[80,23728],{"className":23729,"style":247},[246],[80,23731,23733,23737],{"className":23732},[156],[80,23734],{"className":23735,"style":23736},[160],"height:1.458em;vertical-align:-0.5423em;",[80,23738,23740,23743,23999],{"className":23739},[165],[80,23741],{"className":23742},[235,4746],[80,23744,23746],{"className":23745},[4625],[80,23747,23749,23991],{"className":23748},[178,179],[80,23750,23752,23988],{"className":23751},[183],[80,23753,23756,23886,23894],{"className":23754,"style":23755},[187],"height:0.9157em;",[80,23757,23758,23761],{"style":5013},[80,23759],{"className":23760,"style":4766},[195],[80,23762,23764],{"className":23763},[200,201,202,203],[80,23765,23767,23779,23782,23822,23825,23828,23840,23843,23883],{"className":23766},[165,203],[80,23768,23770,23773,23776],{"className":23769},[7402,203],[80,23771,322],{"className":23772},[203],[80,23774,562],{"className":23775},[203],[80,23777,124],{"className":23778},[203],[80,23780,121],{"className":23781},[235,203],[80,23783,23785,23788],{"className":23784},[165,203],[80,23786,124],{"className":23787},[165,169,203],[80,23789,23791],{"className":23790},[174],[80,23792,23794,23814],{"className":23793},[178,179],[80,23795,23797,23811],{"className":23796},[183],[80,23798,23800],{"className":23799,"style":23313},[187],[80,23801,23802,23805],{"style":23316},[80,23803],{"className":23804,"style":23320},[195],[80,23806,23808],{"className":23807},[200,4802,4803,203],[80,23809,18373],{"className":23810,"style":18420},[165,169,203],[80,23812,222],{"className":23813},[221],[80,23815,23817],{"className":23816},[183],[80,23818,23820],{"className":23819,"style":23336},[187],[80,23821],{},[80,23823,127],{"className":23824},[242,203],[80,23826,4643],{"className":23827},[279,203],[80,23829,23831,23834,23837],{"className":23830},[7402,203],[80,23832,322],{"className":23833},[203],[80,23835,736],{"className":23836},[203],[80,23838,1487],{"className":23839},[203],[80,23841,121],{"className":23842},[235,203],[80,23844,23846,23849],{"className":23845},[165,203],[80,23847,124],{"className":23848},[165,169,203],[80,23850,23852],{"className":23851},[174],[80,23853,23855,23875],{"className":23854},[178,179],[80,23856,23858,23872],{"className":23857},[183],[80,23859,23861],{"className":23860,"style":23313},[187],[80,23862,23863,23866],{"style":23316},[80,23864],{"className":23865,"style":23320},[195],[80,23867,23869],{"className":23868},[200,4802,4803,203],[80,23870,18373],{"className":23871,"style":18420},[165,169,203],[80,23873,222],{"className":23874},[221],[80,23876,23878],{"className":23877},[183],[80,23879,23881],{"className":23880,"style":23336},[187],[80,23882],{},[80,23884,127],{"className":23885},[242,203],[80,23887,23888,23891],{"style":4859},[80,23889],{"className":23890,"style":4766},[195],[80,23892],{"className":23893,"style":4867},[4866],[80,23895,23896,23899],{"style":23413},[80,23897],{"className":23898,"style":4766},[195],[80,23900,23902],{"className":23901},[200,201,202,203],[80,23903,23905,23945,23948],{"className":23904},[165,203],[80,23906,23908,23911],{"className":23907},[165,203],[80,23909,124],{"className":23910},[165,169,203],[80,23912,23914],{"className":23913},[174],[80,23915,23917,23937],{"className":23916},[178,179],[80,23918,23920,23934],{"className":23919},[183],[80,23921,23923],{"className":23922,"style":23313},[187],[80,23924,23925,23928],{"style":23316},[80,23926],{"className":23927,"style":23320},[195],[80,23929,23931],{"className":23930},[200,4802,4803,203],[80,23932,18373],{"className":23933,"style":18420},[165,169,203],[80,23935,222],{"className":23936},[221],[80,23938,23940],{"className":23939},[183],[80,23941,23943],{"className":23942,"style":23336},[187],[80,23944],{},[80,23946,4643],{"className":23947},[279,203],[80,23949,23951,23954],{"className":23950},[165,203],[80,23952,22115],{"className":23953},[165,169,203],[80,23955,23957],{"className":23956},[174],[80,23958,23960,23980],{"className":23959},[178,179],[80,23961,23963,23977],{"className":23962},[183],[80,23964,23966],{"className":23965,"style":23313},[187],[80,23967,23968,23971],{"style":23316},[80,23969],{"className":23970,"style":23320},[195],[80,23972,23974],{"className":23973},[200,4802,4803,203],[80,23975,18373],{"className":23976,"style":18420},[165,169,203],[80,23978,222],{"className":23979},[221],[80,23981,23983],{"className":23982},[183],[80,23984,23986],{"className":23985,"style":23336},[187],[80,23987],{},[80,23989,222],{"className":23990},[221],[80,23992,23994],{"className":23993},[183],[80,23995,23997],{"className":23996,"style":23533},[187],[80,23998],{},[80,24000],{"className":24001},[242,4746],[546,24003,24004,24061],{"align":532},[80,24005,24007,24031],{"className":24006},[83],[80,24008,24010],{"className":24009},[87],[89,24011,24012],{"xmlns":91},[93,24013,24014,24028],{},[96,24015,24016,24018,24020,24022,24024,24026],{},[111,24017,23056],{"stretchy":120},[111,24019,4643],{},[1321,24021,1583],{},[111,24023,114],{"separator":113},[1321,24025,1583],{},[111,24027,23065],{"stretchy":120},[145,24029,24030],{"encoding":147},"[-1, 1]",[80,24032,24034],{"className":24033,"ariaHidden":113},[152],[80,24035,24037,24040,24043,24046,24049,24052,24055,24058],{"className":24036},[156],[80,24038],{"className":24039,"style":2316},[160],[80,24041,23056],{"className":24042},[235],[80,24044,4643],{"className":24045},[165],[80,24047,1583],{"className":24048},[165],[80,24050,114],{"className":24051},[214],[80,24053],{"className":24054,"style":268},[246],[80,24056,1583],{"className":24057},[165],[80,24059,23065],{"className":24060},[242],", mean 0",[527,24063,24064,24067,24367],{},[546,24065,24066],{"align":532},"Z-score",[546,24068,24069],{"align":532},[80,24070,24072,24117],{"className":24071},[83],[80,24073,24075],{"className":24074},[87],[89,24076,24077],{"xmlns":91},[93,24078,24079,24114],{},[96,24080,24081,24087,24089],{},[99,24082,24083,24085],{},[102,24084,124],{},[102,24086,18373],{},[111,24088,22854],{},[4625,24090,24091,24107],{},[96,24092,24093,24099,24101],{},[99,24094,24095,24097],{},[102,24096,124],{},[102,24098,18373],{},[111,24100,4643],{},[99,24102,24103,24105],{},[102,24104,22115],{},[102,24106,18373],{},[99,24108,24109,24112],{},[102,24110,24111],{},"σ",[102,24113,18373],{},[145,24115,24116],{"encoding":147},"x_j \\leftarrow \\frac{x_j - \\mu_j}{\\sigma_j}",[80,24118,24120,24175],{"className":24119,"ariaHidden":113},[152],[80,24121,24123,24126,24166,24169,24172],{"className":24122},[156],[80,24124],{"className":24125,"style":22889},[160],[80,24127,24129,24132],{"className":24128},[165],[80,24130,124],{"className":24131},[165,169],[80,24133,24135],{"className":24134},[174],[80,24136,24138,24158],{"className":24137},[178,179],[80,24139,24141,24155],{"className":24140},[183],[80,24142,24144],{"className":24143,"style":15323},[187],[80,24145,24146,24149],{"style":15326},[80,24147],{"className":24148,"style":196},[195],[80,24150,24152],{"className":24151},[200,201,202,203],[80,24153,18373],{"className":24154,"style":18420},[165,169,203],[80,24156,222],{"className":24157},[221],[80,24159,24161],{"className":24160},[183],[80,24162,24164],{"className":24163,"style":229},[187],[80,24165],{},[80,24167],{"className":24168,"style":247},[246],[80,24170,22854],{"className":24171},[251],[80,24173],{"className":24174,"style":247},[246],[80,24176,24178,24181],{"className":24177},[156],[80,24179],{"className":24180,"style":23736},[160],[80,24182,24184,24187,24364],{"className":24183},[165],[80,24185],{"className":24186},[235,4746],[80,24188,24190],{"className":24189},[4625],[80,24191,24193,24356],{"className":24192},[178,179],[80,24194,24196,24353],{"className":24195},[183],[80,24197,24199,24251,24259],{"className":24198,"style":23755},[187],[80,24200,24201,24204],{"style":5013},[80,24202],{"className":24203,"style":4766},[195],[80,24205,24207],{"className":24206},[200,201,202,203],[80,24208,24210],{"className":24209},[165,203],[80,24211,24213,24216],{"className":24212},[165,203],[80,24214,24111],{"className":24215,"style":834},[165,169,203],[80,24217,24219],{"className":24218},[174],[80,24220,24222,24243],{"className":24221},[178,179],[80,24223,24225,24240],{"className":24224},[183],[80,24226,24228],{"className":24227,"style":23313},[187],[80,24229,24231,24234],{"style":24230},"top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em;",[80,24232],{"className":24233,"style":23320},[195],[80,24235,24237],{"className":24236},[200,4802,4803,203],[80,24238,18373],{"className":24239,"style":18420},[165,169,203],[80,24241,222],{"className":24242},[221],[80,24244,24246],{"className":24245},[183],[80,24247,24249],{"className":24248,"style":23336},[187],[80,24250],{},[80,24252,24253,24256],{"style":4859},[80,24254],{"className":24255,"style":4766},[195],[80,24257],{"className":24258,"style":4867},[4866],[80,24260,24261,24264],{"style":23413},[80,24262],{"className":24263,"style":4766},[195],[80,24265,24267],{"className":24266},[200,201,202,203],[80,24268,24270,24310,24313],{"className":24269},[165,203],[80,24271,24273,24276],{"className":24272},[165,203],[80,24274,124],{"className":24275},[165,169,203],[80,24277,24279],{"className":24278},[174],[80,24280,24282,24302],{"className":24281},[178,179],[80,24283,24285,24299],{"className":24284},[183],[80,24286,24288],{"className":24287,"style":23313},[187],[80,24289,24290,24293],{"style":23316},[80,24291],{"className":24292,"style":23320},[195],[80,24294,24296],{"className":24295},[200,4802,4803,203],[80,24297,18373],{"className":24298,"style":18420},[165,169,203],[80,24300,222],{"className":24301},[221],[80,24303,24305],{"className":24304},[183],[80,24306,24308],{"className":24307,"style":23336},[187],[80,24309],{},[80,24311,4643],{"className":24312},[279,203],[80,24314,24316,24319],{"className":24315},[165,203],[80,24317,22115],{"className":24318},[165,169,203],[80,24320,24322],{"className":24321},[174],[80,24323,24325,24345],{"className":24324},[178,179],[80,24326,24328,24342],{"className":24327},[183],[80,24329,24331],{"className":24330,"style":23313},[187],[80,24332,24333,24336],{"style":23316},[80,24334],{"className":24335,"style":23320},[195],[80,24337,24339],{"className":24338},[200,4802,4803,203],[80,24340,18373],{"className":24341,"style":18420},[165,169,203],[80,24343,222],{"className":24344},[221],[80,24346,24348],{"className":24347},[183],[80,24349,24351],{"className":24350,"style":23336},[187],[80,24352],{},[80,24354,222],{"className":24355},[221],[80,24357,24359],{"className":24358},[183],[80,24360,24362],{"className":24361,"style":23533},[187],[80,24363],{},[80,24365],{"className":24366},[242,4746],[546,24368,24369],{"align":532},"mean 0, std 1",[11,24371,24372,24373,2338,24443,24514,24515,24544,24545,24548],{},"I used z-score from here on, it's the most robust option (it doesn't depend on just two extreme points in the dataset, unlike min-max does). ",[80,24374,24376,24394],{"className":24375},[83],[80,24377,24379],{"className":24378},[87],[89,24380,24381],{"xmlns":91},[93,24382,24383,24391],{},[96,24384,24385],{},[99,24386,24387,24389],{},[102,24388,22115],{},[102,24390,18373],{},[145,24392,24393],{"encoding":147},"\\mu_j",[80,24395,24397],{"className":24396,"ariaHidden":113},[152],[80,24398,24400,24403],{"className":24399},[156],[80,24401],{"className":24402,"style":22889},[160],[80,24404,24406,24409],{"className":24405},[165],[80,24407,22115],{"className":24408},[165,169],[80,24410,24412],{"className":24411},[174],[80,24413,24415,24435],{"className":24414},[178,179],[80,24416,24418,24432],{"className":24417},[183],[80,24419,24421],{"className":24420,"style":15323},[187],[80,24422,24423,24426],{"style":15326},[80,24424],{"className":24425,"style":196},[195],[80,24427,24429],{"className":24428},[200,201,202,203],[80,24430,18373],{"className":24431,"style":18420},[165,169,203],[80,24433,222],{"className":24434},[221],[80,24436,24438],{"className":24437},[183],[80,24439,24441],{"className":24440,"style":229},[187],[80,24442],{},[80,24444,24446,24464],{"className":24445},[83],[80,24447,24449],{"className":24448},[87],[89,24450,24451],{"xmlns":91},[93,24452,24453,24461],{},[96,24454,24455],{},[99,24456,24457,24459],{},[102,24458,24111],{},[102,24460,18373],{},[145,24462,24463],{"encoding":147},"\\sigma_j",[80,24465,24467],{"className":24466,"ariaHidden":113},[152],[80,24468,24470,24473],{"className":24469},[156],[80,24471],{"className":24472,"style":22889},[160],[80,24474,24476,24479],{"className":24475},[165],[80,24477,24111],{"className":24478,"style":834},[165,169],[80,24480,24482],{"className":24481},[174],[80,24483,24485,24506],{"className":24484},[178,179],[80,24486,24488,24503],{"className":24487},[183],[80,24489,24491],{"className":24490,"style":15323},[187],[80,24492,24494,24497],{"style":24493},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[80,24495],{"className":24496,"style":196},[195],[80,24498,24500],{"className":24499},[200,201,202,203],[80,24501,18373],{"className":24502,"style":18420},[165,169,203],[80,24504,222],{"className":24505},[221],[80,24507,24509],{"className":24508},[183],[80,24510,24512],{"className":24511,"style":229},[187],[80,24513],{}," are the mean and standard deviation of feature ",[80,24516,24518,24531],{"className":24517},[83],[80,24519,24521],{"className":24520},[87],[89,24522,24523],{"xmlns":91},[93,24524,24525,24529],{},[96,24526,24527],{},[102,24528,18373],{},[145,24530,18373],{"encoding":147},[80,24532,24534],{"className":24533,"ariaHidden":113},[152],[80,24535,24537,24541],{"className":24536},[156],[80,24538],{"className":24539,"style":24540},[160],"height:0.854em;vertical-align:-0.1944em;",[80,24542,18373],{"className":24543,"style":18420},[165,169],", computed ",[15,24546,24547],{},"only from the training data",". That matters: I store those two numbers and reuse them for any new data that shows up later, I never recompute them.",[2611,24550,24552],{"className":2613,"code":24551,"language":2615,"meta":26,"style":26},"def zscore_normalize_features(X):\n    mu = X.mean(axis=0)      # mean of each column\n    sigma = X.std(axis=0)    # std of each column\n    X_norm = (X - mu) \u002F sigma\n    return X_norm, mu, sigma\n",[65,24553,24554,24559,24564,24569,24574],{"__ignoreMap":26},[80,24555,24556],{"class":2620,"line":33},[80,24557,24558],{},"def zscore_normalize_features(X):\n",[80,24560,24561],{"class":2620,"line":27},[80,24562,24563],{},"    mu = X.mean(axis=0)      # mean of each column\n",[80,24565,24566],{"class":2620,"line":2631},[80,24567,24568],{},"    sigma = X.std(axis=0)    # std of each column\n",[80,24570,24571],{"class":2620,"line":2636},[80,24572,24573],{},"    X_norm = (X - mu) \u002F sigma\n",[80,24575,24576],{"class":2620,"line":2642},[80,24577,24578],{},"    return X_norm, mu, sigma\n",[11,24580,24581],{},"For our 8-house example, size's mean is 1480.1 sqft and its std is 414.7. I normalized it and the range, which used to span hundreds to thousands, turned into something between roughly -1 and 2.",[11,24583,24584,24585,24587],{},"One detail I almost slid past: z-score does ",[15,24586,1294],{}," make the data normally distributed, it only rescales. The shape of the distribution stays exactly the same, only the center and scale change.",[294,24589,24591],{"id":24590},"gradient-descent-with-normalized-features","Gradient descent with normalized features",[11,24593,24594,24595,24646,24647,24719],{},"I ran the same algorithm, on the same 8-house dataset, but with size now z-score normalized, using ",[80,24596,24598,24616],{"className":24597},[83],[80,24599,24601],{"className":24600},[87],[89,24602,24603],{"xmlns":91},[93,24604,24605,24613],{},[96,24606,24607,24609,24611],{},[102,24608,10551],{},[111,24610,130],{},[1321,24612,13534],{},[145,24614,24615],{"encoding":147},"\\alpha = 0.1",[80,24617,24619,24637],{"className":24618,"ariaHidden":113},[152],[80,24620,24622,24625,24628,24631,24634],{"className":24621},[156],[80,24623],{"className":24624,"style":334},[160],[80,24626,10551],{"className":24627,"style":10667},[165,169],[80,24629],{"className":24630,"style":247},[246],[80,24632,130],{"className":24633},[251],[80,24635],{"className":24636,"style":247},[246],[80,24638,24640,24643],{"className":24639},[156],[80,24641],{"className":24642,"style":1614},[160],[80,24644,13534],{"className":24645},[165],", six orders of magnitude bigger than the ",[80,24648,24650,24672],{"className":24649},[83],[80,24651,24653],{"className":24652},[87],[89,24654,24655],{"xmlns":91},[93,24656,24657,24669],{},[96,24658,24659],{},[726,24660,24661,24663],{},[1321,24662,17444],{},[96,24664,24665,24667],{},[111,24666,4643],{},[1321,24668,19842],{},[145,24670,24671],{"encoding":147},"10^{-7}",[80,24673,24675],{"className":24674,"ariaHidden":113},[152],[80,24676,24678,24681,24684],{"className":24677},[156],[80,24679],{"className":24680,"style":1407},[160],[80,24682,1583],{"className":24683},[165],[80,24685,24687,24690],{"className":24686},[165],[80,24688,2071],{"className":24689},[165],[80,24691,24693],{"className":24692},[174],[80,24694,24696],{"className":24695},[178],[80,24697,24699],{"className":24698},[183],[80,24700,24702],{"className":24701,"style":1407},[187],[80,24703,24704,24707],{"style":772},[80,24705],{"className":24706,"style":196},[195],[80,24708,24710],{"className":24709},[200,201,202,203],[80,24711,24713,24716],{"className":24712},[165,203],[80,24714,4643],{"className":24715},[165,203],[80,24717,19842],{"className":24718},[165,203]," values from before:",[13355,24721],{":b-range":3351,":initial-b":2071,":initial-w":2071,":w-range":24722,":x-train":24723,":y-train":20697,"b-label":117,"w-label":24724,":initial-alpha":13534},"[-50, 200]","[-1.273, -0.569, 1.125, 0.591, 1.155, -0.400, -1.486, 0.858]","w (normalized)",[11,24726,24727,24728,24780,24781,24810,24811,24905,24906,24989],{},"After about 50 iterations the cost already flattened near the minimum (I got ",[80,24729,24731,24750],{"className":24730},[83],[80,24732,24734],{"className":24733},[87],[89,24735,24736],{"xmlns":91},[93,24737,24738,24747],{},[96,24739,24740,24742,24744],{},[102,24741,5606],{},[111,24743,9527],{},[1321,24745,24746],{},"919",[145,24748,24749],{"encoding":147},"J \\approx 919",[80,24751,24753,24771],{"className":24752,"ariaHidden":113},[152],[80,24754,24756,24759,24762,24765,24768],{"className":24755},[156],[80,24757],{"className":24758,"style":8672},[160],[80,24760,5606],{"className":24761,"style":5632},[165,169],[80,24763],{"className":24764,"style":247},[246],[80,24766,9527],{"className":24767},[251],[80,24769],{"className":24770,"style":247},[246],[80,24772,24774,24777],{"className":24773},[156],[80,24775],{"className":24776,"style":1614},[160],[80,24778,24746],{"className":24779},[165],", almost identical to the real minimum of ",[80,24782,24784,24798],{"className":24783},[83],[80,24785,24787],{"className":24786},[87],[89,24788,24789],{"xmlns":91},[93,24790,24791,24796],{},[96,24792,24793],{},[1321,24794,24795],{},"919.37",[145,24797,24795],{"encoding":147},[80,24799,24801],{"className":24800,"ariaHidden":113},[152],[80,24802,24804,24807],{"className":24803},[156],[80,24805],{"className":24806,"style":1614},[160],[80,24808,24795],{"className":24809},[165]," I computed directly through a closed-form solution). In the raw example, with ",[80,24812,24814,24840],{"className":24813},[83],[80,24815,24817],{"className":24816},[87],[89,24818,24819],{"xmlns":91},[93,24820,24821,24837],{},[96,24822,24823,24825,24827],{},[102,24824,10551],{},[111,24826,130],{},[726,24828,24829,24831],{},[1321,24830,17444],{},[96,24832,24833,24835],{},[111,24834,4643],{},[1321,24836,19842],{},[145,24838,24839],{"encoding":147},"\\alpha = 10^{-7}",[80,24841,24843,24861],{"className":24842,"ariaHidden":113},[152],[80,24844,24846,24849,24852,24855,24858],{"className":24845},[156],[80,24847],{"className":24848,"style":334},[160],[80,24850,10551],{"className":24851,"style":10667},[165,169],[80,24853],{"className":24854,"style":247},[246],[80,24856,130],{"className":24857},[251],[80,24859],{"className":24860,"style":247},[246],[80,24862,24864,24867,24870],{"className":24863},[156],[80,24865],{"className":24866,"style":1407},[160],[80,24868,1583],{"className":24869},[165],[80,24871,24873,24876],{"className":24872},[165],[80,24874,2071],{"className":24875},[165],[80,24877,24879],{"className":24878},[174],[80,24880,24882],{"className":24881},[178],[80,24883,24885],{"className":24884},[183],[80,24886,24888],{"className":24887,"style":1407},[187],[80,24889,24890,24893],{"style":772},[80,24891],{"className":24892,"style":196},[195],[80,24894,24896],{"className":24895},[200,201,202,203],[80,24897,24899,24902],{"className":24898},[165,203],[80,24900,4643],{"className":24901},[165,203],[80,24903,19842],{"className":24904},[165,203],", cost had barely moved off ",[80,24907,24909,24930],{"className":24908},[83],[80,24910,24912],{"className":24911},[87],[89,24913,24914],{"xmlns":91},[93,24915,24916,24928],{},[96,24917,24918,24920,24922],{},[1321,24919,20615],{},[111,24921,2834],{},[726,24923,24924,24926],{},[1321,24925,17444],{},[1321,24927,13895],{},[145,24929,20626],{"encoding":147},[80,24931,24933,24951],{"className":24932,"ariaHidden":113},[152],[80,24934,24936,24939,24942,24945,24948],{"className":24935},[156],[80,24937],{"className":24938,"style":5303},[160],[80,24940,20615],{"className":24941},[165],[80,24943],{"className":24944,"style":275},[246],[80,24946,2834],{"className":24947},[279],[80,24949],{"className":24950,"style":275},[246],[80,24952,24954,24957,24960],{"className":24953},[156],[80,24955],{"className":24956,"style":1407},[160],[80,24958,1583],{"className":24959},[165],[80,24961,24963,24966],{"className":24962},[165],[80,24964,2071],{"className":24965},[165],[80,24967,24969],{"className":24968},[174],[80,24970,24972],{"className":24971},[178],[80,24973,24975],{"className":24974},[183],[80,24976,24978],{"className":24977,"style":1407},[187],[80,24979,24980,24983],{"style":772},[80,24981],{"className":24982,"style":196},[195],[80,24984,24986],{"className":24985},[200,201,202,203],[80,24987,13895],{"className":24988},[165,203]," after 10 whole iterations. Normalizing didn't change where the minimum is, it completely changed how fast I get there.",[11,24991,24992,24993,25021,25022,25050,25051,25053,25054,25056],{},"I also noticed something small but satisfying to understand: when ",[80,24994,24996,25009],{"className":24995},[83],[80,24997,24999],{"className":24998},[87],[89,25000,25001],{"xmlns":91},[93,25002,25003,25007],{},[96,25004,25005],{},[102,25006,124],{},[145,25008,124],{"encoding":147},[80,25010,25012],{"className":25011,"ariaHidden":113},[152],[80,25013,25015,25018],{"className":25014},[156],[80,25016],{"className":25017,"style":334},[160],[80,25019,124],{"className":25020},[165,169]," is centered at zero (mean zero, thanks to z-score), the optimal ",[80,25023,25025,25038],{"className":25024},[83],[80,25026,25028],{"className":25027},[87],[89,25029,25030],{"xmlns":91},[93,25031,25032,25036],{},[96,25033,25034],{},[102,25035,117],{},[145,25037,117],{"encoding":147},[80,25039,25041],{"className":25040,"ariaHidden":113},[152],[80,25042,25044,25047],{"className":25043},[156],[80,25045],{"className":25046,"style":289},[160],[80,25048,117],{"className":25049},[165,169]," turns out to be exactly the mean of the training prices. That makes sense geometrically: if every ",[65,25052,124],{}," is zero on average, the best line passes through the average ",[65,25055,683],{}," at that point.",[294,25058,25060],{"id":25059},"predicting-a-new-house-and-denormalizing","Predicting a new house and denormalizing",[11,25062,25063],{},"I fit the model with all 4 real features (not just size) on the full 100-house dataset, using the normal equation (the exact closed-form solution, no gradient descent):",[2611,25065,25067],{"className":2613,"code":25066,"language":2615,"meta":26,"style":26},"w = [0.268, -32.90, -67.29, -1.465]   # size, bedrooms, floors, age\nb = 221.50\n",[65,25068,25069,25074],{"__ignoreMap":26},[80,25070,25071],{"class":2620,"line":33},[80,25072,25073],{},"w = [0.268, -32.90, -67.29, -1.465]   # size, bedrooms, floors, age\n",[80,25075,25076],{"class":2620,"line":27},[80,25077,25078],{},"b = 221.50\n",[11,25080,25081,25082,1618],{},"I predicted the target house's price (1200 sqft, 3 bedrooms, 1 floor, 40 years old) and got ",[15,25083,25084],{},"$318.9 thousand",[11,25086,25087,25088,25090],{},"What caught my attention most was the bedrooms coefficient: ",[15,25089,11329],{},". On its own, more bedrooms sounds like a purely good thing, but controlling for the house's size (which is already in the model), more bedrooms in a same-sized house usually means smaller bedrooms, which pulls the price down a bit. That's exactly the kind of thing that only shows up once you look at the coefficients of a model with several features at the same time, you can't see it in a bedrooms-vs-price chart alone.",[11,25092,25093,25094,25165],{},"If I'd fit with normalized features, the weights would come out on a different scale (each ",[80,25095,25097,25115],{"className":25096},[83],[80,25098,25100],{"className":25099},[87],[89,25101,25102],{"xmlns":91},[93,25103,25104,25112],{},[96,25105,25106],{},[99,25107,25108,25110],{},[102,25109,109],{},[102,25111,18373],{},[145,25113,25114],{"encoding":147},"w_j",[80,25116,25118],{"className":25117,"ariaHidden":113},[152],[80,25119,25121,25124],{"className":25120},[156],[80,25122],{"className":25123,"style":22889},[160],[80,25125,25127,25130],{"className":25126},[165],[80,25128,109],{"className":25129,"style":210},[165,169],[80,25131,25133],{"className":25132},[174],[80,25134,25136,25157],{"className":25135},[178,179],[80,25137,25139,25154],{"className":25138},[183],[80,25140,25142],{"className":25141,"style":15323},[187],[80,25143,25145,25148],{"style":25144},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[80,25146],{"className":25147,"style":196},[195],[80,25149,25151],{"className":25150},[200,201,202,203],[80,25152,18373],{"className":25153,"style":18420},[165,169,203],[80,25155,222],{"className":25156},[221],[80,25158,25160],{"className":25159},[183],[80,25161,25163],{"className":25162,"style":229},[187],[80,25164],{}," would represent \"how much the price changes per 1 standard deviation of that feature\", not per 1 real unit). To get back to real-unit coefficients, the denormalization formula is:",[11,25167,25168],{},[80,25169,25171,25352],{"className":25170},[83],[80,25172,25174],{"className":25173},[87],[89,25175,25176],{"xmlns":91},[93,25177,25178,25349],{},[96,25179,25180,25205,25207,25243,25245,25267,25269,25295,25297,25303],{},[7202,25181,25182,25184,25186],{},[102,25183,109],{},[102,25185,18373],{},[96,25187,25188,25190,25192,25194,25197,25199,25201,25203],{},[102,25189,1666],{},[102,25191,1713],{},[102,25193,736],{},[102,25195,25196],{},"g",[102,25198,736],{},[102,25200,1487],{},[102,25202,562],{},[102,25204,1704],{},[111,25206,130],{},[4625,25208,25209,25237],{},[7202,25210,25211,25213,25215],{},[102,25212,109],{},[102,25214,18373],{},[96,25216,25217,25219,25221,25223,25225,25227,25229,25231,25233,25235],{},[102,25218,1487],{},[102,25220,1666],{},[102,25222,1713],{},[102,25224,322],{},[102,25226,562],{},[102,25228,1704],{},[102,25230,736],{},[102,25232,16378],{},[102,25234,1671],{},[102,25236,1663],{},[99,25238,25239,25241],{},[102,25240,24111],{},[102,25242,18373],{},[246,25244],{"width":10580},[726,25246,25247,25249],{},[102,25248,117],{},[96,25250,25251,25253,25255,25257,25259,25261,25263,25265],{},[102,25252,1666],{},[102,25254,1713],{},[102,25256,736],{},[102,25258,25196],{},[102,25260,736],{},[102,25262,1487],{},[102,25264,562],{},[102,25266,1704],{},[111,25268,130],{},[726,25270,25271,25273],{},[102,25272,117],{},[96,25274,25275,25277,25279,25281,25283,25285,25287,25289,25291,25293],{},[102,25276,1487],{},[102,25278,1666],{},[102,25280,1713],{},[102,25282,322],{},[102,25284,562],{},[102,25286,1704],{},[102,25288,736],{},[102,25290,16378],{},[102,25292,1671],{},[102,25294,1663],{},[111,25296,4643],{},[99,25298,25299,25301],{},[111,25300,7206],{},[102,25302,18373],{},[4625,25304,25305,25343],{},[96,25306,25307,25335,25337],{},[7202,25308,25309,25311,25313],{},[102,25310,109],{},[102,25312,18373],{},[96,25314,25315,25317,25319,25321,25323,25325,25327,25329,25331,25333],{},[102,25316,1487],{},[102,25318,1666],{},[102,25320,1713],{},[102,25322,322],{},[102,25324,562],{},[102,25326,1704],{},[102,25328,736],{},[102,25330,16378],{},[102,25332,1671],{},[102,25334,1663],{},[111,25336,15147],{},[99,25338,25339,25341],{},[102,25340,22115],{},[102,25342,18373],{},[99,25344,25345,25347],{},[102,25346,24111],{},[102,25348,18373],{},[145,25350,25351],{"encoding":147},"w_j^{original} = \\frac{w_j^{normalized}}{\\sigma_j} \\qquad b^{original} = b^{normalized} - \\sum_j \\frac{w_j^{normalized} \\cdot \\mu_j}{\\sigma_j}",[80,25353,25355,25442,25690,25759],{"className":25354,"ariaHidden":113},[152],[80,25356,25358,25362,25433,25436,25439],{"className":25357},[156],[80,25359],{"className":25360,"style":25361},[160],"height:1.38em;vertical-align:-0.413em;",[80,25363,25365,25368],{"className":25364},[165],[80,25366,109],{"className":25367,"style":210},[165,169],[80,25369,25371],{"className":25370},[174],[80,25372,25374,25424],{"className":25373},[178,179],[80,25375,25377,25421],{"className":25376},[183],[80,25378,25381,25393],{"className":25379,"style":25380},[187],"height:0.967em;",[80,25382,25384,25387],{"style":25383},"top:-2.4231em;margin-left:-0.0269em;margin-right:0.05em;",[80,25385],{"className":25386,"style":196},[195],[80,25388,25390],{"className":25389},[200,201,202,203],[80,25391,18373],{"className":25392,"style":18420},[165,169,203],[80,25394,25396,25399],{"style":25395},"top:-3.1809em;margin-right:0.05em;",[80,25397],{"className":25398,"style":196},[195],[80,25400,25402],{"className":25401},[200,201,202,203],[80,25403,25405,25408,25411,25414,25418],{"className":25404},[165,203],[80,25406,1851],{"className":25407,"style":1850},[165,169,203],[80,25409,736],{"className":25410},[165,169,203],[80,25412,25196],{"className":25413,"style":834},[165,169,203],[80,25415,25417],{"className":25416},[165,169,203],"ina",[80,25419,1704],{"className":25420,"style":1842},[165,169,203],[80,25422,222],{"className":25423},[221],[80,25425,25427],{"className":25426},[183],[80,25428,25431],{"className":25429,"style":25430},[187],"height:0.413em;",[80,25432],{},[80,25434],{"className":25435,"style":247},[246],[80,25437,130],{"className":25438},[251],[80,25440],{"className":25441,"style":247},[246],[80,25443,25445,25449,25633,25636,25681,25684,25687],{"className":25444},[156],[80,25446],{"className":25447,"style":25448},[160],"height:1.7987em;vertical-align:-0.5423em;",[80,25450,25452,25455,25630],{"className":25451},[165],[80,25453],{"className":25454},[235,4746],[80,25456,25458],{"className":25457},[4625],[80,25459,25461,25622],{"className":25460},[178,179],[80,25462,25464,25619],{"className":25463},[183],[80,25465,25468,25519,25527],{"className":25466,"style":25467},[187],"height:1.2563em;",[80,25469,25470,25473],{"style":5013},[80,25471],{"className":25472,"style":4766},[195],[80,25474,25476],{"className":25475},[200,201,202,203],[80,25477,25479],{"className":25478},[165,203],[80,25480,25482,25485],{"className":25481},[165,203],[80,25483,24111],{"className":25484,"style":834},[165,169,203],[80,25486,25488],{"className":25487},[174],[80,25489,25491,25511],{"className":25490},[178,179],[80,25492,25494,25508],{"className":25493},[183],[80,25495,25497],{"className":25496,"style":23313},[187],[80,25498,25499,25502],{"style":24230},[80,25500],{"className":25501,"style":23320},[195],[80,25503,25505],{"className":25504},[200,4802,4803,203],[80,25506,18373],{"className":25507,"style":18420},[165,169,203],[80,25509,222],{"className":25510},[221],[80,25512,25514],{"className":25513},[183],[80,25515,25517],{"className":25516,"style":23336},[187],[80,25518],{},[80,25520,25521,25524],{"style":4859},[80,25522],{"className":25523,"style":4766},[195],[80,25525],{"className":25526,"style":4867},[4866],[80,25528,25530,25533],{"style":25529},"top:-3.6074em;",[80,25531],{"className":25532,"style":4766},[195],[80,25534,25536],{"className":25535},[200,201,202,203],[80,25537,25539],{"className":25538},[165,203],[80,25540,25542,25545],{"className":25541},[165,203],[80,25543,109],{"className":25544,"style":210},[165,169,203],[80,25546,25548],{"className":25547},[174],[80,25549,25551,25610],{"className":25550},[178,179],[80,25552,25554,25607],{"className":25553},[183],[80,25555,25558,25570],{"className":25556,"style":25557},[187],"height:0.927em;",[80,25559,25561,25564],{"style":25560},"top:-2.214em;margin-left:-0.0269em;margin-right:0.0714em;",[80,25562],{"className":25563,"style":23320},[195],[80,25565,25567],{"className":25566},[200,4802,4803,203],[80,25568,18373],{"className":25569,"style":18420},[165,169,203],[80,25571,25573,25576],{"style":25572},"top:-2.931em;margin-right:0.0714em;",[80,25574],{"className":25575,"style":23320},[195],[80,25577,25579],{"className":25578},[200,4802,4803,203],[80,25580,25582,25585,25588,25592,25595,25598,25601,25604],{"className":25581},[165,203],[80,25583,1487],{"className":25584},[165,169,203],[80,25586,1851],{"className":25587,"style":1850},[165,169,203],[80,25589,25591],{"className":25590},[165,169,203],"ma",[80,25593,1704],{"className":25594,"style":1842},[165,169,203],[80,25596,736],{"className":25597},[165,169,203],[80,25599,16378],{"className":25600,"style":16393},[165,169,203],[80,25602,1671],{"className":25603},[165,169,203],[80,25605,1663],{"className":25606},[165,169,203],[80,25608,222],{"className":25609},[221],[80,25611,25613],{"className":25612},[183],[80,25614,25617],{"className":25615,"style":25616},[187],"height:0.4249em;",[80,25618],{},[80,25620,222],{"className":25621},[221],[80,25623,25625],{"className":25624},[183],[80,25626,25628],{"className":25627,"style":23533},[187],[80,25629],{},[80,25631],{"className":25632},[242,4746],[80,25634],{"className":25635,"style":10763},[246],[80,25637,25639,25642],{"className":25638},[165],[80,25640,117],{"className":25641},[165,169],[80,25643,25645],{"className":25644},[174],[80,25646,25648],{"className":25647},[178],[80,25649,25651],{"className":25650},[183],[80,25652,25655],{"className":25653,"style":25654},[187],"height:0.8491em;",[80,25656,25657,25660],{"style":772},[80,25658],{"className":25659,"style":196},[195],[80,25661,25663],{"className":25662},[200,201,202,203],[80,25664,25666,25669,25672,25675,25678],{"className":25665},[165,203],[80,25667,1851],{"className":25668,"style":1850},[165,169,203],[80,25670,736],{"className":25671},[165,169,203],[80,25673,25196],{"className":25674,"style":834},[165,169,203],[80,25676,25417],{"className":25677},[165,169,203],[80,25679,1704],{"className":25680,"style":1842},[165,169,203],[80,25682],{"className":25683,"style":247},[246],[80,25685,130],{"className":25686},[251],[80,25688],{"className":25689,"style":247},[246],[80,25691,25693,25697,25750,25753,25756],{"className":25692},[156],[80,25694],{"className":25695,"style":25696},[160],"height:0.9324em;vertical-align:-0.0833em;",[80,25698,25700,25703],{"className":25699},[165],[80,25701,117],{"className":25702},[165,169],[80,25704,25706],{"className":25705},[174],[80,25707,25709],{"className":25708},[178],[80,25710,25712],{"className":25711},[183],[80,25713,25715],{"className":25714,"style":25654},[187],[80,25716,25717,25720],{"style":772},[80,25718],{"className":25719,"style":196},[195],[80,25721,25723],{"className":25722},[200,201,202,203],[80,25724,25726,25729,25732,25735,25738,25741,25744,25747],{"className":25725},[165,203],[80,25727,1487],{"className":25728},[165,169,203],[80,25730,1851],{"className":25731,"style":1850},[165,169,203],[80,25733,25591],{"className":25734},[165,169,203],[80,25736,1704],{"className":25737,"style":1842},[165,169,203],[80,25739,736],{"className":25740},[165,169,203],[80,25742,16378],{"className":25743,"style":16393},[165,169,203],[80,25745,1671],{"className":25746},[165,169,203],[80,25748,1663],{"className":25749},[165,169,203],[80,25751],{"className":25752,"style":275},[246],[80,25754,4643],{"className":25755},[279],[80,25757],{"className":25758,"style":275},[246],[80,25760,25762,25765,25807,25810],{"className":25761},[156],[80,25763],{"className":25764,"style":25448},[160],[80,25766,25768,25771],{"className":25767},[7402],[80,25769,7206],{"className":25770,"style":7408},[7402,7406,7407],[80,25772,25774],{"className":25773},[174],[80,25775,25777,25798],{"className":25776},[178,179],[80,25778,25780,25795],{"className":25779},[183],[80,25781,25784],{"className":25782,"style":25783},[187],"height:0.162em;",[80,25785,25786,25789],{"style":7424},[80,25787],{"className":25788,"style":196},[195],[80,25790,25792],{"className":25791},[200,201,202,203],[80,25793,18373],{"className":25794,"style":18420},[165,169,203],[80,25796,222],{"className":25797},[221],[80,25799,25801],{"className":25800},[183],[80,25802,25805],{"className":25803,"style":25804},[187],"height:0.4358em;",[80,25806],{},[80,25808],{"className":25809,"style":268},[246],[80,25811,25813,25816,26027],{"className":25812},[165],[80,25814],{"className":25815},[235,4746],[80,25817,25819],{"className":25818},[4625],[80,25820,25822,26019],{"className":25821},[178,179],[80,25823,25825,26016],{"className":25824},[183],[80,25826,25828,25879,25887],{"className":25827,"style":25467},[187],[80,25829,25830,25833],{"style":5013},[80,25831],{"className":25832,"style":4766},[195],[80,25834,25836],{"className":25835},[200,201,202,203],[80,25837,25839],{"className":25838},[165,203],[80,25840,25842,25845],{"className":25841},[165,203],[80,25843,24111],{"className":25844,"style":834},[165,169,203],[80,25846,25848],{"className":25847},[174],[80,25849,25851,25871],{"className":25850},[178,179],[80,25852,25854,25868],{"className":25853},[183],[80,25855,25857],{"className":25856,"style":23313},[187],[80,25858,25859,25862],{"style":24230},[80,25860],{"className":25861,"style":23320},[195],[80,25863,25865],{"className":25864},[200,4802,4803,203],[80,25866,18373],{"className":25867,"style":18420},[165,169,203],[80,25869,222],{"className":25870},[221],[80,25872,25874],{"className":25873},[183],[80,25875,25877],{"className":25876,"style":23336},[187],[80,25878],{},[80,25880,25881,25884],{"style":4859},[80,25882],{"className":25883,"style":4766},[195],[80,25885],{"className":25886,"style":4867},[4866],[80,25888,25889,25892],{"style":25529},[80,25890],{"className":25891,"style":4766},[195],[80,25893,25895],{"className":25894},[200,201,202,203],[80,25896,25898,25973,25976],{"className":25897},[165,203],[80,25899,25901,25904],{"className":25900},[165,203],[80,25902,109],{"className":25903,"style":210},[165,169,203],[80,25905,25907],{"className":25906},[174],[80,25908,25910,25965],{"className":25909},[178,179],[80,25911,25913,25962],{"className":25912},[183],[80,25914,25916,25927],{"className":25915,"style":25557},[187],[80,25917,25918,25921],{"style":25560},[80,25919],{"className":25920,"style":23320},[195],[80,25922,25924],{"className":25923},[200,4802,4803,203],[80,25925,18373],{"className":25926,"style":18420},[165,169,203],[80,25928,25929,25932],{"style":25572},[80,25930],{"className":25931,"style":23320},[195],[80,25933,25935],{"className":25934},[200,4802,4803,203],[80,25936,25938,25941,25944,25947,25950,25953,25956,25959],{"className":25937},[165,203],[80,25939,1487],{"className":25940},[165,169,203],[80,25942,1851],{"className":25943,"style":1850},[165,169,203],[80,25945,25591],{"className":25946},[165,169,203],[80,25948,1704],{"className":25949,"style":1842},[165,169,203],[80,25951,736],{"className":25952},[165,169,203],[80,25954,16378],{"className":25955,"style":16393},[165,169,203],[80,25957,1671],{"className":25958},[165,169,203],[80,25960,1663],{"className":25961},[165,169,203],[80,25963,222],{"className":25964},[221],[80,25966,25968],{"className":25967},[183],[80,25969,25971],{"className":25970,"style":25616},[187],[80,25972],{},[80,25974,15147],{"className":25975},[279,203],[80,25977,25979,25982],{"className":25978},[165,203],[80,25980,22115],{"className":25981},[165,169,203],[80,25983,25985],{"className":25984},[174],[80,25986,25988,26008],{"className":25987},[178,179],[80,25989,25991,26005],{"className":25990},[183],[80,25992,25994],{"className":25993,"style":23313},[187],[80,25995,25996,25999],{"style":23316},[80,25997],{"className":25998,"style":23320},[195],[80,26000,26002],{"className":26001},[200,4802,4803,203],[80,26003,18373],{"className":26004,"style":18420},[165,169,203],[80,26006,222],{"className":26007},[221],[80,26009,26011],{"className":26010},[183],[80,26012,26014],{"className":26013,"style":23336},[187],[80,26015],{},[80,26017,222],{"className":26018},[221],[80,26020,26022],{"className":26021},[183],[80,26023,26025],{"className":26024,"style":23533},[187],[80,26026],{},[80,26028],{"className":26029},[242,4746],[11,26031,26032,26033,2338,26061,26089,26090,2338,26142,26194],{},"I checked this on our 1-feature example: denormalizing the ",[80,26034,26036,26049],{"className":26035},[83],[80,26037,26039],{"className":26038},[87],[89,26040,26041],{"xmlns":91},[93,26042,26043,26047],{},[96,26044,26045],{},[102,26046,109],{},[145,26048,109],{"encoding":147},[80,26050,26052],{"className":26051,"ariaHidden":113},[152],[80,26053,26055,26058],{"className":26054},[156],[80,26056],{"className":26057,"style":334},[160],[80,26059,109],{"className":26060,"style":210},[165,169],[80,26062,26064,26077],{"className":26063},[83],[80,26065,26067],{"className":26066},[87],[89,26068,26069],{"xmlns":91},[93,26070,26071,26075],{},[96,26072,26073],{},[102,26074,117],{},[145,26076,117],{"encoding":147},[80,26078,26080],{"className":26079,"ariaHidden":113},[152],[80,26081,26083,26086],{"className":26082},[156],[80,26084],{"className":26085,"style":289},[160],[80,26087,117],{"className":26088},[165,169]," that gradient descent found in normalized space, I got ",[80,26091,26093,26112],{"className":26092},[83],[80,26094,26096],{"className":26095},[87],[89,26097,26098],{"xmlns":91},[93,26099,26100,26109],{},[96,26101,26102,26104,26106],{},[102,26103,109],{},[111,26105,130],{},[1321,26107,26108],{},"0.267",[145,26110,26111],{"encoding":147},"w = 0.267",[80,26113,26115,26133],{"className":26114,"ariaHidden":113},[152],[80,26116,26118,26121,26124,26127,26130],{"className":26117},[156],[80,26119],{"className":26120,"style":334},[160],[80,26122,109],{"className":26123,"style":210},[165,169],[80,26125],{"className":26126,"style":247},[246],[80,26128,130],{"className":26129},[251],[80,26131],{"className":26132,"style":247},[246],[80,26134,26136,26139],{"className":26135},[156],[80,26137],{"className":26138,"style":1614},[160],[80,26140,26108],{"className":26141},[165],[80,26143,26145,26164],{"className":26144},[83],[80,26146,26148],{"className":26147},[87],[89,26149,26150],{"xmlns":91},[93,26151,26152,26161],{},[96,26153,26154,26156,26158],{},[102,26155,117],{},[111,26157,130],{},[1321,26159,26160],{},"7.16",[145,26162,26163],{"encoding":147},"b = 7.16",[80,26165,26167,26185],{"className":26166,"ariaHidden":113},[152],[80,26168,26170,26173,26176,26179,26182],{"className":26169},[156],[80,26171],{"className":26172,"style":289},[160],[80,26174,117],{"className":26175},[165,169],[80,26177],{"className":26178,"style":247},[246],[80,26180,130],{"className":26181},[251],[80,26183],{"className":26184,"style":247},[246],[80,26186,26188,26191],{"className":26187},[156],[80,26189],{"className":26190,"style":1614},[160],[80,26192,26160],{"className":26193},[165],", the exact same numbers the closed-form solution gives directly in raw space. Both paths land in the same place, one of them just converged in 50 iterations and the other would need millions.",[294,26196,26198],{"id":26197},"the-contours-the-geometric-picture","The contours: the geometric picture",[11,26200,26201,26202,504,26205,26208],{},"Everything I've described in numbers, I can see at once in a single image. From here on I switched datasets: I'm using the 50-house one that already showed up in the previous posts in this playlist (",[65,26203,26204],{},"square_feet",[65,26206,26207],{},"num_bedrooms",", location score, distance to center), not the 100-house one from the notebook I used earlier. It's a different set of houses, so the correlation numbers here differ from the ones I computed in \"The problem\" section.",[11,26210,26211,26212,26214,26215,26218],{},"Before going abstract (the shape of the ",[73,26213,7678],{},"), it's worth seeing the far more direct, concrete effect first: the shape of the ",[73,26216,26217],{},"data points themselves",". I took size and number of bedrooms from these 50 houses and plotted one against the other, raw and then normalized:",[11,26220,26221],{},[15,26222,26223],{},"Points, without normalizing:",[26225,26226],"housing-scaling-scatter",{"bedrooms-label":18595,"bedrooms-label-normalized":26227,"mode":26228,"size-label":6803,"size-label-normalized":26229},"bedrooms (normalized)","raw","size (normalized)",[11,26231,26232],{},[15,26233,26234],{},"Points, after normalizing (z-score):",[26225,26236],{"bedrooms-label":18595,"bedrooms-label-normalized":26227,"mode":26237,"size-label":6803,"size-label-normalized":26229},"normalized",[11,26239,26240],{},"Notice the cloud of points doesn't change shape, it just shifts and rescales so both axes end up in the same unit (standard deviations instead of sqft\u002Fbedrooms). And it's exactly that change in relative scale between the two axes that makes all the difference further down.",[11,26242,26243],{},"Now the cost map. I fit a model with all 4 features (size, bedrooms, location score, distance to center), fixed two of them at their fitted values, and drew a heatmap varying just size's weight and bedrooms' weight, the same trick the original notebook uses. The red dot marks exactly where my fit landed, and the line right below the chart shows the exact values.",[11,26245,26246],{},[15,26247,26248],{},"Without normalizing:",[26250,26251],"housing-cost-contour",{":b-max":4286,":b-min":17444,":w-max":1323,":w-min":2071,"b-label":26252,"mode":26228,"w-label":26253},"w_bedrooms (raw)","w_size (raw)",[11,26255,26256],{},[15,26257,26258],{},"After normalizing (z-score):",[26250,26260],{":b-max":26261,":b-min":9819,":w-max":9016,":w-min":26262,"b-label":26263,"mode":26237,"w-label":26264},"140","30","w_bedrooms (normalized)","w_size (normalized)",[11,26266,26267],{},"Notice the difference in shape. In the raw contour, the size axis is far more sensitive than the bedrooms axis (a tiny 0.5 change in size's weight already sends the cost skyrocketing, while bedrooms' weight needs to move dozens of units to do the same damage). It's a heavily elongated valley, almost a corridor, and the red dot (the real fit) sits squeezed at the bottom of it. In the normalized contour, both directions end up with similar sensitivity, the shape turns much rounder, and the dot lands near the center of a circle.",[11,26269,26270,26271,26322],{},"It's not a perfect circle (size and number of bedrooms, in this 50-house dataset, still have a small residual correlation, I calculated it and it came out around 0.02, very close to zero but not exactly zero), but the difference in shape is dramatic. And that's not a coincidence or a visual approximation: for a single z-score normalized feature, it's provable that the Hessian becomes exactly the identity matrix, ",[80,26272,26274,26292],{"className":26273},[83],[80,26275,26277],{"className":26276},[87],[89,26278,26279],{"xmlns":91},[93,26280,26281,26289],{},[96,26282,26283,26285,26287],{},[102,26284,22106],{},[111,26286,130],{},[1321,26288,1583],{},[145,26290,26291],{"encoding":147},"\\kappa = 1",[80,26293,26295,26313],{"className":26294,"ariaHidden":113},[152],[80,26296,26298,26301,26304,26307,26310],{"className":26297},[156],[80,26299],{"className":26300,"style":334},[160],[80,26302,22106],{"className":26303},[165,169],[80,26305],{"className":26306,"style":247},[246],[80,26308,130],{"className":26309},[251],[80,26311],{"className":26312,"style":247},[246],[80,26314,26316,26319],{"className":26315},[156],[80,26317],{"className":26318,"style":1614},[160],[80,26320,1583],{"className":26321},[165],", a perfect circle, an exact result and not an approximation. With two real features, the result comes out nearly circular, and the \"nearly\" is literally the size of the correlation between them.",[294,26324,6192],{"id":6191},[521,26326,26327,26337],{},[524,26328,26329],{},[527,26330,26331,26334],{},[530,26332,26333],{"align":532},"What I already knew",[530,26335,26336],{"align":532},"What this post settled",[541,26338,26339,26347,26412],{},[527,26340,26341,26344],{},[546,26342,26343],{"align":532},"Gradient descent finds the minimum on its own",[546,26345,26346],{"align":532},"It only finds it fast if the cost surface isn't a stretched-out corridor",[527,26348,26349,26380],{},[546,26350,26351,26379],{"align":532},[80,26352,26354,26367],{"className":26353},[83],[80,26355,26357],{"className":26356},[87],[89,26358,26359],{"xmlns":91},[93,26360,26361,26365],{},[96,26362,26363],{},[102,26364,10551],{},[145,26366,11037],{"encoding":147},[80,26368,26370],{"className":26369,"ariaHidden":113},[152],[80,26371,26373,26376],{"className":26372},[156],[80,26374],{"className":26375,"style":334},[160],[80,26377,10551],{"className":26378,"style":10667},[165,169]," is the most sensitive parameter I tune",[546,26381,26382,26383,26411],{"align":532},"The \"right\" ",[80,26384,26386,26399],{"className":26385},[83],[80,26387,26389],{"className":26388},[87],[89,26390,26391],{"xmlns":91},[93,26392,26393,26397],{},[96,26394,26395],{},[102,26396,10551],{},[145,26398,11037],{"encoding":147},[80,26400,26402],{"className":26401,"ariaHidden":113},[152],[80,26403,26405,26408],{"className":26404},[156],[80,26406],{"className":26407,"style":334},[160],[80,26409,10551],{"className":26410,"style":10667},[165,169]," depends entirely on the scale of the features, it's not a universal constant",[527,26413,26414,26417],{},[546,26415,26416],{"align":532},"The cost bowl is always convex, for linear regression",[546,26418,26419],{"align":532},"Convex isn't the same as well-conditioned: it can be a deep, round bowl, or a shallow, elongated canyon",[11,26421,6507],{},[299,26423,26424,26430,26495],{},[302,26425,26426,26429],{},[15,26427,26428],{},"The relative scale of the features decides the shape of the cost bowl",", not the optimization algorithm itself.",[302,26431,26432,26465,26466,26494],{},[15,26433,26434,26435,26464],{},"The condition number ",[80,26436,26438,26452],{"className":26437},[83],[80,26439,26441],{"className":26440},[87],[89,26442,26443],{"xmlns":91},[93,26444,26445,26449],{},[96,26446,26447],{},[102,26448,22106],{},[145,26450,26451],{"encoding":147},"\\kappa",[80,26453,26455],{"className":26454,"ariaHidden":113},[152],[80,26456,26458,26461],{"className":26457},[156],[80,26459],{"className":26460,"style":334},[160],[80,26462,22106],{"className":26463},[165,169]," is the metric that quantifies this",": close to 1 is a round, fast-to-descend bowl, much bigger than 1 is a canyon that any fixed ",[80,26467,26469,26482],{"className":26468},[83],[80,26470,26472],{"className":26471},[87],[89,26473,26474],{"xmlns":91},[93,26475,26476,26480],{},[96,26477,26478],{},[102,26479,10551],{},[145,26481,11037],{"encoding":147},[80,26483,26485],{"className":26484,"ariaHidden":113},[152],[80,26486,26488,26491],{"className":26487},[156],[80,26489],{"className":26490,"style":334},[160],[80,26492,10551],{"className":26493,"style":10667},[165,169]," takes forever to cross.",[302,26496,26497,26500],{},[15,26498,26499],{},"Normalizing doesn't change the answer, it changes the path to it",": same minimum, orders of magnitude fewer iterations to get near it.",[294,26502,6716],{"id":6715},[11,26504,16135,26505,26508,26509,26511,26512,2338,26515,26518,26519,26570,26571,26670],{},[562,26506,6725],{"href":6722,"rel":26507},[6724],"). We already saw in ",[562,26510,22335],{"href":4070}," that the raw algorithm, with ",[65,26513,26514],{},"square_feet\u002F100",[65,26516,26517],{},"price\u002F1000",", converges slowly, needing ",[80,26520,26522,26540],{"className":26521},[83],[80,26523,26525],{"className":26524},[87],[89,26526,26527],{"xmlns":91},[93,26528,26529,26537],{},[96,26530,26531,26533,26535],{},[102,26532,10551],{},[111,26534,130],{},[1321,26536,13526],{},[145,26538,26539],{"encoding":147},"\\alpha=0.01",[80,26541,26543,26561],{"className":26542,"ariaHidden":113},[152],[80,26544,26546,26549,26552,26555,26558],{"className":26545},[156],[80,26547],{"className":26548,"style":334},[160],[80,26550,10551],{"className":26551,"style":10667},[165,169],[80,26553],{"className":26554,"style":247},[246],[80,26556,130],{"className":26557},[251],[80,26559],{"className":26560,"style":247},[246],[80,26562,26564,26567],{"className":26563},[156],[80,26565],{"className":26566,"style":1614},[160],[80,26568,13526],{"className":26569},[165]," and 4000 iterations to reach ",[80,26572,26574,26610],{"className":26573},[83],[80,26575,26577],{"className":26576},[87],[89,26578,26579],{"xmlns":91},[93,26580,26581,26607],{},[96,26582,26583,26585,26587,26589,26591,26593,26595,26597,26600,26602,26605],{},[111,26584,121],{"stretchy":120},[102,26586,109],{},[111,26588,114],{"separator":113},[102,26590,117],{},[111,26592,127],{"stretchy":120},[111,26594,130],{},[111,26596,121],{"stretchy":120},[1321,26598,26599],{},"116.5",[111,26601,114],{"separator":113},[1321,26603,26604],{},"398.3",[111,26606,127],{"stretchy":120},[145,26608,26609],{"encoding":147},"(w,b) = (116.5, 398.3)",[80,26611,26613,26646],{"className":26612,"ariaHidden":113},[152],[80,26614,26616,26619,26622,26625,26628,26631,26634,26637,26640,26643],{"className":26615},[156],[80,26617],{"className":26618,"style":2316},[160],[80,26620,121],{"className":26621},[235],[80,26623,109],{"className":26624,"style":210},[165,169],[80,26626,114],{"className":26627},[214],[80,26629],{"className":26630,"style":268},[246],[80,26632,117],{"className":26633},[165,169],[80,26635,127],{"className":26636},[242],[80,26638],{"className":26639,"style":247},[246],[80,26641,130],{"className":26642},[251],[80,26644],{"className":26645,"style":247},[246],[80,26647,26649,26652,26655,26658,26661,26664,26667],{"className":26648},[156],[80,26650],{"className":26651,"style":2316},[160],[80,26653,121],{"className":26654},[235],[80,26656,26599],{"className":26657},[165],[80,26659,114],{"className":26660},[214],[80,26662],{"className":26663,"style":268},[246],[80,26665,26604],{"className":26666},[165],[80,26668,127],{"className":26669},[242],". Now I z-score normalized size before running it:",[2611,26672,26674],{"className":2613,"code":26673,"language":2615,"meta":26,"style":26},"size_norm, mu, sigma = zscore_normalize_features(square_feet)\n\nw, b, J_hist = gradient_descent(\n    size_norm, price,\n    w_in=0, b_in=0,\n    alpha=0.3, num_iters=200,\n    cost_function=compute_cost, gradient_function=compute_gradient)\n\nprint(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[65,26675,26676,26681,26685,26689,26694,26699,26704,26708,26712],{"__ignoreMap":26},[80,26677,26678],{"class":2620,"line":33},[80,26679,26680],{},"size_norm, mu, sigma = zscore_normalize_features(square_feet)\n",[80,26682,26683],{"class":2620,"line":27},[80,26684,2657],{"emptyLinePlaceholder":32},[80,26686,26687],{"class":2620,"line":2631},[80,26688,14544],{},[80,26690,26691],{"class":2620,"line":2636},[80,26692,26693],{},"    size_norm, price,\n",[80,26695,26696],{"class":2620,"line":2642},[80,26697,26698],{},"    w_in=0, b_in=0,\n",[80,26700,26701],{"class":2620,"line":2648},[80,26702,26703],{},"    alpha=0.3, num_iters=200,\n",[80,26705,26706],{"class":2620,"line":2654},[80,26707,14564],{},[80,26709,26710],{"class":2620,"line":2660},[80,26711,2657],{"emptyLinePlaceholder":32},[80,26713,26714],{"class":2620,"line":2666},[80,26715,14573],{},[46,26717,26718],{},[11,26719,26720,3255,26722],{},[15,26721,2693],{},[65,26723,26724],{},"(w, b) found: (88.1, 593.0)",[11,26726,26727,26728,26780,26781,26835,26836,26887],{},"The final cost matches the raw example almost exactly (",[80,26729,26731,26750],{"className":26730},[83],[80,26732,26734],{"className":26733},[87],[89,26735,26736],{"xmlns":91},[93,26737,26738,26747],{},[96,26739,26740,26742,26744],{},[102,26741,5606],{},[111,26743,9527],{},[1321,26745,26746],{},"5189.6",[145,26748,26749],{"encoding":147},"J \\approx 5189.6",[80,26751,26753,26771],{"className":26752,"ariaHidden":113},[152],[80,26754,26756,26759,26762,26765,26768],{"className":26755},[156],[80,26757],{"className":26758,"style":8672},[160],[80,26760,5606],{"className":26761,"style":5632},[165,169],[80,26763],{"className":26764,"style":247},[246],[80,26766,9527],{"className":26767},[251],[80,26769],{"className":26770,"style":247},[246],[80,26772,26774,26777],{"className":26773},[156],[80,26775],{"className":26776,"style":1614},[160],[80,26778,26746],{"className":26779},[165]," in both cases, it's the same minimum), except here I got there in ",[15,26782,26783,26784],{},"200 iterations at ",[80,26785,26787,26805],{"className":26786},[83],[80,26788,26790],{"className":26789},[87],[89,26791,26792],{"xmlns":91},[93,26793,26794,26802],{},[96,26795,26796,26798,26800],{},[102,26797,10551],{},[111,26799,130],{},[1321,26801,13542],{},[145,26803,26804],{"encoding":147},"\\alpha = 0.3",[80,26806,26808,26826],{"className":26807,"ariaHidden":113},[152],[80,26809,26811,26814,26817,26820,26823],{"className":26810},[156],[80,26812],{"className":26813,"style":334},[160],[80,26815,10551],{"className":26816,"style":10667},[165,169],[80,26818],{"className":26819,"style":247},[246],[80,26821,130],{"className":26822},[251],[80,26824],{"className":26825,"style":247},[246],[80,26827,26829,26832],{"className":26828},[156],[80,26830],{"className":26831,"style":1614},[160],[80,26833,13542],{"className":26834},[165],", versus 4000 iterations at ",[80,26837,26839,26857],{"className":26838},[83],[80,26840,26842],{"className":26841},[87],[89,26843,26844],{"xmlns":91},[93,26845,26846,26854],{},[96,26847,26848,26850,26852],{},[102,26849,10551],{},[111,26851,130],{},[1321,26853,13526],{},[145,26855,26856],{"encoding":147},"\\alpha = 0.01",[80,26858,26860,26878],{"className":26859,"ariaHidden":113},[152],[80,26861,26863,26866,26869,26872,26875],{"className":26862},[156],[80,26864],{"className":26865,"style":334},[160],[80,26867,10551],{"className":26868,"style":10667},[165,169],[80,26870],{"className":26871,"style":247},[246],[80,26873,130],{"className":26874},[251],[80,26876],{"className":26877,"style":247},[246],[80,26879,26881,26884],{"className":26880},[156],[80,26882],{"className":26883,"style":1614},[160],[80,26885,13526],{"className":26886},[165]," before. Compare both live:",[14587,26889],{},[26891,26892],"housing-scaled-gradient-descent-simulator",{},[11,26894,26895],{},"Click \"Rodar 100\" a few times on both and time it in your head: one of them already flattened out, the other is still climbing the slope. Same house, same final price, and the only difference between the two simulators is one extra line of code at the very start.",[6949,26897,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":26899},[26900,26901,26902,26903,26905,26907,26908,26909,26910,26911,26912],{"id":18536,"depth":27,"text":18537},{"id":18768,"depth":27,"text":18769},{"id":19658,"depth":27,"text":19659},{"id":20896,"depth":27,"text":26904},"Why exactly there: the 2\u002FL2\u002FL2\u002FL bound and the condition number",{"id":22300,"depth":27,"text":26906},"A practical recipe for picking α\\alphaα",{"id":22777,"depth":27,"text":22778},{"id":24590,"depth":27,"text":24591},{"id":25059,"depth":27,"text":25060},{"id":26197,"depth":27,"text":26198},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716},"2026-08-19","Why the exact same gradient descent that worked fine so far breaks the moment one feature has a wildly different scale from the others, and how a basic statistics trick fixes it for good.",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling",{"title":18482,"description":26914},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling",[26920,26921,26922],"normalization","feature-scaling","learning-rate","p5SOy1hrWlv6-1HdwG-_e2w7LgJ_D2PdPogStNjMZJQ",{"id":26925,"title":26926,"body":26927,"cover":3,"date":26913,"description":30165,"extension":30,"meta":30166,"navigation":32,"order":2660,"path":30167,"playlist":6980,"seo":30168,"status":36,"stem":30169,"tags":30170,"__hash__":30173},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering.md","Feature Engineering and Polynomial Regression",{"type":8,"value":26928,"toc":30151},[26929,26935,26938,26942,26945,27452,27738,27742,27906,27921,27981,27986,28080,28084,28678,28688,28949,29020,29024,29268,29272,29353,29358,29493,29497,29508,29512,29515,29599,29610,29614,29617,29623,29626,29629,29636,29640,29680,29683,30011,30013,30049,30051,30071,30082,30084,30096,30120,30133,30146,30149],[11,26930,26931],{},[57,26932],{"alt":26933,"src":26934},"A \"Flex Tape\" meme: at the top, an excited man labeled \"ME\" next to a jar leaking water captioned \"MODEL WORKING LESS ACCURATELY.\" At the bottom, a hand pressing Flex Tape onto a cracked pane that's still leaking a little water through it, labeled \"FEATURE ENGINEERING\"","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering\u002Fmeme-feature-engineering.jpeg",[11,26936,26937],{},"This is Week 2's fourth lab, and what I expected to be \"just another math trick\" ended up being the post that made me replan the most. The core idea fits in one sentence, but its consequences (including one the original lab never mentions at all) took over the whole post.",[294,26939,26941],{"id":26940},"the-core-idea-linear-in-the-parameters-not-the-feature","The core idea: linear in the parameters, not the feature",[11,26943,26944],{},"Linear regression fits models of the form",[11,26946,26947],{},[80,26948,26950,27042],{"className":26949},[83],[80,26951,26953],{"className":26952},[87],[89,26954,26955],{"xmlns":91},[93,26956,26957,27039],{},[96,26958,26959,26971,26973,26975,26977,26979,26985,26991,26993,26999,27005,27007,27009,27011,27023,27035,27037],{},[99,26960,26961,26963],{},[102,26962,104],{},[96,26964,26965,26967,26969],{},[102,26966,109],{"mathvariant":601},[111,26968,114],{"separator":113},[102,26970,117],{},[111,26972,121],{"stretchy":120},[102,26974,124],{"mathvariant":601},[111,26976,127],{"stretchy":120},[111,26978,130],{},[99,26980,26981,26983],{},[102,26982,109],{},[1321,26984,2071],{},[99,26986,26987,26989],{},[102,26988,124],{},[1321,26990,2071],{},[111,26992,141],{},[99,26994,26995,26997],{},[102,26996,109],{},[1321,26998,1583],{},[99,27000,27001,27003],{},[102,27002,124],{},[1321,27004,1583],{},[111,27006,141],{},[111,27008,8168],{},[111,27010,141],{},[99,27012,27013,27015],{},[102,27014,109],{},[96,27016,27017,27019,27021],{},[102,27018,1487],{},[111,27020,4643],{},[1321,27022,1583],{},[99,27024,27025,27027],{},[102,27026,124],{},[96,27028,27029,27031,27033],{},[102,27030,1487],{},[111,27032,4643],{},[1321,27034,1583],{},[111,27036,141],{},[102,27038,117],{},[145,27040,27041],{"encoding":147},"f_{\\mathbf{w},b}(\\mathbf{x}) = w_0x_0 + w_1x_1 + \\ldots + w_{n-1}x_{n-1} + b",[80,27043,27045,27118,27215,27310,27328,27443],{"className":27044,"ariaHidden":113},[152],[80,27046,27048,27051,27100,27103,27106,27109,27112,27115],{"className":27047},[156],[80,27049],{"className":27050,"style":161},[160],[80,27052,27054,27057],{"className":27053},[165],[80,27055,104],{"className":27056,"style":170},[165,169],[80,27058,27060],{"className":27059},[174],[80,27061,27063,27092],{"className":27062},[178,179],[80,27064,27066,27089],{"className":27065},[183],[80,27067,27069],{"className":27068,"style":188},[187],[80,27070,27071,27074],{"style":191},[80,27072],{"className":27073,"style":196},[195],[80,27075,27077],{"className":27076},[200,201,202,203],[80,27078,27080,27083,27086],{"className":27079},[165,203],[80,27081,109],{"className":27082,"style":700},[165,618,203],[80,27084,114],{"className":27085},[214,203],[80,27087,117],{"className":27088},[165,169,203],[80,27090,222],{"className":27091},[221],[80,27093,27095],{"className":27094},[183],[80,27096,27098],{"className":27097,"style":229},[187],[80,27099],{},[80,27101,121],{"className":27102},[235],[80,27104,124],{"className":27105},[165,618],[80,27107,127],{"className":27108},[242],[80,27110],{"className":27111,"style":247},[246],[80,27113,130],{"className":27114},[251],[80,27116],{"className":27117,"style":247},[246],[80,27119,27121,27125,27166,27206,27209,27212],{"className":27120},[156],[80,27122],{"className":27123,"style":27124},[160],"height:0.7333em;vertical-align:-0.15em;",[80,27126,27128,27131],{"className":27127},[165],[80,27129,109],{"className":27130,"style":210},[165,169],[80,27132,27134],{"className":27133},[174],[80,27135,27137,27158],{"className":27136},[178,179],[80,27138,27140,27155],{"className":27139},[183],[80,27141,27144],{"className":27142,"style":27143},[187],"height:0.3011em;",[80,27145,27146,27149],{"style":25144},[80,27147],{"className":27148,"style":196},[195],[80,27150,27152],{"className":27151},[200,201,202,203],[80,27153,2071],{"className":27154},[165,203],[80,27156,222],{"className":27157},[221],[80,27159,27161],{"className":27160},[183],[80,27162,27164],{"className":27163,"style":15345},[187],[80,27165],{},[80,27167,27169,27172],{"className":27168},[165],[80,27170,124],{"className":27171},[165,169],[80,27173,27175],{"className":27174},[174],[80,27176,27178,27198],{"className":27177},[178,179],[80,27179,27181,27195],{"className":27180},[183],[80,27182,27184],{"className":27183,"style":27143},[187],[80,27185,27186,27189],{"style":15326},[80,27187],{"className":27188,"style":196},[195],[80,27190,27192],{"className":27191},[200,201,202,203],[80,27193,2071],{"className":27194},[165,203],[80,27196,222],{"className":27197},[221],[80,27199,27201],{"className":27200},[183],[80,27202,27204],{"className":27203,"style":15345},[187],[80,27205],{},[80,27207],{"className":27208,"style":275},[246],[80,27210,141],{"className":27211},[279],[80,27213],{"className":27214,"style":275},[246],[80,27216,27218,27221,27261,27301,27304,27307],{"className":27217},[156],[80,27219],{"className":27220,"style":27124},[160],[80,27222,27224,27227],{"className":27223},[165],[80,27225,109],{"className":27226,"style":210},[165,169],[80,27228,27230],{"className":27229},[174],[80,27231,27233,27253],{"className":27232},[178,179],[80,27234,27236,27250],{"className":27235},[183],[80,27237,27239],{"className":27238,"style":27143},[187],[80,27240,27241,27244],{"style":25144},[80,27242],{"className":27243,"style":196},[195],[80,27245,27247],{"className":27246},[200,201,202,203],[80,27248,1583],{"className":27249},[165,203],[80,27251,222],{"className":27252},[221],[80,27254,27256],{"className":27255},[183],[80,27257,27259],{"className":27258,"style":15345},[187],[80,27260],{},[80,27262,27264,27267],{"className":27263},[165],[80,27265,124],{"className":27266},[165,169],[80,27268,27270],{"className":27269},[174],[80,27271,27273,27293],{"className":27272},[178,179],[80,27274,27276,27290],{"className":27275},[183],[80,27277,27279],{"className":27278,"style":27143},[187],[80,27280,27281,27284],{"style":15326},[80,27282],{"className":27283,"style":196},[195],[80,27285,27287],{"className":27286},[200,201,202,203],[80,27288,1583],{"className":27289},[165,203],[80,27291,222],{"className":27292},[221],[80,27294,27296],{"className":27295},[183],[80,27297,27299],{"className":27298,"style":15345},[187],[80,27300],{},[80,27302],{"className":27303,"style":275},[246],[80,27305,141],{"className":27306},[279],[80,27308],{"className":27309,"style":275},[246],[80,27311,27313,27316,27319,27322,27325],{"className":27312},[156],[80,27314],{"className":27315,"style":261},[160],[80,27317,8168],{"className":27318},[7482],[80,27320],{"className":27321,"style":275},[246],[80,27323,141],{"className":27324},[279],[80,27326],{"className":27327,"style":275},[246],[80,27329,27331,27335,27385,27434,27437,27440],{"className":27330},[156],[80,27332],{"className":27333,"style":27334},[160],"height:0.7917em;vertical-align:-0.2083em;",[80,27336,27338,27341],{"className":27337},[165],[80,27339,109],{"className":27340,"style":210},[165,169],[80,27342,27344],{"className":27343},[174],[80,27345,27347,27376],{"className":27346},[178,179],[80,27348,27350,27373],{"className":27349},[183],[80,27351,27353],{"className":27352,"style":27143},[187],[80,27354,27355,27358],{"style":25144},[80,27356],{"className":27357,"style":196},[195],[80,27359,27361],{"className":27360},[200,201,202,203],[80,27362,27364,27367,27370],{"className":27363},[165,203],[80,27365,1487],{"className":27366},[165,169,203],[80,27368,4643],{"className":27369},[279,203],[80,27371,1583],{"className":27372},[165,203],[80,27374,222],{"className":27375},[221],[80,27377,27379],{"className":27378},[183],[80,27380,27383],{"className":27381,"style":27382},[187],"height:0.2083em;",[80,27384],{},[80,27386,27388,27391],{"className":27387},[165],[80,27389,124],{"className":27390},[165,169],[80,27392,27394],{"className":27393},[174],[80,27395,27397,27426],{"className":27396},[178,179],[80,27398,27400,27423],{"className":27399},[183],[80,27401,27403],{"className":27402,"style":27143},[187],[80,27404,27405,27408],{"style":15326},[80,27406],{"className":27407,"style":196},[195],[80,27409,27411],{"className":27410},[200,201,202,203],[80,27412,27414,27417,27420],{"className":27413},[165,203],[80,27415,1487],{"className":27416},[165,169,203],[80,27418,4643],{"className":27419},[279,203],[80,27421,1583],{"className":27422},[165,203],[80,27424,222],{"className":27425},[221],[80,27427,27429],{"className":27428},[183],[80,27430,27432],{"className":27431,"style":27382},[187],[80,27433],{},[80,27435],{"className":27436,"style":275},[246],[80,27438,141],{"className":27439},[279],[80,27441],{"className":27442,"style":275},[246],[80,27444,27446,27449],{"className":27445},[156],[80,27447],{"className":27448,"style":289},[160],[80,27450,117],{"className":27451},[165,169],[11,27453,27454,27455,3255,27458,27500,27501,27572,27573,27631,27632,27676,27677,2338,27705,27733,27734,27737],{},"What makes this \"linear\" is the relationship between the output and the ",[15,27456,27457],{},"parameters",[80,27459,27461,27479],{"className":27460},[83],[80,27462,27464],{"className":27463},[87],[89,27465,27466],{"xmlns":91},[93,27467,27468,27476],{},[96,27469,27470,27472,27474],{},[102,27471,109],{"mathvariant":601},[111,27473,114],{"separator":113},[102,27475,117],{},[145,27477,27478],{"encoding":147},"\\mathbf{w}, b",[80,27480,27482],{"className":27481,"ariaHidden":113},[152],[80,27483,27485,27488,27491,27494,27497],{"className":27484},[156],[80,27486],{"className":27487,"style":1752},[160],[80,27489,109],{"className":27490,"style":700},[165,618],[80,27492,114],{"className":27493},[214],[80,27495],{"className":27496,"style":268},[246],[80,27498,117],{"className":27499},[165,169],", not the relationship between the output and the original variable. Nothing stops ",[80,27502,27504,27522],{"className":27503},[83],[80,27505,27507],{"className":27506},[87],[89,27508,27509],{"xmlns":91},[93,27510,27511,27519],{},[96,27512,27513],{},[99,27514,27515,27517],{},[102,27516,124],{},[1321,27518,1583],{},[145,27520,27521],{"encoding":147},"x_1",[80,27523,27525],{"className":27524,"ariaHidden":113},[152],[80,27526,27528,27532],{"className":27527},[156],[80,27529],{"className":27530,"style":27531},[160],"height:0.5806em;vertical-align:-0.15em;",[80,27533,27535,27538],{"className":27534},[165],[80,27536,124],{"className":27537},[165,169],[80,27539,27541],{"className":27540},[174],[80,27542,27544,27564],{"className":27543},[178,179],[80,27545,27547,27561],{"className":27546},[183],[80,27548,27550],{"className":27549,"style":27143},[187],[80,27551,27552,27555],{"style":15326},[80,27553],{"className":27554,"style":196},[195],[80,27556,27558],{"className":27557},[200,201,202,203],[80,27559,1583],{"className":27560},[165,203],[80,27562,222],{"className":27563},[221],[80,27565,27567],{"className":27566},[183],[80,27568,27570],{"className":27569,"style":15345},[187],[80,27571],{}," from being ",[80,27574,27576,27593],{"className":27575},[83],[80,27577,27579],{"className":27578},[87],[89,27580,27581],{"xmlns":91},[93,27582,27583,27591],{},[96,27584,27585],{},[726,27586,27587,27589],{},[102,27588,124],{},[1321,27590,1323],{},[145,27592,1397],{"encoding":147},[80,27594,27596],{"className":27595,"ariaHidden":113},[152],[80,27597,27599,27602],{"className":27598},[156],[80,27600],{"className":27601,"style":1407},[160],[80,27603,27605,27608],{"className":27604},[165],[80,27606,124],{"className":27607},[165,169],[80,27609,27611],{"className":27610},[174],[80,27612,27614],{"className":27613},[178],[80,27615,27617],{"className":27616},[183],[80,27618,27620],{"className":27619,"style":1407},[187],[80,27621,27622,27625],{"style":772},[80,27623],{"className":27624,"style":196},[195],[80,27626,27628],{"className":27627},[200,201,202,203],[80,27629,1323],{"className":27630},[165,203]," itself, or ",[80,27633,27635,27654],{"className":27634},[83],[80,27636,27638],{"className":27637},[87],[89,27639,27640],{"xmlns":91},[93,27641,27642,27651],{},[96,27643,27644,27647,27649],{},[102,27645,27646],{},"log",[111,27648,20987],{},[102,27650,124],{},[145,27652,27653],{"encoding":147},"\\log x",[80,27655,27657],{"className":27656,"ariaHidden":113},[152],[80,27658,27660,27663,27670,27673],{"className":27659},[156],[80,27661],{"className":27662,"style":1752},[160],[80,27664,27666,27667],{"className":27665},[7402],"lo",[80,27668,25196],{"style":27669},"margin-right:0.0139em;",[80,27671],{"className":27672,"style":268},[246],[80,27674,124],{"className":27675},[165,169],", or the product of two other columns. No matter how I tune ",[80,27678,27680,27693],{"className":27679},[83],[80,27681,27683],{"className":27682},[87],[89,27684,27685],{"xmlns":91},[93,27686,27687,27691],{},[96,27688,27689],{},[102,27690,109],{},[145,27692,109],{"encoding":147},[80,27694,27696],{"className":27695,"ariaHidden":113},[152],[80,27697,27699,27702],{"className":27698},[156],[80,27700],{"className":27701,"style":334},[160],[80,27703,109],{"className":27704,"style":210},[165,169],[80,27706,27708,27721],{"className":27707},[83],[80,27709,27711],{"className":27710},[87],[89,27712,27713],{"xmlns":91},[93,27714,27715,27719],{},[96,27716,27717],{},[102,27718,117],{},[145,27720,117],{"encoding":147},[80,27722,27724],{"className":27723,"ariaHidden":113},[152],[80,27725,27727,27730],{"className":27726},[156],[80,27728],{"className":27729,"style":289},[160],[80,27731,117],{"className":27732},[165,169],", the equation never becomes a curve if the features are straight lines. But if I ",[15,27735,27736],{},"engineer"," a curved feature and plug it in, the exact same machinery fits the curve. That's feature engineering.",[294,27739,27741],{"id":27740},"polynomial-features-in-practice","Polynomial features in practice",[11,27743,27744,27745,27848,27849,27877,27878,1618],{},"I tested this with a simple target: ",[80,27746,27748,27774],{"className":27747},[83],[80,27749,27751],{"className":27750},[87],[89,27752,27753],{"xmlns":91},[93,27754,27755,27771],{},[96,27756,27757,27759,27761,27763,27765],{},[102,27758,683],{},[111,27760,130],{},[1321,27762,1583],{},[111,27764,141],{},[726,27766,27767,27769],{},[102,27768,124],{},[1321,27770,1323],{},[145,27772,27773],{"encoding":147},"y = 1 + x^2",[80,27775,27777,27795,27813],{"className":27776,"ariaHidden":113},[152],[80,27778,27780,27783,27786,27789,27792],{"className":27779},[156],[80,27781],{"className":27782,"style":4514},[160],[80,27784,683],{"className":27785,"style":834},[165,169],[80,27787],{"className":27788,"style":247},[246],[80,27790,130],{"className":27791},[251],[80,27793],{"className":27794,"style":247},[246],[80,27796,27798,27801,27804,27807,27810],{"className":27797},[156],[80,27799],{"className":27800,"style":5303},[160],[80,27802,1583],{"className":27803},[165],[80,27805],{"className":27806,"style":275},[246],[80,27808,141],{"className":27809},[279],[80,27811],{"className":27812,"style":275},[246],[80,27814,27816,27819],{"className":27815},[156],[80,27817],{"className":27818,"style":1407},[160],[80,27820,27822,27825],{"className":27821},[165],[80,27823,124],{"className":27824},[165,169],[80,27826,27828],{"className":27827},[174],[80,27829,27831],{"className":27830},[178],[80,27832,27834],{"className":27833},[183],[80,27835,27837],{"className":27836,"style":1407},[187],[80,27838,27839,27842],{"style":772},[80,27840],{"className":27841,"style":196},[195],[80,27843,27845],{"className":27844},[200,201,202,203],[80,27846,1323],{"className":27847},[165,203],", for ",[80,27850,27852,27865],{"className":27851},[83],[80,27853,27855],{"className":27854},[87],[89,27856,27857],{"xmlns":91},[93,27858,27859,27863],{},[96,27860,27861],{},[102,27862,124],{},[145,27864,124],{"encoding":147},[80,27866,27868],{"className":27867,"ariaHidden":113},[152],[80,27869,27871,27874],{"className":27870},[156],[80,27872],{"className":27873,"style":334},[160],[80,27875,124],{"className":27876},[165,169]," from 0 to 19. First, the naive attempt: feeding the model raw ",[80,27879,27881,27894],{"className":27880},[83],[80,27882,27884],{"className":27883},[87],[89,27885,27886],{"xmlns":91},[93,27887,27888,27892],{},[96,27889,27890],{},[102,27891,124],{},[145,27893,124],{"encoding":147},[80,27895,27897],{"className":27896,"ariaHidden":113},[152],[80,27898,27900,27903],{"className":27899},[156],[80,27901],{"className":27902,"style":334},[160],[80,27904,124],{"className":27905},[165,169],[2611,27907,27909],{"className":2613,"code":27908,"language":2615,"meta":26,"style":26},"x = list(range(20))\ny = [1 + xi**2 for xi in x]\n",[65,27910,27911,27916],{"__ignoreMap":26},[80,27912,27913],{"class":2620,"line":33},[80,27914,27915],{},"x = list(range(20))\n",[80,27917,27918],{"class":2620,"line":27},[80,27919,27920],{},"y = [1 + xi**2 for xi in x]\n",[11,27922,27923,27924,2338,27952,27980],{},"Drag the degree to 1 in the simulator below and notice: no value of ",[80,27925,27927,27940],{"className":27926},[83],[80,27928,27930],{"className":27929},[87],[89,27931,27932],{"xmlns":91},[93,27933,27934,27938],{},[96,27935,27936],{},[102,27937,109],{},[145,27939,109],{"encoding":147},[80,27941,27943],{"className":27942,"ariaHidden":113},[152],[80,27944,27946,27949],{"className":27945},[156],[80,27947],{"className":27948,"style":334},[160],[80,27950,109],{"className":27951,"style":210},[165,169],[80,27953,27955,27968],{"className":27954},[83],[80,27956,27958],{"className":27957},[87],[89,27959,27960],{"xmlns":91},[93,27961,27962,27966],{},[96,27963,27964],{},[102,27965,117],{},[145,27967,117],{"encoding":147},[80,27969,27971],{"className":27970,"ariaHidden":113},[152],[80,27972,27974,27977],{"className":27973},[156],[80,27975],{"className":27976,"style":289},[160],[80,27978,117],{"className":27979},[165,169]," fixes this, because the model is a line and the target is a parabola. Now drag it to 2:",[27982,27983],"polynomial-fit-explorer",{":initial-degree":1583,":max-degree":17445,":x-train":27984,":y-train":27985,"x-label":124,"y-label":683},"[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]","[1,2,5,10,17,26,37,50,65,82,101,122,145,170,197,226,257,290,325,362]",[11,27987,27988,27989,27991,27992,28020,28021,28079],{},"At degree 2 the fit becomes exact, zero RMSE. What I swapped wasn't the algorithm, it was the ",[15,27990,6518],{},": instead of ",[80,27993,27995,28008],{"className":27994},[83],[80,27996,27998],{"className":27997},[87],[89,27999,28000],{"xmlns":91},[93,28001,28002,28006],{},[96,28003,28004],{},[102,28005,124],{},[145,28007,124],{"encoding":147},[80,28009,28011],{"className":28010,"ariaHidden":113},[152],[80,28012,28014,28017],{"className":28013},[156],[80,28015],{"className":28016,"style":334},[160],[80,28018,124],{"className":28019},[165,169],", I fed in ",[80,28022,28024,28041],{"className":28023},[83],[80,28025,28027],{"className":28026},[87],[89,28028,28029],{"xmlns":91},[93,28030,28031,28039],{},[96,28032,28033],{},[726,28034,28035,28037],{},[102,28036,124],{},[1321,28038,1323],{},[145,28040,1397],{"encoding":147},[80,28042,28044],{"className":28043,"ariaHidden":113},[152],[80,28045,28047,28050],{"className":28046},[156],[80,28048],{"className":28049,"style":1407},[160],[80,28051,28053,28056],{"className":28052},[165],[80,28054,124],{"className":28055},[165,169],[80,28057,28059],{"className":28058},[174],[80,28060,28062],{"className":28061},[178],[80,28063,28065],{"className":28064},[183],[80,28066,28068],{"className":28067,"style":1407},[187],[80,28069,28070,28073],{"style":772},[80,28071],{"className":28072,"style":196},[195],[80,28074,28076],{"className":28075},[200,201,202,203],[80,28077,1323],{"className":28078},[165,203]," (plus the rest of the polynomial) and let the same old linear regression find the right weight.",[294,28081,28083],{"id":28082},"choosing-features-without-knowing-the-answer-beforehand","Choosing features, without knowing the answer beforehand",[11,28085,28086,28087,28090,28091,28149,28150,28466,28467,28525,28526,28554,28555,28614,28615,28673,28674,28677],{},"Up there I already ",[15,28088,28089],{},"knew"," the right term was ",[80,28092,28094,28111],{"className":28093},[83],[80,28095,28097],{"className":28096},[87],[89,28098,28099],{"xmlns":91},[93,28100,28101,28109],{},[96,28102,28103],{},[726,28104,28105,28107],{},[102,28106,124],{},[1321,28108,1323],{},[145,28110,1397],{"encoding":147},[80,28112,28114],{"className":28113,"ariaHidden":113},[152],[80,28115,28117,28120],{"className":28116},[156],[80,28118],{"className":28119,"style":1407},[160],[80,28121,28123,28126],{"className":28122},[165],[80,28124,124],{"className":28125},[165,169],[80,28127,28129],{"className":28128},[174],[80,28130,28132],{"className":28131},[178],[80,28133,28135],{"className":28134},[183],[80,28136,28138],{"className":28137,"style":1407},[187],[80,28139,28140,28143],{"style":772},[80,28141],{"className":28142,"style":196},[195],[80,28144,28146],{"className":28145},[200,201,202,203],[80,28147,1323],{"className":28148},[165,203],". In practice, you don't. One strategy is to throw several candidates at it and let the fit decide: I tested ",[80,28151,28153,28209],{"className":28152},[83],[80,28154,28156],{"className":28155},[87],[89,28157,28158],{"xmlns":91},[93,28159,28160,28206],{},[96,28161,28162,28164,28166,28172,28174,28176,28182,28188,28190,28196,28202,28204],{},[102,28163,683],{},[111,28165,130],{},[99,28167,28168,28170],{},[102,28169,109],{},[1321,28171,2071],{},[102,28173,124],{},[111,28175,141],{},[99,28177,28178,28180],{},[102,28179,109],{},[1321,28181,1583],{},[726,28183,28184,28186],{},[102,28185,124],{},[1321,28187,1323],{},[111,28189,141],{},[99,28191,28192,28194],{},[102,28193,109],{},[1321,28195,1323],{},[726,28197,28198,28200],{},[102,28199,124],{},[1321,28201,13895],{},[111,28203,141],{},[102,28205,117],{},[145,28207,28208],{"encoding":147},"y = w_0x + w_1x^2 + w_2x^3 + b",[80,28210,28212,28230,28288,28373,28457],{"className":28211,"ariaHidden":113},[152],[80,28213,28215,28218,28221,28224,28227],{"className":28214},[156],[80,28216],{"className":28217,"style":4514},[160],[80,28219,683],{"className":28220,"style":834},[165,169],[80,28222],{"className":28223,"style":247},[246],[80,28225,130],{"className":28226},[251],[80,28228],{"className":28229,"style":247},[246],[80,28231,28233,28236,28276,28279,28282,28285],{"className":28232},[156],[80,28234],{"className":28235,"style":27124},[160],[80,28237,28239,28242],{"className":28238},[165],[80,28240,109],{"className":28241,"style":210},[165,169],[80,28243,28245],{"className":28244},[174],[80,28246,28248,28268],{"className":28247},[178,179],[80,28249,28251,28265],{"className":28250},[183],[80,28252,28254],{"className":28253,"style":27143},[187],[80,28255,28256,28259],{"style":25144},[80,28257],{"className":28258,"style":196},[195],[80,28260,28262],{"className":28261},[200,201,202,203],[80,28263,2071],{"className":28264},[165,203],[80,28266,222],{"className":28267},[221],[80,28269,28271],{"className":28270},[183],[80,28272,28274],{"className":28273,"style":15345},[187],[80,28275],{},[80,28277,124],{"className":28278},[165,169],[80,28280],{"className":28281,"style":275},[246],[80,28283,141],{"className":28284},[279],[80,28286],{"className":28287,"style":275},[246],[80,28289,28291,28295,28335,28364,28367,28370],{"className":28290},[156],[80,28292],{"className":28293,"style":28294},[160],"height:0.9641em;vertical-align:-0.15em;",[80,28296,28298,28301],{"className":28297},[165],[80,28299,109],{"className":28300,"style":210},[165,169],[80,28302,28304],{"className":28303},[174],[80,28305,28307,28327],{"className":28306},[178,179],[80,28308,28310,28324],{"className":28309},[183],[80,28311,28313],{"className":28312,"style":27143},[187],[80,28314,28315,28318],{"style":25144},[80,28316],{"className":28317,"style":196},[195],[80,28319,28321],{"className":28320},[200,201,202,203],[80,28322,1583],{"className":28323},[165,203],[80,28325,222],{"className":28326},[221],[80,28328,28330],{"className":28329},[183],[80,28331,28333],{"className":28332,"style":15345},[187],[80,28334],{},[80,28336,28338,28341],{"className":28337},[165],[80,28339,124],{"className":28340},[165,169],[80,28342,28344],{"className":28343},[174],[80,28345,28347],{"className":28346},[178],[80,28348,28350],{"className":28349},[183],[80,28351,28353],{"className":28352,"style":1407},[187],[80,28354,28355,28358],{"style":772},[80,28356],{"className":28357,"style":196},[195],[80,28359,28361],{"className":28360},[200,201,202,203],[80,28362,1323],{"className":28363},[165,203],[80,28365],{"className":28366,"style":275},[246],[80,28368,141],{"className":28369},[279],[80,28371],{"className":28372,"style":275},[246],[80,28374,28376,28379,28419,28448,28451,28454],{"className":28375},[156],[80,28377],{"className":28378,"style":28294},[160],[80,28380,28382,28385],{"className":28381},[165],[80,28383,109],{"className":28384,"style":210},[165,169],[80,28386,28388],{"className":28387},[174],[80,28389,28391,28411],{"className":28390},[178,179],[80,28392,28394,28408],{"className":28393},[183],[80,28395,28397],{"className":28396,"style":27143},[187],[80,28398,28399,28402],{"style":25144},[80,28400],{"className":28401,"style":196},[195],[80,28403,28405],{"className":28404},[200,201,202,203],[80,28406,1323],{"className":28407},[165,203],[80,28409,222],{"className":28410},[221],[80,28412,28414],{"className":28413},[183],[80,28415,28417],{"className":28416,"style":15345},[187],[80,28418],{},[80,28420,28422,28425],{"className":28421},[165],[80,28423,124],{"className":28424},[165,169],[80,28426,28428],{"className":28427},[174],[80,28429,28431],{"className":28430},[178],[80,28432,28434],{"className":28433},[183],[80,28435,28437],{"className":28436,"style":1407},[187],[80,28438,28439,28442],{"style":772},[80,28440],{"className":28441,"style":196},[195],[80,28443,28445],{"className":28444},[200,201,202,203],[80,28446,13895],{"className":28447},[165,203],[80,28449],{"className":28450,"style":275},[246],[80,28452,141],{"className":28453},[279],[80,28455],{"className":28456,"style":275},[246],[80,28458,28460,28463],{"className":28459},[156],[80,28461],{"className":28462,"style":289},[160],[80,28464,117],{"className":28465},[165,169]," against a pure ",[80,28468,28470,28487],{"className":28469},[83],[80,28471,28473],{"className":28472},[87],[89,28474,28475],{"xmlns":91},[93,28476,28477,28485],{},[96,28478,28479],{},[726,28480,28481,28483],{},[102,28482,124],{},[1321,28484,1323],{},[145,28486,1397],{"encoding":147},[80,28488,28490],{"className":28489,"ariaHidden":113},[152],[80,28491,28493,28496],{"className":28492},[156],[80,28494],{"className":28495,"style":1407},[160],[80,28497,28499,28502],{"className":28498},[165],[80,28500,124],{"className":28501},[165,169],[80,28503,28505],{"className":28504},[174],[80,28506,28508],{"className":28507},[178],[80,28509,28511],{"className":28510},[183],[80,28512,28514],{"className":28513,"style":1407},[187],[80,28515,28516,28519],{"style":772},[80,28517],{"className":28518,"style":196},[195],[80,28520,28522],{"className":28521},[200,201,202,203],[80,28523,1323],{"className":28524},[165,203]," target, and the solver zeroed out ",[80,28527,28529,28542],{"className":28528},[83],[80,28530,28532],{"className":28531},[87],[89,28533,28534],{"xmlns":91},[93,28535,28536,28540],{},[96,28537,28538],{},[102,28539,124],{},[145,28541,124],{"encoding":147},[80,28543,28545],{"className":28544,"ariaHidden":113},[152],[80,28546,28548,28551],{"className":28547},[156],[80,28549],{"className":28550,"style":334},[160],[80,28552,124],{"className":28553},[165,169],"'s and ",[80,28556,28558,28576],{"className":28557},[83],[80,28559,28561],{"className":28560},[87],[89,28562,28563],{"xmlns":91},[93,28564,28565,28573],{},[96,28566,28567],{},[726,28568,28569,28571],{},[102,28570,124],{},[1321,28572,13895],{},[145,28574,28575],{"encoding":147},"x^3",[80,28577,28579],{"className":28578,"ariaHidden":113},[152],[80,28580,28582,28585],{"className":28581},[156],[80,28583],{"className":28584,"style":1407},[160],[80,28586,28588,28591],{"className":28587},[165],[80,28589,124],{"className":28590},[165,169],[80,28592,28594],{"className":28593},[174],[80,28595,28597],{"className":28596},[178],[80,28598,28600],{"className":28599},[183],[80,28601,28603],{"className":28602,"style":1407},[187],[80,28604,28605,28608],{"style":772},[80,28606],{"className":28607,"style":196},[195],[80,28609,28611],{"className":28610},[200,201,202,203],[80,28612,13895],{"className":28613},[165,203],"'s weights on its own, leaving basically only ",[80,28616,28618,28635],{"className":28617},[83],[80,28619,28621],{"className":28620},[87],[89,28622,28623],{"xmlns":91},[93,28624,28625,28633],{},[96,28626,28627],{},[726,28628,28629,28631],{},[102,28630,124],{},[1321,28632,1323],{},[145,28634,1397],{"encoding":147},[80,28636,28638],{"className":28637,"ariaHidden":113},[152],[80,28639,28641,28644],{"className":28640},[156],[80,28642],{"className":28643,"style":1407},[160],[80,28645,28647,28650],{"className":28646},[165],[80,28648,124],{"className":28649},[165,169],[80,28651,28653],{"className":28652},[174],[80,28654,28656],{"className":28655},[178],[80,28657,28659],{"className":28658},[183],[80,28660,28662],{"className":28661,"style":1407},[187],[80,28663,28664,28667],{"style":772},[80,28665],{"className":28666,"style":196},[195],[80,28668,28670],{"className":28669},[200,201,202,203],[80,28671,1323],{"className":28672},[165,203],"'s standing. (The original notebook runs this same test with gradient descent and only manages to ",[73,28675,28676],{},"shrink"," the wrong weights, not zero them, because convergence never fully finishes. With an exact solution, the result comes out clean.)",[11,28679,28680,28681,28684,28685,28687],{},"Another way to think about it: after creating the features, I'm still doing ",[15,28682,28683],{},"linear"," regression. So the best feature is the one with a ",[15,28686,28683],{}," relationship to the target. That turns into an easy-to-compute Pearson correlation:",[521,28689,28690,28780],{},[524,28691,28692],{},[527,28693,28694,28696],{},[530,28695,18553],{"align":532},[530,28697,28698,28699],{"align":18556},"Correlation with ",[80,28700,28702,28724],{"className":28701},[83],[80,28703,28705],{"className":28704},[87],[89,28706,28707],{"xmlns":91},[93,28708,28709,28721],{},[96,28710,28711,28713,28715],{},[102,28712,683],{},[111,28714,130],{},[726,28716,28717,28719],{},[102,28718,124],{},[1321,28720,1323],{},[145,28722,28723],{"encoding":147},"y=x^2",[80,28725,28727,28745],{"className":28726,"ariaHidden":113},[152],[80,28728,28730,28733,28736,28739,28742],{"className":28729},[156],[80,28731],{"className":28732,"style":4514},[160],[80,28734,683],{"className":28735,"style":834},[165,169],[80,28737],{"className":28738,"style":247},[246],[80,28740,130],{"className":28741},[251],[80,28743],{"className":28744,"style":247},[246],[80,28746,28748,28751],{"className":28747},[156],[80,28749],{"className":28750,"style":1407},[160],[80,28752,28754,28757],{"className":28753},[165],[80,28755,124],{"className":28756},[165,169],[80,28758,28760],{"className":28759},[174],[80,28761,28763],{"className":28762},[178],[80,28764,28766],{"className":28765},[183],[80,28767,28769],{"className":28768,"style":1407},[187],[80,28770,28771,28774],{"style":772},[80,28772],{"className":28773,"style":196},[195],[80,28775,28777],{"className":28776},[200,201,202,203],[80,28778,1323],{"className":28779},[165,203],[541,28781,28782,28817,28884],{},[527,28783,28784,28814],{},[546,28785,28786],{"align":532},[80,28787,28789,28802],{"className":28788},[83],[80,28790,28792],{"className":28791},[87],[89,28793,28794],{"xmlns":91},[93,28795,28796,28800],{},[96,28797,28798],{},[102,28799,124],{},[145,28801,124],{"encoding":147},[80,28803,28805],{"className":28804,"ariaHidden":113},[152],[80,28806,28808,28811],{"className":28807},[156],[80,28809],{"className":28810,"style":334},[160],[80,28812,124],{"className":28813},[165,169],[546,28815,28816],{"align":18556},"0.965",[527,28818,28819,28879],{},[546,28820,28821],{"align":532},[80,28822,28824,28841],{"className":28823},[83],[80,28825,28827],{"className":28826},[87],[89,28828,28829],{"xmlns":91},[93,28830,28831,28839],{},[96,28832,28833],{},[726,28834,28835,28837],{},[102,28836,124],{},[1321,28838,1323],{},[145,28840,1397],{"encoding":147},[80,28842,28844],{"className":28843,"ariaHidden":113},[152],[80,28845,28847,28850],{"className":28846},[156],[80,28848],{"className":28849,"style":1407},[160],[80,28851,28853,28856],{"className":28852},[165],[80,28854,124],{"className":28855},[165,169],[80,28857,28859],{"className":28858},[174],[80,28860,28862],{"className":28861},[178],[80,28863,28865],{"className":28864},[183],[80,28866,28868],{"className":28867,"style":1407},[187],[80,28869,28870,28873],{"style":772},[80,28871],{"className":28872,"style":196},[195],[80,28874,28876],{"className":28875},[200,201,202,203],[80,28877,1323],{"className":28878},[165,203],[546,28880,28881],{"align":18556},[15,28882,28883],{},"1.000",[527,28885,28886,28946],{},[546,28887,28888],{"align":532},[80,28889,28891,28908],{"className":28890},[83],[80,28892,28894],{"className":28893},[87],[89,28895,28896],{"xmlns":91},[93,28897,28898,28906],{},[96,28899,28900],{},[726,28901,28902,28904],{},[102,28903,124],{},[1321,28905,13895],{},[145,28907,28575],{"encoding":147},[80,28909,28911],{"className":28910,"ariaHidden":113},[152],[80,28912,28914,28917],{"className":28913},[156],[80,28915],{"className":28916,"style":1407},[160],[80,28918,28920,28923],{"className":28919},[165],[80,28921,124],{"className":28922},[165,169],[80,28924,28926],{"className":28925},[174],[80,28927,28929],{"className":28928},[178],[80,28930,28932],{"className":28931},[183],[80,28933,28935],{"className":28934,"style":1407},[187],[80,28936,28937,28940],{"style":772},[80,28938],{"className":28939,"style":196},[195],[80,28941,28943],{"className":28942},[200,201,202,203],[80,28944,13895],{"className":28945},[165,203],[546,28947,28948],{"align":18556},"0.986",[11,28950,28951,29009,29010,29012,29013,29016,29017,29019],{},[80,28952,28954,28971],{"className":28953},[83],[80,28955,28957],{"className":28956},[87],[89,28958,28959],{"xmlns":91},[93,28960,28961,28969],{},[96,28962,28963],{},[726,28964,28965,28967],{},[102,28966,124],{},[1321,28968,1323],{},[145,28970,1397],{"encoding":147},[80,28972,28974],{"className":28973,"ariaHidden":113},[152],[80,28975,28977,28980],{"className":28976},[156],[80,28978],{"className":28979,"style":1407},[160],[80,28981,28983,28986],{"className":28982},[165],[80,28984,124],{"className":28985},[165,169],[80,28987,28989],{"className":28988},[174],[80,28990,28992],{"className":28991},[178],[80,28993,28995],{"className":28994},[183],[80,28996,28998],{"className":28997,"style":1407},[187],[80,28999,29000,29003],{"style":772},[80,29001],{"className":29002,"style":196},[195],[80,29004,29006],{"className":29005},[200,201,202,203],[80,29007,1323],{"className":29008},[165,203]," has perfect correlation because it ",[15,29011,75],{}," the target, up to scale. I already mentioned this same heuristic (plot feature against target, look for the line) in ",[562,29014,29015],{"href":26916},"the normalization post",", it's the same idea, just now applied to choose the right ",[73,29018,14750],{}," of the feature, not just its scale.",[294,29021,29023],{"id":29022},"normalizing-again-the-extreme-version","Normalizing, again (the extreme version)",[11,29025,29026,29027,29055,29056,29114,29115,29173,29174,29177,29178,2338,29206,29264,29265,29267],{},"Polynomial features are the most extreme case of different scales I've seen so far: ",[80,29028,29030,29043],{"className":29029},[83],[80,29031,29033],{"className":29032},[87],[89,29034,29035],{"xmlns":91},[93,29036,29037,29041],{},[96,29038,29039],{},[102,29040,124],{},[145,29042,124],{"encoding":147},[80,29044,29046],{"className":29045,"ariaHidden":113},[152],[80,29047,29049,29052],{"className":29048},[156],[80,29050],{"className":29051,"style":334},[160],[80,29053,124],{"className":29054},[165,169]," goes up to 19, ",[80,29057,29059,29076],{"className":29058},[83],[80,29060,29062],{"className":29061},[87],[89,29063,29064],{"xmlns":91},[93,29065,29066,29074],{},[96,29067,29068],{},[726,29069,29070,29072],{},[102,29071,124],{},[1321,29073,1323],{},[145,29075,1397],{"encoding":147},[80,29077,29079],{"className":29078,"ariaHidden":113},[152],[80,29080,29082,29085],{"className":29081},[156],[80,29083],{"className":29084,"style":1407},[160],[80,29086,29088,29091],{"className":29087},[165],[80,29089,124],{"className":29090},[165,169],[80,29092,29094],{"className":29093},[174],[80,29095,29097],{"className":29096},[178],[80,29098,29100],{"className":29099},[183],[80,29101,29103],{"className":29102,"style":1407},[187],[80,29104,29105,29108],{"style":772},[80,29106],{"className":29107,"style":196},[195],[80,29109,29111],{"className":29110},[200,201,202,203],[80,29112,1323],{"className":29113},[165,203]," up to 361, ",[80,29116,29118,29135],{"className":29117},[83],[80,29119,29121],{"className":29120},[87],[89,29122,29123],{"xmlns":91},[93,29124,29125,29133],{},[96,29126,29127],{},[726,29128,29129,29131],{},[102,29130,124],{},[1321,29132,13895],{},[145,29134,28575],{"encoding":147},[80,29136,29138],{"className":29137,"ariaHidden":113},[152],[80,29139,29141,29144],{"className":29140},[156],[80,29142],{"className":29143,"style":1407},[160],[80,29145,29147,29150],{"className":29146},[165],[80,29148,124],{"className":29149},[165,169],[80,29151,29153],{"className":29152},[174],[80,29154,29156],{"className":29155},[178],[80,29157,29159],{"className":29158},[183],[80,29160,29162],{"className":29161,"style":1407},[187],[80,29163,29164,29167],{"style":772},[80,29165],{"className":29166,"style":196},[195],[80,29168,29170],{"className":29169},[200,201,202,203],[80,29171,13895],{"className":29172},[165,203]," up to 6859, a ratio of ",[15,29175,29176],{},"361 times"," between ",[80,29179,29181,29194],{"className":29180},[83],[80,29182,29184],{"className":29183},[87],[89,29185,29186],{"xmlns":91},[93,29187,29188,29192],{},[96,29189,29190],{},[102,29191,124],{},[145,29193,124],{"encoding":147},[80,29195,29197],{"className":29196,"ariaHidden":113},[152],[80,29198,29200,29203],{"className":29199},[156],[80,29201],{"className":29202,"style":334},[160],[80,29204,124],{"className":29205},[165,169],[80,29207,29209,29226],{"className":29208},[83],[80,29210,29212],{"className":29211},[87],[89,29213,29214],{"xmlns":91},[93,29215,29216,29224],{},[96,29217,29218],{},[726,29219,29220,29222],{},[102,29221,124],{},[1321,29223,1323],{},[145,29225,1397],{"encoding":147},[80,29227,29229],{"className":29228,"ariaHidden":113},[152],[80,29230,29232,29235],{"className":29231},[156],[80,29233],{"className":29234,"style":1407},[160],[80,29236,29238,29241],{"className":29237},[165],[80,29239,124],{"className":29240},[165,169],[80,29242,29244],{"className":29243},[174],[80,29245,29247],{"className":29246},[178],[80,29248,29250],{"className":29249},[183],[80,29251,29253],{"className":29252,"style":1407},[187],[80,29254,29255,29258],{"style":772},[80,29256],{"className":29257,"style":196},[195],[80,29259,29261],{"className":29260},[200,201,202,203],[80,29262,1323],{"className":29263},[165,203]," alone. The same trick from ",[562,29266,19687],{"href":26916}," fixes it: z-score each column, computed from training data only. The component I built for this post already normalizes under the hood before solving, exactly like I did back there.",[294,29269,29271],{"id":29270},"a-genuinely-complicated-function","A genuinely complicated function",[11,29273,29274,29275,29352],{},"With feature engineering I can model much wilder things than a parabola. I tested ",[80,29276,29278,29309],{"className":29277},[83],[80,29279,29281],{"className":29280},[87],[89,29282,29283],{"xmlns":91},[93,29284,29285,29306],{},[96,29286,29287,29289,29291,29294,29296,29298,29300,29302,29304],{},[102,29288,683],{},[111,29290,130],{},[102,29292,29293],{},"cos",[111,29295,20987],{},[111,29297,121],{"stretchy":120},[102,29299,124],{},[102,29301,20914],{"mathvariant":10558},[1321,29303,1323],{},[111,29305,127],{"stretchy":120},[145,29307,29308],{"encoding":147},"y = \\cos(x\u002F2)",[80,29310,29312,29330],{"className":29311,"ariaHidden":113},[152],[80,29313,29315,29318,29321,29324,29327],{"className":29314},[156],[80,29316],{"className":29317,"style":4514},[160],[80,29319,683],{"className":29320,"style":834},[165,169],[80,29322],{"className":29323,"style":247},[246],[80,29325,130],{"className":29326},[251],[80,29328],{"className":29329,"style":247},[246],[80,29331,29333,29336,29339,29342,29345,29349],{"className":29332},[156],[80,29334],{"className":29335,"style":2316},[160],[80,29337,29293],{"className":29338},[7402],[80,29340,121],{"className":29341},[235],[80,29343,124],{"className":29344},[165,169],[80,29346,29348],{"className":29347},[165],"\u002F2",[80,29350,127],{"className":29351},[242]," with a polynomial up to degree 13:",[27982,29354],{":initial-degree":1583,":max-degree":29355,":x-train":27984,":y-train":29356,"x-label":124,"y-label":29357},"13","[1.0,0.877583,0.540302,0.070737,-0.416147,-0.801144,-0.989992,-0.936457,-0.653644,-0.210796,0.283662,0.70867,0.96017,0.976588,0.753902,0.346635,-0.1455,-0.602012,-0.91113,-0.997172]","y = cos(x\u002F2)",[11,29359,29360,29361,29456,29457,29485,29486,29489,29490,29492],{},"Drag it up to degree 13: RMSE drops to ",[80,29362,29364,29391],{"className":29363},[83],[80,29365,29367],{"className":29366},[87],[89,29368,29369],{"xmlns":91},[93,29370,29371,29388],{},[96,29372,29373,29376,29378],{},[1321,29374,29375],{},"1.1",[111,29377,2834],{},[726,29379,29380,29382],{},[1321,29381,17444],{},[96,29383,29384,29386],{},[111,29385,4643],{},[1321,29387,15096],{},[145,29389,29390],{"encoding":147},"1.1\\times10^{-5}",[80,29392,29394,29412],{"className":29393,"ariaHidden":113},[152],[80,29395,29397,29400,29403,29406,29409],{"className":29396},[156],[80,29398],{"className":29399,"style":5303},[160],[80,29401,29375],{"className":29402},[165],[80,29404],{"className":29405,"style":275},[246],[80,29407,2834],{"className":29408},[279],[80,29410],{"className":29411,"style":275},[246],[80,29413,29415,29418,29421],{"className":29414},[156],[80,29416],{"className":29417,"style":1407},[160],[80,29419,1583],{"className":29420},[165],[80,29422,29424,29427],{"className":29423},[165],[80,29425,2071],{"className":29426},[165],[80,29428,29430],{"className":29429},[174],[80,29431,29433],{"className":29432},[178],[80,29434,29436],{"className":29435},[183],[80,29437,29439],{"className":29438,"style":1407},[187],[80,29440,29441,29444],{"style":772},[80,29442],{"className":29443,"style":196},[195],[80,29445,29447],{"className":29446},[200,201,202,203],[80,29448,29450,29453],{"className":29449},[165,203],[80,29451,4643],{"className":29452},[165,203],[80,29454,15096],{"className":29455},[165,203],", a nearly perfect fit on the 20 training points. And here I deliberately swapped tools: instead of gradient descent with an ",[80,29458,29460,29473],{"className":29459},[83],[80,29461,29463],{"className":29462},[87],[89,29464,29465],{"xmlns":91},[93,29466,29467,29471],{},[96,29468,29469],{},[102,29470,10551],{},[145,29472,11037],{"encoding":147},[80,29474,29476],{"className":29475,"ariaHidden":113},[152],[80,29477,29479,29482],{"className":29478},[156],[80,29480],{"className":29481,"style":334},[160],[80,29483,10551],{"className":29484,"style":10667},[165,169]," hand-tuned for every degree (what the original notebook does, and it's a lot of work), this component solves via the ",[15,29487,29488],{},"normal equation",", the same exact solution I already used to check results in ",[562,29491,19687],{"href":26916},". That makes exploring many degrees instant, no alpha-hunting needed, leaving all the attention for what actually matters here: what happens once the model gets too flexible.",[294,29494,29496],{"id":29495},"the-elephant-in-the-room-overfitting","The elephant in the room: overfitting",[11,29498,29499,29500,29503,29504,29507],{},"I just fit ",[15,29501,29502],{},"14 parameters"," (13 weights + the bias) to ",[15,29505,29506],{},"20 data points",". The fit looked gorgeous. That should raise an alarm, not a celebration, and that's exactly what the original lab never does.",[2606,29509,29511],{"id":29510},"first-surprise-without-noise-flexibility-isnt-a-sin","First surprise: without noise, flexibility isn't a sin",[11,29513,29514],{},"I split the 20 points into 10 train (even indices) and 10 test (odd indices), with zero noise in the data:",[521,29516,29517,29533],{},[524,29518,29519],{},[527,29520,29521,29524,29527,29530],{},[530,29522,29523],{"align":1884},"Degree",[530,29525,29526],{"align":18556},"Train RMSE",[530,29528,29529],{"align":18556},"Test RMSE",[530,29531,29532],{"align":18556},"test\u002Ftrain ratio",[541,29534,29535,29548,29561,29574,29586],{},[527,29536,29537,29539,29542,29545],{},[546,29538,1583],{"align":1884},[546,29540,29541],{"align":18556},"0.708",[546,29543,29544],{"align":18556},"0.709",[546,29546,29547],{"align":18556},"1.0x",[527,29549,29550,29552,29555,29558],{},[546,29551,13895],{"align":1884},[546,29553,29554],{"align":18556},"0.323",[546,29556,29557],{"align":18556},"0.421",[546,29559,29560],{"align":18556},"1.3x",[527,29562,29563,29565,29568,29571],{},[546,29564,15096],{"align":1884},[546,29566,29567],{"align":18556},"0.043",[546,29569,29570],{"align":18556},"0.141",[546,29572,29573],{"align":18556},"3.3x",[527,29575,29576,29578,29580,29583],{},[546,29577,19842],{"align":1884},[546,29579,13910],{"align":18556},[546,29581,29582],{"align":18556},"0.039",[546,29584,29585],{"align":18556},"21.4x",[527,29587,29588,29590,29593,29596],{},[546,29589,19829],{"align":1884},[546,29591,29592],{"align":18556},"0.000",[546,29594,29595],{"align":18556},"0.009",[546,29597,29598],{"align":18556},"24529x",[11,29600,29601,29602,29605,29606,29609],{},"I noticed something: at degree 9 (10 parameters for 10 training points, the exact boundary for a unique solution), the ratio looks catastrophic (24529 times!), but the ",[15,29603,29604],{},"absolute"," test error stays small (0.009). That contradicts the slogan \"lots of parameters always cause overfitting.\" The correct statement is more subtle: ",[15,29607,29608],{},"overfitting is the model fitting noise",", not simply having many parameters. With no noise to fit, a flexible model interpolates well, even at the extreme edge.",[2606,29611,29613],{"id":29612},"with-noise-the-phenomenon-shows-up-for-real","With noise, the phenomenon shows up for real",[11,29615,29616],{},"Real data always has noise. I added modest noise (standard deviation 0.15) and redid the test, now comparing several degrees at once:",[29618,29619],"train-test-curve-chart",{":points":29620,"test-label":29621,"train-label":29622},"[{\"degree\":1,\"trainRmse\":0.74482,\"testRmse\":0.7491},{\"degree\":2,\"trainRmse\":0.73486,\"testRmse\":0.76026},{\"degree\":3,\"trainRmse\":0.31894,\"testRmse\":0.42316},{\"degree\":4,\"trainRmse\":0.31481,\"testRmse\":0.4589},{\"degree\":5,\"trainRmse\":0.08323,\"testRmse\":0.27673},{\"degree\":6,\"trainRmse\":0.07051,\"testRmse\":0.1808},{\"degree\":7,\"trainRmse\":0.01868,\"testRmse\":0.45143},{\"degree\":8,\"trainRmse\":0.01212,\"testRmse\":0.72573},{\"degree\":9,\"trainRmse\":0.0,\"testRmse\":1.71735}]","test","train",[11,29624,29625],{},"Train error only ever drops (more parameters always fit what's already been seen at least as well, this is nearly a theorem, not a coincidence). Test error drops, hits a minimum around degree 6, then climbs sharply, reaching 1.72 at degree 9, worse than a degree-1 fit. That minimum is what matters, not whichever degree zeroes the training error.",[11,29627,29628],{},"Try it yourself, using only the 10 noisy training points (drag the degree and watch the live test RMSE, computed on the 10 points the fit never saw):",[27982,29630],{":initial-degree":1583,":max-degree":19829,":x-train":29631,":y-train":29632,"x-label":124,"y-label":29633,":x-test":29634,":y-test":29635},"[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]","[1.193228, 0.550253, -0.579973, -1.143308, -0.623747, 0.365632, 0.960921, 0.528028, -0.097393, -0.880685]","y (with noise)","[1, 3, 5, 7, 9, 11, 13, 15, 17, 19]","[1.094999, -0.043944, -0.796443, -1.151981, -0.19079, 0.571574, 0.966876, 0.427335, -0.243645, -1.018878]",[2606,29637,29639],{"id":29638},"extrapolation-where-it-gets-genuinely-dangerous","Extrapolation: where it gets genuinely dangerous",[11,29641,29642,29643,29646,29647,29675,29676,29679],{},"Everything so far was ",[73,29644,29645],{},"inside"," the training range (",[80,29648,29650,29663],{"className":29649},[83],[80,29651,29653],{"className":29652},[87],[89,29654,29655],{"xmlns":91},[93,29656,29657,29661],{},[96,29658,29659],{},[102,29660,124],{},[145,29662,124],{"encoding":147},[80,29664,29666],{"className":29665,"ariaHidden":113},[152],[80,29667,29669,29672],{"className":29668},[156],[80,29670],{"className":29671,"style":334},[160],[80,29673,124],{"className":29674},[165,169]," from 0 to 19). Outside it, the behavior surprised me, and not in the way I expected. Drag the degree to 13 in the simulator below (the green band marks where the training data actually was) and notice: right past the edge, the degree-13 fit stays ",[15,29677,29678],{},"better"," than a low degree for a few points, because it learned the curve's shape with a lot of precision right at the boundary. That's exactly the trap: false confidence. Keep dragging the axis forward, and the same degree 13 that looked safe rockets off to infinity much faster than any low degree.",[27982,29681],{":initial-degree":13895,":max-degree":29355,":x-train":27984,":y-train":29356,"x-label":124,"y-label":29357,":view-max":26262,":view-min":29682},"-3",[11,29684,29685,29686,29738,29739,29774,29775,29804,29805,29834,29835,29886,29887,29922,29923,29952,29953,30010],{},"I computed the numbers to confirm what the eye sees: at ",[80,29687,29689,29708],{"className":29688},[83],[80,29690,29692],{"className":29691},[87],[89,29693,29694],{"xmlns":91},[93,29695,29696,29705],{},[96,29697,29698,29700,29702],{},[102,29699,124],{},[111,29701,130],{},[1321,29703,29704],{},"22",[145,29706,29707],{"encoding":147},"x=22",[80,29709,29711,29729],{"className":29710,"ariaHidden":113},[152],[80,29712,29714,29717,29720,29723,29726],{"className":29713},[156],[80,29715],{"className":29716,"style":334},[160],[80,29718,124],{"className":29719},[165,169],[80,29721],{"className":29722,"style":247},[246],[80,29724,130],{"className":29725},[251],[80,29727],{"className":29728,"style":247},[246],[80,29730,29732,29735],{"className":29731},[156],[80,29733],{"className":29734,"style":1614},[160],[80,29736,29704],{"className":29737},[165],", three units past the boundary, degree 3 already misses by ",[80,29740,29742,29759],{"className":29741},[83],[80,29743,29745],{"className":29744},[87],[89,29746,29747],{"xmlns":91},[93,29748,29749,29756],{},[96,29750,29751,29753],{},[111,29752,4643],{},[1321,29754,29755],{},"5.42",[145,29757,29758],{"encoding":147},"-5.42",[80,29760,29762],{"className":29761,"ariaHidden":113},[152],[80,29763,29765,29768,29771],{"className":29764},[156],[80,29766],{"className":29767,"style":5303},[160],[80,29769,4643],{"className":29770},[165],[80,29772,29755],{"className":29773},[165]," (the true value is ",[80,29776,29778,29792],{"className":29777},[83],[80,29779,29781],{"className":29780},[87],[89,29782,29783],{"xmlns":91},[93,29784,29785,29790],{},[96,29786,29787],{},[1321,29788,29789],{},"0.004",[145,29791,29789],{"encoding":147},[80,29793,29795],{"className":29794,"ariaHidden":113},[152],[80,29796,29798,29801],{"className":29797},[156],[80,29799],{"className":29800,"style":1614},[160],[80,29802,29789],{"className":29803},[165],") while degree 13 lands at ",[80,29806,29808,29822],{"className":29807},[83],[80,29809,29811],{"className":29810},[87],[89,29812,29813],{"xmlns":91},[93,29814,29815,29820],{},[96,29816,29817],{},[1321,29818,29819],{},"0.03",[145,29821,29819],{"encoding":147},[80,29823,29825],{"className":29824,"ariaHidden":113},[152],[80,29826,29828,29831],{"className":29827},[156],[80,29829],{"className":29830,"style":1614},[160],[80,29832,29819],{"className":29833},[165],", nearly perfect. But by ",[80,29836,29838,29856],{"className":29837},[83],[80,29839,29841],{"className":29840},[87],[89,29842,29843],{"xmlns":91},[93,29844,29845,29853],{},[96,29846,29847,29849,29851],{},[102,29848,124],{},[111,29850,130],{},[1321,29852,26262],{},[145,29854,29855],{"encoding":147},"x=30",[80,29857,29859,29877],{"className":29858,"ariaHidden":113},[152],[80,29860,29862,29865,29868,29871,29874],{"className":29861},[156],[80,29863],{"className":29864,"style":334},[160],[80,29866,124],{"className":29867},[165,169],[80,29869],{"className":29870,"style":247},[246],[80,29872,130],{"className":29873},[251],[80,29875],{"className":29876,"style":247},[246],[80,29878,29880,29883],{"className":29879},[156],[80,29881],{"className":29882,"style":1614},[160],[80,29884,26262],{"className":29885},[165],", degree 3 has only grown to ",[80,29888,29890,29907],{"className":29889},[83],[80,29891,29893],{"className":29892},[87],[89,29894,29895],{"xmlns":91},[93,29896,29897,29904],{},[96,29898,29899,29901],{},[111,29900,4643],{},[1321,29902,29903],{},"31",[145,29905,29906],{"encoding":147},"-31",[80,29908,29910],{"className":29909,"ariaHidden":113},[152],[80,29911,29913,29916,29919],{"className":29912},[156],[80,29914],{"className":29915,"style":5303},[160],[80,29917,4643],{"className":29918},[165],[80,29920,29903],{"className":29921},[165]," (bad, but growing slowly), while degree 13 is already at ",[80,29924,29926,29940],{"className":29925},[83],[80,29927,29929],{"className":29928},[87],[89,29930,29931],{"xmlns":91},[93,29932,29933,29938],{},[96,29934,29935],{},[1321,29936,29937],{},"86",[145,29939,29937],{"encoding":147},[80,29941,29943],{"className":29942,"ariaHidden":113},[152],[80,29944,29946,29949],{"className":29945},[156],[80,29947],{"className":29948,"style":1614},[160],[80,29950,29937],{"className":29951},[165],", an absurd value for a function that never leaves ",[80,29954,29956,29980],{"className":29955},[83],[80,29957,29959],{"className":29958},[87],[89,29960,29961],{"xmlns":91},[93,29962,29963,29977],{},[96,29964,29965,29967,29969,29971,29973,29975],{},[111,29966,23056],{"stretchy":120},[111,29968,4643],{},[1321,29970,1583],{},[111,29972,114],{"separator":113},[1321,29974,1583],{},[111,29976,23065],{"stretchy":120},[145,29978,29979],{"encoding":147},"[-1,1]",[80,29981,29983],{"className":29982,"ariaHidden":113},[152],[80,29984,29986,29989,29992,29995,29998,30001,30004,30007],{"className":29985},[156],[80,29987],{"className":29988,"style":2316},[160],[80,29990,23056],{"className":29991},[235],[80,29993,4643],{"className":29994},[165],[80,29996,1583],{"className":29997},[165],[80,29999,114],{"className":30000},[214],[80,30002],{"className":30003,"style":268},[246],[80,30005,1583],{"className":30006},[165],[80,30008,23065],{"className":30009},[242],". The rule of thumb I'm keeping: never trust a polynomial prediction outside the training range, especially at higher degree, because the disaster isn't immediate, it's treacherous.",[294,30012,6192],{"id":6191},[521,30014,30015,30023],{},[524,30016,30017],{},[527,30018,30019,30021],{},[530,30020,26333],{"align":532},[530,30022,26336],{"align":532},[541,30024,30025,30033,30041],{},[527,30026,30027,30030],{},[546,30028,30029],{"align":532},"Linear regression only fits a line",[546,30031,30032],{"align":532},"Engineering new features (powers, logs, ratios), the same regression fits any curve",[527,30034,30035,30038],{},[546,30036,30037],{"align":532},"More parameters fit what's already been seen better",[546,30039,30040],{"align":532},"That's nearly a theorem, and it's exactly why training error can't be used to pick a model",[527,30042,30043,30046],{},[546,30044,30045],{"align":532},"The cost bowl is convex",[546,30047,30048],{"align":532},"Convex doesn't prevent overfitting: the problem isn't optimization, it's the model memorizing noise",[11,30050,6507],{},[299,30052,30053,30059,30065],{},[302,30054,30055,30058],{},[15,30056,30057],{},"Feature engineering doesn't change the algorithm",", it changes the data going into it. The same linear regression learns any curve if you hand it the right feature.",[302,30060,30061,30064],{},[15,30062,30063],{},"Overfitting is about noise, not parameter count",": a flexible model with no noise to fit generalizes well, even at the extreme edge. The danger shows up once there's noise to memorize.",[302,30066,30067,30070],{},[15,30068,30069],{},"Extrapolating with a high-degree polynomial is treacherous",": it can look better than a low degree right past the edge of the data, and still blow up far worse a bit further out.",[11,30072,30073,30074,30077,30078,1618],{},"One question remains: can I use all the flexibility of a high degree ",[15,30075,30076],{},"without"," paying overfitting's price? There's an answer with a continuous dial instead of the discrete choice of degree, called regularization, and it's ",[562,30079,30081],{"href":30080},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-engineering-pitfalls","the subject of the next post",[294,30083,6716],{"id":6715},[11,30085,30086,30087,30091,30092,30095],{},"Same real 50-house dataset from the previous posts. In ",[562,30088,30090],{"href":30089},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls","the normalization bonus post"," I already left a promise: \"feature engineering is the topic of the next lab.\" Following through: I created the feature ",[65,30093,30094],{},"size_per_bedroom = square_feet \u002F num_bedrooms"," and compared the fit with and without it.",[2611,30097,30099],{"className":2613,"code":30098,"language":2615,"meta":26,"style":26},"size_per_bedroom = [s \u002F b for s, b in zip(square_feet, num_bedrooms)]\n\nrmse_before = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center], price)\nrmse_after = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center, size_per_bedroom], price)\n",[65,30100,30101,30106,30110,30115],{"__ignoreMap":26},[80,30102,30103],{"class":2620,"line":33},[80,30104,30105],{},"size_per_bedroom = [s \u002F b for s, b in zip(square_feet, num_bedrooms)]\n",[80,30107,30108],{"class":2620,"line":27},[80,30109,2657],{"emptyLinePlaceholder":32},[80,30111,30112],{"class":2620,"line":2631},[80,30113,30114],{},"rmse_before = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center], price)\n",[80,30116,30117],{"class":2620,"line":2636},[80,30118,30119],{},"rmse_after = fit_eval([square_feet, num_bedrooms, location_score, distance_to_center, size_per_bedroom], price)\n",[46,30121,30122],{},[11,30123,30124,3255,30126,10213,30129,30132],{},[15,30125,2693],{},[65,30127,30128],{},"RMSE without the new feature: 67.25",[65,30130,30131],{},"RMSE with the new feature: 65.13"," (thousand dollars)",[11,30134,30135,30136,30139,30140,30143,30144,1618],{},"A real, if modest, improvement (about 3%). Curious detail: the direct correlation between ",[65,30137,30138],{},"size_per_bedroom"," and price is nearly zero (0.0069, very close to none), so if I'd only looked at the isolated correlation I'd have discarded this feature. It only helps once it joins the model ",[15,30141,30142],{},"together"," with the others, the same kind of hidden effect I already saw with the bedrooms coefficient in ",[562,30145,19687],{"href":26916},[30147,30148],"housing-feature-engineering-explorer",{},[6949,30150,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":30152},[30153,30154,30155,30156,30157,30158,30163,30164],{"id":26940,"depth":27,"text":26941},{"id":27740,"depth":27,"text":27741},{"id":28082,"depth":27,"text":28083},{"id":29022,"depth":27,"text":29023},{"id":29270,"depth":27,"text":29271},{"id":29495,"depth":27,"text":29496,"children":30159},[30160,30161,30162],{"id":29510,"depth":2631,"text":29511},{"id":29612,"depth":2631,"text":29613},{"id":29638,"depth":2631,"text":29639},{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716},"How to make the exact same linear regression fit curves, by engineering new features instead of switching algorithms, and why that forced me to finally confront overfitting head-on.",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering",{"title":26926,"description":30165},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering",[30171,30172,9628],"feature-engineering","polynomial-regression","o5PN2HuTXzVCgWWHeOiFtGRm9a56zwrLaPudpL1oV10",{"id":30175,"title":30176,"body":30177,"cover":3,"date":26913,"description":33727,"extension":30,"meta":33728,"navigation":32,"order":2672,"path":33729,"playlist":6980,"seo":33730,"status":36,"stem":33731,"tags":33732,"__hash__":33735},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn.md","Linear Regression with Scikit-Learn",{"type":8,"value":30178,"toc":33711},[30179,30185,30188,30192,30204,30291,30310,30333,30340,30562,30569,30577,30615,31064,31074,31353,31391,31396,31400,31405,31408,31411,31422,31664,31671,31678,31701,31711,31715,31728,32114,32117,32571,32681,32808,32850,32854,32867,32871,32903,32951,32958,32968,33297,33444,33460,33462,33498,33500,33534,33536,33645,33652,33684,33694,33701,33703,33706,33709],[11,30180,30181],{},[57,30182],{"alt":30183,"src":30184},"A three-panel \"Rick and Morty\" meme: in the first panel, a robot labeled \"scikit-learn\" asks \"WHAT IS MY PURPOSE?\" In the second, Rick answers \"YOU SPLIT THE DATA.\" In the third, the robot says \"OH MY GOD.\"","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn\u002Fmeme-scikitlearn.jpeg",[11,30186,30187],{},"After four posts implementing gradient descent, normalization, and feature engineering entirely by hand, it's finally time to use the real tool. And the first shock was finding out the \"default\" model the course uses here isn't quite what it looks like.",[294,30189,30191],{"id":30190},"the-api-convention-more-important-than-the-model-itself","The API convention, more important than the model itself",[11,30193,30194,30199,30200,30203],{},[562,30195,30198],{"href":30196,"rel":30197},"https:\u002F\u002Fscikit-learn.org\u002F",[6724],"scikit-learn"," ships ready-made, tested implementations of most of what I've already built by hand. But what matters most to learn here isn't the specific model, it's the ",[15,30201,30202],{},"API convention",", because it repeats identically across hundreds of models and transformers:",[521,30205,30206,30218],{},[524,30207,30208],{},[527,30209,30210,30213,30215],{},[530,30211,30212],{"align":532},"Method",[530,30214,7923],{"align":532},[530,30216,30217],{"align":532},"Who has it",[541,30219,30220,30233,30246,30259,30278],{},[527,30221,30222,30227,30230],{},[546,30223,30224],{"align":532},[65,30225,30226],{},".fit(X, y)",[546,30228,30229],{"align":532},"learns parameters from the data",[546,30231,30232],{"align":532},"every estimator",[527,30234,30235,30240,30243],{},[546,30236,30237],{"align":532},[65,30238,30239],{},".predict(X)",[546,30241,30242],{"align":532},"uses the learned parameters to predict",[546,30244,30245],{"align":532},"predictive models",[527,30247,30248,30253,30256],{},[546,30249,30250],{"align":532},[65,30251,30252],{},".transform(X)",[546,30254,30255],{"align":532},"applies an already-learned transformation",[546,30257,30258],{"align":532},"transformers",[527,30260,30261,30266,30276],{},[546,30262,30263],{"align":532},[65,30264,30265],{},".fit_transform(X)",[546,30267,30268,30269,30272,30273],{"align":532},"shortcut for ",[65,30270,30271],{},"fit"," followed by ",[65,30274,30275],{},"transform",[546,30277,30258],{"align":532},[527,30279,30280,30285,30288],{},[546,30281,30282],{"align":532},[65,30283,30284],{},".score(X, y)",[546,30286,30287],{"align":532},"the estimator's default metric (R² for regression)",[546,30289,30290],{"align":532},"almost all of them",[11,30292,30293,30294,504,30297,504,30300,504,30303,504,30306,30309],{},"Learned attributes end with a trailing underscore: ",[65,30295,30296],{},"coef_",[65,30298,30299],{},"intercept_",[65,30301,30302],{},"mean_",[65,30304,30305],{},"scale_",[65,30307,30308],{},"n_iter_",". That trailing underscore is the convention distinguishing \"learned from the data\" from \"configured by me.\"",[11,30311,30312,30313,30315,30316,30319,30320,2338,30322,30325,30326,3255,30329,30332],{},"The golden rule that comes with it: ",[65,30314,30271],{}," only ever gets to see ",[15,30317,30318],{},"training"," data. On test data, only ",[65,30321,30275],{},[65,30323,30324],{},"predict",". Breaking that is data leakage, and it's the most common mistake for anyone starting out (I already hit this note in the ",[562,30327,30328],{"href":30089},"previous two",[562,30330,30331],{"href":30080},"bonus posts",").",[294,30334,30336,30339],{"id":30335},"standardscaler-the-z-score-i-already-built-by-hand",[65,30337,30338],{},"StandardScaler",": the z-score I already built by hand",[11,30341,30342,30344,30345,6529,30347,30481,30482,15648,30510,30561],{},[65,30343,30338],{}," does exactly the math I implemented in ",[562,30346,29015],{"href":26916},[80,30348,30350,30378],{"className":30349},[83],[80,30351,30353],{"className":30352},[87],[89,30354,30355],{"xmlns":91},[93,30356,30357,30375],{},[96,30358,30359,30361,30363],{},[102,30360,124],{},[111,30362,22854],{},[4625,30364,30365,30373],{},[96,30366,30367,30369,30371],{},[102,30368,124],{},[111,30370,4643],{},[102,30372,22115],{},[102,30374,24111],{},[145,30376,30377],{"encoding":147},"x \\leftarrow \\frac{x-\\mu}{\\sigma}",[80,30379,30381,30399],{"className":30380,"ariaHidden":113},[152],[80,30382,30384,30387,30390,30393,30396],{"className":30383},[156],[80,30385],{"className":30386,"style":334},[160],[80,30388,124],{"className":30389},[165,169],[80,30391],{"className":30392,"style":247},[246],[80,30394,22854],{"className":30395},[251],[80,30397],{"className":30398,"style":247},[246],[80,30400,30402,30406],{"className":30401},[156],[80,30403],{"className":30404,"style":30405},[160],"height:1.1994em;vertical-align:-0.345em;",[80,30407,30409,30412,30478],{"className":30408},[165],[80,30410],{"className":30411},[235,4746],[80,30413,30415],{"className":30414},[4625],[80,30416,30418,30470],{"className":30417},[178,179],[80,30419,30421,30467],{"className":30420},[183],[80,30422,30425,30439,30447],{"className":30423,"style":30424},[187],"height:0.8544em;",[80,30426,30427,30430],{"style":5013},[80,30428],{"className":30429,"style":4766},[195],[80,30431,30433],{"className":30432},[200,201,202,203],[80,30434,30436],{"className":30435},[165,203],[80,30437,24111],{"className":30438,"style":834},[165,169,203],[80,30440,30441,30444],{"style":4859},[80,30442],{"className":30443,"style":4766},[195],[80,30445],{"className":30446,"style":4867},[4866],[80,30448,30449,30452],{"style":4870},[80,30450],{"className":30451,"style":4766},[195],[80,30453,30455],{"className":30454},[200,201,202,203],[80,30456,30458,30461,30464],{"className":30457},[165,203],[80,30459,124],{"className":30460},[165,169,203],[80,30462,4643],{"className":30463},[279,203],[80,30465,22115],{"className":30466},[165,169,203],[80,30468,222],{"className":30469},[221],[80,30471,30473],{"className":30472},[183],[80,30474,30476],{"className":30475,"style":7390},[187],[80,30477],{},[80,30479],{"className":30480},[242,4746],", column by column. I checked and it matches my manual z-score digit for digit, including the same population-standard-deviation convention (divide by ",[80,30483,30485,30498],{"className":30484},[83],[80,30486,30488],{"className":30487},[87],[89,30489,30490],{"xmlns":91},[93,30491,30492,30496],{},[96,30493,30494],{},[102,30495,322],{},[145,30497,322],{"encoding":147},[80,30499,30501],{"className":30500,"ariaHidden":113},[152],[80,30502,30504,30507],{"className":30503},[156],[80,30505],{"className":30506,"style":334},[160],[80,30508,322],{"className":30509},[165,169],[80,30511,30513,30531],{"className":30512},[83],[80,30514,30516],{"className":30515},[87],[89,30517,30518],{"xmlns":91},[93,30519,30520,30528],{},[96,30521,30522,30524,30526],{},[102,30523,322],{},[111,30525,4643],{},[1321,30527,1583],{},[145,30529,30530],{"encoding":147},"m-1",[80,30532,30534,30552],{"className":30533,"ariaHidden":113},[152],[80,30535,30537,30540,30543,30546,30549],{"className":30536},[156],[80,30538],{"className":30539,"style":261},[160],[80,30541,322],{"className":30542},[165,169],[80,30544],{"className":30545,"style":275},[246],[80,30547,4643],{"className":30548},[279],[80,30550],{"className":30551,"style":275},[246],[80,30553,30555,30558],{"className":30554},[156],[80,30556],{"className":30557,"style":1614},[160],[80,30559,1583],{"className":30560},[165],") I was already using.",[294,30563,30565,30568],{"id":30564},"sgdregressor-and-what-the-s-means",[65,30566,30567],{},"SGDRegressor"," and what the \"S\" means",[11,30570,30571,30572,30574,30575,1618],{},"The course uses ",[65,30573,30567],{}," without ever explaining the acronym. Worth pausing on, because it's the real difference between this model and the gradient descent I implemented ",[562,30576,15127],{"href":4070},[11,30578,30579,30582,30583,3255,30586,30614],{},[15,30580,30581],{},"Batch gradient descent"," (what I've done so far): every step uses ",[15,30584,30585],{},"all",[80,30587,30589,30602],{"className":30588},[83],[80,30590,30592],{"className":30591},[87],[89,30593,30594],{"xmlns":91},[93,30595,30596,30600],{},[96,30597,30598],{},[102,30599,322],{},[145,30601,322],{"encoding":147},[80,30603,30605],{"className":30604,"ariaHidden":113},[152],[80,30606,30608,30611],{"className":30607},[156],[80,30609],{"className":30610,"style":334},[160],[80,30612,322],{"className":30613},[165,169]," examples to compute the gradient.",[11,30616,30617],{},[80,30618,30620,30720],{"className":30619},[83],[80,30621,30623],{"className":30622},[87],[89,30624,30625],{"xmlns":91},[93,30626,30627,30717],{},[96,30628,30629,30631,30633,30635,30637,30639,30641,30647,30667,30705],{},[102,30630,109],{"mathvariant":601},[111,30632,22854],{},[102,30634,109],{"mathvariant":601},[111,30636,4643],{},[102,30638,10551],{},[134,30640,136],{},[4625,30642,30643,30645],{},[1321,30644,1583],{},[102,30646,322],{},[7202,30648,30649,30651,30659],{},[111,30650,7206],{},[96,30652,30653,30655,30657],{},[102,30654,736],{},[111,30656,130],{},[1321,30658,2071],{},[96,30660,30661,30663,30665],{},[102,30662,322],{},[111,30664,4643],{},[1321,30666,1583],{},[96,30668,30669,30671,30673,30675,30687,30689,30691,30703],{},[111,30670,121],{"fence":113},[102,30672,104],{},[111,30674,121],{"stretchy":120},[726,30676,30677,30679],{},[102,30678,124],{"mathvariant":601},[96,30680,30681,30683,30685],{},[111,30682,121],{"stretchy":120},[102,30684,736],{},[111,30686,127],{"stretchy":120},[111,30688,127],{"stretchy":120},[111,30690,4643],{},[726,30692,30693,30695],{},[102,30694,683],{},[96,30696,30697,30699,30701],{},[111,30698,121],{"stretchy":120},[102,30700,736],{},[111,30702,127],{"stretchy":120},[111,30704,127],{"fence":113},[726,30706,30707,30709],{},[102,30708,124],{"mathvariant":601},[96,30710,30711,30713,30715],{},[111,30712,121],{"stretchy":120},[102,30714,736],{},[111,30716,127],{"stretchy":120},[145,30718,30719],{"encoding":147},"\\mathbf{w} \\leftarrow \\mathbf{w} - \\alpha\\,\\frac{1}{m}\\sum_{i=0}^{m-1}\\left(f(\\mathbf{x}^{(i)}) - y^{(i)}\\right)\\mathbf{x}^{(i)}",[80,30721,30723,30741,30759],{"className":30722,"ariaHidden":113},[152],[80,30724,30726,30729,30732,30735,30738],{"className":30725},[156],[80,30727],{"className":30728,"style":614},[160],[80,30730,109],{"className":30731,"style":700},[165,618],[80,30733],{"className":30734,"style":247},[246],[80,30736,22854],{"className":30737},[251],[80,30739],{"className":30740,"style":247},[246],[80,30742,30744,30747,30750,30753,30756],{"className":30743},[156],[80,30745],{"className":30746,"style":261},[160],[80,30748,109],{"className":30749,"style":700},[165,618],[80,30751],{"className":30752,"style":275},[246],[80,30754,4643],{"className":30755},[279],[80,30757],{"className":30758,"style":275},[246],[80,30760,30762,30765,30768,30771,30839,30842,30911,30914,31023,31026],{"className":30761},[156],[80,30763],{"className":30764,"style":11708},[160],[80,30766,10551],{"className":30767,"style":10667},[165,169],[80,30769],{"className":30770,"style":268},[246],[80,30772,30774,30777,30836],{"className":30773},[165],[80,30775],{"className":30776},[235,4746],[80,30778,30780],{"className":30779},[4625],[80,30781,30783,30828],{"className":30782},[178,179],[80,30784,30786,30825],{"className":30785},[183],[80,30787,30789,30803,30811],{"className":30788,"style":5010},[187],[80,30790,30791,30794],{"style":5013},[80,30792],{"className":30793,"style":4766},[195],[80,30795,30797],{"className":30796},[200,201,202,203],[80,30798,30800],{"className":30799},[165,203],[80,30801,322],{"className":30802},[165,169,203],[80,30804,30805,30808],{"style":4859},[80,30806],{"className":30807,"style":4766},[195],[80,30809],{"className":30810,"style":4867},[4866],[80,30812,30813,30816],{"style":5042},[80,30814],{"className":30815,"style":4766},[195],[80,30817,30819],{"className":30818},[200,201,202,203],[80,30820,30822],{"className":30821},[165,203],[80,30823,1583],{"className":30824},[165,203],[80,30826,222],{"className":30827},[221],[80,30829,30831],{"className":30830},[183],[80,30832,30834],{"className":30833,"style":7390},[187],[80,30835],{},[80,30837],{"className":30838},[242,4746],[80,30840],{"className":30841,"style":268},[246],[80,30843,30845,30848],{"className":30844},[7402],[80,30846,7206],{"className":30847,"style":7408},[7402,7406,7407],[80,30849,30851],{"className":30850},[174],[80,30852,30854,30903],{"className":30853},[178,179],[80,30855,30857,30900],{"className":30856},[183],[80,30858,30860,30880],{"className":30859,"style":7421},[187],[80,30861,30862,30865],{"style":7424},[80,30863],{"className":30864,"style":196},[195],[80,30866,30868],{"className":30867},[200,201,202,203],[80,30869,30871,30874,30877],{"className":30870},[165,203],[80,30872,736],{"className":30873},[165,169,203],[80,30875,130],{"className":30876},[251,203],[80,30878,2071],{"className":30879},[165,203],[80,30881,30882,30885],{"style":7445},[80,30883],{"className":30884,"style":196},[195],[80,30886,30888],{"className":30887},[200,201,202,203],[80,30889,30891,30894,30897],{"className":30890},[165,203],[80,30892,322],{"className":30893},[165,169,203],[80,30895,4643],{"className":30896},[279,203],[80,30898,1583],{"className":30899},[165,203],[80,30901,222],{"className":30902},[221],[80,30904,30906],{"className":30905},[183],[80,30907,30909],{"className":30908,"style":7473},[187],[80,30910],{},[80,30912],{"className":30913,"style":268},[246],[80,30915,30917,30923,30926,30929,30967,30970,30973,30976,30979,31017],{"className":30916},[7482],[80,30918,30920],{"className":30919,"style":7490},[235,7489],[80,30921,121],{"className":30922},[7494,4803],[80,30924,104],{"className":30925,"style":170},[165,169],[80,30927,121],{"className":30928},[235],[80,30930,30932,30935],{"className":30931},[165],[80,30933,124],{"className":30934},[165,618],[80,30936,30938],{"className":30937},[174],[80,30939,30941],{"className":30940},[178],[80,30942,30944],{"className":30943},[183],[80,30945,30947],{"className":30946,"style":751},[187],[80,30948,30949,30952],{"style":772},[80,30950],{"className":30951,"style":196},[195],[80,30953,30955],{"className":30954},[200,201,202,203],[80,30956,30958,30961,30964],{"className":30957},[165,203],[80,30959,121],{"className":30960},[235,203],[80,30962,736],{"className":30963},[165,169,203],[80,30965,127],{"className":30966},[242,203],[80,30968,127],{"className":30969},[242],[80,30971],{"className":30972,"style":275},[246],[80,30974,4643],{"className":30975},[279],[80,30977],{"className":30978,"style":275},[246],[80,30980,30982,30985],{"className":30981},[165],[80,30983,683],{"className":30984,"style":834},[165,169],[80,30986,30988],{"className":30987},[174],[80,30989,30991],{"className":30990},[178],[80,30992,30994],{"className":30993},[183],[80,30995,30997],{"className":30996,"style":751},[187],[80,30998,30999,31002],{"style":772},[80,31000],{"className":31001,"style":196},[195],[80,31003,31005],{"className":31004},[200,201,202,203],[80,31006,31008,31011,31014],{"className":31007},[165,203],[80,31009,121],{"className":31010},[235,203],[80,31012,736],{"className":31013},[165,169,203],[80,31015,127],{"className":31016},[242,203],[80,31018,31020],{"className":31019,"style":7490},[242,7489],[80,31021,127],{"className":31022},[7494,4803],[80,31024],{"className":31025,"style":268},[246],[80,31027,31029,31032],{"className":31028},[165],[80,31030,124],{"className":31031},[165,618],[80,31033,31035],{"className":31034},[174],[80,31036,31038],{"className":31037},[178],[80,31039,31041],{"className":31040},[183],[80,31042,31044],{"className":31043,"style":751},[187],[80,31045,31046,31049],{"style":772},[80,31047],{"className":31048,"style":196},[195],[80,31050,31052],{"className":31051},[200,201,202,203],[80,31053,31055,31058,31061],{"className":31054},[165,203],[80,31056,121],{"className":31057},[235,203],[80,31059,736],{"className":31060},[165,169,203],[80,31062,127],{"className":31063},[242,203],[11,31065,31066,31069,31070,31073],{},[15,31067,31068],{},"Stochastic gradient descent (SGD)",": every step uses a ",[15,31071,31072],{},"single"," example, picked at random.",[11,31075,31076],{},[80,31077,31079,31151],{"className":31078},[83],[80,31080,31082],{"className":31081},[87],[89,31083,31084],{"xmlns":91},[93,31085,31086,31148],{},[96,31087,31088,31090,31092,31094,31096,31098,31136],{},[102,31089,109],{"mathvariant":601},[111,31091,22854],{},[102,31093,109],{"mathvariant":601},[111,31095,4643],{},[102,31097,10551],{},[96,31099,31100,31102,31104,31106,31118,31120,31122,31134],{},[111,31101,121],{"fence":113},[102,31103,104],{},[111,31105,121],{"stretchy":120},[726,31107,31108,31110],{},[102,31109,124],{"mathvariant":601},[96,31111,31112,31114,31116],{},[111,31113,121],{"stretchy":120},[102,31115,736],{},[111,31117,127],{"stretchy":120},[111,31119,127],{"stretchy":120},[111,31121,4643],{},[726,31123,31124,31126],{},[102,31125,683],{},[96,31127,31128,31130,31132],{},[111,31129,121],{"stretchy":120},[102,31131,736],{},[111,31133,127],{"stretchy":120},[111,31135,127],{"fence":113},[726,31137,31138,31140],{},[102,31139,124],{"mathvariant":601},[96,31141,31142,31144,31146],{},[111,31143,121],{"stretchy":120},[102,31145,736],{},[111,31147,127],{"stretchy":120},[145,31149,31150],{"encoding":147},"\\mathbf{w} \\leftarrow \\mathbf{w} - \\alpha\\left(f(\\mathbf{x}^{(i)}) - y^{(i)}\\right)\\mathbf{x}^{(i)}",[80,31152,31154,31172,31190],{"className":31153,"ariaHidden":113},[152],[80,31155,31157,31160,31163,31166,31169],{"className":31156},[156],[80,31158],{"className":31159,"style":614},[160],[80,31161,109],{"className":31162,"style":700},[165,618],[80,31164],{"className":31165,"style":247},[246],[80,31167,22854],{"className":31168},[251],[80,31170],{"className":31171,"style":247},[246],[80,31173,31175,31178,31181,31184,31187],{"className":31174},[156],[80,31176],{"className":31177,"style":261},[160],[80,31179,109],{"className":31180,"style":700},[165,618],[80,31182],{"className":31183,"style":275},[246],[80,31185,4643],{"className":31186},[279],[80,31188],{"className":31189,"style":275},[246],[80,31191,31193,31197,31200,31203,31312,31315],{"className":31192},[156],[80,31194],{"className":31195,"style":31196},[160],"height:1.238em;vertical-align:-0.35em;",[80,31198,10551],{"className":31199,"style":10667},[165,169],[80,31201],{"className":31202,"style":268},[246],[80,31204,31206,31212,31215,31218,31256,31259,31262,31265,31268,31306],{"className":31205},[7482],[80,31207,31209],{"className":31208,"style":7490},[235,7489],[80,31210,121],{"className":31211},[7494,4803],[80,31213,104],{"className":31214,"style":170},[165,169],[80,31216,121],{"className":31217},[235],[80,31219,31221,31224],{"className":31220},[165],[80,31222,124],{"className":31223},[165,618],[80,31225,31227],{"className":31226},[174],[80,31228,31230],{"className":31229},[178],[80,31231,31233],{"className":31232},[183],[80,31234,31236],{"className":31235,"style":751},[187],[80,31237,31238,31241],{"style":772},[80,31239],{"className":31240,"style":196},[195],[80,31242,31244],{"className":31243},[200,201,202,203],[80,31245,31247,31250,31253],{"className":31246},[165,203],[80,31248,121],{"className":31249},[235,203],[80,31251,736],{"className":31252},[165,169,203],[80,31254,127],{"className":31255},[242,203],[80,31257,127],{"className":31258},[242],[80,31260],{"className":31261,"style":275},[246],[80,31263,4643],{"className":31264},[279],[80,31266],{"className":31267,"style":275},[246],[80,31269,31271,31274],{"className":31270},[165],[80,31272,683],{"className":31273,"style":834},[165,169],[80,31275,31277],{"className":31276},[174],[80,31278,31280],{"className":31279},[178],[80,31281,31283],{"className":31282},[183],[80,31284,31286],{"className":31285,"style":751},[187],[80,31287,31288,31291],{"style":772},[80,31289],{"className":31290,"style":196},[195],[80,31292,31294],{"className":31293},[200,201,202,203],[80,31295,31297,31300,31303],{"className":31296},[165,203],[80,31298,121],{"className":31299},[235,203],[80,31301,736],{"className":31302},[165,169,203],[80,31304,127],{"className":31305},[242,203],[80,31307,31309],{"className":31308,"style":7490},[242,7489],[80,31310,127],{"className":31311},[7494,4803],[80,31313],{"className":31314,"style":268},[246],[80,31316,31318,31321],{"className":31317},[165],[80,31319,124],{"className":31320},[165,618],[80,31322,31324],{"className":31323},[174],[80,31325,31327],{"className":31326},[178],[80,31328,31330],{"className":31329},[183],[80,31331,31333],{"className":31332,"style":751},[187],[80,31334,31335,31338],{"style":772},[80,31336],{"className":31337,"style":196},[195],[80,31339,31341],{"className":31340},[200,201,202,203],[80,31342,31344,31347,31350],{"className":31343},[165,203],[80,31345,121],{"className":31346},[235,203],[80,31348,736],{"className":31349},[165,169,203],[80,31351,127],{"className":31352},[242,203],[11,31354,31355,31356,31384,31385,31388,31389,3774],{},"Dirt-cheap step, noisy direction, a zigzag path, but ",[80,31357,31359,31372],{"className":31358},[83],[80,31360,31362],{"className":31361},[87],[89,31363,31364],{"xmlns":91},[93,31365,31366,31370],{},[96,31367,31368],{},[102,31369,322],{},[145,31371,322],{"encoding":147},[80,31373,31375],{"className":31374,"ariaHidden":113},[152],[80,31376,31378,31381],{"className":31377},[156],[80,31379],{"className":31380,"style":334},[160],[80,31382,322],{"className":31383},[165,169]," updates for the price of one batch step. I built an SGD engine from scratch (the same ",[65,31386,31387],{},"GradientDescentSimulator"," as always, just swapping the gear underneath) so you can see the difference live, on the same 8-house dataset from ",[562,31390,29015],{"href":26916},[11,31392,31393],{},[15,31394,31395],{},"Batch:",[13355,31397],{":alpha-slider-max":20699,":alpha-slider-min":20700,":alpha-slider-step":20700,":b-range":20694,":initial-alpha":31398,":initial-b":2071,":initial-w":2071,":w-range":20695,":x-train":20696,":y-train":20697,"b-label":117,"mode":31399,"w-label":109},"8e-7","batch",[11,31401,31402],{},[15,31403,31404],{},"Stochastic:",[13355,31406],{":alpha-slider-max":20699,":alpha-slider-min":20700,":alpha-slider-step":20700,":b-range":20694,":initial-alpha":31398,":initial-b":2071,":initial-w":2071,":w-range":20695,":x-train":20696,":y-train":20697,"b-label":117,"mode":31407,"w-label":109},"stochastic",[11,31409,31410],{},"Click \"Rodar 100\" on both with the same default alpha. After 50 steps, batch already has cost near 922 (very close to the real minimum, 919), while stochastic, with the same number of steps but each one only seeing one house at a time, is still bouncing around 7102. The stochastic path on the chart is visibly \"messier\" too, back and forth instead of a smooth descent.",[11,31412,31413,31414,31417,31418,31421],{},"One full pass through the examples is called an ",[15,31415,31416],{},"epoch",". In scikit-learn, ",[65,31419,31420],{},"max_iter"," counts epochs, not individual updates.",[521,31423,31424,31436],{},[524,31425,31426],{},[527,31427,31428,31430,31433],{},[530,31429],{"align":532},[530,31431,31432],{"align":532},"Batch GD",[530,31434,31435],{"align":532},"SGD",[541,31437,31438,31538,31575,31586,31597],{},[527,31439,31440,31443,31492],{},[546,31441,31442],{"align":532},"cost per update",[546,31444,31445],{"align":532},[80,31446,31448,31471],{"className":31447},[83],[80,31449,31451],{"className":31450},[87],[89,31452,31453],{"xmlns":91},[93,31454,31455,31468],{},[96,31456,31457,31460,31462,31464,31466],{},[102,31458,31459],{},"O",[111,31461,121],{"stretchy":120},[102,31463,322],{},[102,31465,1487],{},[111,31467,127],{"stretchy":120},[145,31469,31470],{"encoding":147},"O(mn)",[80,31472,31474],{"className":31473,"ariaHidden":113},[152],[80,31475,31477,31480,31483,31486,31489],{"className":31476},[156],[80,31478],{"className":31479,"style":2316},[160],[80,31481,31459],{"className":31482,"style":1850},[165,169],[80,31484,121],{"className":31485},[235],[80,31487,1321],{"className":31488},[165,169],[80,31490,127],{"className":31491},[242],[546,31493,31494],{"align":532},[80,31495,31497,31517],{"className":31496},[83],[80,31498,31500],{"className":31499},[87],[89,31501,31502],{"xmlns":91},[93,31503,31504,31514],{},[96,31505,31506,31508,31510,31512],{},[102,31507,31459],{},[111,31509,121],{"stretchy":120},[102,31511,1487],{},[111,31513,127],{"stretchy":120},[145,31515,31516],{"encoding":147},"O(n)",[80,31518,31520],{"className":31519,"ariaHidden":113},[152],[80,31521,31523,31526,31529,31532,31535],{"className":31522},[156],[80,31524],{"className":31525,"style":2316},[160],[80,31527,31459],{"className":31528,"style":1850},[165,169],[80,31530,121],{"className":31531},[235],[80,31533,1487],{"className":31534},[165,169],[80,31536,127],{"className":31537},[242],[527,31539,31540,31543,31545],{},[546,31541,31542],{"align":532},"updates per epoch",[546,31544,1583],{"align":532},[546,31546,31547],{"align":532},[80,31548,31550,31563],{"className":31549},[83],[80,31551,31553],{"className":31552},[87],[89,31554,31555],{"xmlns":91},[93,31556,31557,31561],{},[96,31558,31559],{},[102,31560,322],{},[145,31562,322],{"encoding":147},[80,31564,31566],{"className":31565,"ariaHidden":113},[152],[80,31567,31569,31572],{"className":31568},[156],[80,31570],{"className":31571,"style":334},[160],[80,31573,322],{"className":31574},[165,169],[527,31576,31577,31580,31583],{},[546,31578,31579],{"align":532},"trajectory",[546,31581,31582],{"align":532},"smooth",[546,31584,31585],{"align":532},"noisy",[527,31587,31588,31591,31594],{},[546,31589,31590],{"align":532},"deterministic?",[546,31592,31593],{"align":532},"yes",[546,31595,31596],{"align":532},"no (depends on the sampled order)",[527,31598,31599,31602,31633],{},[546,31600,31601],{"align":532},"good when",[546,31603,31604,31605],{"align":532},"small\u002Fmedium ",[80,31606,31608,31621],{"className":31607},[83],[80,31609,31611],{"className":31610},[87],[89,31612,31613],{"xmlns":91},[93,31614,31615,31619],{},[96,31616,31617],{},[102,31618,322],{},[145,31620,322],{"encoding":147},[80,31622,31624],{"className":31623,"ariaHidden":113},[152],[80,31625,31627,31630],{"className":31626},[156],[80,31628],{"className":31629,"style":334},[160],[80,31631,322],{"className":31632},[165,169],[546,31634,31635,31636],{"align":532},"very large ",[80,31637,31639,31652],{"className":31638},[83],[80,31640,31642],{"className":31641},[87],[89,31643,31644],{"xmlns":91},[93,31645,31646,31650],{},[96,31647,31648],{},[102,31649,322],{},[145,31651,322],{"encoding":147},[80,31653,31655],{"className":31654,"ariaHidden":113},[152],[80,31656,31658,31661],{"className":31657},[156],[80,31659],{"className":31660,"style":334},[160],[80,31662,322],{"className":31663},[165,169],[11,31665,31666,31667,31670],{},"With a constant learning rate, SGD never fully stops trembling around the minimum. That's why scikit-learn defaults to a rate that ",[15,31668,31669],{},"decreases"," over training.",[294,31672,31674,31675,31677],{"id":31673},"two-things-max_iter-hides","Two things ",[65,31676,31420],{}," hides",[11,31679,31680,31681,2338,31684,31686,31687,2338,31690,31693,31694,31696,31697,31700],{},"I ask for ",[65,31682,31683],{},"max_iter=1000",[65,31685,30567],{}," usually stops well before that. Not a bug: there's early stopping (",[65,31688,31689],{},"tol",[65,31691,31692],{},"n_iter_no_change",") that detects when the loss has genuinely stopped improving and halts on its own. ",[65,31695,31420],{}," is a ",[15,31698,31699],{},"ceiling",", not a target.",[11,31702,31703,31704,31707,31708,31710],{},"And without fixing ",[65,31705,31706],{},"random_state",", every ",[65,31709,30271],{}," samples a different order for the examples, and the result changes run to run. The variation is usually small, but real, especially on smaller or harder datasets. Fixing the seed is what makes the result reproducible.",[294,31712,31714],{"id":31713},"the-surprise-by-default-this-is-ridge-not-plain-least-squares","The surprise: by default, this is Ridge, not plain least squares",[11,31716,31717,31718,31720,31721,2338,31724,31727],{},"This is the one that caught me most off guard. ",[65,31719,30567],{},"'s defaults are ",[65,31722,31723],{},"penalty='l2'",[65,31725,31726],{},"alpha=0.0001",". In other words, by default it doesn't minimize",[11,31729,31730],{},[80,31731,31733,31817],{"className":31732},[83],[80,31734,31736],{"className":31735},[87],[89,31737,31738],{"xmlns":91},[93,31739,31740,31814],{},[96,31741,31742,31744,31746,31748,31750,31752,31754,31756,31766,31772],{},[102,31743,5606],{},[111,31745,121],{"stretchy":120},[102,31747,109],{"mathvariant":601},[111,31749,114],{"separator":113},[102,31751,117],{},[111,31753,127],{"stretchy":120},[111,31755,130],{},[4625,31757,31758,31760],{},[1321,31759,1583],{},[96,31761,31762,31764],{},[1321,31763,1323],{},[102,31765,322],{},[99,31767,31768,31770],{},[111,31769,7206],{},[102,31771,736],{},[726,31773,31774,31812],{},[96,31775,31776,31778,31780,31782,31794,31796,31798,31810],{},[111,31777,121],{"fence":113},[102,31779,104],{},[111,31781,121],{"stretchy":120},[726,31783,31784,31786],{},[102,31785,124],{"mathvariant":601},[96,31787,31788,31790,31792],{},[111,31789,121],{"stretchy":120},[102,31791,736],{},[111,31793,127],{"stretchy":120},[111,31795,127],{"stretchy":120},[111,31797,4643],{},[726,31799,31800,31802],{},[102,31801,683],{},[96,31803,31804,31806,31808],{},[111,31805,121],{"stretchy":120},[102,31807,736],{},[111,31809,127],{"stretchy":120},[111,31811,127],{"fence":113},[1321,31813,1323],{},[145,31815,31816],{"encoding":147},"J(\\mathbf{w},b) = \\frac{1}{2m}\\sum_i \\left(f(\\mathbf{x}^{(i)}) - y^{(i)}\\right)^2",[80,31818,31820,31856],{"className":31819,"ariaHidden":113},[152],[80,31821,31823,31826,31829,31832,31835,31838,31841,31844,31847,31850,31853],{"className":31822},[156],[80,31824],{"className":31825,"style":2316},[160],[80,31827,5606],{"className":31828,"style":5632},[165,169],[80,31830,121],{"className":31831},[235],[80,31833,109],{"className":31834,"style":700},[165,618],[80,31836,114],{"className":31837},[214],[80,31839],{"className":31840,"style":268},[246],[80,31842,117],{"className":31843},[165,169],[80,31845,127],{"className":31846},[242],[80,31848],{"className":31849,"style":247},[246],[80,31851,130],{"className":31852},[251],[80,31854],{"className":31855,"style":247},[246],[80,31857,31859,31862,31933,31936,31976,31979],{"className":31858},[156],[80,31860],{"className":31861,"style":7323},[160],[80,31863,31865,31868,31930],{"className":31864},[165],[80,31866],{"className":31867},[235,4746],[80,31869,31871],{"className":31870},[4625],[80,31872,31874,31922],{"className":31873},[178,179],[80,31875,31877,31919],{"className":31876},[183],[80,31878,31880,31897,31905],{"className":31879,"style":5010},[187],[80,31881,31882,31885],{"style":5013},[80,31883],{"className":31884,"style":4766},[195],[80,31886,31888],{"className":31887},[200,201,202,203],[80,31889,31891,31894],{"className":31890},[165,203],[80,31892,1323],{"className":31893},[165,203],[80,31895,322],{"className":31896},[165,169,203],[80,31898,31899,31902],{"style":4859},[80,31900],{"className":31901,"style":4766},[195],[80,31903],{"className":31904,"style":4867},[4866],[80,31906,31907,31910],{"style":5042},[80,31908],{"className":31909,"style":4766},[195],[80,31911,31913],{"className":31912},[200,201,202,203],[80,31914,31916],{"className":31915},[165,203],[80,31917,1583],{"className":31918},[165,203],[80,31920,222],{"className":31921},[221],[80,31923,31925],{"className":31924},[183],[80,31926,31928],{"className":31927,"style":7390},[187],[80,31929],{},[80,31931],{"className":31932},[242,4746],[80,31934],{"className":31935,"style":268},[246],[80,31937,31939,31942],{"className":31938},[7402],[80,31940,7206],{"className":31941,"style":7408},[7402,7406,7407],[80,31943,31945],{"className":31944},[174],[80,31946,31948,31968],{"className":31947},[178,179],[80,31949,31951,31965],{"className":31950},[183],[80,31952,31954],{"className":31953,"style":25783},[187],[80,31955,31956,31959],{"style":7424},[80,31957],{"className":31958,"style":196},[195],[80,31960,31962],{"className":31961},[200,201,202,203],[80,31963,736],{"className":31964},[165,169,203],[80,31966,222],{"className":31967},[221],[80,31969,31971],{"className":31970},[183],[80,31972,31974],{"className":31973,"style":7473},[187],[80,31975],{},[80,31977],{"className":31978,"style":268},[246],[80,31980,31982,32091],{"className":31981},[7482],[80,31983,31985,31991,31994,31997,32035,32038,32041,32044,32047,32085],{"className":31984},[7482],[80,31986,31988],{"className":31987,"style":7490},[235,7489],[80,31989,121],{"className":31990},[7494,4803],[80,31992,104],{"className":31993,"style":170},[165,169],[80,31995,121],{"className":31996},[235],[80,31998,32000,32003],{"className":31999},[165],[80,32001,124],{"className":32002},[165,618],[80,32004,32006],{"className":32005},[174],[80,32007,32009],{"className":32008},[178],[80,32010,32012],{"className":32011},[183],[80,32013,32015],{"className":32014,"style":751},[187],[80,32016,32017,32020],{"style":772},[80,32018],{"className":32019,"style":196},[195],[80,32021,32023],{"className":32022},[200,201,202,203],[80,32024,32026,32029,32032],{"className":32025},[165,203],[80,32027,121],{"className":32028},[235,203],[80,32030,736],{"className":32031},[165,169,203],[80,32033,127],{"className":32034},[242,203],[80,32036,127],{"className":32037},[242],[80,32039],{"className":32040,"style":275},[246],[80,32042,4643],{"className":32043},[279],[80,32045],{"className":32046,"style":275},[246],[80,32048,32050,32053],{"className":32049},[165],[80,32051,683],{"className":32052,"style":834},[165,169],[80,32054,32056],{"className":32055},[174],[80,32057,32059],{"className":32058},[178],[80,32060,32062],{"className":32061},[183],[80,32063,32065],{"className":32064,"style":751},[187],[80,32066,32067,32070],{"style":772},[80,32068],{"className":32069,"style":196},[195],[80,32071,32073],{"className":32072},[200,201,202,203],[80,32074,32076,32079,32082],{"className":32075},[165,203],[80,32077,121],{"className":32078},[235,203],[80,32080,736],{"className":32081},[165,169,203],[80,32083,127],{"className":32084},[242,203],[80,32086,32088],{"className":32087,"style":7490},[242,7489],[80,32089,127],{"className":32090},[7494,4803],[80,32092,32094],{"className":32093},[174],[80,32095,32097],{"className":32096},[178],[80,32098,32100],{"className":32099},[183],[80,32101,32103],{"className":32102,"style":7653},[187],[80,32104,32105,32108],{"style":7656},[80,32106],{"className":32107,"style":196},[195],[80,32109,32111],{"className":32110},[200,201,202,203],[80,32112,1323],{"className":32113},[165,203],[11,32115,32116],{},"it minimizes",[11,32118,32119],{},[80,32120,32122,32221],{"className":32121},[83],[80,32123,32125],{"className":32124},[87],[89,32126,32127],{"xmlns":91},[93,32128,32129,32218],{},[96,32130,32131,32133,32135,32137,32139,32141,32143,32145,32155,32161,32203,32205,32207,32210,32212],{},[102,32132,5606],{},[111,32134,121],{"stretchy":120},[102,32136,109],{"mathvariant":601},[111,32138,114],{"separator":113},[102,32140,117],{},[111,32142,127],{"stretchy":120},[111,32144,130],{},[4625,32146,32147,32149],{},[1321,32148,1583],{},[96,32150,32151,32153],{},[1321,32152,1323],{},[102,32154,322],{},[99,32156,32157,32159],{},[111,32158,7206],{},[102,32160,736],{},[726,32162,32163,32201],{},[96,32164,32165,32167,32169,32171,32183,32185,32187,32199],{},[111,32166,121],{"fence":113},[102,32168,104],{},[111,32170,121],{"stretchy":120},[726,32172,32173,32175],{},[102,32174,124],{"mathvariant":601},[96,32176,32177,32179,32181],{},[111,32178,121],{"stretchy":120},[102,32180,736],{},[111,32182,127],{"stretchy":120},[111,32184,127],{"stretchy":120},[111,32186,4643],{},[726,32188,32189,32191],{},[102,32190,683],{},[96,32192,32193,32195,32197],{},[111,32194,121],{"stretchy":120},[102,32196,736],{},[111,32198,127],{"stretchy":120},[111,32200,127],{"fence":113},[1321,32202,1323],{},[111,32204,141],{},[102,32206,10551],{},[102,32208,32209],{"mathvariant":10558},"∥",[102,32211,109],{"mathvariant":601},[726,32213,32214,32216],{},[102,32215,32209],{"mathvariant":10558},[1321,32217,1323],{},[145,32219,32220],{"encoding":147},"J(\\mathbf{w},b) = \\frac{1}{2m}\\sum_i \\left(f(\\mathbf{x}^{(i)}) - y^{(i)}\\right)^2 + \\alpha\\|\\mathbf{w}\\|^2",[80,32222,32224,32260,32527],{"className":32223,"ariaHidden":113},[152],[80,32225,32227,32230,32233,32236,32239,32242,32245,32248,32251,32254,32257],{"className":32226},[156],[80,32228],{"className":32229,"style":2316},[160],[80,32231,5606],{"className":32232,"style":5632},[165,169],[80,32234,121],{"className":32235},[235],[80,32237,109],{"className":32238,"style":700},[165,618],[80,32240,114],{"className":32241},[214],[80,32243],{"className":32244,"style":268},[246],[80,32246,117],{"className":32247},[165,169],[80,32249,127],{"className":32250},[242],[80,32252],{"className":32253,"style":247},[246],[80,32255,130],{"className":32256},[251],[80,32258],{"className":32259,"style":247},[246],[80,32261,32263,32266,32337,32340,32380,32383,32518,32521,32524],{"className":32262},[156],[80,32264],{"className":32265,"style":7323},[160],[80,32267,32269,32272,32334],{"className":32268},[165],[80,32270],{"className":32271},[235,4746],[80,32273,32275],{"className":32274},[4625],[80,32276,32278,32326],{"className":32277},[178,179],[80,32279,32281,32323],{"className":32280},[183],[80,32282,32284,32301,32309],{"className":32283,"style":5010},[187],[80,32285,32286,32289],{"style":5013},[80,32287],{"className":32288,"style":4766},[195],[80,32290,32292],{"className":32291},[200,201,202,203],[80,32293,32295,32298],{"className":32294},[165,203],[80,32296,1323],{"className":32297},[165,203],[80,32299,322],{"className":32300},[165,169,203],[80,32302,32303,32306],{"style":4859},[80,32304],{"className":32305,"style":4766},[195],[80,32307],{"className":32308,"style":4867},[4866],[80,32310,32311,32314],{"style":5042},[80,32312],{"className":32313,"style":4766},[195],[80,32315,32317],{"className":32316},[200,201,202,203],[80,32318,32320],{"className":32319},[165,203],[80,32321,1583],{"className":32322},[165,203],[80,32324,222],{"className":32325},[221],[80,32327,32329],{"className":32328},[183],[80,32330,32332],{"className":32331,"style":7390},[187],[80,32333],{},[80,32335],{"className":32336},[242,4746],[80,32338],{"className":32339,"style":268},[246],[80,32341,32343,32346],{"className":32342},[7402],[80,32344,7206],{"className":32345,"style":7408},[7402,7406,7407],[80,32347,32349],{"className":32348},[174],[80,32350,32352,32372],{"className":32351},[178,179],[80,32353,32355,32369],{"className":32354},[183],[80,32356,32358],{"className":32357,"style":25783},[187],[80,32359,32360,32363],{"style":7424},[80,32361],{"className":32362,"style":196},[195],[80,32364,32366],{"className":32365},[200,201,202,203],[80,32367,736],{"className":32368},[165,169,203],[80,32370,222],{"className":32371},[221],[80,32373,32375],{"className":32374},[183],[80,32376,32378],{"className":32377,"style":7473},[187],[80,32379],{},[80,32381],{"className":32382,"style":268},[246],[80,32384,32386,32495],{"className":32385},[7482],[80,32387,32389,32395,32398,32401,32439,32442,32445,32448,32451,32489],{"className":32388},[7482],[80,32390,32392],{"className":32391,"style":7490},[235,7489],[80,32393,121],{"className":32394},[7494,4803],[80,32396,104],{"className":32397,"style":170},[165,169],[80,32399,121],{"className":32400},[235],[80,32402,32404,32407],{"className":32403},[165],[80,32405,124],{"className":32406},[165,618],[80,32408,32410],{"className":32409},[174],[80,32411,32413],{"className":32412},[178],[80,32414,32416],{"className":32415},[183],[80,32417,32419],{"className":32418,"style":751},[187],[80,32420,32421,32424],{"style":772},[80,32422],{"className":32423,"style":196},[195],[80,32425,32427],{"className":32426},[200,201,202,203],[80,32428,32430,32433,32436],{"className":32429},[165,203],[80,32431,121],{"className":32432},[235,203],[80,32434,736],{"className":32435},[165,169,203],[80,32437,127],{"className":32438},[242,203],[80,32440,127],{"className":32441},[242],[80,32443],{"className":32444,"style":275},[246],[80,32446,4643],{"className":32447},[279],[80,32449],{"className":32450,"style":275},[246],[80,32452,32454,32457],{"className":32453},[165],[80,32455,683],{"className":32456,"style":834},[165,169],[80,32458,32460],{"className":32459},[174],[80,32461,32463],{"className":32462},[178],[80,32464,32466],{"className":32465},[183],[80,32467,32469],{"className":32468,"style":751},[187],[80,32470,32471,32474],{"style":772},[80,32472],{"className":32473,"style":196},[195],[80,32475,32477],{"className":32476},[200,201,202,203],[80,32478,32480,32483,32486],{"className":32479},[165,203],[80,32481,121],{"className":32482},[235,203],[80,32484,736],{"className":32485},[165,169,203],[80,32487,127],{"className":32488},[242,203],[80,32490,32492],{"className":32491,"style":7490},[242,7489],[80,32493,127],{"className":32494},[7494,4803],[80,32496,32498],{"className":32497},[174],[80,32499,32501],{"className":32500},[178],[80,32502,32504],{"className":32503},[183],[80,32505,32507],{"className":32506,"style":7653},[187],[80,32508,32509,32512],{"style":7656},[80,32510],{"className":32511,"style":196},[195],[80,32513,32515],{"className":32514},[200,201,202,203],[80,32516,1323],{"className":32517},[165,203],[80,32519],{"className":32520,"style":275},[246],[80,32522,141],{"className":32523},[279],[80,32525],{"className":32526,"style":275},[246],[80,32528,32530,32533,32536,32539,32542],{"className":32529},[156],[80,32531],{"className":32532,"style":8187},[160],[80,32534,10551],{"className":32535,"style":10667},[165,169],[80,32537,32209],{"className":32538},[165],[80,32540,109],{"className":32541,"style":700},[165,618],[80,32543,32545,32548],{"className":32544},[165],[80,32546,32209],{"className":32547},[165],[80,32549,32551],{"className":32550},[174],[80,32552,32554],{"className":32553},[178],[80,32555,32557],{"className":32556},[183],[80,32558,32560],{"className":32559,"style":1407},[187],[80,32561,32562,32565],{"style":772},[80,32563],{"className":32564,"style":196},[195],[80,32566,32568],{"className":32567},[200,201,202,203],[80,32569,1323],{"className":32570},[165,203],[11,32572,32573,32574,9083,32577,32628,32629,32680],{},"That extra term is exactly the L2 regularization I built from scratch in ",[562,32575,32576],{"href":30080},"the previous bonus post",[80,32578,32580,32598],{"className":32579},[83],[80,32581,32583],{"className":32582},[87],[89,32584,32585],{"xmlns":91},[93,32586,32587,32595],{},[96,32588,32589,32591,32593],{},[102,32590,10551],{},[111,32592,130],{},[1321,32594,13510],{},[145,32596,32597],{"encoding":147},"\\alpha=0.0001",[80,32599,32601,32619],{"className":32600,"ariaHidden":113},[152],[80,32602,32604,32607,32610,32613,32616],{"className":32603},[156],[80,32605],{"className":32606,"style":334},[160],[80,32608,10551],{"className":32609,"style":10667},[165,169],[80,32611],{"className":32612,"style":247},[246],[80,32614,130],{"className":32615},[251],[80,32617],{"className":32618,"style":247},[246],[80,32620,32622,32625],{"className":32621},[156],[80,32623],{"className":32624,"style":1614},[160],[80,32626,13510],{"className":32627},[165]," (the default) the effect is too small to notice on this dataset, but the mechanism is real. I fit with ",[80,32630,32632,32650],{"className":32631},[83],[80,32633,32635],{"className":32634},[87],[89,32636,32637],{"xmlns":91},[93,32638,32639,32647],{},[96,32640,32641,32643,32645],{},[102,32642,10551],{},[111,32644,130],{},[1321,32646,1583],{},[145,32648,32649],{"encoding":147},"\\alpha=1",[80,32651,32653,32671],{"className":32652,"ariaHidden":113},[152],[80,32654,32656,32659,32662,32665,32668],{"className":32655},[156],[80,32657],{"className":32658,"style":334},[160],[80,32660,10551],{"className":32661,"style":10667},[165,169],[80,32663],{"className":32664,"style":247},[246],[80,32666,130],{"className":32667},[251],[80,32669],{"className":32670,"style":247},[246],[80,32672,32674,32677],{"className":32673},[156],[80,32675],{"className":32676,"style":1614},[160],[80,32678,1583],{"className":32679},[165]," (much stronger, just to make it visible) on our 100-house dataset:",[521,32682,32683,32734],{},[524,32684,32685],{},[527,32686,32687,32690,32693],{},[530,32688,32689],{"align":532},"Model",[530,32691,32692],{"align":18556},"RMSE",[530,32694,32695],{"align":18556},[80,32696,32698,32716],{"className":32697},[83],[80,32699,32701],{"className":32700},[87],[89,32702,32703],{"xmlns":91},[93,32704,32705,32713],{},[96,32706,32707,32709,32711],{},[102,32708,32209],{"mathvariant":10558},[102,32710,109],{"mathvariant":601},[102,32712,32209],{"mathvariant":10558},[145,32714,32715],{"encoding":147},"\\|\\mathbf{w}\\|",[80,32717,32719],{"className":32718,"ariaHidden":113},[152],[80,32720,32722,32725,32728,32731],{"className":32721},[156],[80,32723],{"className":32724,"style":2316},[160],[80,32726,32209],{"className":32727},[165],[80,32729,109],{"className":32730,"style":700},[165,618],[80,32732,32209],{"className":32733},[165],[541,32735,32736,32747],{},[527,32737,32738,32741,32744],{},[546,32739,32740],{"align":532},"No regularization",[546,32742,32743],{"align":18556},"20.96",[546,32745,32746],{"align":18556},"123.25",[527,32748,32749,32802,32805],{},[546,32750,32751,32752,127],{"align":532},"Ridge (",[80,32753,32755,32772],{"className":32754},[83],[80,32756,32758],{"className":32757},[87],[89,32759,32760],{"xmlns":91},[93,32761,32762,32770],{},[96,32763,32764,32766,32768],{},[102,32765,10551],{},[111,32767,130],{},[1321,32769,1583],{},[145,32771,32649],{"encoding":147},[80,32773,32775,32793],{"className":32774,"ariaHidden":113},[152],[80,32776,32778,32781,32784,32787,32790],{"className":32777},[156],[80,32779],{"className":32780,"style":334},[160],[80,32782,10551],{"className":32783,"style":10667},[165,169],[80,32785],{"className":32786,"style":247},[246],[80,32788,130],{"className":32789},[251],[80,32791],{"className":32792,"style":247},[246],[80,32794,32796,32799],{"className":32795},[156],[80,32797],{"className":32798,"style":1614},[160],[80,32800,1583],{"className":32801},[165],[546,32803,32804],{"align":18556},"21.04",[546,32806,32807],{"align":18556},"120.39",[11,32809,32810,32811,20101,32814,32817,32818,32821,32822,1618],{},"The weight shrinks, the training error gets a little worse, exactly the trade-off I already saw in ",[562,32812,32813],{"href":30080},"the Ridge post",[15,32815,32816],{},"Know the defaults of the library you're using",": an ",[65,32819,32820],{},"SGDRegressor()"," called with no arguments isn't \"raw linear regression,\" it's Ridge with a small, discrete ",[80,32823,32825,32838],{"className":32824},[83],[80,32826,32828],{"className":32827},[87],[89,32829,32830],{"xmlns":91},[93,32831,32832,32836],{},[96,32833,32834],{},[102,32835,10551],{},[145,32837,11037],{"encoding":147},[80,32839,32841],{"className":32840,"ariaHidden":113},[152],[80,32842,32844,32847],{"className":32843},[156],[80,32845],{"className":32846,"style":334},[160],[80,32848,10551],{"className":32849,"style":10667},[165,169],[294,32851,32853],{"id":32852},"predicting-and-a-fragile-way-to-compare","Predicting, and a fragile way to compare",[11,32855,32856,32857,32860,32861,1291,32864,32866],{},"Two mathematically equivalent calculations can differ in the last bit of precision because of floating-point operation order. Comparing predictions with ",[65,32858,32859],{},"=="," is fragile (",[65,32862,32863],{},"0.1 + 0.2 == 0.3",[65,32865,120],{}," in any language using 64-bit floats, for the exact same reason). The right way is to check the difference falls within a small tolerance, not demand exact equality.",[294,32868,32870],{"id":32869},"metrics-how-good-is-this-model-really","Metrics: how good is this model, really",[11,32872,32873,32874,32902],{},"Up to now I'd used the cost ",[80,32875,32877,32890],{"className":32876},[83],[80,32878,32880],{"className":32879},[87],[89,32881,32882],{"xmlns":91},[93,32883,32884,32888],{},[96,32885,32886],{},[102,32887,5606],{},[145,32889,5606],{"encoding":147},[80,32891,32893],{"className":32892,"ariaHidden":113},[152],[80,32894,32896,32899],{"className":32895},[156],[80,32897],{"className":32898,"style":8672},[160],[80,32900,5606],{"className":32901,"style":5632},[165,169]," to train, but never a metric meant to be read by a person. I fit the model with all 4 features on the full 100-house dataset and computed three standard metrics:",[521,32904,32905,32918],{},[524,32906,32907],{},[527,32908,32909,32912,32915],{},[530,32910,32911],{"align":532},"Metric",[530,32913,32914],{"align":18556},"Value",[530,32916,32917],{"align":532},"What it measures",[541,32919,32920,32930,32940],{},[527,32921,32922,32924,32927],{},[546,32923,32692],{"align":532},[546,32925,32926],{"align":18556},"20.96 thousand US$",[546,32928,32929],{"align":532},"typical error, punishes big misses disproportionately",[527,32931,32932,32934,32937],{},[546,32933,16927],{"align":532},[546,32935,32936],{"align":18556},"16.91 thousand US$",[546,32938,32939],{"align":532},"mean absolute error, more robust to outliers",[527,32941,32942,32945,32948],{},[546,32943,32944],{"align":532},"R²",[546,32946,32947],{"align":18556},"0.9594",[546,32949,32950],{"align":532},"fraction of price's variance explained by the model",[11,32952,32953,32954,32957],{},"For reference: a \"dumb\" model that always guesses the mean has an RMSE of 104.07. Ours misses ",[15,32955,32956],{},"4.96 times less",". That's the real payoff of having a model, not just the R² number alone.",[294,32959,32961,32963,32964,32967],{"id":32960},"sgdregressor-or-linearregression-the-real-choice",[65,32962,30567],{}," or ",[65,32965,32966],{},"LinearRegression","? The real choice",[11,32969,32970,32971,32973,32974,32977,32978,32980,32981,33169,33170,33173,33174,20101,33248,33250,33251,33296],{},"The course presents ",[65,32972,30567],{}," as \"scikit-learn's linear regression,\" but in practice it's the ",[15,32975,32976],{},"least"," common choice. ",[65,32979,32966],{}," solves the normal equation in closed form, ",[80,32982,32984,33031],{"className":32983},[83],[80,32985,32987],{"className":32986},[87],[89,32988,32989],{"xmlns":91},[93,32990,32991,33028],{},[96,32992,32993,32997,32999,33001,33008,33010,33020,33026],{},[102,32994,32996],{"mathvariant":32995},"bold-italic","θ",[111,32998,130],{},[111,33000,121],{"stretchy":120},[726,33002,33003,33005],{},[102,33004,15519],{"mathvariant":601},[102,33006,33007],{"mathvariant":10558},"⊤",[102,33009,15519],{"mathvariant":601},[726,33011,33012,33014],{},[111,33013,127],{"stretchy":120},[96,33015,33016,33018],{},[111,33017,4643],{},[1321,33019,1583],{},[726,33021,33022,33024],{},[102,33023,15519],{"mathvariant":601},[102,33025,33007],{"mathvariant":10558},[102,33027,683],{"mathvariant":601},[145,33029,33030],{"encoding":147},"\\boldsymbol{\\theta} = (\\mathbf{X}^\\top\\mathbf{X})^{-1}\\mathbf{X}^\\top\\mathbf{y}",[80,33032,33034,33060],{"className":33033,"ariaHidden":113},[152],[80,33035,33037,33040,33051,33054,33057],{"className":33036},[156],[80,33038],{"className":33039,"style":289},[160],[80,33041,33043],{"className":33042},[165],[80,33044,33046],{"className":33045},[165],[80,33047,32996],{"className":33048,"style":33050},[165,33049],"boldsymbol","margin-right:0.0319em;",[80,33052],{"className":33053,"style":247},[246],[80,33055,130],{"className":33056},[251],[80,33058],{"className":33059,"style":247},[246],[80,33061,33063,33067,33070,33099,33102,33137,33166],{"className":33062},[156],[80,33064],{"className":33065,"style":33066},[160],"height:1.0991em;vertical-align:-0.25em;",[80,33068,121],{"className":33069},[235],[80,33071,33073,33076],{"className":33072},[165],[80,33074,15519],{"className":33075},[165,618],[80,33077,33079],{"className":33078},[174],[80,33080,33082],{"className":33081},[178],[80,33083,33085],{"className":33084},[183],[80,33086,33088],{"className":33087,"style":25654},[187],[80,33089,33090,33093],{"style":772},[80,33091],{"className":33092,"style":196},[195],[80,33094,33096],{"className":33095},[200,201,202,203],[80,33097,33007],{"className":33098},[165,203],[80,33100,15519],{"className":33101},[165,618],[80,33103,33105,33108],{"className":33104},[242],[80,33106,127],{"className":33107},[242],[80,33109,33111],{"className":33110},[174],[80,33112,33114],{"className":33113},[178],[80,33115,33117],{"className":33116},[183],[80,33118,33120],{"className":33119,"style":1407},[187],[80,33121,33122,33125],{"style":772},[80,33123],{"className":33124,"style":196},[195],[80,33126,33128],{"className":33127},[200,201,202,203],[80,33129,33131,33134],{"className":33130},[165,203],[80,33132,4643],{"className":33133},[165,203],[80,33135,1583],{"className":33136},[165,203],[80,33138,33140,33143],{"className":33139},[165],[80,33141,15519],{"className":33142},[165,618],[80,33144,33146],{"className":33145},[174],[80,33147,33149],{"className":33148},[178],[80,33150,33152],{"className":33151},[183],[80,33153,33155],{"className":33154,"style":25654},[187],[80,33156,33157,33160],{"style":772},[80,33158],{"className":33159,"style":196},[195],[80,33161,33163],{"className":33162},[200,201,202,203],[80,33164,33007],{"className":33165},[165,203],[80,33167,683],{"className":33168,"style":700},[165,618],", no iteration, no learning rate, no random seed. The cost grows with the ",[15,33171,33172],{},"cube"," of the number of features, ",[80,33175,33177,33201],{"className":33176},[83],[80,33178,33180],{"className":33179},[87],[89,33181,33182],{"xmlns":91},[93,33183,33184,33198],{},[96,33185,33186,33188,33190,33196],{},[102,33187,31459],{},[111,33189,121],{"stretchy":120},[726,33191,33192,33194],{},[102,33193,1487],{},[1321,33195,13895],{},[111,33197,127],{"stretchy":120},[145,33199,33200],{"encoding":147},"O(n^3)",[80,33202,33204],{"className":33203,"ariaHidden":113},[152],[80,33205,33207,33210,33213,33216,33245],{"className":33206},[156],[80,33208],{"className":33209,"style":8187},[160],[80,33211,31459],{"className":33212,"style":1850},[165,169],[80,33214,121],{"className":33215},[235],[80,33217,33219,33222],{"className":33218},[165],[80,33220,1487],{"className":33221},[165,169],[80,33223,33225],{"className":33224},[174],[80,33226,33228],{"className":33227},[178],[80,33229,33231],{"className":33230},[183],[80,33232,33234],{"className":33233,"style":1407},[187],[80,33235,33236,33239],{"style":772},[80,33237],{"className":33238,"style":196},[195],[80,33240,33242],{"className":33241},[200,201,202,203],[80,33243,13895],{"className":33244},[165,203],[80,33246,127],{"className":33247},[242],[65,33249,30567],{}," iterates, costs ",[80,33252,33254,33275],{"className":33253},[83],[80,33255,33257],{"className":33256},[87],[89,33258,33259],{"xmlns":91},[93,33260,33261,33273],{},[96,33262,33263,33265,33267,33269,33271],{},[102,33264,31459],{},[111,33266,121],{"stretchy":120},[102,33268,322],{},[102,33270,1487],{},[111,33272,127],{"stretchy":120},[145,33274,31470],{"encoding":147},[80,33276,33278],{"className":33277,"ariaHidden":113},[152],[80,33279,33281,33284,33287,33290,33293],{"className":33280},[156],[80,33282],{"className":33283,"style":2316},[160],[80,33285,31459],{"className":33286,"style":1850},[165,169],[80,33288,121],{"className":33289},[235],[80,33291,1321],{"className":33292},[165,169],[80,33294,127],{"className":33295},[242]," per epoch, and never needs everything loaded into memory at once.",[521,33298,33299,33314],{},[524,33300,33301],{},[527,33302,33303,33306,33310],{},[530,33304,33305],{"align":532},"Criterion",[530,33307,33308],{"align":532},[65,33309,32966],{},[530,33311,33312],{"align":532},[65,33313,30567],{},[541,33315,33316,33327,33338,33351,33421,33435],{},[527,33317,33318,33321,33324],{},[546,33319,33320],{"align":532},"accuracy",[546,33322,33323],{"align":532},"exact solution",[546,33325,33326],{"align":532},"approximate",[527,33328,33329,33332,33335],{},[546,33330,33331],{"align":532},"determinism",[546,33333,33334],{"align":532},"full",[546,33336,33337],{"align":532},"depends on the seed",[527,33339,33340,33343,33346],{},[546,33341,33342],{"align":532},"needs normalizing?",[546,33344,33345],{"align":532},"no",[546,33347,33348],{"align":532},[15,33349,33350],{},"yes, mandatorily",[527,33352,33353,33356,33418],{},[546,33354,33355],{"align":532},"huge feature count",[546,33357,33358,33359,127],{"align":532},"gets expensive (",[80,33360,33362,33380],{"className":33361},[83],[80,33363,33365],{"className":33364},[87],[89,33366,33367],{"xmlns":91},[93,33368,33369,33377],{},[96,33370,33371],{},[726,33372,33373,33375],{},[102,33374,1487],{},[1321,33376,13895],{},[145,33378,33379],{"encoding":147},"n^3",[80,33381,33383],{"className":33382,"ariaHidden":113},[152],[80,33384,33386,33389],{"className":33385},[156],[80,33387],{"className":33388,"style":1407},[160],[80,33390,33392,33395],{"className":33391},[165],[80,33393,1487],{"className":33394},[165,169],[80,33396,33398],{"className":33397},[174],[80,33399,33401],{"className":33400},[178],[80,33402,33404],{"className":33403},[183],[80,33405,33407],{"className":33406,"style":1407},[187],[80,33408,33409,33412],{"style":772},[80,33410],{"className":33411,"style":196},[195],[80,33413,33415],{"className":33414},[200,201,202,203],[80,33416,13895],{"className":33417},[165,203],[546,33419,33420],{"align":532},"fine",[527,33422,33423,33426,33429],{},[546,33424,33425],{"align":532},"huge example count",[546,33427,33428],{"align":532},"needs to fit in memory",[546,33430,33431,33432],{"align":532},"scales well, supports ",[65,33433,33434],{},"partial_fit",[527,33436,33437,33440,33442],{},[546,33438,33439],{"align":532},"incremental learning",[546,33441,33345],{"align":532},[546,33443,31593],{"align":532},[11,33445,33446,33449,33450,33452,33453,33456,33457,33459],{},[15,33447,33448],{},"Rule of thumb:"," I default to ",[65,33451,32966],{}," (or ",[65,33454,33455],{},"Ridge","). I only reach for ",[65,33458,30567],{}," when the data doesn't fit in memory, arrives as a stream, or the feature count is huge.",[294,33461,6192],{"id":6191},[521,33463,33464,33472],{},[524,33465,33466],{},[527,33467,33468,33470],{},[530,33469,26333],{"align":532},[530,33471,26336],{"align":532},[541,33473,33474,33482,33490],{},[527,33475,33476,33479],{},[546,33477,33478],{"align":532},"Gradient descent uses the whole dataset every step",[546,33480,33481],{"align":532},"There's a version that uses one example at a time, cheaper per step, noisier",[527,33483,33484,33487],{},[546,33485,33486],{"align":532},"I implemented all of this by hand up to now",[546,33488,33489],{"align":532},"The field's standard library does the same thing, with the same API convention across hundreds of models",[527,33491,33492,33495],{},[546,33493,33494],{"align":532},"Ridge is a choice I made explicitly",[546,33496,33497],{"align":532},"Scikit-learn's \"default\" stochastic gradient model already ships with Ridge turned on, without saying so",[11,33499,6507],{},[299,33501,33502,33517,33523],{},[302,33503,33504,33516],{},[15,33505,15115,33506,20914,33508,20914,33510,20914,33512,33515],{},[65,33507,30271],{},[65,33509,30324],{},[65,33511,30275],{},[65,33513,33514],{},"score"," convention matters more than memorizing one specific model",", it repeats across the whole library.",[302,33518,33519,33522],{},[15,33520,33521],{},"SGD trades precision for speed per step",": same general direction, a cheaper, noisier path.",[302,33524,33525,6529,33528,33530,33531,33533],{},[15,33526,33527],{},"Know the defaults of what you're using",[65,33529,31723],{}," turned on by default in ",[65,33532,30567],{}," is exactly the kind of detail that changes what your code is actually doing.",[294,33535,6716],{"id":6715},[11,33537,33538,33539,2338,33541,33543,33544,33640,33641,33644],{},"Same real housing dataset from the previous posts. I already know batch gradient descent, with ",[65,33540,26514],{},[65,33542,26517],{},", alpha=0.01, needs 4000 full iterations to reach ",[80,33545,33547,33580],{"className":33546},[83],[80,33548,33550],{"className":33549},[87],[89,33551,33552],{"xmlns":91},[93,33553,33554,33578],{},[96,33555,33556,33558,33560,33562,33564,33566,33568,33570,33572,33574,33576],{},[111,33557,121],{"stretchy":120},[102,33559,109],{},[111,33561,114],{"separator":113},[102,33563,117],{},[111,33565,127],{"stretchy":120},[111,33567,130],{},[111,33569,121],{"stretchy":120},[1321,33571,26599],{},[111,33573,114],{"separator":113},[1321,33575,26604],{},[111,33577,127],{"stretchy":120},[145,33579,26609],{"encoding":147},[80,33581,33583,33616],{"className":33582,"ariaHidden":113},[152],[80,33584,33586,33589,33592,33595,33598,33601,33604,33607,33610,33613],{"className":33585},[156],[80,33587],{"className":33588,"style":2316},[160],[80,33590,121],{"className":33591},[235],[80,33593,109],{"className":33594,"style":210},[165,169],[80,33596,114],{"className":33597},[214],[80,33599],{"className":33600,"style":268},[246],[80,33602,117],{"className":33603},[165,169],[80,33605,127],{"className":33606},[242],[80,33608],{"className":33609,"style":247},[246],[80,33611,130],{"className":33612},[251],[80,33614],{"className":33615,"style":247},[246],[80,33617,33619,33622,33625,33628,33631,33634,33637],{"className":33618},[156],[80,33620],{"className":33621,"style":2316},[160],[80,33623,121],{"className":33624},[235],[80,33626,26599],{"className":33627},[165],[80,33629,114],{"className":33630},[214],[80,33632],{"className":33633,"style":268},[246],[80,33635,26604],{"className":33636},[165],[80,33638,127],{"className":33639},[242],", final cost 5189.72. Each one of those 4000 iterations looks at all 50 houses, so that's ",[15,33642,33643],{},"200 thousand"," example evaluations total.",[11,33646,33647,33648,33651],{},"I ran the stochastic version with the same alpha, but counting ",[15,33649,33650],{},"individual steps"," instead of full iterations:",[2611,33653,33655],{"className":2613,"code":33654,"language":2615,"meta":26,"style":26},"w, b, hist = sgd_gradient_descent(\n    square_feet_norm, price,\n    w_in=0, b_in=0,\n    alpha=0.01, num_steps=4000)  # 4000 examples seen, not 4000 full passes\n\nprint(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[65,33656,33657,33662,33667,33671,33676,33680],{"__ignoreMap":26},[80,33658,33659],{"class":2620,"line":33},[80,33660,33661],{},"w, b, hist = sgd_gradient_descent(\n",[80,33663,33664],{"class":2620,"line":27},[80,33665,33666],{},"    square_feet_norm, price,\n",[80,33668,33669],{"class":2620,"line":2631},[80,33670,26698],{},[80,33672,33673],{"class":2620,"line":2636},[80,33674,33675],{},"    alpha=0.01, num_steps=4000)  # 4000 examples seen, not 4000 full passes\n",[80,33677,33678],{"class":2620,"line":2642},[80,33679,2657],{"emptyLinePlaceholder":32},[80,33681,33682],{"class":2620,"line":2648},[80,33683,14573],{},[46,33685,33686],{},[11,33687,33688,3255,33690,33693],{},[15,33689,2693],{},[65,33691,33692],{},"(w, b) found: (112.1, 396.9)",", final cost 5233.79",[11,33695,33696,33697,33700],{},"Practically the same result as batch (cost 5233.79 versus 5189.72), using ",[15,33698,33699],{},"50 times fewer"," example evaluations (4000 versus 200 thousand). Compare both live:",[14587,33702],{},[33704,33705],"housing-stochastic-gradient-descent-simulator",{},[11,33707,33708],{},"Click \"Rodar 2000\" on each and notice: the stochastic one reaches a good neighborhood much faster in terms of total work, even with the messier path on the chart.",[6949,33710,6951],{},{"title":26,"searchDepth":27,"depth":27,"links":33712},[33713,33714,33716,33718,33720,33721,33722,33723,33725,33726],{"id":30190,"depth":27,"text":30191},{"id":30335,"depth":27,"text":33715},"StandardScaler: the z-score I already built by hand",{"id":30564,"depth":27,"text":33717},"SGDRegressor and what the \"S\" means",{"id":31673,"depth":27,"text":33719},"Two things max_iter hides",{"id":31713,"depth":27,"text":31714},{"id":32852,"depth":27,"text":32853},{"id":32869,"depth":27,"text":32870},{"id":32960,"depth":27,"text":33724},"SGDRegressor or LinearRegression? The real choice",{"id":6191,"depth":27,"text":6192},{"id":6715,"depth":27,"text":6716},"After building gradient descent, normalization, and feature engineering by hand, I finally use scikit-learn, and find out the course's 'default' model hides two surprises nobody warns you about.",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn",{"title":30176,"description":33727},"en\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn",[30198,33733,33734],"stochastic-gradient-descent","sgd","yR3cXoVOUASqEpiV0kbz68EbkdzyppXZRqzgG0_FqX8",1787338982715]