[{"data":1,"prerenderedAt":8665},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling":3,"post-en-machine-learning-specialization-w2-lab03-feature-scaling":4},"\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling",{"id":5,"title":6,"body":7,"cover":8650,"date":8651,"description":8652,"extension":8653,"meta":8654,"navigation":8408,"order":8427,"path":8655,"playlist":8656,"seo":8657,"status":8658,"stem":8659,"tags":8660,"__hash__":8664},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling.md","Feature Scaling and Learning Rate",{"type":8,"value":9,"toc":8635},"minimark",[10,18,86,91,98,210,217,220,224,227,230,328,331,335,956,1269,1301,1305,1308,1330,1459,2351,2354,2364,2558,2602,2605,2872,3028,3558,3760,3975,4007,4013,4114,4279,4456,4460,4492,6058,6237,6270,6273,6280,6284,6413,6419,6684,6751,6755,6758,6773,6779,6786,6861,7738,7903,7907,7917,7929,7934,7940,7945,7948,7951,7954,7959,7965,7970,7978,7981,8036,8040,8135,8138,8217,8221,8394,8448,8458,8622,8625,8628,8631],[11,12,13],"p",{},[14,15],"img",{"alt":16,"src":17},"A confused kid meme with open hands and the caption \"what do you mean I need scaling?\"","\u002Fimages\u002Fposts\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling\u002Fmeme-feature-scaling.jpeg",[11,19,20,21,25,26,31,32,36,37,85],{},"I picked up the optional lab from Course 1, Week 2 (",[22,23,24],"code",{},"C1_W2_Lab03",") expecting just another follow-up post, and ended up finding the topic that annoyed me the most in the whole specialization so far: the exact gradient descent I built in ",[27,28,30],"a",{"href":29},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab04-gradient-descent","the last post",", same algorithm, zero code changes, simply ",[33,34,35],"strong",{},"stops working"," depending on the unit I use to measure a feature. I swapped meters for centimeters and the algorithm diverged. That's not a bug, it's math, and after this post you'll never pick an ",[38,39,42,65],"span",{"className":40},[41],"katex",[38,43,46],{"className":44},[45],"katex-mathml",[47,48,50],"math",{"xmlns":49},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[51,52,53,60],"semantics",{},[54,55,56],"mrow",{},[57,58,59],"mi",{},"α",[61,62,64],"annotation",{"encoding":63},"application\u002Fx-tex","\\alpha",[38,66,70],{"className":67,"ariaHidden":69},[68],"katex-html","true",[38,71,74,79],{"className":72},[73],"base",[38,75],{"className":76,"style":78},[77],"strut","height:0.4306em;",[38,80,59],{"className":81,"style":84},[82,83],"mord","mathnormal","margin-right:0.0037em;"," in the dark again without understanding why.",[87,88,90],"h2",{"id":89},"the-problem-4-features-wildly-different-scales","The problem: 4 features, wildly different scales",[11,92,93,94,97],{},"The original notebook uses a 100-house dataset (derived from the Ames Housing dataset, the same one used in the course) with 4 features: size in sqft, number of bedrooms, number of floors, and age. The goal is to predict the price of a ",[33,95,96],{},"1200 sqft, 3-bedroom, 1-floor, 40-year-old"," house.",[99,100,101,128],"table",{},[102,103,104],"thead",{},[105,106,107,112,116,119,122,125],"tr",{},[108,109,111],"th",{"align":110},"left","Feature",[108,113,115],{"align":114},"right","Min",[108,117,118],{"align":114},"Max",[108,120,121],{"align":114},"Mean",[108,123,124],{"align":114},"Std",[108,126,127],{"align":114},"Range",[129,130,131,152,171,190],"tbody",{},[105,132,133,137,140,143,146,149],{},[134,135,136],"td",{"align":110},"size (sqft)",[134,138,139],{"align":114},"788",[134,141,142],{"align":114},"3194",[134,144,145],{"align":114},"1413.7",[134,147,148],{"align":114},"412.2",[134,150,151],{"align":114},"2406",[105,153,154,157,160,163,166,169],{},[134,155,156],{"align":110},"bedrooms",[134,158,159],{"align":114},"0",[134,161,162],{"align":114},"4",[134,164,165],{"align":114},"2.7",[134,167,168],{"align":114},"0.7",[134,170,162],{"align":114},[105,172,173,176,179,182,185,188],{},[134,174,175],{"align":110},"floors",[134,177,178],{"align":114},"1",[134,180,181],{"align":114},"2",[134,183,184],{"align":114},"1.4",[134,186,187],{"align":114},"0.5",[134,189,178],{"align":114},[105,191,192,195,198,201,204,207],{},[134,193,194],{"align":110},"age (years)",[134,196,197],{"align":114},"12",[134,199,200],{"align":114},"107",[134,202,203],{"align":114},"38.6",[134,205,206],{"align":114},"25.8",[134,208,209],{"align":114},"95",[11,211,212,213,216],{},"I noticed this number the moment I ran the cell: the feature with the biggest range (size) is ",[33,214,215],{},"2406 times"," bigger than the one with the smallest range (floors). I kept that number in mind, because it's the root of everything that follows.",[11,218,219],{},"Before jumping into modeling, I looked at each feature against price, one at a time:",[221,222],"feature-scaling-overview",{"age-label":194,"bedrooms-label":156,"floors-label":175,"price-label":223,"size-label":136},"price (thousand US$)",[11,225,226],{},"Size carries real signal (the cloud of points clearly climbs from left to right). Bedrooms and floors are discrete and noisy, you can see 2-bedroom houses costing more than 3-bedroom ones. Age pulls price down, but with a lot of scatter. That doesn't mean bedrooms and floors are useless, just that alone they explain little, their value shows up once they join the model together with the others (the negative bedrooms coefficient I compute further down is exactly that kind of hidden effect).",[11,228,229],{},"I also computed the correlation between every pair of features (and between each feature and price):",[99,231,232,251],{},[102,233,234],{},[105,235,236,238,241,243,245,248],{},[108,237],{"align":110},[108,239,240],{"align":114},"size",[108,242,156],{"align":114},[108,244,175],{"align":114},[108,246,247],{"align":114},"age",[108,249,250],{"align":114},"price",[129,252,253,274,293,311],{},[105,254,255,259,262,265,268,271],{},[134,256,257],{"align":110},[33,258,240],{},[134,260,261],{"align":114},"1.00",[134,263,264],{"align":114},"0.56",[134,266,267],{"align":114},"0.60",[134,269,270],{"align":114},"-0.27",[134,272,273],{"align":114},"0.86",[105,275,276,280,282,284,287,290],{},[134,277,278],{"align":110},[33,279,156],{},[134,281,264],{"align":114},[134,283,261],{"align":114},[134,285,286],{"align":114},"0.38",[134,288,289],{"align":114},"-0.06",[134,291,292],{"align":114},"0.29",[105,294,295,299,301,303,305,308],{},[134,296,297],{"align":110},[33,298,175],{},[134,300,267],{"align":114},[134,302,286],{"align":114},[134,304,261],{"align":114},[134,306,307],{"align":114},"-0.20",[134,309,310],{"align":114},"0.32",[105,312,313,317,319,321,323,325],{},[134,314,315],{"align":110},[33,316,247],{},[134,318,270],{"align":114},[134,320,289],{"align":114},[134,322,307],{"align":114},[134,324,261],{"align":114},[134,326,327],{"align":114},"-0.58",[11,329,330],{},"Size is by far the feature most correlated with price (0.86), which matches the chart above. And notice size and bedrooms aren't independent (0.56), bigger houses tend to have more bedrooms, which is kind of obvious once you stop to think about it, but it's always good to confirm with a number instead of a guess.",[87,332,334],{"id":333},"quick-recap-with-a-single-feature","Quick recap, with a single feature",[11,336,337,338,528,529,951,952,955],{},"To keep the foundation I already built, I'll simplify to 1 feature (size) for this first part, exactly like the previous posts: ",[38,339,341,393],{"className":340},[41],[38,342,344],{"className":343},[45],[47,345,346],{"xmlns":49},[51,347,348,390],{},[54,349,350,368,372,375,378,381,383,385,388],{},[351,352,353,356],"msub",{},[57,354,355],{},"f",[54,357,358,361,365],{},[57,359,360],{},"w",[362,363,364],"mo",{"separator":69},",",[57,366,367],{},"b",[362,369,371],{"stretchy":370},"false","(",[57,373,374],{},"x",[362,376,377],{"stretchy":370},")",[362,379,380],{},"=",[57,382,360],{},[57,384,374],{},[362,386,387],{},"+",[57,389,367],{},[61,391,392],{"encoding":63},"f_{w,b}(x) = wx + b",[38,394,396,494,518],{"className":395,"ariaHidden":69},[68],[38,397,399,403,471,475,478,482,487,491],{"className":398},[73],[38,400],{"className":401,"style":402},[77],"height:1.0361em;vertical-align:-0.2861em;",[38,404,406,410],{"className":405},[82],[38,407,355],{"className":408,"style":409},[82,83],"margin-right:0.1076em;",[38,411,414],{"className":412},[413],"msupsub",[38,415,419,462],{"className":416},[417,418],"vlist-t","vlist-t2",[38,420,423,457],{"className":421},[422],"vlist-r",[38,424,428],{"className":425,"style":427},[426],"vlist","height:0.3361em;",[38,429,431,436],{"style":430},"top:-2.55em;margin-left:-0.1076em;margin-right:0.05em;",[38,432],{"className":433,"style":435},[434],"pstrut","height:2.7em;",[38,437,443],{"className":438},[439,440,441,442],"sizing","reset-size6","size3","mtight",[38,444,446,450,454],{"className":445},[82,442],[38,447,360],{"className":448,"style":449},[82,83,442],"margin-right:0.0269em;",[38,451,364],{"className":452},[453,442],"mpunct",[38,455,367],{"className":456},[82,83,442],[38,458,461],{"className":459},[460],"vlist-s","​",[38,463,465],{"className":464},[422],[38,466,469],{"className":467,"style":468},[426],"height:0.2861em;",[38,470],{},[38,472,371],{"className":473},[474],"mopen",[38,476,374],{"className":477},[82,83],[38,479,377],{"className":480},[481],"mclose",[38,483],{"className":484,"style":486},[485],"mspace","margin-right:0.2778em;",[38,488,380],{"className":489},[490],"mrel",[38,492],{"className":493,"style":486},[485],[38,495,497,501,504,507,511,515],{"className":496},[73],[38,498],{"className":499,"style":500},[77],"height:0.6667em;vertical-align:-0.0833em;",[38,502,360],{"className":503,"style":449},[82,83],[38,505,374],{"className":506},[82,83],[38,508],{"className":509,"style":510},[485],"margin-right:0.2222em;",[38,512,387],{"className":513},[514],"mbin",[38,516],{"className":517,"style":510},[485],[38,519,521,525],{"className":520},[73],[38,522],{"className":523,"style":524},[77],"height:0.6944em;",[38,526,367],{"className":527},[82,83],", cost ",[38,530,532,629],{"className":531},[41],[38,533,535],{"className":534},[45],[47,536,537],{"xmlns":49},[51,538,539,626],{},[54,540,541,544,546,548,550,552,554,556,569,572,574,586,588,602,604,607,620],{},[57,542,543],{},"J",[362,545,371],{"stretchy":370},[57,547,360],{},[362,549,364],{"separator":69},[57,551,367],{},[362,553,377],{"stretchy":370},[362,555,380],{},[557,558,559,562],"mfrac",{},[560,561,178],"mn",{},[54,563,564,566],{},[560,565,181],{},[57,567,568],{},"m",[362,570,571],{},"∑",[362,573,371],{"stretchy":370},[351,575,576,578],{},[57,577,355],{},[54,579,580,582,584],{},[57,581,360],{},[362,583,364],{"separator":69},[57,585,367],{},[362,587,371],{"stretchy":370},[589,590,591,593],"msup",{},[57,592,374],{},[54,594,595,597,600],{},[362,596,371],{"stretchy":370},[57,598,599],{},"i",[362,601,377],{"stretchy":370},[362,603,377],{"stretchy":370},[362,605,606],{},"−",[589,608,609,612],{},[57,610,611],{},"y",[54,613,614,616,618],{},[362,615,371],{"stretchy":370},[57,617,599],{},[362,619,377],{"stretchy":370},[589,621,622,624],{},[362,623,377],{"stretchy":370},[560,625,181],{},[61,627,628],{"encoding":63},"J(w,b) = \\frac{1}{2m}\\sum(f_{w,b}(x^{(i)}) - y^{(i)})^2",[38,630,632,671,875],{"className":631,"ariaHidden":69},[68],[38,633,635,639,643,646,649,652,656,659,662,665,668],{"className":634},[73],[38,636],{"className":637,"style":638},[77],"height:1em;vertical-align:-0.25em;",[38,640,543],{"className":641,"style":642},[82,83],"margin-right:0.0962em;",[38,644,371],{"className":645},[474],[38,647,360],{"className":648,"style":449},[82,83],[38,650,364],{"className":651},[453],[38,653],{"className":654,"style":655},[485],"margin-right:0.1667em;",[38,657,367],{"className":658},[82,83],[38,660,377],{"className":661},[481],[38,663],{"className":664,"style":486},[485],[38,666,380],{"className":667},[490],[38,669],{"className":670,"style":486},[485],[38,672,674,678,758,761,768,771,820,823,863,866,869,872],{"className":673},[73],[38,675],{"className":676,"style":677},[77],"height:1.233em;vertical-align:-0.345em;",[38,679,681,685,755],{"className":680},[82],[38,682],{"className":683},[474,684],"nulldelimiter",[38,686,688],{"className":687},[557],[38,689,691,746],{"className":690},[417,418],[38,692,694,743],{"className":693},[422],[38,695,698,717,728],{"className":696,"style":697},[426],"height:0.8451em;",[38,699,701,705],{"style":700},"top:-2.655em;",[38,702],{"className":703,"style":704},[434],"height:3em;",[38,706,708],{"className":707},[439,440,441,442],[38,709,711,714],{"className":710},[82,442],[38,712,181],{"className":713},[82,442],[38,715,568],{"className":716},[82,83,442],[38,718,720,723],{"style":719},"top:-3.23em;",[38,721],{"className":722,"style":704},[434],[38,724],{"className":725,"style":727},[726],"frac-line","border-bottom-width:0.04em;",[38,729,731,734],{"style":730},"top:-3.394em;",[38,732],{"className":733,"style":704},[434],[38,735,737],{"className":736},[439,440,441,442],[38,738,740],{"className":739},[82,442],[38,741,178],{"className":742},[82,442],[38,744,461],{"className":745},[460],[38,747,749],{"className":748},[422],[38,750,753],{"className":751,"style":752},[426],"height:0.345em;",[38,754],{},[38,756],{"className":757},[481,684],[38,759],{"className":760,"style":655},[485],[38,762,571],{"className":763,"style":767},[764,765,766],"mop","op-symbol","small-op","position:relative;top:0em;",[38,769,371],{"className":770},[474],[38,772,774,777],{"className":773},[82],[38,775,355],{"className":776,"style":409},[82,83],[38,778,780],{"className":779},[413],[38,781,783,812],{"className":782},[417,418],[38,784,786,809],{"className":785},[422],[38,787,789],{"className":788,"style":427},[426],[38,790,791,794],{"style":430},[38,792],{"className":793,"style":435},[434],[38,795,797],{"className":796},[439,440,441,442],[38,798,800,803,806],{"className":799},[82,442],[38,801,360],{"className":802,"style":449},[82,83,442],[38,804,364],{"className":805},[453,442],[38,807,367],{"className":808},[82,83,442],[38,810,461],{"className":811},[460],[38,813,815],{"className":814},[422],[38,816,818],{"className":817,"style":468},[426],[38,819],{},[38,821,371],{"className":822},[474],[38,824,826,829],{"className":825},[82],[38,827,374],{"className":828},[82,83],[38,830,832],{"className":831},[413],[38,833,835],{"className":834},[417],[38,836,838],{"className":837},[422],[38,839,842],{"className":840,"style":841},[426],"height:0.888em;",[38,843,845,848],{"style":844},"top:-3.063em;margin-right:0.05em;",[38,846],{"className":847,"style":435},[434],[38,849,851],{"className":850},[439,440,441,442],[38,852,854,857,860],{"className":853},[82,442],[38,855,371],{"className":856},[474,442],[38,858,599],{"className":859},[82,83,442],[38,861,377],{"className":862},[481,442],[38,864,377],{"className":865},[481],[38,867],{"className":868,"style":510},[485],[38,870,606],{"className":871},[514],[38,873],{"className":874,"style":510},[485],[38,876,878,882,921],{"className":877},[73],[38,879],{"className":880,"style":881},[77],"height:1.138em;vertical-align:-0.25em;",[38,883,885,889],{"className":884},[82],[38,886,611],{"className":887,"style":888},[82,83],"margin-right:0.0359em;",[38,890,892],{"className":891},[413],[38,893,895],{"className":894},[417],[38,896,898],{"className":897},[422],[38,899,901],{"className":900,"style":841},[426],[38,902,903,906],{"style":844},[38,904],{"className":905,"style":435},[434],[38,907,909],{"className":908},[439,440,441,442],[38,910,912,915,918],{"className":911},[82,442],[38,913,371],{"className":914},[474,442],[38,916,599],{"className":917},[82,83,442],[38,919,377],{"className":920},[481,442],[38,922,924,927],{"className":923},[481],[38,925,377],{"className":926},[481],[38,928,930],{"className":929},[413],[38,931,933],{"className":932},[417],[38,934,936],{"className":935},[422],[38,937,940],{"className":938,"style":939},[426],"height:0.8141em;",[38,941,942,945],{"style":844},[38,943],{"className":944,"style":435},[434],[38,946,948],{"className":947},[439,440,441,442],[38,949,181],{"className":950},[82,442],". The gradient descent I built ",[27,953,954],{"href":29},"two posts ago"," doesn't change a single line:",[11,957,958],{},[38,959,961,1026],{"className":960},[41],[38,962,964],{"className":963},[45],[47,965,966],{"xmlns":49},[51,967,968,1023],{},[54,969,970,972,974,976,978,980,996,999,1001,1003,1005,1007,1009],{},[57,971,360],{},[362,973,380],{},[57,975,360],{},[362,977,606],{},[57,979,59],{},[557,981,982,990],{},[54,983,984,988],{},[57,985,987],{"mathvariant":986},"normal","∂",[57,989,543],{},[54,991,992,994],{},[57,993,987],{"mathvariant":986},[57,995,360],{},[485,997],{"width":998},"2em",[57,1000,367],{},[362,1002,380],{},[57,1004,367],{},[362,1006,606],{},[57,1008,59],{},[557,1010,1011,1017],{},[54,1012,1013,1015],{},[57,1014,987],{"mathvariant":986},[57,1016,543],{},[54,1018,1019,1021],{},[57,1020,987],{"mathvariant":986},[57,1022,367],{},[61,1024,1025],{"encoding":63},"w = w - \\alpha \\frac{\\partial J}{\\partial w} \\qquad b = b - \\alpha \\frac{\\partial J}{\\partial b}",[38,1027,1029,1047,1065,1167,1186],{"className":1028,"ariaHidden":69},[68],[38,1030,1032,1035,1038,1041,1044],{"className":1031},[73],[38,1033],{"className":1034,"style":78},[77],[38,1036,360],{"className":1037,"style":449},[82,83],[38,1039],{"className":1040,"style":486},[485],[38,1042,380],{"className":1043},[490],[38,1045],{"className":1046,"style":486},[485],[38,1048,1050,1053,1056,1059,1062],{"className":1049},[73],[38,1051],{"className":1052,"style":500},[77],[38,1054,360],{"className":1055,"style":449},[82,83],[38,1057],{"className":1058,"style":510},[485],[38,1060,606],{"className":1061},[514],[38,1063],{"className":1064,"style":510},[485],[38,1066,1068,1072,1075,1151,1155,1158,1161,1164],{"className":1067},[73],[38,1069],{"className":1070,"style":1071},[77],"height:1.2251em;vertical-align:-0.345em;",[38,1073,59],{"className":1074,"style":84},[82,83],[38,1076,1078,1081,1148],{"className":1077},[82],[38,1079],{"className":1080},[474,684],[38,1082,1084],{"className":1083},[557],[38,1085,1087,1140],{"className":1086},[417,418],[38,1088,1090,1137],{"className":1089},[422],[38,1091,1094,1112,1120],{"className":1092,"style":1093},[426],"height:0.8801em;",[38,1095,1096,1099],{"style":700},[38,1097],{"className":1098,"style":704},[434],[38,1100,1102],{"className":1101},[439,440,441,442],[38,1103,1105,1109],{"className":1104},[82,442],[38,1106,987],{"className":1107,"style":1108},[82,442],"margin-right:0.0556em;",[38,1110,360],{"className":1111,"style":449},[82,83,442],[38,1113,1114,1117],{"style":719},[38,1115],{"className":1116,"style":704},[434],[38,1118],{"className":1119,"style":727},[726],[38,1121,1122,1125],{"style":730},[38,1123],{"className":1124,"style":704},[434],[38,1126,1128],{"className":1127},[439,440,441,442],[38,1129,1131,1134],{"className":1130},[82,442],[38,1132,987],{"className":1133,"style":1108},[82,442],[38,1135,543],{"className":1136,"style":642},[82,83,442],[38,1138,461],{"className":1139},[460],[38,1141,1143],{"className":1142},[422],[38,1144,1146],{"className":1145,"style":752},[426],[38,1147],{},[38,1149],{"className":1150},[481,684],[38,1152],{"className":1153,"style":1154},[485],"margin-right:2em;",[38,1156,367],{"className":1157},[82,83],[38,1159],{"className":1160,"style":486},[485],[38,1162,380],{"className":1163},[490],[38,1165],{"className":1166,"style":486},[485],[38,1168,1170,1174,1177,1180,1183],{"className":1169},[73],[38,1171],{"className":1172,"style":1173},[77],"height:0.7778em;vertical-align:-0.0833em;",[38,1175,367],{"className":1176},[82,83],[38,1178],{"className":1179,"style":510},[485],[38,1181,606],{"className":1182},[514],[38,1184],{"className":1185,"style":510},[485],[38,1187,1189,1192,1195],{"className":1188},[73],[38,1190],{"className":1191,"style":1071},[77],[38,1193,59],{"className":1194,"style":84},[82,83],[38,1196,1198,1201,1266],{"className":1197},[82],[38,1199],{"className":1200},[474,684],[38,1202,1204],{"className":1203},[557],[38,1205,1207,1258],{"className":1206},[417,418],[38,1208,1210,1255],{"className":1209},[422],[38,1211,1213,1230,1238],{"className":1212,"style":1093},[426],[38,1214,1215,1218],{"style":700},[38,1216],{"className":1217,"style":704},[434],[38,1219,1221],{"className":1220},[439,440,441,442],[38,1222,1224,1227],{"className":1223},[82,442],[38,1225,987],{"className":1226,"style":1108},[82,442],[38,1228,367],{"className":1229},[82,83,442],[38,1231,1232,1235],{"style":719},[38,1233],{"className":1234,"style":704},[434],[38,1236],{"className":1237,"style":727},[726],[38,1239,1240,1243],{"style":730},[38,1241],{"className":1242,"style":704},[434],[38,1244,1246],{"className":1245},[439,440,441,442],[38,1247,1249,1252],{"className":1248},[82,442],[38,1250,987],{"className":1251,"style":1108},[82,442],[38,1253,543],{"className":1254,"style":642},[82,83,442],[38,1256,461],{"className":1257},[460],[38,1259,1261],{"className":1260},[422],[38,1262,1264],{"className":1263,"style":752},[426],[38,1265],{},[38,1267],{"className":1268},[481,684],[11,1270,1271,1272,1300],{},"The real notebook runs this with all 4 features at once (one ",[38,1273,1275,1288],{"className":1274},[41],[38,1276,1278],{"className":1277},[45],[47,1279,1280],{"xmlns":49},[51,1281,1282,1286],{},[54,1283,1284],{},[57,1285,360],{},[61,1287,360],{"encoding":63},[38,1289,1291],{"className":1290,"ariaHidden":69},[68],[38,1292,1294,1297],{"className":1293},[73],[38,1295],{"className":1296,"style":78},[77],[38,1298,360],{"className":1299,"style":449},[82,83]," per feature), but the core lesson shows up completely already with a single feature, and it's much easier to visualize on a 2D chart. Keep this simplification in mind, it comes back later when I show the result with the real 4 features.",[87,1302,1304],{"id":1303},"gradient-descent-diverges-on-its-own","Gradient descent diverges on its own",[11,1306,1307],{},"I picked 8 real houses from the notebook's dataset (size in sqft, price in thousands of dollars) to test this by hand:",[1309,1310,1315],"pre",{"className":1311,"code":1312,"language":1313,"meta":1314,"style":1314},"language-python shiki shiki-themes github-light github-dark","x_train = [952, 1244, 1947, 1725, 1959, 1314, 864, 1836]\ny_train = [271.5, 300, 509.8, 394, 540, 415, 230, 560]\n","python","",[22,1316,1317,1324],{"__ignoreMap":1314},[38,1318,1321],{"class":1319,"line":1320},"line",1,[38,1322,1323],{},"x_train = [952, 1244, 1947, 1725, 1959, 1314, 864, 1836]\n",[38,1325,1327],{"class":1319,"line":1326},2,[38,1328,1329],{},"y_train = [271.5, 300, 509.8, 394, 540, 415, 230, 560]\n",[11,1331,1332,1333,1336,1337,1340,1341,1369,1370,1458],{},"I ran the same ",[22,1334,1335],{},"gradient_descent"," from ",[27,1338,1339],{"href":29},"the previous post"," with three different ",[38,1342,1344,1357],{"className":1343},[41],[38,1345,1347],{"className":1346},[45],[47,1348,1349],{"xmlns":49},[51,1350,1351,1355],{},[54,1352,1353],{},[57,1354,59],{},[61,1356,64],{"encoding":63},[38,1358,1360],{"className":1359,"ariaHidden":69},[68],[38,1361,1363,1366],{"className":1362},[73],[38,1364],{"className":1365,"style":78},[77],[38,1367,59],{"className":1368,"style":84},[82,83]," values, just 10 iterations, starting at ",[38,1371,1373,1399],{"className":1372},[41],[38,1374,1376],{"className":1375},[45],[47,1377,1378],{"xmlns":49},[51,1379,1380,1396],{},[54,1381,1382,1384,1386,1388,1390,1392,1394],{},[57,1383,360],{},[362,1385,380],{},[560,1387,159],{},[362,1389,364],{"separator":69},[57,1391,367],{},[362,1393,380],{},[560,1395,159],{},[61,1397,1398],{"encoding":63},"w=0, b=0",[38,1400,1402,1420,1448],{"className":1401,"ariaHidden":69},[68],[38,1403,1405,1408,1411,1414,1417],{"className":1404},[73],[38,1406],{"className":1407,"style":78},[77],[38,1409,360],{"className":1410,"style":449},[82,83],[38,1412],{"className":1413,"style":486},[485],[38,1415,380],{"className":1416},[490],[38,1418],{"className":1419,"style":486},[485],[38,1421,1423,1427,1430,1433,1436,1439,1442,1445],{"className":1422},[73],[38,1424],{"className":1425,"style":1426},[77],"height:0.8889em;vertical-align:-0.1944em;",[38,1428,159],{"className":1429},[82],[38,1431,364],{"className":1432},[453],[38,1434],{"className":1435,"style":655},[485],[38,1437,367],{"className":1438},[82,83],[38,1440],{"className":1441,"style":486},[485],[38,1443,380],{"className":1444},[490],[38,1446],{"className":1447,"style":486},[485],[38,1449,1451,1455],{"className":1450},[73],[38,1452],{"className":1453,"style":1454},[77],"height:0.6444em;",[38,1456,159],{"className":1457},[82],":",[1460,1461,1462,1765,2059],"blockquote",{},[11,1463,1464,1588,1589,1674,1675,1761,1762,1764],{},[33,1465,1466,1467,1458],{},"Output with ",[38,1468,1470,1504],{"className":1469},[41],[38,1471,1473],{"className":1472},[45],[47,1474,1475],{"xmlns":49},[51,1476,1477,1501],{},[54,1478,1479,1481,1483,1486,1489],{},[57,1480,59],{},[362,1482,380],{},[560,1484,1485],{},"9",[362,1487,1488],{},"×",[589,1490,1491,1494],{},[560,1492,1493],{},"10",[54,1495,1496,1498],{},[362,1497,606],{},[560,1499,1500],{},"7",[61,1502,1503],{"encoding":63},"\\alpha = 9\\times10^{-7}",[38,1505,1507,1525,1544],{"className":1506,"ariaHidden":69},[68],[38,1508,1510,1513,1516,1519,1522],{"className":1509},[73],[38,1511],{"className":1512,"style":78},[77],[38,1514,59],{"className":1515,"style":84},[82,83],[38,1517],{"className":1518,"style":486},[485],[38,1520,380],{"className":1521},[490],[38,1523],{"className":1524,"style":486},[485],[38,1526,1528,1532,1535,1538,1541],{"className":1527},[73],[38,1529],{"className":1530,"style":1531},[77],"height:0.7278em;vertical-align:-0.0833em;",[38,1533,1485],{"className":1534},[82],[38,1536],{"className":1537,"style":510},[485],[38,1539,1488],{"className":1540},[514],[38,1542],{"className":1543,"style":510},[485],[38,1545,1547,1550,1553],{"className":1546},[73],[38,1548],{"className":1549,"style":939},[77],[38,1551,178],{"className":1552},[82],[38,1554,1556,1559],{"className":1555},[82],[38,1557,159],{"className":1558},[82],[38,1560,1562],{"className":1561},[413],[38,1563,1565],{"className":1564},[417],[38,1566,1568],{"className":1567},[422],[38,1569,1571],{"className":1570,"style":939},[426],[38,1572,1573,1576],{"style":844},[38,1574],{"className":1575,"style":435},[434],[38,1577,1579],{"className":1578},[439,440,441,442],[38,1580,1582,1585],{"className":1581},[82,442],[38,1583,606],{"className":1584},[82,442],[38,1586,1500],{"className":1587},[82,442]," cost climbs every iteration, from ",[38,1590,1592,1615],{"className":1591},[41],[38,1593,1595],{"className":1594},[45],[47,1596,1597],{"xmlns":49},[51,1598,1599,1612],{},[54,1600,1601,1604,1606],{},[560,1602,1603],{},"8.8",[362,1605,1488],{},[589,1607,1608,1610],{},[560,1609,1493],{},[560,1611,162],{},[61,1613,1614],{"encoding":63},"8.8\\times10^4",[38,1616,1618,1636],{"className":1617,"ariaHidden":69},[68],[38,1619,1621,1624,1627,1630,1633],{"className":1620},[73],[38,1622],{"className":1623,"style":1531},[77],[38,1625,1603],{"className":1626},[82],[38,1628],{"className":1629,"style":510},[485],[38,1631,1488],{"className":1632},[514],[38,1634],{"className":1635,"style":510},[485],[38,1637,1639,1642,1645],{"className":1638},[73],[38,1640],{"className":1641,"style":939},[77],[38,1643,178],{"className":1644},[82],[38,1646,1648,1651],{"className":1647},[82],[38,1649,159],{"className":1650},[82],[38,1652,1654],{"className":1653},[413],[38,1655,1657],{"className":1656},[417],[38,1658,1660],{"className":1659},[422],[38,1661,1663],{"className":1662,"style":939},[426],[38,1664,1665,1668],{"style":844},[38,1666],{"className":1667,"style":435},[434],[38,1669,1671],{"className":1670},[439,440,441,442],[38,1672,162],{"className":1673},[82,442]," to ",[38,1676,1678,1702],{"className":1677},[41],[38,1679,1681],{"className":1680},[45],[47,1682,1683],{"xmlns":49},[51,1684,1685,1699],{},[54,1686,1687,1690,1692],{},[560,1688,1689],{},"9.4",[362,1691,1488],{},[589,1693,1694,1696],{},[560,1695,1493],{},[560,1697,1698],{},"5",[61,1700,1701],{"encoding":63},"9.4\\times10^5",[38,1703,1705,1723],{"className":1704,"ariaHidden":69},[68],[38,1706,1708,1711,1714,1717,1720],{"className":1707},[73],[38,1709],{"className":1710,"style":1531},[77],[38,1712,1689],{"className":1713},[82],[38,1715],{"className":1716,"style":510},[485],[38,1718,1488],{"className":1719},[514],[38,1721],{"className":1722,"style":510},[485],[38,1724,1726,1729,1732],{"className":1725},[73],[38,1727],{"className":1728,"style":939},[77],[38,1730,178],{"className":1731},[82],[38,1733,1735,1738],{"className":1734},[82],[38,1736,159],{"className":1737},[82],[38,1739,1741],{"className":1740},[413],[38,1742,1744],{"className":1743},[417],[38,1745,1747],{"className":1746},[422],[38,1748,1750],{"className":1749,"style":939},[426],[38,1751,1752,1755],{"style":844},[38,1753],{"className":1754,"style":435},[434],[38,1756,1758],{"className":1757},[439,440,441,442],[38,1759,1698],{"className":1760},[82,442],". ",[22,1763,360],{}," flips sign constantly.",[11,1766,1767,1886,1887,1674,1970,2055,2056,2058],{},[33,1768,1466,1769,1458],{},[38,1770,1772,1803],{"className":1771},[41],[38,1773,1775],{"className":1774},[45],[47,1776,1777],{"xmlns":49},[51,1778,1779,1800],{},[54,1780,1781,1783,1785,1788,1790],{},[57,1782,59],{},[362,1784,380],{},[560,1786,1787],{},"8",[362,1789,1488],{},[589,1791,1792,1794],{},[560,1793,1493],{},[54,1795,1796,1798],{},[362,1797,606],{},[560,1799,1500],{},[61,1801,1802],{"encoding":63},"\\alpha = 8\\times10^{-7}",[38,1804,1806,1824,1842],{"className":1805,"ariaHidden":69},[68],[38,1807,1809,1812,1815,1818,1821],{"className":1808},[73],[38,1810],{"className":1811,"style":78},[77],[38,1813,59],{"className":1814,"style":84},[82,83],[38,1816],{"className":1817,"style":486},[485],[38,1819,380],{"className":1820},[490],[38,1822],{"className":1823,"style":486},[485],[38,1825,1827,1830,1833,1836,1839],{"className":1826},[73],[38,1828],{"className":1829,"style":1531},[77],[38,1831,1787],{"className":1832},[82],[38,1834],{"className":1835,"style":510},[485],[38,1837,1488],{"className":1838},[514],[38,1840],{"className":1841,"style":510},[485],[38,1843,1845,1848,1851],{"className":1844},[73],[38,1846],{"className":1847,"style":939},[77],[38,1849,178],{"className":1850},[82],[38,1852,1854,1857],{"className":1853},[82],[38,1855,159],{"className":1856},[82],[38,1858,1860],{"className":1859},[413],[38,1861,1863],{"className":1862},[417],[38,1864,1866],{"className":1865},[422],[38,1867,1869],{"className":1868,"style":939},[426],[38,1870,1871,1874],{"style":844},[38,1872],{"className":1873,"style":435},[434],[38,1875,1877],{"className":1876},[439,440,441,442],[38,1878,1880,1883],{"className":1879},[82,442],[38,1881,606],{"className":1882},[82,442],[38,1884,1500],{"className":1885},[82,442]," cost drops from ",[38,1888,1890,1911],{"className":1889},[41],[38,1891,1893],{"className":1892},[45],[47,1894,1895],{"xmlns":49},[51,1896,1897,1909],{},[54,1898,1899,1901,1903],{},[560,1900,1603],{},[362,1902,1488],{},[589,1904,1905,1907],{},[560,1906,1493],{},[560,1908,162],{},[61,1910,1614],{"encoding":63},[38,1912,1914,1932],{"className":1913,"ariaHidden":69},[68],[38,1915,1917,1920,1923,1926,1929],{"className":1916},[73],[38,1918],{"className":1919,"style":1531},[77],[38,1921,1603],{"className":1922},[82],[38,1924],{"className":1925,"style":510},[485],[38,1927,1488],{"className":1928},[514],[38,1930],{"className":1931,"style":510},[485],[38,1933,1935,1938,1941],{"className":1934},[73],[38,1936],{"className":1937,"style":939},[77],[38,1939,178],{"className":1940},[82],[38,1942,1944,1947],{"className":1943},[82],[38,1945,159],{"className":1946},[82],[38,1948,1950],{"className":1949},[413],[38,1951,1953],{"className":1952},[417],[38,1954,1956],{"className":1955},[422],[38,1957,1959],{"className":1958,"style":939},[426],[38,1960,1961,1964],{"style":844},[38,1962],{"className":1963,"style":435},[434],[38,1965,1967],{"className":1966},[439,440,441,442],[38,1968,162],{"className":1969},[82,442],[38,1971,1973,1996],{"className":1972},[41],[38,1974,1976],{"className":1975},[45],[47,1977,1978],{"xmlns":49},[51,1979,1980,1993],{},[54,1981,1982,1984,1986],{},[560,1983,1689],{},[362,1985,1488],{},[589,1987,1988,1990],{},[560,1989,1493],{},[560,1991,1992],{},"3",[61,1994,1995],{"encoding":63},"9.4\\times10^3",[38,1997,1999,2017],{"className":1998,"ariaHidden":69},[68],[38,2000,2002,2005,2008,2011,2014],{"className":2001},[73],[38,2003],{"className":2004,"style":1531},[77],[38,2006,1689],{"className":2007},[82],[38,2009],{"className":2010,"style":510},[485],[38,2012,1488],{"className":2013},[514],[38,2015],{"className":2016,"style":510},[485],[38,2018,2020,2023,2026],{"className":2019},[73],[38,2021],{"className":2022,"style":939},[77],[38,2024,178],{"className":2025},[82],[38,2027,2029,2032],{"className":2028},[82],[38,2030,159],{"className":2031},[82],[38,2033,2035],{"className":2034},[413],[38,2036,2038],{"className":2037},[417],[38,2039,2041],{"className":2040},[422],[38,2042,2044],{"className":2043,"style":939},[426],[38,2045,2046,2049],{"style":844},[38,2047],{"className":2048,"style":435},[434],[38,2050,2052],{"className":2051},[439,440,441,442],[38,2053,1992],{"className":2054},[82,442],", but ",[22,2057,360],{}," still oscillates (0 → 0.51 → 0.06 → 0.46 → 0.10...).",[11,2060,2061,1886,2179,1674,2262,2347,2348,2350],{},[33,2062,1466,2063,1458],{},[38,2064,2066,2096],{"className":2065},[41],[38,2067,2069],{"className":2068},[45],[47,2070,2071],{"xmlns":49},[51,2072,2073,2093],{},[54,2074,2075,2077,2079,2081,2083],{},[57,2076,59],{},[362,2078,380],{},[560,2080,178],{},[362,2082,1488],{},[589,2084,2085,2087],{},[560,2086,1493],{},[54,2088,2089,2091],{},[362,2090,606],{},[560,2092,1500],{},[61,2094,2095],{"encoding":63},"\\alpha = 1\\times10^{-7}",[38,2097,2099,2117,2135],{"className":2098,"ariaHidden":69},[68],[38,2100,2102,2105,2108,2111,2114],{"className":2101},[73],[38,2103],{"className":2104,"style":78},[77],[38,2106,59],{"className":2107,"style":84},[82,83],[38,2109],{"className":2110,"style":486},[485],[38,2112,380],{"className":2113},[490],[38,2115],{"className":2116,"style":486},[485],[38,2118,2120,2123,2126,2129,2132],{"className":2119},[73],[38,2121],{"className":2122,"style":1531},[77],[38,2124,178],{"className":2125},[82],[38,2127],{"className":2128,"style":510},[485],[38,2130,1488],{"className":2131},[514],[38,2133],{"className":2134,"style":510},[485],[38,2136,2138,2141,2144],{"className":2137},[73],[38,2139],{"className":2140,"style":939},[77],[38,2142,178],{"className":2143},[82],[38,2145,2147,2150],{"className":2146},[82],[38,2148,159],{"className":2149},[82],[38,2151,2153],{"className":2152},[413],[38,2154,2156],{"className":2155},[417],[38,2157,2159],{"className":2158},[422],[38,2160,2162],{"className":2161,"style":939},[426],[38,2163,2164,2167],{"style":844},[38,2165],{"className":2166,"style":435},[434],[38,2168,2170],{"className":2169},[439,440,441,442],[38,2171,2173,2176],{"className":2172},[82,442],[38,2174,606],{"className":2175},[82,442],[38,2177,1500],{"className":2178},[82,442],[38,2180,2182,2203],{"className":2181},[41],[38,2183,2185],{"className":2184},[45],[47,2186,2187],{"xmlns":49},[51,2188,2189,2201],{},[54,2190,2191,2193,2195],{},[560,2192,1603],{},[362,2194,1488],{},[589,2196,2197,2199],{},[560,2198,1493],{},[560,2200,162],{},[61,2202,1614],{"encoding":63},[38,2204,2206,2224],{"className":2205,"ariaHidden":69},[68],[38,2207,2209,2212,2215,2218,2221],{"className":2208},[73],[38,2210],{"className":2211,"style":1531},[77],[38,2213,1603],{"className":2214},[82],[38,2216],{"className":2217,"style":510},[485],[38,2219,1488],{"className":2220},[514],[38,2222],{"className":2223,"style":510},[485],[38,2225,2227,2230,2233],{"className":2226},[73],[38,2228],{"className":2229,"style":939},[77],[38,2231,178],{"className":2232},[82],[38,2234,2236,2239],{"className":2235},[82],[38,2237,159],{"className":2238},[82],[38,2240,2242],{"className":2241},[413],[38,2243,2245],{"className":2244},[417],[38,2246,2248],{"className":2247},[422],[38,2249,2251],{"className":2250,"style":939},[426],[38,2252,2253,2256],{"style":844},[38,2254],{"className":2255,"style":435},[434],[38,2257,2259],{"className":2258},[439,440,441,442],[38,2260,162],{"className":2261},[82,442],[38,2263,2265,2288],{"className":2264},[41],[38,2266,2268],{"className":2267},[45],[47,2269,2270],{"xmlns":49},[51,2271,2272,2285],{},[54,2273,2274,2277,2279],{},[560,2275,2276],{},"1.3",[362,2278,1488],{},[589,2280,2281,2283],{},[560,2282,1493],{},[560,2284,1992],{},[61,2286,2287],{"encoding":63},"1.3\\times10^3",[38,2289,2291,2309],{"className":2290,"ariaHidden":69},[68],[38,2292,2294,2297,2300,2303,2306],{"className":2293},[73],[38,2295],{"className":2296,"style":1531},[77],[38,2298,2276],{"className":2299},[82],[38,2301],{"className":2302,"style":510},[485],[38,2304,1488],{"className":2305},[514],[38,2307],{"className":2308,"style":510},[485],[38,2310,2312,2315,2318],{"className":2311},[73],[38,2313],{"className":2314,"style":939},[77],[38,2316,178],{"className":2317},[82],[38,2319,2321,2324],{"className":2320},[82],[38,2322,159],{"className":2323},[82],[38,2325,2327],{"className":2326},[413],[38,2328,2330],{"className":2329},[417],[38,2331,2333],{"className":2332},[422],[38,2334,2336],{"className":2335,"style":939},[426],[38,2337,2338,2341],{"style":844},[38,2339],{"className":2340,"style":435},[434],[38,2342,2344],{"className":2343},[439,440,441,442],[38,2345,1992],{"className":2346},[82,442],", ",[22,2349,360],{}," climbs straight up without oscillating (0 → 0.06 → 0.11 → 0.15...).",[11,2352,2353],{},"Try it yourself in the simulator below. I started with a fairly small alpha on purpose, drag the slider up slowly and notice exactly where the cost flips from \"dropping\" to \"climbing\":",[2355,2356],"gradient-descent-simulator",{":b-range":2357,":initial-b":159,":initial-w":159,":w-range":2358,":x-train":2359,":y-train":2360,"b-label":367,"w-label":360,":alpha-presets":2361,":alpha-slider-max":2362,":alpha-slider-min":2363,":alpha-slider-step":2363},"[-250, 250]","[0, 0.6]","[952, 1244, 1947, 1725, 1959, 1314, 864, 1836]","[271.5, 300, 509.8, 394, 540, 415, 230, 560]","[1e-7, 3e-7, 8e-7, 9e-7, 9.9e-7]","2e-6","1e-8",[11,2365,2366,2367,2461,2462,2557],{},"The pattern I saw matches exactly what the notebook describes for the full dataset (with all 100 houses and all 4 features, the threshold sits between ",[38,2368,2370,2396],{"className":2369},[41],[38,2371,2373],{"className":2372},[45],[47,2374,2375],{"xmlns":49},[51,2376,2377,2393],{},[54,2378,2379,2381,2383],{},[560,2380,1485],{},[362,2382,1488],{},[589,2384,2385,2387],{},[560,2386,1493],{},[54,2388,2389,2391],{},[362,2390,606],{},[560,2392,1500],{},[61,2394,2395],{"encoding":63},"9\\times10^{-7}",[38,2397,2399,2417],{"className":2398,"ariaHidden":69},[68],[38,2400,2402,2405,2408,2411,2414],{"className":2401},[73],[38,2403],{"className":2404,"style":1531},[77],[38,2406,1485],{"className":2407},[82],[38,2409],{"className":2410,"style":510},[485],[38,2412,1488],{"className":2413},[514],[38,2415],{"className":2416,"style":510},[485],[38,2418,2420,2423,2426],{"className":2419},[73],[38,2421],{"className":2422,"style":939},[77],[38,2424,178],{"className":2425},[82],[38,2427,2429,2432],{"className":2428},[82],[38,2430,159],{"className":2431},[82],[38,2433,2435],{"className":2434},[413],[38,2436,2438],{"className":2437},[417],[38,2439,2441],{"className":2440},[422],[38,2442,2444],{"className":2443,"style":939},[426],[38,2445,2446,2449],{"style":844},[38,2447],{"className":2448,"style":435},[434],[38,2450,2452],{"className":2451},[439,440,441,442],[38,2453,2455,2458],{"className":2454},[82,442],[38,2456,606],{"className":2457},[82,442],[38,2459,1500],{"className":2460},[82,442]," and ",[38,2463,2465,2492],{"className":2464},[41],[38,2466,2468],{"className":2467},[45],[47,2469,2470],{"xmlns":49},[51,2471,2472,2489],{},[54,2473,2474,2477,2479],{},[560,2475,2476],{},"9.9",[362,2478,1488],{},[589,2480,2481,2483],{},[560,2482,1493],{},[54,2484,2485,2487],{},[362,2486,606],{},[560,2488,1500],{},[61,2490,2491],{"encoding":63},"9.9\\times10^{-7}",[38,2493,2495,2513],{"className":2494,"ariaHidden":69},[68],[38,2496,2498,2501,2504,2507,2510],{"className":2497},[73],[38,2499],{"className":2500,"style":1531},[77],[38,2502,2476],{"className":2503},[82],[38,2505],{"className":2506,"style":510},[485],[38,2508,1488],{"className":2509},[514],[38,2511],{"className":2512,"style":510},[485],[38,2514,2516,2519,2522],{"className":2515},[73],[38,2517],{"className":2518,"style":939},[77],[38,2520,178],{"className":2521},[82],[38,2523,2525,2528],{"className":2524},[82],[38,2526,159],{"className":2527},[82],[38,2529,2531],{"className":2530},[413],[38,2532,2534],{"className":2533},[417],[38,2535,2537],{"className":2536},[422],[38,2538,2540],{"className":2539,"style":939},[426],[38,2541,2542,2545],{"style":844},[38,2543],{"className":2544,"style":435},[434],[38,2546,2548],{"className":2547},[439,440,441,442],[38,2549,2551,2554],{"className":2550},[82,442],[38,2552,606],{"className":2553},[82,442],[38,2555,1500],{"className":2556},[82,442],", very close to what I found here with just size and a smaller subset). That's not a coincidence, and the next section explains exactly why.",[87,2559,2561,2562,2601],{"id":2560},"why-exactly-there-the-2l2l2l-bound-and-the-condition-number","Why exactly there: the ",[38,2563,2565,2585],{"className":2564},[41],[38,2566,2568],{"className":2567},[45],[47,2569,2570],{"xmlns":49},[51,2571,2572,2582],{},[54,2573,2574,2576,2579],{},[560,2575,181],{},[57,2577,2578],{"mathvariant":986},"\u002F",[57,2580,2581],{},"L",[61,2583,2584],{"encoding":63},"2\u002FL",[38,2586,2588],{"className":2587,"ariaHidden":69},[68],[38,2589,2591,2594,2598],{"className":2590},[73],[38,2592],{"className":2593,"style":638},[77],[38,2595,2597],{"className":2596},[82],"2\u002F",[38,2599,2581],{"className":2600},[82,83]," bound and the condition number",[11,2603,2604],{},"This isn't trial and error, it can be calculated. For a quadratic cost function like ours, gradient descent converges if and only if",[11,2606,2607],{},[38,2608,2610,2664],{"className":2609},[41],[38,2611,2613],{"className":2612},[45],[47,2614,2615],{"xmlns":49},[51,2616,2617,2661],{},[54,2618,2619,2621,2624,2626,2628,2634,2636,2638,2640,2642,2653,2655,2659],{},[560,2620,159],{},[362,2622,2623],{},"\u003C",[57,2625,59],{},[362,2627,2623],{},[557,2629,2630,2632],{},[560,2631,181],{},[57,2633,2581],{},[362,2635,364],{"separator":69},[485,2637],{"width":998},[57,2639,2581],{},[362,2641,380],{},[351,2643,2644,2647,2650],{},[57,2645,2646],{},"λ",[57,2648,2649],{},"max",[362,2651,2652],{},"⁡",[362,2654,371],{"stretchy":370},[57,2656,2658],{"mathvariant":2657},"bold","H",[362,2660,377],{"stretchy":370},[61,2662,2663],{"encoding":63},"0 \u003C \\alpha \u003C \\frac{2}{L}, \\qquad L = \\lambda_{\\max}(\\mathbf{H})",[38,2665,2667,2686,2705,2801],{"className":2666,"ariaHidden":69},[68],[38,2668,2670,2674,2677,2680,2683],{"className":2669},[73],[38,2671],{"className":2672,"style":2673},[77],"height:0.6835em;vertical-align:-0.0391em;",[38,2675,159],{"className":2676},[82],[38,2678],{"className":2679,"style":486},[485],[38,2681,2623],{"className":2682},[490],[38,2684],{"className":2685,"style":486},[485],[38,2687,2689,2693,2696,2699,2702],{"className":2688},[73],[38,2690],{"className":2691,"style":2692},[77],"height:0.5782em;vertical-align:-0.0391em;",[38,2694,59],{"className":2695,"style":84},[82,83],[38,2697],{"className":2698,"style":486},[485],[38,2700,2623],{"className":2701},[490],[38,2703],{"className":2704,"style":486},[485],[38,2706,2708,2712,2780,2783,2786,2789,2792,2795,2798],{"className":2707},[73],[38,2709],{"className":2710,"style":2711},[77],"height:1.1901em;vertical-align:-0.345em;",[38,2713,2715,2718,2777],{"className":2714},[82],[38,2716],{"className":2717},[474,684],[38,2719,2721],{"className":2720},[557],[38,2722,2724,2769],{"className":2723},[417,418],[38,2725,2727,2766],{"className":2726},[422],[38,2728,2730,2744,2752],{"className":2729,"style":697},[426],[38,2731,2732,2735],{"style":700},[38,2733],{"className":2734,"style":704},[434],[38,2736,2738],{"className":2737},[439,440,441,442],[38,2739,2741],{"className":2740},[82,442],[38,2742,2581],{"className":2743},[82,83,442],[38,2745,2746,2749],{"style":719},[38,2747],{"className":2748,"style":704},[434],[38,2750],{"className":2751,"style":727},[726],[38,2753,2754,2757],{"style":730},[38,2755],{"className":2756,"style":704},[434],[38,2758,2760],{"className":2759},[439,440,441,442],[38,2761,2763],{"className":2762},[82,442],[38,2764,181],{"className":2765},[82,442],[38,2767,461],{"className":2768},[460],[38,2770,2772],{"className":2771},[422],[38,2773,2775],{"className":2774,"style":752},[426],[38,2776],{},[38,2778],{"className":2779},[481,684],[38,2781,364],{"className":2782},[453],[38,2784],{"className":2785,"style":1154},[485],[38,2787],{"className":2788,"style":655},[485],[38,2790,2581],{"className":2791},[82,83],[38,2793],{"className":2794,"style":486},[485],[38,2796,380],{"className":2797},[490],[38,2799],{"className":2800,"style":486},[485],[38,2802,2804,2807,2862,2865,2869],{"className":2803},[73],[38,2805],{"className":2806,"style":638},[77],[38,2808,2810,2813],{"className":2809},[82],[38,2811,2646],{"className":2812},[82,83],[38,2814,2816],{"className":2815},[413],[38,2817,2819,2853],{"className":2818},[417,418],[38,2820,2822,2850],{"className":2821},[422],[38,2823,2826],{"className":2824,"style":2825},[426],"height:0.1514em;",[38,2827,2829,2832],{"style":2828},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[38,2830],{"className":2831,"style":435},[434],[38,2833,2835],{"className":2834},[439,440,441,442],[38,2836,2838],{"className":2837},[82,442],[38,2839,2841,2844,2847],{"className":2840},[764,442],[38,2842,568],{"className":2843},[442],[38,2845,27],{"className":2846},[442],[38,2848,374],{"className":2849},[442],[38,2851,461],{"className":2852},[460],[38,2854,2856],{"className":2855},[422],[38,2857,2860],{"className":2858,"style":2859},[426],"height:0.15em;",[38,2861],{},[38,2863,371],{"className":2864},[474],[38,2866,2658],{"className":2867},[82,2868],"mathbf",[38,2870,377],{"className":2871},[481],[11,2873,2874,2875,2905,2906,2911,2912,2997,2998,3027],{},"where ",[38,2876,2878,2892],{"className":2877},[41],[38,2879,2881],{"className":2880},[45],[47,2882,2883],{"xmlns":49},[51,2884,2885,2889],{},[54,2886,2887],{},[57,2888,2658],{"mathvariant":2657},[61,2890,2891],{"encoding":63},"\\mathbf{H}",[38,2893,2895],{"className":2894,"ariaHidden":69},[68],[38,2896,2898,2902],{"className":2897},[73],[38,2899],{"className":2900,"style":2901},[77],"height:0.6861em;",[38,2903,2658],{"className":2904},[82,2868]," is the ",[2907,2908,2910],"glossary-term",{"definition":2909},"the matrix of second derivatives of the cost function, it describes the bowl's curvature in every direction","Hessian"," of the cost and ",[38,2913,2915,2935],{"className":2914},[41],[38,2916,2918],{"className":2917},[45],[47,2919,2920],{"xmlns":49},[51,2921,2922,2932],{},[54,2923,2924],{},[351,2925,2926,2928,2930],{},[57,2927,2646],{},[57,2929,2649],{},[362,2931,2652],{},[61,2933,2934],{"encoding":63},"\\lambda_{\\max}",[38,2936,2938],{"className":2937,"ariaHidden":69},[68],[38,2939,2941,2945],{"className":2940},[73],[38,2942],{"className":2943,"style":2944},[77],"height:0.8444em;vertical-align:-0.15em;",[38,2946,2948,2951],{"className":2947},[82],[38,2949,2646],{"className":2950},[82,83],[38,2952,2954],{"className":2953},[413],[38,2955,2957,2989],{"className":2956},[417,418],[38,2958,2960,2986],{"className":2959},[422],[38,2961,2963],{"className":2962,"style":2825},[426],[38,2964,2965,2968],{"style":2828},[38,2966],{"className":2967,"style":435},[434],[38,2969,2971],{"className":2970},[439,440,441,442],[38,2972,2974],{"className":2973},[82,442],[38,2975,2977,2980,2983],{"className":2976},[764,442],[38,2978,568],{"className":2979},[442],[38,2981,27],{"className":2982},[442],[38,2984,374],{"className":2985},[442],[38,2987,461],{"className":2988},[460],[38,2990,2992],{"className":2991},[422],[38,2993,2995],{"className":2994,"style":2859},[426],[38,2996],{}," is its largest eigenvalue. ",[38,2999,3001,3014],{"className":3000},[41],[38,3002,3004],{"className":3003},[45],[47,3005,3006],{"xmlns":49},[51,3007,3008,3012],{},[54,3009,3010],{},[57,3011,2581],{},[61,3013,2581],{"encoding":63},[38,3015,3017],{"className":3016,"ariaHidden":69},[68],[38,3018,3020,3024],{"className":3019},[73],[38,3021],{"className":3022,"style":3023},[77],"height:0.6833em;",[38,3025,2581],{"className":3026},[82,83]," is the bowl's maximum curvature: the sharper the steepest direction, the smaller the step allowed before it overshoots to the other side of the valley.",[11,3029,3030,3031,3082,3083,2461,3111,3139,3140,3249,3250,3368,3369,3462,3463,3557],{},"I computed the Hessian for our 8-house example (a ",[38,3032,3034,3052],{"className":3033},[41],[38,3035,3037],{"className":3036},[45],[47,3038,3039],{"xmlns":49},[51,3040,3041,3049],{},[54,3042,3043,3045,3047],{},[560,3044,181],{},[362,3046,1488],{},[560,3048,181],{},[61,3050,3051],{"encoding":63},"2\\times2",[38,3053,3055,3073],{"className":3054,"ariaHidden":69},[68],[38,3056,3058,3061,3064,3067,3070],{"className":3057},[73],[38,3059],{"className":3060,"style":1531},[77],[38,3062,181],{"className":3063},[82],[38,3065],{"className":3066,"style":510},[485],[38,3068,1488],{"className":3069},[514],[38,3071],{"className":3072,"style":510},[485],[38,3074,3076,3079],{"className":3075},[73],[38,3077],{"className":3078,"style":1454},[77],[38,3080,181],{"className":3081},[82]," calculation, since it's just ",[38,3084,3086,3099],{"className":3085},[41],[38,3087,3089],{"className":3088},[45],[47,3090,3091],{"xmlns":49},[51,3092,3093,3097],{},[54,3094,3095],{},[57,3096,360],{},[61,3098,360],{"encoding":63},[38,3100,3102],{"className":3101,"ariaHidden":69},[68],[38,3103,3105,3108],{"className":3104},[73],[38,3106],{"className":3107,"style":78},[77],[38,3109,360],{"className":3110,"style":449},[82,83],[38,3112,3114,3127],{"className":3113},[41],[38,3115,3117],{"className":3116},[45],[47,3118,3119],{"xmlns":49},[51,3120,3121,3125],{},[54,3122,3123],{},[57,3124,367],{},[61,3126,367],{"encoding":63},[38,3128,3130],{"className":3129,"ariaHidden":69},[68],[38,3131,3133,3136],{"className":3132},[73],[38,3134],{"className":3135,"style":524},[77],[38,3137,367],{"className":3138},[82,83],") and got ",[38,3141,3143,3172],{"className":3142},[41],[38,3144,3146],{"className":3145},[45],[47,3147,3148],{"xmlns":49},[51,3149,3150,3169],{},[54,3151,3152,3154,3157,3160,3162],{},[57,3153,2581],{},[362,3155,3156],{},"≈",[560,3158,3159],{},"2.36",[362,3161,1488],{},[589,3163,3164,3166],{},[560,3165,1493],{},[560,3167,3168],{},"6",[61,3170,3171],{"encoding":63},"L \\approx 2.36\\times10^6",[38,3173,3175,3193,3211],{"className":3174,"ariaHidden":69},[68],[38,3176,3178,3181,3184,3187,3190],{"className":3177},[73],[38,3179],{"className":3180,"style":3023},[77],[38,3182,2581],{"className":3183},[82,83],[38,3185],{"className":3186,"style":486},[485],[38,3188,3156],{"className":3189},[490],[38,3191],{"className":3192,"style":486},[485],[38,3194,3196,3199,3202,3205,3208],{"className":3195},[73],[38,3197],{"className":3198,"style":1531},[77],[38,3200,3159],{"className":3201},[82],[38,3203],{"className":3204,"style":510},[485],[38,3206,1488],{"className":3207},[514],[38,3209],{"className":3210,"style":510},[485],[38,3212,3214,3217,3220],{"className":3213},[73],[38,3215],{"className":3216,"style":939},[77],[38,3218,178],{"className":3219},[82],[38,3221,3223,3226],{"className":3222},[82],[38,3224,159],{"className":3225},[82],[38,3227,3229],{"className":3228},[413],[38,3230,3232],{"className":3231},[417],[38,3233,3235],{"className":3234},[422],[38,3236,3238],{"className":3237,"style":939},[426],[38,3239,3240,3243],{"style":844},[38,3241],{"className":3242,"style":435},[434],[38,3244,3246],{"className":3245},[439,440,441,442],[38,3247,3168],{"className":3248},[82,442],", giving a critical ",[38,3251,3253,3284],{"className":3252},[41],[38,3254,3256],{"className":3255},[45],[47,3257,3258],{"xmlns":49},[51,3259,3260,3281],{},[54,3261,3262,3264,3266,3269,3271],{},[57,3263,59],{},[362,3265,3156],{},[560,3267,3268],{},"8.46",[362,3270,1488],{},[589,3272,3273,3275],{},[560,3274,1493],{},[54,3276,3277,3279],{},[362,3278,606],{},[560,3280,1500],{},[61,3282,3283],{"encoding":63},"\\alpha \\approx 8.46\\times10^{-7}",[38,3285,3287,3306,3324],{"className":3286,"ariaHidden":69},[68],[38,3288,3290,3294,3297,3300,3303],{"className":3289},[73],[38,3291],{"className":3292,"style":3293},[77],"height:0.4831em;",[38,3295,59],{"className":3296,"style":84},[82,83],[38,3298],{"className":3299,"style":486},[485],[38,3301,3156],{"className":3302},[490],[38,3304],{"className":3305,"style":486},[485],[38,3307,3309,3312,3315,3318,3321],{"className":3308},[73],[38,3310],{"className":3311,"style":1531},[77],[38,3313,3268],{"className":3314},[82],[38,3316],{"className":3317,"style":510},[485],[38,3319,1488],{"className":3320},[514],[38,3322],{"className":3323,"style":510},[485],[38,3325,3327,3330,3333],{"className":3326},[73],[38,3328],{"className":3329,"style":939},[77],[38,3331,178],{"className":3332},[82],[38,3334,3336,3339],{"className":3335},[82],[38,3337,159],{"className":3338},[82],[38,3340,3342],{"className":3341},[413],[38,3343,3345],{"className":3344},[417],[38,3346,3348],{"className":3347},[422],[38,3349,3351],{"className":3350,"style":939},[426],[38,3352,3353,3356],{"style":844},[38,3354],{"className":3355,"style":435},[434],[38,3357,3359],{"className":3358},[439,440,441,442],[38,3360,3362,3365],{"className":3361},[82,442],[38,3363,606],{"className":3364},[82,442],[38,3366,1500],{"className":3367},[82,442],". That matches what I saw in the simulator: ",[38,3370,3372,3397],{"className":3371},[41],[38,3373,3375],{"className":3374},[45],[47,3376,3377],{"xmlns":49},[51,3378,3379,3395],{},[54,3380,3381,3383,3385],{},[560,3382,1485],{},[362,3384,1488],{},[589,3386,3387,3389],{},[560,3388,1493],{},[54,3390,3391,3393],{},[362,3392,606],{},[560,3394,1500],{},[61,3396,2395],{"encoding":63},[38,3398,3400,3418],{"className":3399,"ariaHidden":69},[68],[38,3401,3403,3406,3409,3412,3415],{"className":3402},[73],[38,3404],{"className":3405,"style":1531},[77],[38,3407,1485],{"className":3408},[82],[38,3410],{"className":3411,"style":510},[485],[38,3413,1488],{"className":3414},[514],[38,3416],{"className":3417,"style":510},[485],[38,3419,3421,3424,3427],{"className":3420},[73],[38,3422],{"className":3423,"style":939},[77],[38,3425,178],{"className":3426},[82],[38,3428,3430,3433],{"className":3429},[82],[38,3431,159],{"className":3432},[82],[38,3434,3436],{"className":3435},[413],[38,3437,3439],{"className":3438},[417],[38,3440,3442],{"className":3441},[422],[38,3443,3445],{"className":3444,"style":939},[426],[38,3446,3447,3450],{"style":844},[38,3448],{"className":3449,"style":435},[434],[38,3451,3453],{"className":3452},[439,440,441,442],[38,3454,3456,3459],{"className":3455},[82,442],[38,3457,606],{"className":3458},[82,442],[38,3460,1500],{"className":3461},[82,442]," crosses that threshold and diverges, ",[38,3464,3466,3492],{"className":3465},[41],[38,3467,3469],{"className":3468},[45],[47,3470,3471],{"xmlns":49},[51,3472,3473,3489],{},[54,3474,3475,3477,3479],{},[560,3476,1787],{},[362,3478,1488],{},[589,3480,3481,3483],{},[560,3482,1493],{},[54,3484,3485,3487],{},[362,3486,606],{},[560,3488,1500],{},[61,3490,3491],{"encoding":63},"8\\times10^{-7}",[38,3493,3495,3513],{"className":3494,"ariaHidden":69},[68],[38,3496,3498,3501,3504,3507,3510],{"className":3497},[73],[38,3499],{"className":3500,"style":1531},[77],[38,3502,1787],{"className":3503},[82],[38,3505],{"className":3506,"style":510},[485],[38,3508,1488],{"className":3509},[514],[38,3511],{"className":3512,"style":510},[485],[38,3514,3516,3519,3522],{"className":3515},[73],[38,3517],{"className":3518,"style":939},[77],[38,3520,178],{"className":3521},[82],[38,3523,3525,3528],{"className":3524},[82],[38,3526,159],{"className":3527},[82],[38,3529,3531],{"className":3530},[413],[38,3532,3534],{"className":3533},[417],[38,3535,3537],{"className":3536},[422],[38,3538,3540],{"className":3539,"style":939},[426],[38,3541,3542,3545],{"style":844},[38,3543],{"className":3544,"style":435},[434],[38,3546,3548],{"className":3547},[439,440,441,442],[38,3549,3551,3554],{"className":3550},[82,442],[38,3552,606],{"className":3553},[82,442],[38,3555,1500],{"className":3556},[82,442]," stays under it and converges.",[11,3559,3560,3561,3612,3613,3641,3642,3759],{},"The original notebook runs this same calculation with the real 4 features (the Hessian becomes ",[38,3562,3564,3582],{"className":3563},[41],[38,3565,3567],{"className":3566},[45],[47,3568,3569],{"xmlns":49},[51,3570,3571,3579],{},[54,3572,3573,3575,3577],{},[560,3574,1698],{},[362,3576,1488],{},[560,3578,1698],{},[61,3580,3581],{"encoding":63},"5\\times5",[38,3583,3585,3603],{"className":3584,"ariaHidden":69},[68],[38,3586,3588,3591,3594,3597,3600],{"className":3587},[73],[38,3589],{"className":3590,"style":1531},[77],[38,3592,1698],{"className":3593},[82],[38,3595],{"className":3596,"style":510},[485],[38,3598,1488],{"className":3599},[514],[38,3601],{"className":3602,"style":510},[485],[38,3604,3606,3609],{"className":3605},[73],[38,3607],{"className":3608,"style":1454},[77],[38,3610,1698],{"className":3611},[82],", counting ",[38,3614,3616,3629],{"className":3615},[41],[38,3617,3619],{"className":3618},[45],[47,3620,3621],{"xmlns":49},[51,3622,3623,3627],{},[54,3624,3625],{},[57,3626,367],{},[61,3628,367],{"encoding":63},[38,3630,3632],{"className":3631,"ariaHidden":69},[68],[38,3633,3635,3638],{"className":3634},[73],[38,3636],{"className":3637,"style":524},[77],[38,3639,367],{"className":3640},[82,83],") and lands on a critical ",[38,3643,3645,3676],{"className":3644},[41],[38,3646,3648],{"className":3647},[45],[47,3649,3650],{"xmlns":49},[51,3651,3652,3673],{},[54,3653,3654,3656,3658,3661,3663],{},[57,3655,59],{},[362,3657,3156],{},[560,3659,3660],{},"9.22",[362,3662,1488],{},[589,3664,3665,3667],{},[560,3666,1493],{},[54,3668,3669,3671],{},[362,3670,606],{},[560,3672,1500],{},[61,3674,3675],{"encoding":63},"\\alpha \\approx 9.22\\times10^{-7}",[38,3677,3679,3697,3715],{"className":3678,"ariaHidden":69},[68],[38,3680,3682,3685,3688,3691,3694],{"className":3681},[73],[38,3683],{"className":3684,"style":3293},[77],[38,3686,59],{"className":3687,"style":84},[82,83],[38,3689],{"className":3690,"style":486},[485],[38,3692,3156],{"className":3693},[490],[38,3695],{"className":3696,"style":486},[485],[38,3698,3700,3703,3706,3709,3712],{"className":3699},[73],[38,3701],{"className":3702,"style":1531},[77],[38,3704,3660],{"className":3705},[82],[38,3707],{"className":3708,"style":510},[485],[38,3710,1488],{"className":3711},[514],[38,3713],{"className":3714,"style":510},[485],[38,3716,3718,3721,3724],{"className":3717},[73],[38,3719],{"className":3720,"style":939},[77],[38,3722,178],{"className":3723},[82],[38,3725,3727,3730],{"className":3726},[82],[38,3728,159],{"className":3729},[82],[38,3731,3733],{"className":3732},[413],[38,3734,3736],{"className":3735},[417],[38,3737,3739],{"className":3738},[422],[38,3740,3742],{"className":3741,"style":939},[426],[38,3743,3744,3747],{"style":844},[38,3745],{"className":3746,"style":435},[434],[38,3748,3750],{"className":3749},[439,440,441,442],[38,3751,3753,3756],{"className":3752},[82,442],[38,3754,606],{"className":3755},[82,442],[38,3757,1500],{"className":3758},[82,442],", practically the same number I found by simplifying to 1 feature. That makes sense: size is the feature with the most absurd scale, so it alone already dominates most of the bowl's curvature.",[11,3761,3762,3763,3767,3768,3831,3832,3862,3863,3970,3971,3974],{},"There's a second number that comes out of this same calculation, the ",[2907,3764,3766],{"definition":3765},"the ratio between the largest and smallest curvature of the cost bowl, kappa = L\u002Fmu, and the bigger it is, the more elongated the bowl and the slower gradient descent moves","condition number"," ",[38,3769,3771,3795],{"className":3770},[41],[38,3772,3774],{"className":3773},[45],[47,3775,3776],{"xmlns":49},[51,3777,3778,3792],{},[54,3779,3780,3783,3785,3787,3789],{},[57,3781,3782],{},"κ",[362,3784,380],{},[57,3786,2581],{},[57,3788,2578],{"mathvariant":986},[57,3790,3791],{},"μ",[61,3793,3794],{"encoding":63},"\\kappa = L\u002F\\mu",[38,3796,3798,3816],{"className":3797,"ariaHidden":69},[68],[38,3799,3801,3804,3807,3810,3813],{"className":3800},[73],[38,3802],{"className":3803,"style":78},[77],[38,3805,3782],{"className":3806},[82,83],[38,3808],{"className":3809,"style":486},[485],[38,3811,380],{"className":3812},[490],[38,3814],{"className":3815,"style":486},[485],[38,3817,3819,3822,3825,3828],{"className":3818},[73],[38,3820],{"className":3821,"style":638},[77],[38,3823,2581],{"className":3824},[82,83],[38,3826,2578],{"className":3827},[82],[38,3829,3791],{"className":3830},[82,83]," (where ",[38,3833,3835,3849],{"className":3834},[41],[38,3836,3838],{"className":3837},[45],[47,3839,3840],{"xmlns":49},[51,3841,3842,3846],{},[54,3843,3844],{},[57,3845,3791],{},[61,3847,3848],{"encoding":63},"\\mu",[38,3850,3852],{"className":3851,"ariaHidden":69},[68],[38,3853,3855,3859],{"className":3854},[73],[38,3856],{"className":3857,"style":3858},[77],"height:0.625em;vertical-align:-0.1944em;",[38,3860,3791],{"className":3861},[82,83]," is the smallest curvature). It measures how \"stretched\" the bowl is. I found ",[38,3864,3866,3893],{"className":3865},[41],[38,3867,3869],{"className":3868},[45],[47,3870,3871],{"xmlns":49},[51,3872,3873,3890],{},[54,3874,3875,3877,3879,3882,3884],{},[57,3876,3782],{},[362,3878,3156],{},[560,3880,3881],{},"5.8",[362,3883,1488],{},[589,3885,3886,3888],{},[560,3887,1493],{},[560,3889,1500],{},[61,3891,3892],{"encoding":63},"\\kappa \\approx 5.8\\times10^7",[38,3894,3896,3914,3932],{"className":3895,"ariaHidden":69},[68],[38,3897,3899,3902,3905,3908,3911],{"className":3898},[73],[38,3900],{"className":3901,"style":3293},[77],[38,3903,3782],{"className":3904},[82,83],[38,3906],{"className":3907,"style":486},[485],[38,3909,3156],{"className":3910},[490],[38,3912],{"className":3913,"style":486},[485],[38,3915,3917,3920,3923,3926,3929],{"className":3916},[73],[38,3918],{"className":3919,"style":1531},[77],[38,3921,3881],{"className":3922},[82],[38,3924],{"className":3925,"style":510},[485],[38,3927,1488],{"className":3928},[514],[38,3930],{"className":3931,"style":510},[485],[38,3933,3935,3938,3941],{"className":3934},[73],[38,3936],{"className":3937,"style":939},[77],[38,3939,178],{"className":3940},[82],[38,3942,3944,3947],{"className":3943},[82],[38,3945,159],{"className":3946},[82],[38,3948,3950],{"className":3949},[413],[38,3951,3953],{"className":3952},[417],[38,3954,3956],{"className":3955},[422],[38,3957,3959],{"className":3958,"style":939},[426],[38,3960,3961,3964],{"style":844},[38,3962],{"className":3963,"style":435},[434],[38,3965,3967],{"className":3966},[439,440,441,442],[38,3968,1500],{"className":3969},[82,442]," for the full 100-house dataset with all 4 features. I also computed how many iterations would be needed, in the best case, just to reduce the error by 10x: ",[33,3972,3973],{},"around 67 million",". That's when I fully understood why \"lower alpha a bit and wait longer\" isn't a real solution here.",[87,3976,3978,3979],{"id":3977},"a-practical-recipe-for-picking-αalphaα","A practical recipe for picking ",[38,3980,3982,3995],{"className":3981},[41],[38,3983,3985],{"className":3984},[45],[47,3986,3987],{"xmlns":49},[51,3988,3989,3993],{},[54,3990,3991],{},[57,3992,59],{},[61,3994,64],{"encoding":63},[38,3996,3998],{"className":3997,"ariaHidden":69},[68],[38,3999,4001,4004],{"className":4000},[73],[38,4002],{"className":4003,"style":78},[77],[38,4005,59],{"className":4006,"style":84},[82,83],[11,4008,4009,4010,1458],{},"I won't always want to compute an eigenvalue. The recipe Andrew Ng teaches, and that I've already been using since ",[27,4011,4012],{"href":29},"the gradient descent post",[4014,4015,4016,4073,4079,4082],"ol",{},[4017,4018,4019,4020,4072],"li",{},"Start small, like ",[38,4021,4023,4042],{"className":4022},[41],[38,4024,4026],{"className":4025},[45],[47,4027,4028],{"xmlns":49},[51,4029,4030,4039],{},[54,4031,4032,4034,4036],{},[57,4033,59],{},[362,4035,380],{},[560,4037,4038],{},"0.001",[61,4040,4041],{"encoding":63},"\\alpha = 0.001",[38,4043,4045,4063],{"className":4044,"ariaHidden":69},[68],[38,4046,4048,4051,4054,4057,4060],{"className":4047},[73],[38,4049],{"className":4050,"style":78},[77],[38,4052,59],{"className":4053,"style":84},[82,83],[38,4055],{"className":4056,"style":486},[485],[38,4058,380],{"className":4059},[490],[38,4061],{"className":4062,"style":486},[485],[38,4064,4066,4069],{"className":4065},[73],[38,4067],{"className":4068,"style":1454},[77],[38,4070,4038],{"className":4071},[82],".",[4017,4074,4075,4076],{},"Multiply by roughly 3 each attempt: ",[22,4077,4078],{},"0.001, 0.003, 0.01, 0.03, 0.1...",[4017,4080,4081],{},"Run a few iterations and look at the cost-vs-iteration chart.",[4017,4083,4084,4085,4113],{},"Pick the largest ",[38,4086,4088,4101],{"className":4087},[41],[38,4089,4091],{"className":4090},[45],[47,4092,4093],{"xmlns":49},[51,4094,4095,4099],{},[54,4096,4097],{},[57,4098,59],{},[61,4100,64],{"encoding":63},[38,4102,4104],{"className":4103,"ariaHidden":69},[68],[38,4105,4107,4110],{"className":4106},[73],[38,4108],{"className":4109,"style":78},[77],[38,4111,59],{"className":4112,"style":84},[82,83]," that still gives a smooth, decreasing curve.",[99,4115,4116,4126],{},[102,4117,4118],{},[105,4119,4120,4123],{},[108,4121,4122],{"align":110},"What shows up on the chart",[108,4124,4125],{"align":110},"Diagnosis",[129,4127,4128,4170,4206,4242],{},[105,4129,4130,4139],{},[134,4131,4132,4133,2578,4136],{"align":110},"Cost climbs, or turns into ",[22,4134,4135],{},"nan",[22,4137,4138],{},"inf",[134,4140,4141,4169],{"align":110},[38,4142,4144,4157],{"className":4143},[41],[38,4145,4147],{"className":4146},[45],[47,4148,4149],{"xmlns":49},[51,4150,4151,4155],{},[54,4152,4153],{},[57,4154,59],{},[61,4156,64],{"encoding":63},[38,4158,4160],{"className":4159,"ariaHidden":69},[68],[38,4161,4163,4166],{"className":4162},[73],[38,4164],{"className":4165,"style":78},[77],[38,4167,59],{"className":4168,"style":84},[82,83]," too big",[105,4171,4172,4175],{},[134,4173,4174],{"align":110},"Cost drops but jagged, with spikes",[134,4176,4177,4205],{"align":110},[38,4178,4180,4193],{"className":4179},[41],[38,4181,4183],{"className":4182},[45],[47,4184,4185],{"xmlns":49},[51,4186,4187,4191],{},[54,4188,4189],{},[57,4190,59],{},[61,4192,64],{"encoding":63},[38,4194,4196],{"className":4195,"ariaHidden":69},[68],[38,4197,4199,4202],{"className":4198},[73],[38,4200],{"className":4201,"style":78},[77],[38,4203,59],{"className":4204,"style":84},[82,83]," at the limit, lower it",[105,4207,4208,4211],{},[134,4209,4210],{"align":110},"Cost drops in an almost straight, slow line",[134,4212,4213,4241],{"align":110},[38,4214,4216,4229],{"className":4215},[41],[38,4217,4219],{"className":4218},[45],[47,4220,4221],{"xmlns":49},[51,4222,4223,4227],{},[54,4224,4225],{},[57,4226,59],{},[61,4228,64],{"encoding":63},[38,4230,4232],{"className":4231,"ariaHidden":69},[68],[38,4233,4235,4238],{"className":4234},[73],[38,4236],{"className":4237,"style":78},[77],[38,4239,59],{"className":4240,"style":84},[82,83]," too small",[105,4243,4244,4247],{},[134,4245,4246],{"align":110},"Cost drops fast and then flattens",[134,4248,4249,4250,4278],{"align":110},"good ",[38,4251,4253,4266],{"className":4252},[41],[38,4254,4256],{"className":4255},[45],[47,4257,4258],{"xmlns":49},[51,4259,4260,4264],{},[54,4261,4262],{},[57,4263,59],{},[61,4265,64],{"encoding":63},[38,4267,4269],{"className":4268,"ariaHidden":69},[68],[38,4270,4272,4275],{"className":4271},[73],[38,4273],{"className":4274,"style":78},[77],[38,4276,59],{"className":4277,"style":84},[82,83],", converged",[11,4280,4281,4282,4310,4311,2461,4383,4455],{},"I ran this sweep on the full dataset and found the usable range of ",[38,4283,4285,4298],{"className":4284},[41],[38,4286,4288],{"className":4287},[45],[47,4289,4290],{"xmlns":49},[51,4291,4292,4296],{},[54,4293,4294],{},[57,4295,59],{},[61,4297,64],{"encoding":63},[38,4299,4301],{"className":4300,"ariaHidden":69},[68],[38,4302,4304,4307],{"className":4303},[73],[38,4305],{"className":4306,"style":78},[77],[38,4308,59],{"className":4309,"style":84},[82,83]," (with no scaling at all) sits between ",[38,4312,4314,4336],{"className":4313},[41],[38,4315,4317],{"className":4316},[45],[47,4318,4319],{"xmlns":49},[51,4320,4321,4333],{},[54,4322,4323],{},[589,4324,4325,4327],{},[560,4326,1493],{},[54,4328,4329,4331],{},[362,4330,606],{},[560,4332,1787],{},[61,4334,4335],{"encoding":63},"10^{-8}",[38,4337,4339],{"className":4338,"ariaHidden":69},[68],[38,4340,4342,4345,4348],{"className":4341},[73],[38,4343],{"className":4344,"style":939},[77],[38,4346,178],{"className":4347},[82],[38,4349,4351,4354],{"className":4350},[82],[38,4352,159],{"className":4353},[82],[38,4355,4357],{"className":4356},[413],[38,4358,4360],{"className":4359},[417],[38,4361,4363],{"className":4362},[422],[38,4364,4366],{"className":4365,"style":939},[426],[38,4367,4368,4371],{"style":844},[38,4369],{"className":4370,"style":435},[434],[38,4372,4374],{"className":4373},[439,440,441,442],[38,4375,4377,4380],{"className":4376},[82,442],[38,4378,606],{"className":4379},[82,442],[38,4381,1787],{"className":4382},[82,442],[38,4384,4386,4408],{"className":4385},[41],[38,4387,4389],{"className":4388},[45],[47,4390,4391],{"xmlns":49},[51,4392,4393,4405],{},[54,4394,4395],{},[589,4396,4397,4399],{},[560,4398,1493],{},[54,4400,4401,4403],{},[362,4402,606],{},[560,4404,3168],{},[61,4406,4407],{"encoding":63},"10^{-6}",[38,4409,4411],{"className":4410,"ariaHidden":69},[68],[38,4412,4414,4417,4420],{"className":4413},[73],[38,4415],{"className":4416,"style":939},[77],[38,4418,178],{"className":4419},[82],[38,4421,4423,4426],{"className":4422},[82],[38,4424,159],{"className":4425},[82],[38,4427,4429],{"className":4428},[413],[38,4430,4432],{"className":4431},[417],[38,4433,4435],{"className":4434},[422],[38,4436,4438],{"className":4437,"style":939},[426],[38,4439,4440,4443],{"style":844},[38,4441],{"className":4442,"style":435},[434],[38,4444,4446],{"className":4445},[439,440,441,442],[38,4447,4449,4452],{"className":4448},[82,442],[38,4450,606],{"className":4451},[82,442],[38,4453,3168],{"className":4454},[82,442],". An extremely narrow range, and one that depends entirely on the unit I used to measure size. If I'd measured in square meters instead of sqft, every one of these numbers would shift. That's fragile, and it's not how I want to train any model.",[87,4457,4459],{"id":4458},"the-scaling-techniques","The scaling techniques",[11,4461,4462,4463,4491],{},"The fix isn't hunting for a better ",[38,4464,4466,4479],{"className":4465},[41],[38,4467,4469],{"className":4468},[45],[47,4470,4471],{"xmlns":49},[51,4472,4473,4477],{},[54,4474,4475],{},[57,4476,59],{},[61,4478,64],{"encoding":63},[38,4480,4482],{"className":4481,"ariaHidden":69},[68],[38,4483,4485,4488],{"className":4484},[73],[38,4486],{"className":4487,"style":78},[77],[38,4489,59],{"className":4490,"style":84},[82,83],", it's fixing the scale of the features before running anything. There are a few ways to do that:",[99,4493,4494,4507],{},[102,4495,4496],{},[105,4497,4498,4501,4504],{},[108,4499,4500],{"align":110},"Technique",[108,4502,4503],{"align":110},"Formula",[108,4505,4506],{"align":110},"Resulting range",[129,4508,4509,4781,5280,5750],{},[105,4510,4511,4514,4725],{},[134,4512,4513],{"align":110},"Divide by max",[134,4515,4516],{"align":110},[38,4517,4519,4563],{"className":4518},[41],[38,4520,4522],{"className":4521},[45],[47,4523,4524],{"xmlns":49},[51,4525,4526,4560],{},[54,4527,4528,4535,4538,4544,4546,4548,4550,4552,4558],{},[351,4529,4530,4532],{},[57,4531,374],{},[57,4533,4534],{},"j",[362,4536,4537],{},"←",[351,4539,4540,4542],{},[57,4541,374],{},[57,4543,4534],{},[57,4545,2578],{"mathvariant":986},[57,4547,2649],{},[362,4549,2652],{},[362,4551,371],{"stretchy":370},[351,4553,4554,4556],{},[57,4555,374],{},[57,4557,4534],{},[362,4559,377],{"stretchy":370},[61,4561,4562],{"encoding":63},"x_j \\leftarrow x_j \u002F \\max(x_j)",[38,4564,4566,4624],{"className":4565,"ariaHidden":69},[68],[38,4567,4569,4573,4615,4618,4621],{"className":4568},[73],[38,4570],{"className":4571,"style":4572},[77],"height:0.7167em;vertical-align:-0.2861em;",[38,4574,4576,4579],{"className":4575},[82],[38,4577,374],{"className":4578},[82,83],[38,4580,4582],{"className":4581},[413],[38,4583,4585,4607],{"className":4584},[417,418],[38,4586,4588,4604],{"className":4587},[422],[38,4589,4592],{"className":4590,"style":4591},[426],"height:0.3117em;",[38,4593,4594,4597],{"style":2828},[38,4595],{"className":4596,"style":435},[434],[38,4598,4600],{"className":4599},[439,440,441,442],[38,4601,4534],{"className":4602,"style":4603},[82,83,442],"margin-right:0.0572em;",[38,4605,461],{"className":4606},[460],[38,4608,4610],{"className":4609},[422],[38,4611,4613],{"className":4612,"style":468},[426],[38,4614],{},[38,4616],{"className":4617,"style":486},[485],[38,4619,4537],{"className":4620},[490],[38,4622],{"className":4623,"style":486},[485],[38,4625,4627,4630,4670,4673,4676,4679,4682,4722],{"className":4626},[73],[38,4628],{"className":4629,"style":402},[77],[38,4631,4633,4636],{"className":4632},[82],[38,4634,374],{"className":4635},[82,83],[38,4637,4639],{"className":4638},[413],[38,4640,4642,4662],{"className":4641},[417,418],[38,4643,4645,4659],{"className":4644},[422],[38,4646,4648],{"className":4647,"style":4591},[426],[38,4649,4650,4653],{"style":2828},[38,4651],{"className":4652,"style":435},[434],[38,4654,4656],{"className":4655},[439,440,441,442],[38,4657,4534],{"className":4658,"style":4603},[82,83,442],[38,4660,461],{"className":4661},[460],[38,4663,4665],{"className":4664},[422],[38,4666,4668],{"className":4667,"style":468},[426],[38,4669],{},[38,4671,2578],{"className":4672},[82],[38,4674],{"className":4675,"style":655},[485],[38,4677,2649],{"className":4678},[764],[38,4680,371],{"className":4681},[474],[38,4683,4685,4688],{"className":4684},[82],[38,4686,374],{"className":4687},[82,83],[38,4689,4691],{"className":4690},[413],[38,4692,4694,4714],{"className":4693},[417,418],[38,4695,4697,4711],{"className":4696},[422],[38,4698,4700],{"className":4699,"style":4591},[426],[38,4701,4702,4705],{"style":2828},[38,4703],{"className":4704,"style":435},[434],[38,4706,4708],{"className":4707},[439,440,441,442],[38,4709,4534],{"className":4710,"style":4603},[82,83,442],[38,4712,461],{"className":4713},[460],[38,4715,4717],{"className":4716},[422],[38,4718,4720],{"className":4719,"style":468},[426],[38,4721],{},[38,4723,377],{"className":4724},[481],[134,4726,4727],{"align":110},[38,4728,4730,4754],{"className":4729},[41],[38,4731,4733],{"className":4732},[45],[47,4734,4735],{"xmlns":49},[51,4736,4737,4751],{},[54,4738,4739,4742,4744,4746,4748],{},[362,4740,4741],{"stretchy":370},"[",[560,4743,159],{},[362,4745,364],{"separator":69},[560,4747,178],{},[362,4749,4750],{"stretchy":370},"]",[61,4752,4753],{"encoding":63},"[0, 1]",[38,4755,4757],{"className":4756,"ariaHidden":69},[68],[38,4758,4760,4763,4766,4769,4772,4775,4778],{"className":4759},[73],[38,4761],{"className":4762,"style":638},[77],[38,4764,4741],{"className":4765},[474],[38,4767,159],{"className":4768},[82],[38,4770,364],{"className":4771},[453],[38,4773],{"className":4774,"style":655},[485],[38,4776,178],{"className":4777},[82],[38,4779,4750],{"className":4780},[481],[105,4782,4783,4786,5227],{},[134,4784,4785],{"align":110},"Min-max",[134,4787,4788],{"align":110},[38,4789,4791,4870],{"className":4790},[41],[38,4792,4794],{"className":4793},[45],[47,4795,4796],{"xmlns":49},[51,4797,4798,4867],{},[54,4799,4800,4806,4808],{},[351,4801,4802,4804],{},[57,4803,374],{},[57,4805,4534],{},[362,4807,4537],{},[557,4809,4810,4835],{},[54,4811,4812,4818,4820,4823,4825,4827,4833],{},[351,4813,4814,4816],{},[57,4815,374],{},[57,4817,4534],{},[362,4819,606],{},[57,4821,4822],{},"min",[362,4824,2652],{},[362,4826,371],{"stretchy":370},[351,4828,4829,4831],{},[57,4830,374],{},[57,4832,4534],{},[362,4834,377],{"stretchy":370},[54,4836,4837,4839,4841,4843,4849,4851,4853,4855,4857,4859,4865],{},[57,4838,2649],{},[362,4840,2652],{},[362,4842,371],{"stretchy":370},[351,4844,4845,4847],{},[57,4846,374],{},[57,4848,4534],{},[362,4850,377],{"stretchy":370},[362,4852,606],{},[57,4854,4822],{},[362,4856,2652],{},[362,4858,371],{"stretchy":370},[351,4860,4861,4863],{},[57,4862,374],{},[57,4864,4534],{},[362,4866,377],{"stretchy":370},[61,4868,4869],{"encoding":63},"x_j \\leftarrow \\frac{x_j - \\min(x_j)}{\\max(x_j) - \\min(x_j)}",[38,4871,4873,4928],{"className":4872,"ariaHidden":69},[68],[38,4874,4876,4879,4919,4922,4925],{"className":4875},[73],[38,4877],{"className":4878,"style":4572},[77],[38,4880,4882,4885],{"className":4881},[82],[38,4883,374],{"className":4884},[82,83],[38,4886,4888],{"className":4887},[413],[38,4889,4891,4911],{"className":4890},[417,418],[38,4892,4894,4908],{"className":4893},[422],[38,4895,4897],{"className":4896,"style":4591},[426],[38,4898,4899,4902],{"style":2828},[38,4900],{"className":4901,"style":435},[434],[38,4903,4905],{"className":4904},[439,440,441,442],[38,4906,4534],{"className":4907,"style":4603},[82,83,442],[38,4909,461],{"className":4910},[460],[38,4912,4914],{"className":4913},[422],[38,4915,4917],{"className":4916,"style":468},[426],[38,4918],{},[38,4920],{"className":4921,"style":486},[485],[38,4923,4537],{"className":4924},[490],[38,4926],{"className":4927,"style":486},[485],[38,4929,4931,4935],{"className":4930},[73],[38,4932],{"className":4933,"style":4934},[77],"height:1.5746em;vertical-align:-0.5423em;",[38,4936,4938,4941,5224],{"className":4937},[82],[38,4939],{"className":4940},[474,684],[38,4942,4944],{"className":4943},[557],[38,4945,4947,5215],{"className":4946},[417,418],[38,4948,4950,5212],{"className":4949},[422],[38,4951,4954,5091,5099],{"className":4952,"style":4953},[426],"height:1.0323em;",[38,4955,4956,4959],{"style":700},[38,4957],{"className":4958,"style":704},[434],[38,4960,4962],{"className":4961},[439,440,441,442],[38,4963,4965,4977,4980,5026,5029,5032,5045,5048,5088],{"className":4964},[82,442],[38,4966,4968,4971,4974],{"className":4967},[764,442],[38,4969,568],{"className":4970},[442],[38,4972,27],{"className":4973},[442],[38,4975,374],{"className":4976},[442],[38,4978,371],{"className":4979},[474,442],[38,4981,4983,4986],{"className":4982},[82,442],[38,4984,374],{"className":4985},[82,83,442],[38,4987,4989],{"className":4988},[413],[38,4990,4992,5017],{"className":4991},[417,418],[38,4993,4995,5014],{"className":4994},[422],[38,4996,4999],{"className":4997,"style":4998},[426],"height:0.3281em;",[38,5000,5002,5006],{"style":5001},"top:-2.357em;margin-left:0em;margin-right:0.0714em;",[38,5003],{"className":5004,"style":5005},[434],"height:2.5em;",[38,5007,5011],{"className":5008},[439,5009,5010,442],"reset-size3","size1",[38,5012,4534],{"className":5013,"style":4603},[82,83,442],[38,5015,461],{"className":5016},[460],[38,5018,5020],{"className":5019},[422],[38,5021,5024],{"className":5022,"style":5023},[426],"height:0.2819em;",[38,5025],{},[38,5027,377],{"className":5028},[481,442],[38,5030,606],{"className":5031},[514,442],[38,5033,5035,5038,5041],{"className":5034},[764,442],[38,5036,568],{"className":5037},[442],[38,5039,599],{"className":5040},[442],[38,5042,5044],{"className":5043},[442],"n",[38,5046,371],{"className":5047},[474,442],[38,5049,5051,5054],{"className":5050},[82,442],[38,5052,374],{"className":5053},[82,83,442],[38,5055,5057],{"className":5056},[413],[38,5058,5060,5080],{"className":5059},[417,418],[38,5061,5063,5077],{"className":5062},[422],[38,5064,5066],{"className":5065,"style":4998},[426],[38,5067,5068,5071],{"style":5001},[38,5069],{"className":5070,"style":5005},[434],[38,5072,5074],{"className":5073},[439,5009,5010,442],[38,5075,4534],{"className":5076,"style":4603},[82,83,442],[38,5078,461],{"className":5079},[460],[38,5081,5083],{"className":5082},[422],[38,5084,5086],{"className":5085,"style":5023},[426],[38,5087],{},[38,5089,377],{"className":5090},[481,442],[38,5092,5093,5096],{"style":719},[38,5094],{"className":5095,"style":704},[434],[38,5097],{"className":5098,"style":727},[726],[38,5100,5102,5105],{"style":5101},"top:-3.5073em;",[38,5103],{"className":5104,"style":704},[434],[38,5106,5108],{"className":5107},[439,440,441,442],[38,5109,5111,5151,5154,5166,5169,5209],{"className":5110},[82,442],[38,5112,5114,5117],{"className":5113},[82,442],[38,5115,374],{"className":5116},[82,83,442],[38,5118,5120],{"className":5119},[413],[38,5121,5123,5143],{"className":5122},[417,418],[38,5124,5126,5140],{"className":5125},[422],[38,5127,5129],{"className":5128,"style":4998},[426],[38,5130,5131,5134],{"style":5001},[38,5132],{"className":5133,"style":5005},[434],[38,5135,5137],{"className":5136},[439,5009,5010,442],[38,5138,4534],{"className":5139,"style":4603},[82,83,442],[38,5141,461],{"className":5142},[460],[38,5144,5146],{"className":5145},[422],[38,5147,5149],{"className":5148,"style":5023},[426],[38,5150],{},[38,5152,606],{"className":5153},[514,442],[38,5155,5157,5160,5163],{"className":5156},[764,442],[38,5158,568],{"className":5159},[442],[38,5161,599],{"className":5162},[442],[38,5164,5044],{"className":5165},[442],[38,5167,371],{"className":5168},[474,442],[38,5170,5172,5175],{"className":5171},[82,442],[38,5173,374],{"className":5174},[82,83,442],[38,5176,5178],{"className":5177},[413],[38,5179,5181,5201],{"className":5180},[417,418],[38,5182,5184,5198],{"className":5183},[422],[38,5185,5187],{"className":5186,"style":4998},[426],[38,5188,5189,5192],{"style":5001},[38,5190],{"className":5191,"style":5005},[434],[38,5193,5195],{"className":5194},[439,5009,5010,442],[38,5196,4534],{"className":5197,"style":4603},[82,83,442],[38,5199,461],{"className":5200},[460],[38,5202,5204],{"className":5203},[422],[38,5205,5207],{"className":5206,"style":5023},[426],[38,5208],{},[38,5210,377],{"className":5211},[481,442],[38,5213,461],{"className":5214},[460],[38,5216,5218],{"className":5217},[422],[38,5219,5222],{"className":5220,"style":5221},[426],"height:0.5423em;",[38,5223],{},[38,5225],{"className":5226},[481,684],[134,5228,5229],{"align":110},[38,5230,5232,5253],{"className":5231},[41],[38,5233,5235],{"className":5234},[45],[47,5236,5237],{"xmlns":49},[51,5238,5239,5251],{},[54,5240,5241,5243,5245,5247,5249],{},[362,5242,4741],{"stretchy":370},[560,5244,159],{},[362,5246,364],{"separator":69},[560,5248,178],{},[362,5250,4750],{"stretchy":370},[61,5252,4753],{"encoding":63},[38,5254,5256],{"className":5255,"ariaHidden":69},[68],[38,5257,5259,5262,5265,5268,5271,5274,5277],{"className":5258},[73],[38,5260],{"className":5261,"style":638},[77],[38,5263,4741],{"className":5264},[474],[38,5266,159],{"className":5267},[82],[38,5269,364],{"className":5270},[453],[38,5272],{"className":5273,"style":655},[485],[38,5275,178],{"className":5276},[82],[38,5278,4750],{"className":5279},[481],[105,5281,5282,5285,5690],{},[134,5283,5284],{"align":110},"Mean normalization",[134,5286,5287],{"align":110},[38,5288,5290,5360],{"className":5289},[41],[38,5291,5293],{"className":5292},[45],[47,5294,5295],{"xmlns":49},[51,5296,5297,5357],{},[54,5298,5299,5305,5307],{},[351,5300,5301,5303],{},[57,5302,374],{},[57,5304,4534],{},[362,5306,4537],{},[557,5308,5309,5325],{},[54,5310,5311,5317,5319],{},[351,5312,5313,5315],{},[57,5314,374],{},[57,5316,4534],{},[362,5318,606],{},[351,5320,5321,5323],{},[57,5322,3791],{},[57,5324,4534],{},[54,5326,5327,5329,5331,5333,5339,5341,5343,5345,5347,5349,5355],{},[57,5328,2649],{},[362,5330,2652],{},[362,5332,371],{"stretchy":370},[351,5334,5335,5337],{},[57,5336,374],{},[57,5338,4534],{},[362,5340,377],{"stretchy":370},[362,5342,606],{},[57,5344,4822],{},[362,5346,2652],{},[362,5348,371],{"stretchy":370},[351,5350,5351,5353],{},[57,5352,374],{},[57,5354,4534],{},[362,5356,377],{"stretchy":370},[61,5358,5359],{"encoding":63},"x_j \\leftarrow \\frac{x_j - \\mu_j}{\\max(x_j) - \\min(x_j)}",[38,5361,5363,5418],{"className":5362,"ariaHidden":69},[68],[38,5364,5366,5369,5409,5412,5415],{"className":5365},[73],[38,5367],{"className":5368,"style":4572},[77],[38,5370,5372,5375],{"className":5371},[82],[38,5373,374],{"className":5374},[82,83],[38,5376,5378],{"className":5377},[413],[38,5379,5381,5401],{"className":5380},[417,418],[38,5382,5384,5398],{"className":5383},[422],[38,5385,5387],{"className":5386,"style":4591},[426],[38,5388,5389,5392],{"style":2828},[38,5390],{"className":5391,"style":435},[434],[38,5393,5395],{"className":5394},[439,440,441,442],[38,5396,4534],{"className":5397,"style":4603},[82,83,442],[38,5399,461],{"className":5400},[460],[38,5402,5404],{"className":5403},[422],[38,5405,5407],{"className":5406,"style":468},[426],[38,5408],{},[38,5410],{"className":5411,"style":486},[485],[38,5413,4537],{"className":5414},[490],[38,5416],{"className":5417,"style":486},[485],[38,5419,5421,5425],{"className":5420},[73],[38,5422],{"className":5423,"style":5424},[77],"height:1.458em;vertical-align:-0.5423em;",[38,5426,5428,5431,5687],{"className":5427},[82],[38,5429],{"className":5430},[474,684],[38,5432,5434],{"className":5433},[557],[38,5435,5437,5679],{"className":5436},[417,418],[38,5438,5440,5676],{"className":5439},[422],[38,5441,5444,5574,5582],{"className":5442,"style":5443},[426],"height:0.9157em;",[38,5445,5446,5449],{"style":700},[38,5447],{"className":5448,"style":704},[434],[38,5450,5452],{"className":5451},[439,440,441,442],[38,5453,5455,5467,5470,5510,5513,5516,5528,5531,5571],{"className":5454},[82,442],[38,5456,5458,5461,5464],{"className":5457},[764,442],[38,5459,568],{"className":5460},[442],[38,5462,27],{"className":5463},[442],[38,5465,374],{"className":5466},[442],[38,5468,371],{"className":5469},[474,442],[38,5471,5473,5476],{"className":5472},[82,442],[38,5474,374],{"className":5475},[82,83,442],[38,5477,5479],{"className":5478},[413],[38,5480,5482,5502],{"className":5481},[417,418],[38,5483,5485,5499],{"className":5484},[422],[38,5486,5488],{"className":5487,"style":4998},[426],[38,5489,5490,5493],{"style":5001},[38,5491],{"className":5492,"style":5005},[434],[38,5494,5496],{"className":5495},[439,5009,5010,442],[38,5497,4534],{"className":5498,"style":4603},[82,83,442],[38,5500,461],{"className":5501},[460],[38,5503,5505],{"className":5504},[422],[38,5506,5508],{"className":5507,"style":5023},[426],[38,5509],{},[38,5511,377],{"className":5512},[481,442],[38,5514,606],{"className":5515},[514,442],[38,5517,5519,5522,5525],{"className":5518},[764,442],[38,5520,568],{"className":5521},[442],[38,5523,599],{"className":5524},[442],[38,5526,5044],{"className":5527},[442],[38,5529,371],{"className":5530},[474,442],[38,5532,5534,5537],{"className":5533},[82,442],[38,5535,374],{"className":5536},[82,83,442],[38,5538,5540],{"className":5539},[413],[38,5541,5543,5563],{"className":5542},[417,418],[38,5544,5546,5560],{"className":5545},[422],[38,5547,5549],{"className":5548,"style":4998},[426],[38,5550,5551,5554],{"style":5001},[38,5552],{"className":5553,"style":5005},[434],[38,5555,5557],{"className":5556},[439,5009,5010,442],[38,5558,4534],{"className":5559,"style":4603},[82,83,442],[38,5561,461],{"className":5562},[460],[38,5564,5566],{"className":5565},[422],[38,5567,5569],{"className":5568,"style":5023},[426],[38,5570],{},[38,5572,377],{"className":5573},[481,442],[38,5575,5576,5579],{"style":719},[38,5577],{"className":5578,"style":704},[434],[38,5580],{"className":5581,"style":727},[726],[38,5583,5584,5587],{"style":5101},[38,5585],{"className":5586,"style":704},[434],[38,5588,5590],{"className":5589},[439,440,441,442],[38,5591,5593,5633,5636],{"className":5592},[82,442],[38,5594,5596,5599],{"className":5595},[82,442],[38,5597,374],{"className":5598},[82,83,442],[38,5600,5602],{"className":5601},[413],[38,5603,5605,5625],{"className":5604},[417,418],[38,5606,5608,5622],{"className":5607},[422],[38,5609,5611],{"className":5610,"style":4998},[426],[38,5612,5613,5616],{"style":5001},[38,5614],{"className":5615,"style":5005},[434],[38,5617,5619],{"className":5618},[439,5009,5010,442],[38,5620,4534],{"className":5621,"style":4603},[82,83,442],[38,5623,461],{"className":5624},[460],[38,5626,5628],{"className":5627},[422],[38,5629,5631],{"className":5630,"style":5023},[426],[38,5632],{},[38,5634,606],{"className":5635},[514,442],[38,5637,5639,5642],{"className":5638},[82,442],[38,5640,3791],{"className":5641},[82,83,442],[38,5643,5645],{"className":5644},[413],[38,5646,5648,5668],{"className":5647},[417,418],[38,5649,5651,5665],{"className":5650},[422],[38,5652,5654],{"className":5653,"style":4998},[426],[38,5655,5656,5659],{"style":5001},[38,5657],{"className":5658,"style":5005},[434],[38,5660,5662],{"className":5661},[439,5009,5010,442],[38,5663,4534],{"className":5664,"style":4603},[82,83,442],[38,5666,461],{"className":5667},[460],[38,5669,5671],{"className":5670},[422],[38,5672,5674],{"className":5673,"style":5023},[426],[38,5675],{},[38,5677,461],{"className":5678},[460],[38,5680,5682],{"className":5681},[422],[38,5683,5685],{"className":5684,"style":5221},[426],[38,5686],{},[38,5688],{"className":5689},[481,684],[134,5691,5692,5749],{"align":110},[38,5693,5695,5719],{"className":5694},[41],[38,5696,5698],{"className":5697},[45],[47,5699,5700],{"xmlns":49},[51,5701,5702,5716],{},[54,5703,5704,5706,5708,5710,5712,5714],{},[362,5705,4741],{"stretchy":370},[362,5707,606],{},[560,5709,178],{},[362,5711,364],{"separator":69},[560,5713,178],{},[362,5715,4750],{"stretchy":370},[61,5717,5718],{"encoding":63},"[-1, 1]",[38,5720,5722],{"className":5721,"ariaHidden":69},[68],[38,5723,5725,5728,5731,5734,5737,5740,5743,5746],{"className":5724},[73],[38,5726],{"className":5727,"style":638},[77],[38,5729,4741],{"className":5730},[474],[38,5732,606],{"className":5733},[82],[38,5735,178],{"className":5736},[82],[38,5738,364],{"className":5739},[453],[38,5741],{"className":5742,"style":655},[485],[38,5744,178],{"className":5745},[82],[38,5747,4750],{"className":5748},[481],", mean 0",[105,5751,5752,5755,6055],{},[134,5753,5754],{"align":110},"Z-score",[134,5756,5757],{"align":110},[38,5758,5760,5805],{"className":5759},[41],[38,5761,5763],{"className":5762},[45],[47,5764,5765],{"xmlns":49},[51,5766,5767,5802],{},[54,5768,5769,5775,5777],{},[351,5770,5771,5773],{},[57,5772,374],{},[57,5774,4534],{},[362,5776,4537],{},[557,5778,5779,5795],{},[54,5780,5781,5787,5789],{},[351,5782,5783,5785],{},[57,5784,374],{},[57,5786,4534],{},[362,5788,606],{},[351,5790,5791,5793],{},[57,5792,3791],{},[57,5794,4534],{},[351,5796,5797,5800],{},[57,5798,5799],{},"σ",[57,5801,4534],{},[61,5803,5804],{"encoding":63},"x_j \\leftarrow \\frac{x_j - \\mu_j}{\\sigma_j}",[38,5806,5808,5863],{"className":5807,"ariaHidden":69},[68],[38,5809,5811,5814,5854,5857,5860],{"className":5810},[73],[38,5812],{"className":5813,"style":4572},[77],[38,5815,5817,5820],{"className":5816},[82],[38,5818,374],{"className":5819},[82,83],[38,5821,5823],{"className":5822},[413],[38,5824,5826,5846],{"className":5825},[417,418],[38,5827,5829,5843],{"className":5828},[422],[38,5830,5832],{"className":5831,"style":4591},[426],[38,5833,5834,5837],{"style":2828},[38,5835],{"className":5836,"style":435},[434],[38,5838,5840],{"className":5839},[439,440,441,442],[38,5841,4534],{"className":5842,"style":4603},[82,83,442],[38,5844,461],{"className":5845},[460],[38,5847,5849],{"className":5848},[422],[38,5850,5852],{"className":5851,"style":468},[426],[38,5853],{},[38,5855],{"className":5856,"style":486},[485],[38,5858,4537],{"className":5859},[490],[38,5861],{"className":5862,"style":486},[485],[38,5864,5866,5869],{"className":5865},[73],[38,5867],{"className":5868,"style":5424},[77],[38,5870,5872,5875,6052],{"className":5871},[82],[38,5873],{"className":5874},[474,684],[38,5876,5878],{"className":5877},[557],[38,5879,5881,6044],{"className":5880},[417,418],[38,5882,5884,6041],{"className":5883},[422],[38,5885,5887,5939,5947],{"className":5886,"style":5443},[426],[38,5888,5889,5892],{"style":700},[38,5890],{"className":5891,"style":704},[434],[38,5893,5895],{"className":5894},[439,440,441,442],[38,5896,5898],{"className":5897},[82,442],[38,5899,5901,5904],{"className":5900},[82,442],[38,5902,5799],{"className":5903,"style":888},[82,83,442],[38,5905,5907],{"className":5906},[413],[38,5908,5910,5931],{"className":5909},[417,418],[38,5911,5913,5928],{"className":5912},[422],[38,5914,5916],{"className":5915,"style":4998},[426],[38,5917,5919,5922],{"style":5918},"top:-2.357em;margin-left:-0.0359em;margin-right:0.0714em;",[38,5920],{"className":5921,"style":5005},[434],[38,5923,5925],{"className":5924},[439,5009,5010,442],[38,5926,4534],{"className":5927,"style":4603},[82,83,442],[38,5929,461],{"className":5930},[460],[38,5932,5934],{"className":5933},[422],[38,5935,5937],{"className":5936,"style":5023},[426],[38,5938],{},[38,5940,5941,5944],{"style":719},[38,5942],{"className":5943,"style":704},[434],[38,5945],{"className":5946,"style":727},[726],[38,5948,5949,5952],{"style":5101},[38,5950],{"className":5951,"style":704},[434],[38,5953,5955],{"className":5954},[439,440,441,442],[38,5956,5958,5998,6001],{"className":5957},[82,442],[38,5959,5961,5964],{"className":5960},[82,442],[38,5962,374],{"className":5963},[82,83,442],[38,5965,5967],{"className":5966},[413],[38,5968,5970,5990],{"className":5969},[417,418],[38,5971,5973,5987],{"className":5972},[422],[38,5974,5976],{"className":5975,"style":4998},[426],[38,5977,5978,5981],{"style":5001},[38,5979],{"className":5980,"style":5005},[434],[38,5982,5984],{"className":5983},[439,5009,5010,442],[38,5985,4534],{"className":5986,"style":4603},[82,83,442],[38,5988,461],{"className":5989},[460],[38,5991,5993],{"className":5992},[422],[38,5994,5996],{"className":5995,"style":5023},[426],[38,5997],{},[38,5999,606],{"className":6000},[514,442],[38,6002,6004,6007],{"className":6003},[82,442],[38,6005,3791],{"className":6006},[82,83,442],[38,6008,6010],{"className":6009},[413],[38,6011,6013,6033],{"className":6012},[417,418],[38,6014,6016,6030],{"className":6015},[422],[38,6017,6019],{"className":6018,"style":4998},[426],[38,6020,6021,6024],{"style":5001},[38,6022],{"className":6023,"style":5005},[434],[38,6025,6027],{"className":6026},[439,5009,5010,442],[38,6028,4534],{"className":6029,"style":4603},[82,83,442],[38,6031,461],{"className":6032},[460],[38,6034,6036],{"className":6035},[422],[38,6037,6039],{"className":6038,"style":5023},[426],[38,6040],{},[38,6042,461],{"className":6043},[460],[38,6045,6047],{"className":6046},[422],[38,6048,6050],{"className":6049,"style":5221},[426],[38,6051],{},[38,6053],{"className":6054},[481,684],[134,6056,6057],{"align":110},"mean 0, std 1",[11,6059,6060,6061,2461,6131,6202,6203,6232,6233,6236],{},"I used z-score from here on, it's the most robust option (it doesn't depend on just two extreme points in the dataset, unlike min-max does). ",[38,6062,6064,6082],{"className":6063},[41],[38,6065,6067],{"className":6066},[45],[47,6068,6069],{"xmlns":49},[51,6070,6071,6079],{},[54,6072,6073],{},[351,6074,6075,6077],{},[57,6076,3791],{},[57,6078,4534],{},[61,6080,6081],{"encoding":63},"\\mu_j",[38,6083,6085],{"className":6084,"ariaHidden":69},[68],[38,6086,6088,6091],{"className":6087},[73],[38,6089],{"className":6090,"style":4572},[77],[38,6092,6094,6097],{"className":6093},[82],[38,6095,3791],{"className":6096},[82,83],[38,6098,6100],{"className":6099},[413],[38,6101,6103,6123],{"className":6102},[417,418],[38,6104,6106,6120],{"className":6105},[422],[38,6107,6109],{"className":6108,"style":4591},[426],[38,6110,6111,6114],{"style":2828},[38,6112],{"className":6113,"style":435},[434],[38,6115,6117],{"className":6116},[439,440,441,442],[38,6118,4534],{"className":6119,"style":4603},[82,83,442],[38,6121,461],{"className":6122},[460],[38,6124,6126],{"className":6125},[422],[38,6127,6129],{"className":6128,"style":468},[426],[38,6130],{},[38,6132,6134,6152],{"className":6133},[41],[38,6135,6137],{"className":6136},[45],[47,6138,6139],{"xmlns":49},[51,6140,6141,6149],{},[54,6142,6143],{},[351,6144,6145,6147],{},[57,6146,5799],{},[57,6148,4534],{},[61,6150,6151],{"encoding":63},"\\sigma_j",[38,6153,6155],{"className":6154,"ariaHidden":69},[68],[38,6156,6158,6161],{"className":6157},[73],[38,6159],{"className":6160,"style":4572},[77],[38,6162,6164,6167],{"className":6163},[82],[38,6165,5799],{"className":6166,"style":888},[82,83],[38,6168,6170],{"className":6169},[413],[38,6171,6173,6194],{"className":6172},[417,418],[38,6174,6176,6191],{"className":6175},[422],[38,6177,6179],{"className":6178,"style":4591},[426],[38,6180,6182,6185],{"style":6181},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[38,6183],{"className":6184,"style":435},[434],[38,6186,6188],{"className":6187},[439,440,441,442],[38,6189,4534],{"className":6190,"style":4603},[82,83,442],[38,6192,461],{"className":6193},[460],[38,6195,6197],{"className":6196},[422],[38,6198,6200],{"className":6199,"style":468},[426],[38,6201],{}," are the mean and standard deviation of feature ",[38,6204,6206,6219],{"className":6205},[41],[38,6207,6209],{"className":6208},[45],[47,6210,6211],{"xmlns":49},[51,6212,6213,6217],{},[54,6214,6215],{},[57,6216,4534],{},[61,6218,4534],{"encoding":63},[38,6220,6222],{"className":6221,"ariaHidden":69},[68],[38,6223,6225,6229],{"className":6224},[73],[38,6226],{"className":6227,"style":6228},[77],"height:0.854em;vertical-align:-0.1944em;",[38,6230,4534],{"className":6231,"style":4603},[82,83],", computed ",[33,6234,6235],{},"only from the training data",". That matters: I store those two numbers and reuse them for any new data that shows up later, I never recompute them.",[1309,6238,6240],{"className":1311,"code":6239,"language":1313,"meta":1314,"style":1314},"def zscore_normalize_features(X):\n    mu = X.mean(axis=0)      # mean of each column\n    sigma = X.std(axis=0)    # std of each column\n    X_norm = (X - mu) \u002F sigma\n    return X_norm, mu, sigma\n",[22,6241,6242,6247,6252,6258,6264],{"__ignoreMap":1314},[38,6243,6244],{"class":1319,"line":1320},[38,6245,6246],{},"def zscore_normalize_features(X):\n",[38,6248,6249],{"class":1319,"line":1326},[38,6250,6251],{},"    mu = X.mean(axis=0)      # mean of each column\n",[38,6253,6255],{"class":1319,"line":6254},3,[38,6256,6257],{},"    sigma = X.std(axis=0)    # std of each column\n",[38,6259,6261],{"class":1319,"line":6260},4,[38,6262,6263],{},"    X_norm = (X - mu) \u002F sigma\n",[38,6265,6267],{"class":1319,"line":6266},5,[38,6268,6269],{},"    return X_norm, mu, sigma\n",[11,6271,6272],{},"For our 8-house example, size's mean is 1480.1 sqft and its std is 414.7. I normalized it and the range, which used to span hundreds to thousands, turned into something between roughly -1 and 2.",[11,6274,6275,6276,6279],{},"One detail I almost slid past: z-score does ",[33,6277,6278],{},"not"," make the data normally distributed, it only rescales. The shape of the distribution stays exactly the same, only the center and scale change.",[87,6281,6283],{"id":6282},"gradient-descent-with-normalized-features","Gradient descent with normalized features",[11,6285,6286,6287,6339,6340,6412],{},"I ran the same algorithm, on the same 8-house dataset, but with size now z-score normalized, using ",[38,6288,6290,6309],{"className":6289},[41],[38,6291,6293],{"className":6292},[45],[47,6294,6295],{"xmlns":49},[51,6296,6297,6306],{},[54,6298,6299,6301,6303],{},[57,6300,59],{},[362,6302,380],{},[560,6304,6305],{},"0.1",[61,6307,6308],{"encoding":63},"\\alpha = 0.1",[38,6310,6312,6330],{"className":6311,"ariaHidden":69},[68],[38,6313,6315,6318,6321,6324,6327],{"className":6314},[73],[38,6316],{"className":6317,"style":78},[77],[38,6319,59],{"className":6320,"style":84},[82,83],[38,6322],{"className":6323,"style":486},[485],[38,6325,380],{"className":6326},[490],[38,6328],{"className":6329,"style":486},[485],[38,6331,6333,6336],{"className":6332},[73],[38,6334],{"className":6335,"style":1454},[77],[38,6337,6305],{"className":6338},[82],", six orders of magnitude bigger than the ",[38,6341,6343,6365],{"className":6342},[41],[38,6344,6346],{"className":6345},[45],[47,6347,6348],{"xmlns":49},[51,6349,6350,6362],{},[54,6351,6352],{},[589,6353,6354,6356],{},[560,6355,1493],{},[54,6357,6358,6360],{},[362,6359,606],{},[560,6361,1500],{},[61,6363,6364],{"encoding":63},"10^{-7}",[38,6366,6368],{"className":6367,"ariaHidden":69},[68],[38,6369,6371,6374,6377],{"className":6370},[73],[38,6372],{"className":6373,"style":939},[77],[38,6375,178],{"className":6376},[82],[38,6378,6380,6383],{"className":6379},[82],[38,6381,159],{"className":6382},[82],[38,6384,6386],{"className":6385},[413],[38,6387,6389],{"className":6388},[417],[38,6390,6392],{"className":6391},[422],[38,6393,6395],{"className":6394,"style":939},[426],[38,6396,6397,6400],{"style":844},[38,6398],{"className":6399,"style":435},[434],[38,6401,6403],{"className":6402},[439,440,441,442],[38,6404,6406,6409],{"className":6405},[82,442],[38,6407,606],{"className":6408},[82,442],[38,6410,1500],{"className":6411},[82,442]," values from before:",[2355,6414],{":b-range":6415,":initial-b":159,":initial-w":159,":w-range":6416,":x-train":6417,":y-train":2360,"b-label":367,"w-label":6418,":initial-alpha":6305},"[300, 500]","[-50, 200]","[-1.273, -0.569, 1.125, 0.591, 1.155, -0.400, -1.486, 0.858]","w (normalized)",[11,6420,6421,6422,6474,6475,6504,6505,6599,6600,6683],{},"After about 50 iterations the cost already flattened near the minimum (I got ",[38,6423,6425,6444],{"className":6424},[41],[38,6426,6428],{"className":6427},[45],[47,6429,6430],{"xmlns":49},[51,6431,6432,6441],{},[54,6433,6434,6436,6438],{},[57,6435,543],{},[362,6437,3156],{},[560,6439,6440],{},"919",[61,6442,6443],{"encoding":63},"J \\approx 919",[38,6445,6447,6465],{"className":6446,"ariaHidden":69},[68],[38,6448,6450,6453,6456,6459,6462],{"className":6449},[73],[38,6451],{"className":6452,"style":3023},[77],[38,6454,543],{"className":6455,"style":642},[82,83],[38,6457],{"className":6458,"style":486},[485],[38,6460,3156],{"className":6461},[490],[38,6463],{"className":6464,"style":486},[485],[38,6466,6468,6471],{"className":6467},[73],[38,6469],{"className":6470,"style":1454},[77],[38,6472,6440],{"className":6473},[82],", almost identical to the real minimum of ",[38,6476,6478,6492],{"className":6477},[41],[38,6479,6481],{"className":6480},[45],[47,6482,6483],{"xmlns":49},[51,6484,6485,6490],{},[54,6486,6487],{},[560,6488,6489],{},"919.37",[61,6491,6489],{"encoding":63},[38,6493,6495],{"className":6494,"ariaHidden":69},[68],[38,6496,6498,6501],{"className":6497},[73],[38,6499],{"className":6500,"style":1454},[77],[38,6502,6489],{"className":6503},[82]," I computed directly through a closed-form solution). In the raw example, with ",[38,6506,6508,6534],{"className":6507},[41],[38,6509,6511],{"className":6510},[45],[47,6512,6513],{"xmlns":49},[51,6514,6515,6531],{},[54,6516,6517,6519,6521],{},[57,6518,59],{},[362,6520,380],{},[589,6522,6523,6525],{},[560,6524,1493],{},[54,6526,6527,6529],{},[362,6528,606],{},[560,6530,1500],{},[61,6532,6533],{"encoding":63},"\\alpha = 10^{-7}",[38,6535,6537,6555],{"className":6536,"ariaHidden":69},[68],[38,6538,6540,6543,6546,6549,6552],{"className":6539},[73],[38,6541],{"className":6542,"style":78},[77],[38,6544,59],{"className":6545,"style":84},[82,83],[38,6547],{"className":6548,"style":486},[485],[38,6550,380],{"className":6551},[490],[38,6553],{"className":6554,"style":486},[485],[38,6556,6558,6561,6564],{"className":6557},[73],[38,6559],{"className":6560,"style":939},[77],[38,6562,178],{"className":6563},[82],[38,6565,6567,6570],{"className":6566},[82],[38,6568,159],{"className":6569},[82],[38,6571,6573],{"className":6572},[413],[38,6574,6576],{"className":6575},[417],[38,6577,6579],{"className":6578},[422],[38,6580,6582],{"className":6581,"style":939},[426],[38,6583,6584,6587],{"style":844},[38,6585],{"className":6586,"style":435},[434],[38,6588,6590],{"className":6589},[439,440,441,442],[38,6591,6593,6596],{"className":6592},[82,442],[38,6594,606],{"className":6595},[82,442],[38,6597,1500],{"className":6598},[82,442],", cost had barely moved off ",[38,6601,6603,6624],{"className":6602},[41],[38,6604,6606],{"className":6605},[45],[47,6607,6608],{"xmlns":49},[51,6609,6610,6622],{},[54,6611,6612,6614,6616],{},[560,6613,2276],{},[362,6615,1488],{},[589,6617,6618,6620],{},[560,6619,1493],{},[560,6621,1992],{},[61,6623,2287],{"encoding":63},[38,6625,6627,6645],{"className":6626,"ariaHidden":69},[68],[38,6628,6630,6633,6636,6639,6642],{"className":6629},[73],[38,6631],{"className":6632,"style":1531},[77],[38,6634,2276],{"className":6635},[82],[38,6637],{"className":6638,"style":510},[485],[38,6640,1488],{"className":6641},[514],[38,6643],{"className":6644,"style":510},[485],[38,6646,6648,6651,6654],{"className":6647},[73],[38,6649],{"className":6650,"style":939},[77],[38,6652,178],{"className":6653},[82],[38,6655,6657,6660],{"className":6656},[82],[38,6658,159],{"className":6659},[82],[38,6661,6663],{"className":6662},[413],[38,6664,6666],{"className":6665},[417],[38,6667,6669],{"className":6668},[422],[38,6670,6672],{"className":6671,"style":939},[426],[38,6673,6674,6677],{"style":844},[38,6675],{"className":6676,"style":435},[434],[38,6678,6680],{"className":6679},[439,440,441,442],[38,6681,1992],{"className":6682},[82,442]," after 10 whole iterations. Normalizing didn't change where the minimum is, it completely changed how fast I get there.",[11,6685,6686,6687,6715,6716,6744,6745,6747,6748,6750],{},"I also noticed something small but satisfying to understand: when ",[38,6688,6690,6703],{"className":6689},[41],[38,6691,6693],{"className":6692},[45],[47,6694,6695],{"xmlns":49},[51,6696,6697,6701],{},[54,6698,6699],{},[57,6700,374],{},[61,6702,374],{"encoding":63},[38,6704,6706],{"className":6705,"ariaHidden":69},[68],[38,6707,6709,6712],{"className":6708},[73],[38,6710],{"className":6711,"style":78},[77],[38,6713,374],{"className":6714},[82,83]," is centered at zero (mean zero, thanks to z-score), the optimal ",[38,6717,6719,6732],{"className":6718},[41],[38,6720,6722],{"className":6721},[45],[47,6723,6724],{"xmlns":49},[51,6725,6726,6730],{},[54,6727,6728],{},[57,6729,367],{},[61,6731,367],{"encoding":63},[38,6733,6735],{"className":6734,"ariaHidden":69},[68],[38,6736,6738,6741],{"className":6737},[73],[38,6739],{"className":6740,"style":524},[77],[38,6742,367],{"className":6743},[82,83]," turns out to be exactly the mean of the training prices. That makes sense geometrically: if every ",[22,6746,374],{}," is zero on average, the best line passes through the average ",[22,6749,611],{}," at that point.",[87,6752,6754],{"id":6753},"predicting-a-new-house-and-denormalizing","Predicting a new house and denormalizing",[11,6756,6757],{},"I fit the model with all 4 real features (not just size) on the full 100-house dataset, using the normal equation (the exact closed-form solution, no gradient descent):",[1309,6759,6761],{"className":1311,"code":6760,"language":1313,"meta":1314,"style":1314},"w = [0.268, -32.90, -67.29, -1.465]   # size, bedrooms, floors, age\nb = 221.50\n",[22,6762,6763,6768],{"__ignoreMap":1314},[38,6764,6765],{"class":1319,"line":1320},[38,6766,6767],{},"w = [0.268, -32.90, -67.29, -1.465]   # size, bedrooms, floors, age\n",[38,6769,6770],{"class":1319,"line":1326},[38,6771,6772],{},"b = 221.50\n",[11,6774,6775,6776,4072],{},"I predicted the target house's price (1200 sqft, 3 bedrooms, 1 floor, 40 years old) and got ",[33,6777,6778],{},"$318.9 thousand",[11,6780,6781,6782,6785],{},"What caught my attention most was the bedrooms coefficient: ",[33,6783,6784],{},"negative",". On its own, more bedrooms sounds like a purely good thing, but controlling for the house's size (which is already in the model), more bedrooms in a same-sized house usually means smaller bedrooms, which pulls the price down a bit. That's exactly the kind of thing that only shows up once you look at the coefficients of a model with several features at the same time, you can't see it in a bedrooms-vs-price chart alone.",[11,6787,6788,6789,6860],{},"If I'd fit with normalized features, the weights would come out on a different scale (each ",[38,6790,6792,6810],{"className":6791},[41],[38,6793,6795],{"className":6794},[45],[47,6796,6797],{"xmlns":49},[51,6798,6799,6807],{},[54,6800,6801],{},[351,6802,6803,6805],{},[57,6804,360],{},[57,6806,4534],{},[61,6808,6809],{"encoding":63},"w_j",[38,6811,6813],{"className":6812,"ariaHidden":69},[68],[38,6814,6816,6819],{"className":6815},[73],[38,6817],{"className":6818,"style":4572},[77],[38,6820,6822,6825],{"className":6821},[82],[38,6823,360],{"className":6824,"style":449},[82,83],[38,6826,6828],{"className":6827},[413],[38,6829,6831,6852],{"className":6830},[417,418],[38,6832,6834,6849],{"className":6833},[422],[38,6835,6837],{"className":6836,"style":4591},[426],[38,6838,6840,6843],{"style":6839},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[38,6841],{"className":6842,"style":435},[434],[38,6844,6846],{"className":6845},[439,440,441,442],[38,6847,4534],{"className":6848,"style":4603},[82,83,442],[38,6850,461],{"className":6851},[460],[38,6853,6855],{"className":6854},[422],[38,6856,6858],{"className":6857,"style":468},[426],[38,6859],{}," would represent \"how much the price changes per 1 standard deviation of that feature\", not per 1 real unit). To get back to real-unit coefficients, the denormalization formula is:",[11,6862,6863],{},[38,6864,6866,7055],{"className":6865},[41],[38,6867,6869],{"className":6868},[45],[47,6870,6871],{"xmlns":49},[51,6872,6873,7052],{},[54,6874,6875,6904,6906,6945,6947,6969,6971,6997,6999,7005],{},[6876,6877,6878,6880,6882],"msubsup",{},[57,6879,360],{},[57,6881,4534],{},[54,6883,6884,6887,6890,6892,6895,6897,6899,6901],{},[57,6885,6886],{},"o",[57,6888,6889],{},"r",[57,6891,599],{},[57,6893,6894],{},"g",[57,6896,599],{},[57,6898,5044],{},[57,6900,27],{},[57,6902,6903],{},"l",[362,6905,380],{},[557,6907,6908,6939],{},[6876,6909,6910,6912,6914],{},[57,6911,360],{},[57,6913,4534],{},[54,6915,6916,6918,6920,6922,6924,6926,6928,6930,6933,6936],{},[57,6917,5044],{},[57,6919,6886],{},[57,6921,6889],{},[57,6923,568],{},[57,6925,27],{},[57,6927,6903],{},[57,6929,599],{},[57,6931,6932],{},"z",[57,6934,6935],{},"e",[57,6937,6938],{},"d",[351,6940,6941,6943],{},[57,6942,5799],{},[57,6944,4534],{},[485,6946],{"width":998},[589,6948,6949,6951],{},[57,6950,367],{},[54,6952,6953,6955,6957,6959,6961,6963,6965,6967],{},[57,6954,6886],{},[57,6956,6889],{},[57,6958,599],{},[57,6960,6894],{},[57,6962,599],{},[57,6964,5044],{},[57,6966,27],{},[57,6968,6903],{},[362,6970,380],{},[589,6972,6973,6975],{},[57,6974,367],{},[54,6976,6977,6979,6981,6983,6985,6987,6989,6991,6993,6995],{},[57,6978,5044],{},[57,6980,6886],{},[57,6982,6889],{},[57,6984,568],{},[57,6986,27],{},[57,6988,6903],{},[57,6990,599],{},[57,6992,6932],{},[57,6994,6935],{},[57,6996,6938],{},[362,6998,606],{},[351,7000,7001,7003],{},[362,7002,571],{},[57,7004,4534],{},[557,7006,7007,7046],{},[54,7008,7009,7037,7040],{},[6876,7010,7011,7013,7015],{},[57,7012,360],{},[57,7014,4534],{},[54,7016,7017,7019,7021,7023,7025,7027,7029,7031,7033,7035],{},[57,7018,5044],{},[57,7020,6886],{},[57,7022,6889],{},[57,7024,568],{},[57,7026,27],{},[57,7028,6903],{},[57,7030,599],{},[57,7032,6932],{},[57,7034,6935],{},[57,7036,6938],{},[362,7038,7039],{},"⋅",[351,7041,7042,7044],{},[57,7043,3791],{},[57,7045,4534],{},[351,7047,7048,7050],{},[57,7049,5799],{},[57,7051,4534],{},[61,7053,7054],{"encoding":63},"w_j^{original} = \\frac{w_j^{normalized}}{\\sigma_j} \\qquad b^{original} = b^{normalized} - \\sum_j \\frac{w_j^{normalized} \\cdot \\mu_j}{\\sigma_j}",[38,7056,7058,7148,7397,7466],{"className":7057,"ariaHidden":69},[68],[38,7059,7061,7065,7139,7142,7145],{"className":7060},[73],[38,7062],{"className":7063,"style":7064},[77],"height:1.38em;vertical-align:-0.413em;",[38,7066,7068,7071],{"className":7067},[82],[38,7069,360],{"className":7070,"style":449},[82,83],[38,7072,7074],{"className":7073},[413],[38,7075,7077,7130],{"className":7076},[417,418],[38,7078,7080,7127],{"className":7079},[422],[38,7081,7084,7096],{"className":7082,"style":7083},[426],"height:0.967em;",[38,7085,7087,7090],{"style":7086},"top:-2.4231em;margin-left:-0.0269em;margin-right:0.05em;",[38,7088],{"className":7089,"style":435},[434],[38,7091,7093],{"className":7092},[439,440,441,442],[38,7094,4534],{"className":7095,"style":4603},[82,83,442],[38,7097,7099,7102],{"style":7098},"top:-3.1809em;margin-right:0.05em;",[38,7100],{"className":7101,"style":435},[434],[38,7103,7105],{"className":7104},[439,440,441,442],[38,7106,7108,7113,7116,7119,7123],{"className":7107},[82,442],[38,7109,7112],{"className":7110,"style":7111},[82,83,442],"margin-right:0.0278em;","or",[38,7114,599],{"className":7115},[82,83,442],[38,7117,6894],{"className":7118,"style":888},[82,83,442],[38,7120,7122],{"className":7121},[82,83,442],"ina",[38,7124,6903],{"className":7125,"style":7126},[82,83,442],"margin-right:0.0197em;",[38,7128,461],{"className":7129},[460],[38,7131,7133],{"className":7132},[422],[38,7134,7137],{"className":7135,"style":7136},[426],"height:0.413em;",[38,7138],{},[38,7140],{"className":7141,"style":486},[485],[38,7143,380],{"className":7144},[490],[38,7146],{"className":7147,"style":486},[485],[38,7149,7151,7155,7340,7343,7388,7391,7394],{"className":7150},[73],[38,7152],{"className":7153,"style":7154},[77],"height:1.7987em;vertical-align:-0.5423em;",[38,7156,7158,7161,7337],{"className":7157},[82],[38,7159],{"className":7160},[474,684],[38,7162,7164],{"className":7163},[557],[38,7165,7167,7329],{"className":7166},[417,418],[38,7168,7170,7326],{"className":7169},[422],[38,7171,7174,7225,7233],{"className":7172,"style":7173},[426],"height:1.2563em;",[38,7175,7176,7179],{"style":700},[38,7177],{"className":7178,"style":704},[434],[38,7180,7182],{"className":7181},[439,440,441,442],[38,7183,7185],{"className":7184},[82,442],[38,7186,7188,7191],{"className":7187},[82,442],[38,7189,5799],{"className":7190,"style":888},[82,83,442],[38,7192,7194],{"className":7193},[413],[38,7195,7197,7217],{"className":7196},[417,418],[38,7198,7200,7214],{"className":7199},[422],[38,7201,7203],{"className":7202,"style":4998},[426],[38,7204,7205,7208],{"style":5918},[38,7206],{"className":7207,"style":5005},[434],[38,7209,7211],{"className":7210},[439,5009,5010,442],[38,7212,4534],{"className":7213,"style":4603},[82,83,442],[38,7215,461],{"className":7216},[460],[38,7218,7220],{"className":7219},[422],[38,7221,7223],{"className":7222,"style":5023},[426],[38,7224],{},[38,7226,7227,7230],{"style":719},[38,7228],{"className":7229,"style":704},[434],[38,7231],{"className":7232,"style":727},[726],[38,7234,7236,7239],{"style":7235},"top:-3.6074em;",[38,7237],{"className":7238,"style":704},[434],[38,7240,7242],{"className":7241},[439,440,441,442],[38,7243,7245],{"className":7244},[82,442],[38,7246,7248,7251],{"className":7247},[82,442],[38,7249,360],{"className":7250,"style":449},[82,83,442],[38,7252,7254],{"className":7253},[413],[38,7255,7257,7317],{"className":7256},[417,418],[38,7258,7260,7314],{"className":7259},[422],[38,7261,7264,7276],{"className":7262,"style":7263},[426],"height:0.927em;",[38,7265,7267,7270],{"style":7266},"top:-2.214em;margin-left:-0.0269em;margin-right:0.0714em;",[38,7268],{"className":7269,"style":5005},[434],[38,7271,7273],{"className":7272},[439,5009,5010,442],[38,7274,4534],{"className":7275,"style":4603},[82,83,442],[38,7277,7279,7282],{"style":7278},"top:-2.931em;margin-right:0.0714em;",[38,7280],{"className":7281,"style":5005},[434],[38,7283,7285],{"className":7284},[439,5009,5010,442],[38,7286,7288,7291,7294,7298,7301,7304,7308,7311],{"className":7287},[82,442],[38,7289,5044],{"className":7290},[82,83,442],[38,7292,7112],{"className":7293,"style":7111},[82,83,442],[38,7295,7297],{"className":7296},[82,83,442],"ma",[38,7299,6903],{"className":7300,"style":7126},[82,83,442],[38,7302,599],{"className":7303},[82,83,442],[38,7305,6932],{"className":7306,"style":7307},[82,83,442],"margin-right:0.044em;",[38,7309,6935],{"className":7310},[82,83,442],[38,7312,6938],{"className":7313},[82,83,442],[38,7315,461],{"className":7316},[460],[38,7318,7320],{"className":7319},[422],[38,7321,7324],{"className":7322,"style":7323},[426],"height:0.4249em;",[38,7325],{},[38,7327,461],{"className":7328},[460],[38,7330,7332],{"className":7331},[422],[38,7333,7335],{"className":7334,"style":5221},[426],[38,7336],{},[38,7338],{"className":7339},[481,684],[38,7341],{"className":7342,"style":1154},[485],[38,7344,7346,7349],{"className":7345},[82],[38,7347,367],{"className":7348},[82,83],[38,7350,7352],{"className":7351},[413],[38,7353,7355],{"className":7354},[417],[38,7356,7358],{"className":7357},[422],[38,7359,7362],{"className":7360,"style":7361},[426],"height:0.8491em;",[38,7363,7364,7367],{"style":844},[38,7365],{"className":7366,"style":435},[434],[38,7368,7370],{"className":7369},[439,440,441,442],[38,7371,7373,7376,7379,7382,7385],{"className":7372},[82,442],[38,7374,7112],{"className":7375,"style":7111},[82,83,442],[38,7377,599],{"className":7378},[82,83,442],[38,7380,6894],{"className":7381,"style":888},[82,83,442],[38,7383,7122],{"className":7384},[82,83,442],[38,7386,6903],{"className":7387,"style":7126},[82,83,442],[38,7389],{"className":7390,"style":486},[485],[38,7392,380],{"className":7393},[490],[38,7395],{"className":7396,"style":486},[485],[38,7398,7400,7404,7457,7460,7463],{"className":7399},[73],[38,7401],{"className":7402,"style":7403},[77],"height:0.9324em;vertical-align:-0.0833em;",[38,7405,7407,7410],{"className":7406},[82],[38,7408,367],{"className":7409},[82,83],[38,7411,7413],{"className":7412},[413],[38,7414,7416],{"className":7415},[417],[38,7417,7419],{"className":7418},[422],[38,7420,7422],{"className":7421,"style":7361},[426],[38,7423,7424,7427],{"style":844},[38,7425],{"className":7426,"style":435},[434],[38,7428,7430],{"className":7429},[439,440,441,442],[38,7431,7433,7436,7439,7442,7445,7448,7451,7454],{"className":7432},[82,442],[38,7434,5044],{"className":7435},[82,83,442],[38,7437,7112],{"className":7438,"style":7111},[82,83,442],[38,7440,7297],{"className":7441},[82,83,442],[38,7443,6903],{"className":7444,"style":7126},[82,83,442],[38,7446,599],{"className":7447},[82,83,442],[38,7449,6932],{"className":7450,"style":7307},[82,83,442],[38,7452,6935],{"className":7453},[82,83,442],[38,7455,6938],{"className":7456},[82,83,442],[38,7458],{"className":7459,"style":510},[485],[38,7461,606],{"className":7462},[514],[38,7464],{"className":7465,"style":510},[485],[38,7467,7469,7472,7515,7518],{"className":7468},[73],[38,7470],{"className":7471,"style":7154},[77],[38,7473,7475,7478],{"className":7474},[764],[38,7476,571],{"className":7477,"style":767},[764,765,766],[38,7479,7481],{"className":7480},[413],[38,7482,7484,7506],{"className":7483},[417,418],[38,7485,7487,7503],{"className":7486},[422],[38,7488,7491],{"className":7489,"style":7490},[426],"height:0.162em;",[38,7492,7494,7497],{"style":7493},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[38,7495],{"className":7496,"style":435},[434],[38,7498,7500],{"className":7499},[439,440,441,442],[38,7501,4534],{"className":7502,"style":4603},[82,83,442],[38,7504,461],{"className":7505},[460],[38,7507,7509],{"className":7508},[422],[38,7510,7513],{"className":7511,"style":7512},[426],"height:0.4358em;",[38,7514],{},[38,7516],{"className":7517,"style":655},[485],[38,7519,7521,7524,7735],{"className":7520},[82],[38,7522],{"className":7523},[474,684],[38,7525,7527],{"className":7526},[557],[38,7528,7530,7727],{"className":7529},[417,418],[38,7531,7533,7724],{"className":7532},[422],[38,7534,7536,7587,7595],{"className":7535,"style":7173},[426],[38,7537,7538,7541],{"style":700},[38,7539],{"className":7540,"style":704},[434],[38,7542,7544],{"className":7543},[439,440,441,442],[38,7545,7547],{"className":7546},[82,442],[38,7548,7550,7553],{"className":7549},[82,442],[38,7551,5799],{"className":7552,"style":888},[82,83,442],[38,7554,7556],{"className":7555},[413],[38,7557,7559,7579],{"className":7558},[417,418],[38,7560,7562,7576],{"className":7561},[422],[38,7563,7565],{"className":7564,"style":4998},[426],[38,7566,7567,7570],{"style":5918},[38,7568],{"className":7569,"style":5005},[434],[38,7571,7573],{"className":7572},[439,5009,5010,442],[38,7574,4534],{"className":7575,"style":4603},[82,83,442],[38,7577,461],{"className":7578},[460],[38,7580,7582],{"className":7581},[422],[38,7583,7585],{"className":7584,"style":5023},[426],[38,7586],{},[38,7588,7589,7592],{"style":719},[38,7590],{"className":7591,"style":704},[434],[38,7593],{"className":7594,"style":727},[726],[38,7596,7597,7600],{"style":7235},[38,7598],{"className":7599,"style":704},[434],[38,7601,7603],{"className":7602},[439,440,441,442],[38,7604,7606,7681,7684],{"className":7605},[82,442],[38,7607,7609,7612],{"className":7608},[82,442],[38,7610,360],{"className":7611,"style":449},[82,83,442],[38,7613,7615],{"className":7614},[413],[38,7616,7618,7673],{"className":7617},[417,418],[38,7619,7621,7670],{"className":7620},[422],[38,7622,7624,7635],{"className":7623,"style":7263},[426],[38,7625,7626,7629],{"style":7266},[38,7627],{"className":7628,"style":5005},[434],[38,7630,7632],{"className":7631},[439,5009,5010,442],[38,7633,4534],{"className":7634,"style":4603},[82,83,442],[38,7636,7637,7640],{"style":7278},[38,7638],{"className":7639,"style":5005},[434],[38,7641,7643],{"className":7642},[439,5009,5010,442],[38,7644,7646,7649,7652,7655,7658,7661,7664,7667],{"className":7645},[82,442],[38,7647,5044],{"className":7648},[82,83,442],[38,7650,7112],{"className":7651,"style":7111},[82,83,442],[38,7653,7297],{"className":7654},[82,83,442],[38,7656,6903],{"className":7657,"style":7126},[82,83,442],[38,7659,599],{"className":7660},[82,83,442],[38,7662,6932],{"className":7663,"style":7307},[82,83,442],[38,7665,6935],{"className":7666},[82,83,442],[38,7668,6938],{"className":7669},[82,83,442],[38,7671,461],{"className":7672},[460],[38,7674,7676],{"className":7675},[422],[38,7677,7679],{"className":7678,"style":7323},[426],[38,7680],{},[38,7682,7039],{"className":7683},[514,442],[38,7685,7687,7690],{"className":7686},[82,442],[38,7688,3791],{"className":7689},[82,83,442],[38,7691,7693],{"className":7692},[413],[38,7694,7696,7716],{"className":7695},[417,418],[38,7697,7699,7713],{"className":7698},[422],[38,7700,7702],{"className":7701,"style":4998},[426],[38,7703,7704,7707],{"style":5001},[38,7705],{"className":7706,"style":5005},[434],[38,7708,7710],{"className":7709},[439,5009,5010,442],[38,7711,4534],{"className":7712,"style":4603},[82,83,442],[38,7714,461],{"className":7715},[460],[38,7717,7719],{"className":7718},[422],[38,7720,7722],{"className":7721,"style":5023},[426],[38,7723],{},[38,7725,461],{"className":7726},[460],[38,7728,7730],{"className":7729},[422],[38,7731,7733],{"className":7732,"style":5221},[426],[38,7734],{},[38,7736],{"className":7737},[481,684],[11,7739,7740,7741,2461,7769,7797,7798,2461,7850,7902],{},"I checked this on our 1-feature example: denormalizing the ",[38,7742,7744,7757],{"className":7743},[41],[38,7745,7747],{"className":7746},[45],[47,7748,7749],{"xmlns":49},[51,7750,7751,7755],{},[54,7752,7753],{},[57,7754,360],{},[61,7756,360],{"encoding":63},[38,7758,7760],{"className":7759,"ariaHidden":69},[68],[38,7761,7763,7766],{"className":7762},[73],[38,7764],{"className":7765,"style":78},[77],[38,7767,360],{"className":7768,"style":449},[82,83],[38,7770,7772,7785],{"className":7771},[41],[38,7773,7775],{"className":7774},[45],[47,7776,7777],{"xmlns":49},[51,7778,7779,7783],{},[54,7780,7781],{},[57,7782,367],{},[61,7784,367],{"encoding":63},[38,7786,7788],{"className":7787,"ariaHidden":69},[68],[38,7789,7791,7794],{"className":7790},[73],[38,7792],{"className":7793,"style":524},[77],[38,7795,367],{"className":7796},[82,83]," that gradient descent found in normalized space, I got ",[38,7799,7801,7820],{"className":7800},[41],[38,7802,7804],{"className":7803},[45],[47,7805,7806],{"xmlns":49},[51,7807,7808,7817],{},[54,7809,7810,7812,7814],{},[57,7811,360],{},[362,7813,380],{},[560,7815,7816],{},"0.267",[61,7818,7819],{"encoding":63},"w = 0.267",[38,7821,7823,7841],{"className":7822,"ariaHidden":69},[68],[38,7824,7826,7829,7832,7835,7838],{"className":7825},[73],[38,7827],{"className":7828,"style":78},[77],[38,7830,360],{"className":7831,"style":449},[82,83],[38,7833],{"className":7834,"style":486},[485],[38,7836,380],{"className":7837},[490],[38,7839],{"className":7840,"style":486},[485],[38,7842,7844,7847],{"className":7843},[73],[38,7845],{"className":7846,"style":1454},[77],[38,7848,7816],{"className":7849},[82],[38,7851,7853,7872],{"className":7852},[41],[38,7854,7856],{"className":7855},[45],[47,7857,7858],{"xmlns":49},[51,7859,7860,7869],{},[54,7861,7862,7864,7866],{},[57,7863,367],{},[362,7865,380],{},[560,7867,7868],{},"7.16",[61,7870,7871],{"encoding":63},"b = 7.16",[38,7873,7875,7893],{"className":7874,"ariaHidden":69},[68],[38,7876,7878,7881,7884,7887,7890],{"className":7877},[73],[38,7879],{"className":7880,"style":524},[77],[38,7882,367],{"className":7883},[82,83],[38,7885],{"className":7886,"style":486},[485],[38,7888,380],{"className":7889},[490],[38,7891],{"className":7892,"style":486},[485],[38,7894,7896,7899],{"className":7895},[73],[38,7897],{"className":7898,"style":1454},[77],[38,7900,7868],{"className":7901},[82],", the exact same numbers the closed-form solution gives directly in raw space. Both paths land in the same place, one of them just converged in 50 iterations and the other would need millions.",[87,7904,7906],{"id":7905},"the-contours-the-geometric-picture","The contours: the geometric picture",[11,7908,7909,7910,2347,7913,7916],{},"Everything I've described in numbers, I can see at once in a single image. From here on I switched datasets: I'm using the 50-house one that already showed up in the previous posts in this playlist (",[22,7911,7912],{},"square_feet",[22,7914,7915],{},"num_bedrooms",", location score, distance to center), not the 100-house one from the notebook I used earlier. It's a different set of houses, so the correlation numbers here differ from the ones I computed in \"The problem\" section.",[11,7918,7919,7920,7924,7925,7928],{},"Before going abstract (the shape of the ",[7921,7922,7923],"em",{},"cost","), it's worth seeing the far more direct, concrete effect first: the shape of the ",[7921,7926,7927],{},"data points themselves",". I took size and number of bedrooms from these 50 houses and plotted one against the other, raw and then normalized:",[11,7930,7931],{},[33,7932,7933],{},"Points, without normalizing:",[7935,7936],"housing-scaling-scatter",{"bedrooms-label":156,"bedrooms-label-normalized":7937,"mode":7938,"size-label":136,"size-label-normalized":7939},"bedrooms (normalized)","raw","size (normalized)",[11,7941,7942],{},[33,7943,7944],{},"Points, after normalizing (z-score):",[7935,7946],{"bedrooms-label":156,"bedrooms-label-normalized":7937,"mode":7947,"size-label":136,"size-label-normalized":7939},"normalized",[11,7949,7950],{},"Notice the cloud of points doesn't change shape, it just shifts and rescales so both axes end up in the same unit (standard deviations instead of sqft\u002Fbedrooms). And it's exactly that change in relative scale between the two axes that makes all the difference further down.",[11,7952,7953],{},"Now the cost map. I fit a model with all 4 features (size, bedrooms, location score, distance to center), fixed two of them at their fitted values, and drew a heatmap varying just size's weight and bedrooms' weight, the same trick the original notebook uses. The red dot marks exactly where my fit landed, and the line right below the chart shows the exact values.",[11,7955,7956],{},[33,7957,7958],{},"Without normalizing:",[7960,7961],"housing-cost-contour",{":b-max":7962,":b-min":1493,":w-max":181,":w-min":159,"b-label":7963,"mode":7938,"w-label":7964},"100","w_bedrooms (raw)","w_size (raw)",[11,7966,7967],{},[33,7968,7969],{},"After normalizing (z-score):",[7960,7971],{":b-max":7972,":b-min":7973,":w-max":7974,":w-min":7975,"b-label":7976,"mode":7947,"w-label":7977},"140","20","150","30","w_bedrooms (normalized)","w_size (normalized)",[11,7979,7980],{},"Notice the difference in shape. In the raw contour, the size axis is far more sensitive than the bedrooms axis (a tiny 0.5 change in size's weight already sends the cost skyrocketing, while bedrooms' weight needs to move dozens of units to do the same damage). It's a heavily elongated valley, almost a corridor, and the red dot (the real fit) sits squeezed at the bottom of it. In the normalized contour, both directions end up with similar sensitivity, the shape turns much rounder, and the dot lands near the center of a circle.",[11,7982,7983,7984,8035],{},"It's not a perfect circle (size and number of bedrooms, in this 50-house dataset, still have a small residual correlation, I calculated it and it came out around 0.02, very close to zero but not exactly zero), but the difference in shape is dramatic. And that's not a coincidence or a visual approximation: for a single z-score normalized feature, it's provable that the Hessian becomes exactly the identity matrix, ",[38,7985,7987,8005],{"className":7986},[41],[38,7988,7990],{"className":7989},[45],[47,7991,7992],{"xmlns":49},[51,7993,7994,8002],{},[54,7995,7996,7998,8000],{},[57,7997,3782],{},[362,7999,380],{},[560,8001,178],{},[61,8003,8004],{"encoding":63},"\\kappa = 1",[38,8006,8008,8026],{"className":8007,"ariaHidden":69},[68],[38,8009,8011,8014,8017,8020,8023],{"className":8010},[73],[38,8012],{"className":8013,"style":78},[77],[38,8015,3782],{"className":8016},[82,83],[38,8018],{"className":8019,"style":486},[485],[38,8021,380],{"className":8022},[490],[38,8024],{"className":8025,"style":486},[485],[38,8027,8029,8032],{"className":8028},[73],[38,8030],{"className":8031,"style":1454},[77],[38,8033,178],{"className":8034},[82],", a perfect circle, an exact result and not an approximation. With two real features, the result comes out nearly circular, and the \"nearly\" is literally the size of the correlation between them.",[87,8037,8039],{"id":8038},"wrapping-up","Wrapping up",[99,8041,8042,8052],{},[102,8043,8044],{},[105,8045,8046,8049],{},[108,8047,8048],{"align":110},"What I already knew",[108,8050,8051],{"align":110},"What this post settled",[129,8053,8054,8062,8127],{},[105,8055,8056,8059],{},[134,8057,8058],{"align":110},"Gradient descent finds the minimum on its own",[134,8060,8061],{"align":110},"It only finds it fast if the cost surface isn't a stretched-out corridor",[105,8063,8064,8095],{},[134,8065,8066,8094],{"align":110},[38,8067,8069,8082],{"className":8068},[41],[38,8070,8072],{"className":8071},[45],[47,8073,8074],{"xmlns":49},[51,8075,8076,8080],{},[54,8077,8078],{},[57,8079,59],{},[61,8081,64],{"encoding":63},[38,8083,8085],{"className":8084,"ariaHidden":69},[68],[38,8086,8088,8091],{"className":8087},[73],[38,8089],{"className":8090,"style":78},[77],[38,8092,59],{"className":8093,"style":84},[82,83]," is the most sensitive parameter I tune",[134,8096,8097,8098,8126],{"align":110},"The \"right\" ",[38,8099,8101,8114],{"className":8100},[41],[38,8102,8104],{"className":8103},[45],[47,8105,8106],{"xmlns":49},[51,8107,8108,8112],{},[54,8109,8110],{},[57,8111,59],{},[61,8113,64],{"encoding":63},[38,8115,8117],{"className":8116,"ariaHidden":69},[68],[38,8118,8120,8123],{"className":8119},[73],[38,8121],{"className":8122,"style":78},[77],[38,8124,59],{"className":8125,"style":84},[82,83]," depends entirely on the scale of the features, it's not a universal constant",[105,8128,8129,8132],{},[134,8130,8131],{"align":110},"The cost bowl is always convex, for linear regression",[134,8133,8134],{"align":110},"Convex isn't the same as well-conditioned: it can be a deep, round bowl, or a shallow, elongated canyon",[11,8136,8137],{},"Three takeaways:",[4014,8139,8140,8146,8211],{},[4017,8141,8142,8145],{},[33,8143,8144],{},"The relative scale of the features decides the shape of the cost bowl",", not the optimization algorithm itself.",[4017,8147,8148,8181,8182,8210],{},[33,8149,8150,8151,8180],{},"The condition number ",[38,8152,8154,8168],{"className":8153},[41],[38,8155,8157],{"className":8156},[45],[47,8158,8159],{"xmlns":49},[51,8160,8161,8165],{},[54,8162,8163],{},[57,8164,3782],{},[61,8166,8167],{"encoding":63},"\\kappa",[38,8169,8171],{"className":8170,"ariaHidden":69},[68],[38,8172,8174,8177],{"className":8173},[73],[38,8175],{"className":8176,"style":78},[77],[38,8178,3782],{"className":8179},[82,83]," is the metric that quantifies this",": close to 1 is a round, fast-to-descend bowl, much bigger than 1 is a canyon that any fixed ",[38,8183,8185,8198],{"className":8184},[41],[38,8186,8188],{"className":8187},[45],[47,8189,8190],{"xmlns":49},[51,8191,8192,8196],{},[54,8193,8194],{},[57,8195,59],{},[61,8197,64],{"encoding":63},[38,8199,8201],{"className":8200,"ariaHidden":69},[68],[38,8202,8204,8207],{"className":8203},[73],[38,8205],{"className":8206,"style":78},[77],[38,8208,59],{"className":8209,"style":84},[82,83]," takes forever to cross.",[4017,8212,8213,8216],{},[33,8214,8215],{},"Normalizing doesn't change the answer, it changes the path to it",": same minimum, orders of magnitude fewer iterations to get near it.",[87,8218,8220],{"id":8219},"practical-application","Practical application",[11,8222,8223,8224,8230,8231,8233,8234,2461,8237,8240,8241,8293,8294,8393],{},"Same real housing dataset from the previous posts (",[27,8225,8229],{"href":8226,"rel":8227},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fdenkuznetz\u002Fhousing-prices-regression",[8228],"nofollow","Housing Prices Regression, Kaggle","). We already saw in ",[27,8232,4012],{"href":29}," that the raw algorithm, with ",[22,8235,8236],{},"square_feet\u002F100",[22,8238,8239],{},"price\u002F1000",", converges slowly, needing ",[38,8242,8244,8263],{"className":8243},[41],[38,8245,8247],{"className":8246},[45],[47,8248,8249],{"xmlns":49},[51,8250,8251,8260],{},[54,8252,8253,8255,8257],{},[57,8254,59],{},[362,8256,380],{},[560,8258,8259],{},"0.01",[61,8261,8262],{"encoding":63},"\\alpha=0.01",[38,8264,8266,8284],{"className":8265,"ariaHidden":69},[68],[38,8267,8269,8272,8275,8278,8281],{"className":8268},[73],[38,8270],{"className":8271,"style":78},[77],[38,8273,59],{"className":8274,"style":84},[82,83],[38,8276],{"className":8277,"style":486},[485],[38,8279,380],{"className":8280},[490],[38,8282],{"className":8283,"style":486},[485],[38,8285,8287,8290],{"className":8286},[73],[38,8288],{"className":8289,"style":1454},[77],[38,8291,8259],{"className":8292},[82]," and 4000 iterations to reach ",[38,8295,8297,8333],{"className":8296},[41],[38,8298,8300],{"className":8299},[45],[47,8301,8302],{"xmlns":49},[51,8303,8304,8330],{},[54,8305,8306,8308,8310,8312,8314,8316,8318,8320,8323,8325,8328],{},[362,8307,371],{"stretchy":370},[57,8309,360],{},[362,8311,364],{"separator":69},[57,8313,367],{},[362,8315,377],{"stretchy":370},[362,8317,380],{},[362,8319,371],{"stretchy":370},[560,8321,8322],{},"116.5",[362,8324,364],{"separator":69},[560,8326,8327],{},"398.3",[362,8329,377],{"stretchy":370},[61,8331,8332],{"encoding":63},"(w,b) = (116.5, 398.3)",[38,8334,8336,8369],{"className":8335,"ariaHidden":69},[68],[38,8337,8339,8342,8345,8348,8351,8354,8357,8360,8363,8366],{"className":8338},[73],[38,8340],{"className":8341,"style":638},[77],[38,8343,371],{"className":8344},[474],[38,8346,360],{"className":8347,"style":449},[82,83],[38,8349,364],{"className":8350},[453],[38,8352],{"className":8353,"style":655},[485],[38,8355,367],{"className":8356},[82,83],[38,8358,377],{"className":8359},[481],[38,8361],{"className":8362,"style":486},[485],[38,8364,380],{"className":8365},[490],[38,8367],{"className":8368,"style":486},[485],[38,8370,8372,8375,8378,8381,8384,8387,8390],{"className":8371},[73],[38,8373],{"className":8374,"style":638},[77],[38,8376,371],{"className":8377},[474],[38,8379,8322],{"className":8380},[82],[38,8382,364],{"className":8383},[453],[38,8385],{"className":8386,"style":655},[485],[38,8388,8327],{"className":8389},[82],[38,8391,377],{"className":8392},[481],". Now I z-score normalized size before running it:",[1309,8395,8397],{"className":1311,"code":8396,"language":1313,"meta":1314,"style":1314},"size_norm, mu, sigma = zscore_normalize_features(square_feet)\n\nw, b, J_hist = gradient_descent(\n    size_norm, price,\n    w_in=0, b_in=0,\n    alpha=0.3, num_iters=200,\n    cost_function=compute_cost, gradient_function=compute_gradient)\n\nprint(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[22,8398,8399,8404,8410,8415,8420,8425,8431,8437,8442],{"__ignoreMap":1314},[38,8400,8401],{"class":1319,"line":1320},[38,8402,8403],{},"size_norm, mu, sigma = zscore_normalize_features(square_feet)\n",[38,8405,8406],{"class":1319,"line":1326},[38,8407,8409],{"emptyLinePlaceholder":8408},true,"\n",[38,8411,8412],{"class":1319,"line":6254},[38,8413,8414],{},"w, b, J_hist = gradient_descent(\n",[38,8416,8417],{"class":1319,"line":6260},[38,8418,8419],{},"    size_norm, price,\n",[38,8421,8422],{"class":1319,"line":6266},[38,8423,8424],{},"    w_in=0, b_in=0,\n",[38,8426,8428],{"class":1319,"line":8427},6,[38,8429,8430],{},"    alpha=0.3, num_iters=200,\n",[38,8432,8434],{"class":1319,"line":8433},7,[38,8435,8436],{},"    cost_function=compute_cost, gradient_function=compute_gradient)\n",[38,8438,8440],{"class":1319,"line":8439},8,[38,8441,8409],{"emptyLinePlaceholder":8408},[38,8443,8445],{"class":1319,"line":8444},9,[38,8446,8447],{},"print(f\"(w, b) found: ({w:.1f}, {b:.1f})\")\n",[1460,8449,8450],{},[11,8451,8452,3767,8455],{},[33,8453,8454],{},"Output:",[22,8456,8457],{},"(w, b) found: (88.1, 593.0)",[11,8459,8460,8461,8513,8514,8569,8570,8621],{},"The final cost matches the raw example almost exactly (",[38,8462,8464,8483],{"className":8463},[41],[38,8465,8467],{"className":8466},[45],[47,8468,8469],{"xmlns":49},[51,8470,8471,8480],{},[54,8472,8473,8475,8477],{},[57,8474,543],{},[362,8476,3156],{},[560,8478,8479],{},"5189.6",[61,8481,8482],{"encoding":63},"J \\approx 5189.6",[38,8484,8486,8504],{"className":8485,"ariaHidden":69},[68],[38,8487,8489,8492,8495,8498,8501],{"className":8488},[73],[38,8490],{"className":8491,"style":3023},[77],[38,8493,543],{"className":8494,"style":642},[82,83],[38,8496],{"className":8497,"style":486},[485],[38,8499,3156],{"className":8500},[490],[38,8502],{"className":8503,"style":486},[485],[38,8505,8507,8510],{"className":8506},[73],[38,8508],{"className":8509,"style":1454},[77],[38,8511,8479],{"className":8512},[82]," in both cases, it's the same minimum), except here I got there in ",[33,8515,8516,8517],{},"200 iterations at ",[38,8518,8520,8539],{"className":8519},[41],[38,8521,8523],{"className":8522},[45],[47,8524,8525],{"xmlns":49},[51,8526,8527,8536],{},[54,8528,8529,8531,8533],{},[57,8530,59],{},[362,8532,380],{},[560,8534,8535],{},"0.3",[61,8537,8538],{"encoding":63},"\\alpha = 0.3",[38,8540,8542,8560],{"className":8541,"ariaHidden":69},[68],[38,8543,8545,8548,8551,8554,8557],{"className":8544},[73],[38,8546],{"className":8547,"style":78},[77],[38,8549,59],{"className":8550,"style":84},[82,83],[38,8552],{"className":8553,"style":486},[485],[38,8555,380],{"className":8556},[490],[38,8558],{"className":8559,"style":486},[485],[38,8561,8563,8566],{"className":8562},[73],[38,8564],{"className":8565,"style":1454},[77],[38,8567,8535],{"className":8568},[82],", versus 4000 iterations at ",[38,8571,8573,8591],{"className":8572},[41],[38,8574,8576],{"className":8575},[45],[47,8577,8578],{"xmlns":49},[51,8579,8580,8588],{},[54,8581,8582,8584,8586],{},[57,8583,59],{},[362,8585,380],{},[560,8587,8259],{},[61,8589,8590],{"encoding":63},"\\alpha = 0.01",[38,8592,8594,8612],{"className":8593,"ariaHidden":69},[68],[38,8595,8597,8600,8603,8606,8609],{"className":8596},[73],[38,8598],{"className":8599,"style":78},[77],[38,8601,59],{"className":8602,"style":84},[82,83],[38,8604],{"className":8605,"style":486},[485],[38,8607,380],{"className":8608},[490],[38,8610],{"className":8611,"style":486},[485],[38,8613,8615,8618],{"className":8614},[73],[38,8616],{"className":8617,"style":1454},[77],[38,8619,8259],{"className":8620},[82]," before. Compare both live:",[8623,8624],"housing-gradient-descent-simulator",{},[8626,8627],"housing-scaled-gradient-descent-simulator",{},[11,8629,8630],{},"Click \"Rodar 100\" a few times on both and time it in your head: one of them already flattened out, the other is still climbing the slope. Same house, same final price, and the only difference between the two simulators is one extra line of code at the very start.",[8632,8633,8634],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":1314,"searchDepth":1326,"depth":1326,"links":8636},[8637,8638,8639,8640,8642,8644,8645,8646,8647,8648,8649],{"id":89,"depth":1326,"text":90},{"id":333,"depth":1326,"text":334},{"id":1303,"depth":1326,"text":1304},{"id":2560,"depth":1326,"text":8641},"Why exactly there: the 2\u002FL2\u002FL2\u002FL bound and the condition number",{"id":3977,"depth":1326,"text":8643},"A practical recipe for picking α\\alphaα",{"id":4458,"depth":1326,"text":4459},{"id":6282,"depth":1326,"text":6283},{"id":6753,"depth":1326,"text":6754},{"id":7905,"depth":1326,"text":7906},{"id":8038,"depth":1326,"text":8039},{"id":8219,"depth":1326,"text":8220},null,"2026-08-19","Why the exact same gradient descent that worked fine so far breaks the moment one feature has a wildly different scale from the others, and how a basic statistics trick fixes it for good.","md",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling","machine-learning-specialization",{"title":6,"description":8652},"published","en\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling",[8661,8662,8663],"normalization","feature-scaling","learning-rate","p5SOy1hrWlv6-1HdwG-_e2w7LgJ_D2PdPogStNjMZJQ",1787338984816]