[{"data":1,"prerenderedAt":2186},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls":3,"post-en-machine-learning-specialization-feature-scaling-pitfalls":4},"\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls",{"id":5,"title":6,"body":7,"cover":2171,"date":2172,"description":2173,"extension":2174,"meta":2175,"navigation":733,"order":747,"path":2176,"playlist":2177,"seo":2178,"status":2179,"stem":2180,"tags":2181,"__hash__":2185},"posts\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls.md","Just a Little Extra: Feature Scaling Pitfalls",{"type":8,"value":9,"toc":2157},"minimark",[10,20,25,109,112,135,145,181,199,203,335,350,359,398,412,444,448,483,489,554,558,561,576,692,704,709,763,773,870,874,877,882,1069,1337,1341,1426,1661,1665,1799,1866,1898,2001,2088,2092,2099,2153],[11,12,13,14,19],"p",{},"This post doesn't come from any required section of the lab, it's what was left over after I split off the essentials of normalization into ",[15,16,18],"a",{"href":17},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling","the main post",". These are the traps I'd only run into on a real project, outside the comfort of an educational notebook.",[21,22,24],"h2",{"id":23},"data-leakage-normalizing-before-splitting-train-and-test","Data leakage: normalizing before splitting train and test",[11,26,27,28,75,76,108],{},"The most common mistake of all. If I compute ",[29,30,33,56],"span",{"className":31},[32],"katex",[29,34,37],{"className":35},[36],"katex-mathml",[38,39,41],"math",{"xmlns":40},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[42,43,44,51],"semantics",{},[45,46,47],"mrow",{},[48,49,50],"mi",{},"μ",[52,53,55],"annotation",{"encoding":54},"application\u002Fx-tex","\\mu",[29,57,61],{"className":58,"ariaHidden":60},[59],"katex-html","true",[29,62,65,70],{"className":63},[64],"base",[29,66],{"className":67,"style":69},[68],"strut","height:0.625em;vertical-align:-0.1944em;",[29,71,50],{"className":72},[73,74],"mord","mathnormal"," and ",[29,77,79,94],{"className":78},[32],[29,80,82],{"className":81},[36],[38,83,84],{"xmlns":40},[42,85,86,91],{},[45,87,88],{},[48,89,90],{},"σ",[52,92,93],{"encoding":54},"\\sigma",[29,95,97],{"className":96,"ariaHidden":60},[59],[29,98,100,104],{"className":99},[64],[29,101],{"className":102,"style":103},[68],"height:0.4306em;",[29,105,90],{"className":106,"style":107},[73,74],"margin-right:0.0359em;"," using the whole dataset (train and test together) before splitting, information from the test set leaks into training, and my validation metric ends up looking better than it actually is, without me noticing.",[11,110,111],{},"I split the 100-house dataset into 80 train \u002F 20 test and compared size's mean computed both ways:",[113,114,119],"pre",{"className":115,"code":116,"language":117,"meta":118,"style":118},"language-python shiki shiki-themes github-light github-dark","mu_all = X_train.mean(axis=0)          # WRONG: uses train + test\nmu_train_only = X_train[i_tr].mean(axis=0)  # RIGHT: train only\n","python","",[120,121,122,129],"code",{"__ignoreMap":118},[29,123,126],{"class":124,"line":125},"line",1,[29,127,128],{},"mu_all = X_train.mean(axis=0)          # WRONG: uses train + test\n",[29,130,132],{"class":124,"line":131},2,[29,133,134],{},"mu_train_only = X_train[i_tr].mean(axis=0)  # RIGHT: train only\n",[136,137,138],"blockquote",{},[11,139,140,144],{},[141,142,143],"strong",{},"Output:"," mean with everything combined = 1413.7, train-only mean = 1385.3, a difference of 28.4.",[11,146,147,148,176,177,180],{},"That doesn't look like much on its own, so I planted a worse scenario: a giant 9000 sqft house showing up only in the test set. If I computed ",[29,149,151,164],{"className":150},[32],[29,152,154],{"className":153},[36],[38,155,156],{"xmlns":40},[42,157,158,162],{},[45,159,160],{},[48,161,50],{},[52,163,55],{"encoding":54},[29,165,167],{"className":166,"ariaHidden":60},[59],[29,168,170,173],{"className":169},[64],[29,171],{"className":172,"style":69},[68],[29,174,50],{"className":175},[73,74]," using the whole dataset (including that \"invisible\" house that should still be locked away in test), the mean would jump to 1488.8, a shift of over 100 units caused by ",[141,178,179],{},"one single row"," I shouldn't have even looked at yet.",[11,182,183,186,187,190,191,194,195,198],{},[141,184,185],{},"The rule I carry with me:"," ",[120,188,189],{},"fit"," only on training data, ",[120,192,193],{},"transform"," on everyone. This applies to anything that learns statistics from the data before using it (normalization, dimensionality reduction, feature selection), not just linear regression. scikit-learn's ",[120,196,197],{},"Pipeline"," (shown further down) exists basically to make this mistake impossible to make by accident.",[21,200,202],{"id":201},"constant-feature-division-by-zero","Constant feature: division by zero",[11,204,205,206,326,327,330,331,334],{},"If a column has ",[29,207,209,237],{"className":208},[32],[29,210,212],{"className":211},[36],[38,213,214],{"xmlns":40},[42,215,216,234],{},[45,217,218,226,230],{},[219,220,221,223],"msub",{},[48,222,90],{},[48,224,225],{},"j",[227,228,229],"mo",{},"=",[231,232,233],"mn",{},"0",[52,235,236],{"encoding":54},"\\sigma_j = 0",[29,238,240,316],{"className":239,"ariaHidden":60},[59],[29,241,243,247,304,309,313],{"className":242},[64],[29,244],{"className":245,"style":246},[68],"height:0.7167em;vertical-align:-0.2861em;",[29,248,250,253],{"className":249},[73],[29,251,90],{"className":252,"style":107},[73,74],[29,254,257],{"className":255},[256],"msupsub",[29,258,262,295],{"className":259},[260,261],"vlist-t","vlist-t2",[29,263,266,290],{"className":264},[265],"vlist-r",[29,267,271],{"className":268,"style":270},[269],"vlist","height:0.3117em;",[29,272,274,279],{"style":273},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[29,275],{"className":276,"style":278},[277],"pstrut","height:2.7em;",[29,280,286],{"className":281},[282,283,284,285],"sizing","reset-size6","size3","mtight",[29,287,225],{"className":288,"style":289},[73,74,285],"margin-right:0.0572em;",[29,291,294],{"className":292},[293],"vlist-s","​",[29,296,298],{"className":297},[265],[29,299,302],{"className":300,"style":301},[269],"height:0.2861em;",[29,303],{},[29,305],{"className":306,"style":308},[307],"mspace","margin-right:0.2778em;",[29,310,229],{"className":311},[312],"mrel",[29,314],{"className":315,"style":308},[307],[29,317,319,323],{"className":318},[64],[29,320],{"className":321,"style":322},[68],"height:0.6444em;",[29,324,233],{"className":325},[73]," (everyone shares the same value), the z-score formula divides by zero and produces ",[120,328,329],{},"inf"," or ",[120,332,333],{},"nan",".",[113,336,338],{"className":115,"code":337,"language":117,"meta":118,"style":118},"mu, sigma = 1413.71, 0.0   # sigma is zero, constant column\n(1500 - mu) \u002F sigma\n",[120,339,340,345],{"__ignoreMap":118},[29,341,342],{"class":124,"line":125},[29,343,344],{},"mu, sigma = 1413.71, 0.0   # sigma is zero, constant column\n",[29,346,347],{"class":124,"line":131},[29,348,349],{},"(1500 - mu) \u002F sigma\n",[136,351,352],{},[11,353,354,186,357],{},[141,355,356],{},"Output without protection:",[120,358,329],{},[113,360,362],{"className":115,"code":361,"language":117,"meta":118,"style":118},"def safe_zscore(X, eps=1e-12):\n    mu = X.mean(axis=0)\n    sigma = X.std(axis=0)\n    constant = sigma \u003C eps\n    sigma_safe = np.where(constant, 1.0, sigma)  # forces sigma=1 on constant columns\n    return (X - mu) \u002F sigma_safe, mu, sigma_safe\n",[120,363,364,369,374,380,386,392],{"__ignoreMap":118},[29,365,366],{"class":124,"line":125},[29,367,368],{},"def safe_zscore(X, eps=1e-12):\n",[29,370,371],{"class":124,"line":131},[29,372,373],{},"    mu = X.mean(axis=0)\n",[29,375,377],{"class":124,"line":376},3,[29,378,379],{},"    sigma = X.std(axis=0)\n",[29,381,383],{"class":124,"line":382},4,[29,384,385],{},"    constant = sigma \u003C eps\n",[29,387,389],{"class":124,"line":388},5,[29,390,391],{},"    sigma_safe = np.where(constant, 1.0, sigma)  # forces sigma=1 on constant columns\n",[29,393,395],{"class":124,"line":394},6,[29,396,397],{},"    return (X - mu) \u002F sigma_safe, mu, sigma_safe\n",[136,399,400],{},[11,401,402,186,405,408,409,411],{},[141,403,404],{},"Output with protection:",[120,406,407],{},"86.29"," (the constant column becomes zero instead of ",[120,410,329],{},", without breaking the rest of the calculation)",[11,413,414,415,443],{},"A constant column carries zero information for the model either way, so locking ",[29,416,418,431],{"className":417},[32],[29,419,421],{"className":420},[36],[38,422,423],{"xmlns":40},[42,424,425,429],{},[45,426,427],{},[48,428,90],{},[52,430,93],{"encoding":54},[29,432,434],{"className":433,"ariaHidden":60},[59],[29,435,437,440],{"className":436},[64],[29,438],{"className":439,"style":103},[68],[29,441,90],{"className":442,"style":107},[73,74]," at 1 is just a safe way to \"turn off\" that feature without crashing the whole program.",[21,445,447],{"id":446},"when-i-do-not-normalize","When I do NOT normalize",[449,450,451,458,473],"ul",{},[452,453,454,457],"li",{},[141,455,456],{},"Decision trees, Random Forest, Gradient Boosting:"," these models split by thresholds on each isolated feature. Scale doesn't matter to them at all, normalizing just wastes processing time.",[452,459,460,467,468,472],{},[141,461,462,463,466],{},"Normal equation \u002F ",[120,464,465],{},"LinearRegression",":"," solved in closed form, no iterative step involved. It doesn't ",[469,470,471],"em",{},"need"," normalization (though very extreme scales can still cause numerical precision issues).",[452,474,475,478,479,482],{},[141,476,477],{},"When the unit matters for interpretation:"," if I need to say \"each extra sqft is worth $X\", I have to denormalize the coefficients back (I did that in ",[15,480,481],{"href":17},"the main post's"," denormalization section).",[11,484,485,488],{},[141,486,487],{},"I always normalize"," for: gradient descent, SVM, k-NN, k-means, PCA, neural networks, and any model with L1\u002FL2 regularization (otherwise the penalty unfairly falls on small-scale features).",[11,490,491,492,521,522,550,551,553],{},"What about the target ",[29,493,495,509],{"className":494},[32],[29,496,498],{"className":497},[36],[38,499,500],{"xmlns":40},[42,501,502,507],{},[45,503,504],{},[48,505,506],{},"y",[52,508,506],{"encoding":54},[29,510,512],{"className":511,"ariaHidden":60},[59],[29,513,515,518],{"className":514},[64],[29,516],{"className":517,"style":69},[68],[29,519,506],{"className":520,"style":107},[73,74],"? Not required for linear regression with gradient descent, but it helps if ",[29,523,525,538],{"className":524},[32],[29,526,528],{"className":527},[36],[38,529,530],{"xmlns":40},[42,531,532,536],{},[45,533,534],{},[48,535,506],{},[52,537,506],{"encoding":54},[29,539,541],{"className":540,"ariaHidden":60},[59],[29,542,544,547],{"className":543},[64],[29,545],{"className":546,"style":69},[68],[29,548,506],{"className":549,"style":107},[73,74]," has a huge magnitude (avoids ",[120,552,329],{}," in the cost). If I normalize the target, I need to remember to denormalize the predictions before reporting any metric to anyone.",[21,555,557],{"id":556},"checking-against-an-exact-solution","Checking against an exact solution",[11,559,560],{},"Is my gradient descent implementation correct? I don't have to trust it blindly. Linear regression has a closed-form solution (the normal equation), so I can compare my iterative result against the exact answer.",[113,562,564],{"className":115,"code":563,"language":117,"meta":118,"style":118},"theta_exact = np.linalg.lstsq(X_aug, y_train, rcond=None)[0]\nw_exact, b_exact = theta_exact[:4], theta_exact[4]\n",[120,565,566,571],{"__ignoreMap":118},[29,567,568],{"class":124,"line":125},[29,569,570],{},"theta_exact = np.linalg.lstsq(X_aug, y_train, rcond=None)[0]\n",[29,572,573],{"class":124,"line":131},[29,574,575],{},"w_exact, b_exact = theta_exact[:4], theta_exact[4]\n",[11,577,578,579,581,582,636,637,691],{},"In ",[15,580,18],{"href":17}," I already showed that, with a single feature, denormalizing my gradient descent's result (",[29,583,585,605],{"className":584},[32],[29,586,588],{"className":587},[36],[38,589,590],{"xmlns":40},[42,591,592,602],{},[45,593,594,597,599],{},[48,595,596],{},"w",[227,598,229],{},[231,600,601],{},"0.267",[52,603,604],{"encoding":54},"w = 0.267",[29,606,608,627],{"className":607,"ariaHidden":60},[59],[29,609,611,614,618,621,624],{"className":610},[64],[29,612],{"className":613,"style":103},[68],[29,615,596],{"className":616,"style":617},[73,74],"margin-right:0.0269em;",[29,619],{"className":620,"style":308},[307],[29,622,229],{"className":623},[312],[29,625],{"className":626,"style":308},[307],[29,628,630,633],{"className":629},[64],[29,631],{"className":632,"style":322},[68],[29,634,601],{"className":635},[73],", ",[29,638,640,660],{"className":639},[32],[29,641,643],{"className":642},[36],[38,644,645],{"xmlns":40},[42,646,647,657],{},[45,648,649,652,654],{},[48,650,651],{},"b",[227,653,229],{},[231,655,656],{},"7.16",[52,658,659],{"encoding":54},"b = 7.16",[29,661,663,682],{"className":662,"ariaHidden":60},[59],[29,664,666,670,673,676,679],{"className":665},[64],[29,667],{"className":668,"style":669},[68],"height:0.6944em;",[29,671,651],{"className":672},[73,74],[29,674],{"className":675,"style":308},[307],[29,677,229],{"className":678},[312],[29,680],{"className":681,"style":308},[307],[29,683,685,688],{"className":684},[64],[29,686],{"className":687,"style":322},[68],[29,689,656],{"className":690},[73],") matches digit for digit what the closed-form formula gives directly in raw space. I ran the same check with all 4 real features: the theoretical minimum cost from the normal equation and my gradient descent's cost after 1000 iterations end up a negligible distance apart. Both calculations agree, which gives me a lot more confidence that I didn't write a silent bug into the implementation.",[136,693,694],{},[11,695,696,697,699,700,703],{},"I don't have scikit-learn installed in this environment to run a third comparison method (",[120,698,465],{},"\u002F",[120,701,702],{},"SGDRegressor","), but the principle is the same as in the original notebook: running the same calculation through independent methods and checking they all agree is the cheapest way to catch an implementation bug before trusting a number.",[11,705,706,707,466],{},"The idiomatic way to do all of this in production is a ",[120,708,197],{},[113,710,712],{"className":115,"code":711,"language":117,"meta":118,"style":118},"from sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import SGDRegressor\nfrom sklearn.pipeline import make_pipeline\n\npipe = make_pipeline(\n    StandardScaler(),                                   # equivalent to my zscore_normalize_features\n    SGDRegressor(max_iter=2000, tol=1e-6, eta0=0.1),\n)\npipe.fit(X_train, y_train)\n",[120,713,714,719,724,729,735,740,745,751,757],{"__ignoreMap":118},[29,715,716],{"class":124,"line":125},[29,717,718],{},"from sklearn.preprocessing import StandardScaler\n",[29,720,721],{"class":124,"line":131},[29,722,723],{},"from sklearn.linear_model import SGDRegressor\n",[29,725,726],{"class":124,"line":376},[29,727,728],{},"from sklearn.pipeline import make_pipeline\n",[29,730,731],{"class":124,"line":382},[29,732,734],{"emptyLinePlaceholder":733},true,"\n",[29,736,737],{"class":124,"line":388},[29,738,739],{},"pipe = make_pipeline(\n",[29,741,742],{"class":124,"line":394},[29,743,744],{},"    StandardScaler(),                                   # equivalent to my zscore_normalize_features\n",[29,746,748],{"class":124,"line":747},7,[29,749,750],{},"    SGDRegressor(max_iter=2000, tol=1e-6, eta0=0.1),\n",[29,752,754],{"class":124,"line":753},8,[29,755,756],{},")\n",[29,758,760],{"class":124,"line":759},9,[29,761,762],{},"pipe.fit(X_train, y_train)\n",[11,764,765,766,768,769,772],{},"The ",[120,767,197],{}," guarantees the ",[120,770,771],{},"StandardScaler"," only gets fit on training data in each validation fold, eliminating data leakage automatically. It's the correct way to do this in practice, instead of normalizing by hand like I did in this post to understand the mechanics underneath.",[11,774,775,781,782,869],{},[141,776,777,778,780],{},"Why learn gradient descent, if ",[120,779,465],{}," solves it exactly?"," Because the normal equation only exists for linear models, and costs ",[29,783,785,816],{"className":784},[32],[29,786,788],{"className":787},[36],[38,789,790],{"xmlns":40},[42,791,792,813],{},[45,793,794,797,801,810],{},[48,795,796],{},"O",[227,798,800],{"stretchy":799},"false","(",[802,803,804,807],"msup",{},[48,805,806],{},"n",[231,808,809],{},"3",[227,811,812],{"stretchy":799},")",[52,814,815],{"encoding":54},"O(n^3)",[29,817,819],{"className":818,"ariaHidden":60},[59],[29,820,822,826,830,834,865],{"className":821},[64],[29,823],{"className":824,"style":825},[68],"height:1.0641em;vertical-align:-0.25em;",[29,827,796],{"className":828,"style":829},[73,74],"margin-right:0.0278em;",[29,831,800],{"className":832},[833],"mopen",[29,835,837,840],{"className":836},[73],[29,838,806],{"className":839},[73,74],[29,841,843],{"className":842},[256],[29,844,846],{"className":845},[260],[29,847,849],{"className":848},[265],[29,850,853],{"className":851,"style":852},[269],"height:0.8141em;",[29,854,856,859],{"style":855},"top:-3.063em;margin-right:0.05em;",[29,857],{"className":858,"style":278},[277],[29,860,862],{"className":861},[282,283,284,285],[29,863,809],{"className":864},[73,285],[29,866,812],{"className":867},[868],"mclose"," to invert the matrix, infeasible with millions of features. Gradient descent is the algorithm that actually trains neural networks, logistic regression, SVMs, and pretty much everything else. Linear regression is just the playground where you can watch the mechanics work with an exact answer to check against.",[21,871,873],{"id":872},"exercises","Exercises",[11,875,876],{},"Try them before opening the answer.",[878,879,881],"h3",{"id":880},"exercise-1-sensitivity-to-units","Exercise 1: sensitivity to units",[11,883,884,885,888,889,1016,1017,334],{},"Convert ",[120,886,887],{},"size"," from sqft to square meters (1 sqft = 0.092903 m²) and run gradient descent on the raw data with ",[29,890,892,928],{"className":891},[32],[29,893,895],{"className":894},[36],[38,896,897],{"xmlns":40},[42,898,899,925],{},[45,900,901,904,906,909,912],{},[48,902,903],{},"α",[227,905,229],{},[231,907,908],{},"9",[227,910,911],{},"×",[802,913,914,917],{},[231,915,916],{},"10",[45,918,919,922],{},[227,920,921],{},"−",[231,923,924],{},"7",[52,926,927],{"encoding":54},"\\alpha = 9\\times10^{-7}",[29,929,931,950,971],{"className":930,"ariaHidden":60},[59],[29,932,934,937,941,944,947],{"className":933},[64],[29,935],{"className":936,"style":103},[68],[29,938,903],{"className":939,"style":940},[73,74],"margin-right:0.0037em;",[29,942],{"className":943,"style":308},[307],[29,945,229],{"className":946},[312],[29,948],{"className":949,"style":308},[307],[29,951,953,957,960,964,968],{"className":952},[64],[29,954],{"className":955,"style":956},[68],"height:0.7278em;vertical-align:-0.0833em;",[29,958,908],{"className":959},[73],[29,961],{"className":962,"style":963},[307],"margin-right:0.2222em;",[29,965,911],{"className":966},[967],"mbin",[29,969],{"className":970,"style":963},[307],[29,972,974,977,981],{"className":973},[64],[29,975],{"className":976,"style":852},[68],[29,978,980],{"className":979},[73],"1",[29,982,984,987],{"className":983},[73],[29,985,233],{"className":986},[73],[29,988,990],{"className":989},[256],[29,991,993],{"className":992},[260],[29,994,996],{"className":995},[265],[29,997,999],{"className":998,"style":852},[269],[29,1000,1001,1004],{"style":855},[29,1002],{"className":1003,"style":278},[277],[29,1005,1007],{"className":1006},[282,283,284,285],[29,1008,1010,1013],{"className":1009},[73,285],[29,1011,921],{"className":1012},[73,285],[29,1014,924],{"className":1015},[73,285],". What happens? Then normalize and run with ",[29,1018,1020,1039],{"className":1019},[32],[29,1021,1023],{"className":1022},[36],[38,1024,1025],{"xmlns":40},[42,1026,1027,1036],{},[45,1028,1029,1031,1033],{},[48,1030,903],{},[227,1032,229],{},[231,1034,1035],{},"0.1",[52,1037,1038],{"encoding":54},"\\alpha = 0.1",[29,1040,1042,1060],{"className":1041,"ariaHidden":60},[59],[29,1043,1045,1048,1051,1054,1057],{"className":1044},[64],[29,1046],{"className":1047,"style":103},[68],[29,1049,903],{"className":1050,"style":940},[73,74],[29,1052],{"className":1053,"style":308},[307],[29,1055,229],{"className":1056},[312],[29,1058],{"className":1059,"style":308},[307],[29,1061,1063,1066],{"className":1062},[64],[29,1064],{"className":1065,"style":322},[68],[29,1067,1035],{"className":1068},[73],[1070,1071,1072,1078],"details",{},[1073,1074,1075],"summary",{},[651,1076,1077],{},"Answer",[11,1079,1080,1081,1111,1112,1141,1142,1170,1171,1174,1175,1336],{},"On the raw data, changing the unit shifts size's scale by roughly 10x, which changes ",[29,1082,1084,1098],{"className":1083},[32],[29,1085,1087],{"className":1086},[36],[38,1088,1089],{"xmlns":40},[42,1090,1091,1096],{},[45,1092,1093],{},[48,1094,1095],{},"L",[52,1097,1095],{"encoding":54},[29,1099,1101],{"className":1100,"ariaHidden":60},[59],[29,1102,1104,1108],{"className":1103},[64],[29,1105],{"className":1106,"style":1107},[68],"height:0.6833em;",[29,1109,1095],{"className":1110},[73,74]," and therefore the critical ",[29,1113,1115,1129],{"className":1114},[32],[29,1116,1118],{"className":1117},[36],[38,1119,1120],{"xmlns":40},[42,1121,1122,1126],{},[45,1123,1124],{},[48,1125,903],{},[52,1127,1128],{"encoding":54},"\\alpha",[29,1130,1132],{"className":1131,"ariaHidden":60},[59],[29,1133,1135,1138],{"className":1134},[64],[29,1136],{"className":1137,"style":103},[68],[29,1139,903],{"className":1140,"style":940},[73,74],". An ",[29,1143,1145,1158],{"className":1144},[32],[29,1146,1148],{"className":1147},[36],[38,1149,1150],{"xmlns":40},[42,1151,1152,1156],{},[45,1153,1154],{},[48,1155,903],{},[52,1157,1128],{"encoding":54},[29,1159,1161],{"className":1160,"ariaHidden":60},[59],[29,1162,1164,1167],{"className":1163},[64],[29,1165],{"className":1166,"style":103},[68],[29,1168,903],{"className":1169,"style":940},[73,74]," that worked before might start diverging or become painfully slow. On the normalized data ",[141,1172,1173],{},"nothing changes",": z-score is invariant to a linear unit change, because ",[29,1176,1178,1232],{"className":1177},[32],[29,1179,1181],{"className":1180},[36],[38,1182,1183],{"xmlns":40},[42,1184,1185,1229],{},[45,1186,1187,1189,1191,1194,1196,1198,1200,1202,1205,1207,1209,1211,1213,1215,1217,1219,1221,1223,1225,1227],{},[227,1188,800],{"stretchy":799},[48,1190,15],{},[48,1192,1193],{},"x",[227,1195,921],{},[48,1197,15],{},[48,1199,50],{},[227,1201,812],{"stretchy":799},[48,1203,699],{"mathvariant":1204},"normal",[227,1206,800],{"stretchy":799},[48,1208,15],{},[48,1210,90],{},[227,1212,812],{"stretchy":799},[227,1214,229],{},[227,1216,800],{"stretchy":799},[48,1218,1193],{},[227,1220,921],{},[48,1222,50],{},[227,1224,812],{"stretchy":799},[48,1226,699],{"mathvariant":1204},[48,1228,90],{},[52,1230,1231],{"encoding":54},"(ax - a\\mu)\u002F(a\\sigma) = (x-\\mu)\u002F\\sigma",[29,1233,1235,1260,1297,1318],{"className":1234,"ariaHidden":60},[59],[29,1236,1238,1242,1245,1248,1251,1254,1257],{"className":1237},[64],[29,1239],{"className":1240,"style":1241},[68],"height:1em;vertical-align:-0.25em;",[29,1243,800],{"className":1244},[833],[29,1246,15],{"className":1247},[73,74],[29,1249,1193],{"className":1250},[73,74],[29,1252],{"className":1253,"style":963},[307],[29,1255,921],{"className":1256},[967],[29,1258],{"className":1259,"style":963},[307],[29,1261,1263,1266,1269,1272,1275,1278,1281,1285,1288,1291,1294],{"className":1262},[64],[29,1264],{"className":1265,"style":1241},[68],[29,1267,15],{"className":1268},[73,74],[29,1270,50],{"className":1271},[73,74],[29,1273,812],{"className":1274},[868],[29,1276,699],{"className":1277},[73],[29,1279,800],{"className":1280},[833],[29,1282,1284],{"className":1283,"style":107},[73,74],"aσ",[29,1286,812],{"className":1287},[868],[29,1289],{"className":1290,"style":308},[307],[29,1292,229],{"className":1293},[312],[29,1295],{"className":1296,"style":308},[307],[29,1298,1300,1303,1306,1309,1312,1315],{"className":1299},[64],[29,1301],{"className":1302,"style":1241},[68],[29,1304,800],{"className":1305},[833],[29,1307,1193],{"className":1308},[73,74],[29,1310],{"className":1311,"style":963},[307],[29,1313,921],{"className":1314},[967],[29,1316],{"className":1317,"style":963},[307],[29,1319,1321,1324,1327,1330,1333],{"className":1320},[64],[29,1322],{"className":1323,"style":1241},[68],[29,1325,50],{"className":1326},[73,74],[29,1328,812],{"className":1329},[868],[29,1331,699],{"className":1332},[73],[29,1334,90],{"className":1335,"style":107},[73,74],". That's the strongest argument in favor of normalizing: the result stops depending on an arbitrary choice of unit.",[878,1338,1340],{"id":1339},"exercise-2-min-max-vs-z-score","Exercise 2: min-max vs z-score",[11,1342,1343,1344,1394,1395,1425],{},"Normalize with min-max instead of z-score and run gradient descent with ",[29,1345,1347,1364],{"className":1346},[32],[29,1348,1350],{"className":1349},[36],[38,1351,1352],{"xmlns":40},[42,1353,1354,1362],{},[45,1355,1356,1358,1360],{},[48,1357,903],{},[227,1359,229],{},[231,1361,1035],{},[52,1363,1038],{"encoding":54},[29,1365,1367,1385],{"className":1366,"ariaHidden":60},[59],[29,1368,1370,1373,1376,1379,1382],{"className":1369},[64],[29,1371],{"className":1372,"style":103},[68],[29,1374,903],{"className":1375,"style":940},[73,74],[29,1377],{"className":1378,"style":308},[307],[29,1380,229],{"className":1381},[312],[29,1383],{"className":1384,"style":308},[307],[29,1386,1388,1391],{"className":1387},[64],[29,1389],{"className":1390,"style":322},[68],[29,1392,1035],{"className":1393},[73],", 1000 iterations. Compare the final cost and ",[29,1396,1398,1413],{"className":1397},[32],[29,1399,1401],{"className":1400},[36],[38,1402,1403],{"xmlns":40},[42,1404,1405,1410],{},[45,1406,1407],{},[48,1408,1409],{},"κ",[52,1411,1412],{"encoding":54},"\\kappa",[29,1414,1416],{"className":1415,"ariaHidden":60},[59],[29,1417,1419,1422],{"className":1418},[64],[29,1420],{"className":1421,"style":103},[68],[29,1423,1409],{"className":1424},[73,74],". Then insert an outlier (a 20000 sqft house) and repeat.",[1070,1427,1428,1432,1651],{},[1073,1429,1430],{},[651,1431,1077],{},[11,1433,1434,1435,636,1492,1520,1521,1592,1593,1621,1622,1650],{},"Without the outlier, min-max works almost as well as z-score (features in ",[29,1436,1438,1463],{"className":1437},[32],[29,1439,1441],{"className":1440},[36],[38,1442,1443],{"xmlns":40},[42,1444,1445,1460],{},[45,1446,1447,1450,1452,1455,1457],{},[227,1448,1449],{"stretchy":799},"[",[231,1451,233],{},[227,1453,1454],{"separator":60},",",[231,1456,980],{},[227,1458,1459],{"stretchy":799},"]",[52,1461,1462],{"encoding":54},"[0,1]",[29,1464,1466],{"className":1465,"ariaHidden":60},[59],[29,1467,1469,1472,1475,1478,1482,1486,1489],{"className":1468},[64],[29,1470],{"className":1471,"style":1241},[68],[29,1473,1449],{"className":1474},[833],[29,1476,233],{"className":1477},[73],[29,1479,1454],{"className":1480},[1481],"mpunct",[29,1483],{"className":1484,"style":1485},[307],"margin-right:0.1667em;",[29,1487,980],{"className":1488},[73],[29,1490,1459],{"className":1491},[868],[29,1493,1495,1508],{"className":1494},[32],[29,1496,1498],{"className":1497},[36],[38,1499,1500],{"xmlns":40},[42,1501,1502,1506],{},[45,1503,1504],{},[48,1505,1409],{},[52,1507,1412],{"encoding":54},[29,1509,1511],{"className":1510,"ariaHidden":60},[59],[29,1512,1514,1517],{"className":1513},[64],[29,1515],{"className":1516,"style":103},[68],[29,1518,1409],{"className":1519},[73,74]," much smaller than raw). But notice min-max doesn't center at zero, which leaves a residual correlation between the ",[29,1522,1524,1542],{"className":1523},[32],[29,1525,1527],{"className":1526},[36],[38,1528,1529],{"xmlns":40},[42,1530,1531,1539],{},[45,1532,1533],{},[219,1534,1535,1537],{},[48,1536,596],{},[48,1538,225],{},[52,1540,1541],{"encoding":54},"w_j",[29,1543,1545],{"className":1544,"ariaHidden":60},[59],[29,1546,1548,1551],{"className":1547},[64],[29,1549],{"className":1550,"style":246},[68],[29,1552,1554,1557],{"className":1553},[73],[29,1555,596],{"className":1556,"style":617},[73,74],[29,1558,1560],{"className":1559},[256],[29,1561,1563,1584],{"className":1562},[260,261],[29,1564,1566,1581],{"className":1565},[265],[29,1567,1569],{"className":1568,"style":270},[269],[29,1570,1572,1575],{"style":1571},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[29,1573],{"className":1574,"style":278},[277],[29,1576,1578],{"className":1577},[282,283,284,285],[29,1579,225],{"className":1580,"style":289},[73,74,285],[29,1582,294],{"className":1583},[293],[29,1585,1587],{"className":1586},[265],[29,1588,1590],{"className":1589,"style":301},[269],[29,1591],{},"'s and ",[29,1594,1596,1609],{"className":1595},[32],[29,1597,1599],{"className":1598},[36],[38,1600,1601],{"xmlns":40},[42,1602,1603,1607],{},[45,1604,1605],{},[48,1606,651],{},[52,1608,651],{"encoding":54},[29,1610,1612],{"className":1611,"ariaHidden":60},[59],[29,1613,1615,1618],{"className":1614},[64],[29,1616],{"className":1617,"style":669},[68],[29,1619,651],{"className":1620},[73,74],", and ",[29,1623,1625,1638],{"className":1624},[32],[29,1626,1628],{"className":1627},[36],[38,1629,1630],{"xmlns":40},[42,1631,1632,1636],{},[45,1633,1634],{},[48,1635,1409],{},[52,1637,1412],{"encoding":54},[29,1639,1641],{"className":1640,"ariaHidden":60},[59],[29,1642,1644,1647],{"className":1643},[64],[29,1645],{"className":1646,"style":103},[68],[29,1648,1409],{"className":1649},[73,74]," tends to come out worse than z-score's.",[11,1652,1653,1654,1656,1657,1660],{},"With the outlier, min-max collapses: ",[120,1655,887],{}," becomes nearly 0 for every normal house and 1 only for the outlier, destroying the feature's resolution for the real houses. Z-score also suffers (mean and std aren't robust to outliers), but much less. A genuinely robust fix: ",[120,1658,1659],{},"sklearn.preprocessing.RobustScaler"," (uses median and IQR instead of mean and std).",[878,1662,1664],{"id":1663},"exercise-3-implement-a-stopping-criterion","Exercise 3: implement a stopping criterion",[11,1666,1667,1668,1671,1672,1675,1676,334],{},"My ",[120,1669,1670],{},"gradient_descent"," always runs the full ",[120,1673,1674],{},"num_iters"," iterations. Add early stopping when ",[29,1677,1679,1715],{"className":1678},[32],[29,1680,1682],{"className":1681},[36],[38,1683,1684],{"xmlns":40},[42,1685,1686,1712],{},[45,1687,1688,1691,1694,1697,1704,1707],{},[48,1689,1690],{"mathvariant":1204},"∥",[48,1692,1693],{"mathvariant":1204},"∇",[48,1695,1696],{},"J",[219,1698,1699,1701],{},[48,1700,1690],{"mathvariant":1204},[48,1702,1703],{"mathvariant":1204},"∞",[227,1705,1706],{},"\u003C",[1708,1709,1711],"mtext",{"mathvariant":1710},"monospace","tol",[52,1713,1714],{"encoding":54},"\\|\\nabla J\\|_\\infty \u003C \\texttt{tol}",[29,1716,1718,1784],{"className":1717,"ariaHidden":60},[59],[29,1719,1721,1724,1728,1732,1775,1778,1781],{"className":1720},[64],[29,1722],{"className":1723,"style":1241},[68],[29,1725,1727],{"className":1726},[73],"∥∇",[29,1729,1696],{"className":1730,"style":1731},[73,74],"margin-right:0.0962em;",[29,1733,1735,1738],{"className":1734},[73],[29,1736,1690],{"className":1737},[73],[29,1739,1741],{"className":1740},[256],[29,1742,1744,1766],{"className":1743},[260,261],[29,1745,1747,1763],{"className":1746},[265],[29,1748,1751],{"className":1749,"style":1750},[269],"height:0.1514em;",[29,1752,1754,1757],{"style":1753},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[29,1755],{"className":1756,"style":278},[277],[29,1758,1760],{"className":1759},[282,283,284,285],[29,1761,1703],{"className":1762},[73,285],[29,1764,294],{"className":1765},[293],[29,1767,1769],{"className":1768},[265],[29,1770,1773],{"className":1771,"style":1772},[269],"height:0.15em;",[29,1774],{},[29,1776],{"className":1777,"style":308},[307],[29,1779,1706],{"className":1780},[312],[29,1782],{"className":1783,"style":308},[307],[29,1785,1787,1791],{"className":1786},[64],[29,1788],{"className":1789,"style":1790},[68],"height:0.6111em;",[29,1792,1795],{"className":1793},[73,1794],"text",[29,1796,1711],{"className":1797},[73,1798],"texttt",[1070,1800,1801,1805,1830],{},[1073,1802,1803],{},[651,1804,1077],{},[113,1806,1808],{"className":115,"code":1807,"language":117,"meta":118,"style":118},"grad_norm = max(np.max(np.abs(dj_dw)), abs(dj_db))\nif grad_norm \u003C tol:\n    print(f\"Converged at iteration {i}: |grad|_inf = {grad_norm:.2e}\")\n    break\n",[120,1809,1810,1815,1820,1825],{"__ignoreMap":118},[29,1811,1812],{"class":124,"line":125},[29,1813,1814],{},"grad_norm = max(np.max(np.abs(dj_dw)), abs(dj_db))\n",[29,1816,1817],{"class":124,"line":131},[29,1818,1819],{},"if grad_norm \u003C tol:\n",[29,1821,1822],{"class":124,"line":376},[29,1823,1824],{},"    print(f\"Converged at iteration {i}: |grad|_inf = {grad_norm:.2e}\")\n",[29,1826,1827],{"class":124,"line":382},[29,1828,1829],{},"    break\n",[11,1831,1832,1833,1836,1837,1865],{},"A cost-improvement criterion (",[120,1834,1835],{},"abs(J_prev - J_new) \u002F max(abs(J_prev), 1e-12) \u003C tol",") is also common, but it carries a risk: with too small an ",[29,1838,1840,1853],{"className":1839},[32],[29,1841,1843],{"className":1842},[36],[38,1844,1845],{"xmlns":40},[42,1846,1847,1851],{},[45,1848,1849],{},[48,1850,903],{},[52,1852,1128],{"encoding":54},[29,1854,1856],{"className":1855,"ariaHidden":60},[59],[29,1857,1859,1862],{"className":1858},[64],[29,1860],{"className":1861,"style":103},[68],[29,1863,903],{"className":1864,"style":940},[73,74],", the cost also improves very little per iteration, and the algorithm stops thinking it converged without actually having converged. The gradient-based criterion is more reliable.",[878,1867,1869,1870],{"id":1868},"exercise-4-predict-the-critical-αalphaα","Exercise 4: predict the critical ",[29,1871,1873,1886],{"className":1872},[32],[29,1874,1876],{"className":1875},[36],[38,1877,1878],{"xmlns":40},[42,1879,1880,1884],{},[45,1881,1882],{},[48,1883,903],{},[52,1885,1128],{"encoding":54},[29,1887,1889],{"className":1888,"ariaHidden":60},[59],[29,1890,1892,1895],{"className":1891},[64],[29,1893],{"className":1894,"style":103},[68],[29,1896,903],{"className":1897,"style":940},[73,74],[11,1899,1900,1901,1929,1930,75,1965,2000],{},"Without running gradient descent, compute the critical ",[29,1902,1904,1917],{"className":1903},[32],[29,1905,1907],{"className":1906},[36],[38,1908,1909],{"xmlns":40},[42,1910,1911,1915],{},[45,1912,1913],{},[48,1914,903],{},[52,1916,1128],{"encoding":54},[29,1918,1920],{"className":1919,"ariaHidden":60},[59],[29,1921,1923,1926],{"className":1922},[64],[29,1924],{"className":1925,"style":103},[68],[29,1927,903],{"className":1928,"style":940},[73,74]," for min-max normalized data. Then confirm it empirically by running at ",[29,1931,1933,1950],{"className":1932},[32],[29,1934,1936],{"className":1935},[36],[38,1937,1938],{"xmlns":40},[42,1939,1940,1947],{},[45,1941,1942,1945],{},[231,1943,1944],{},"0.9",[227,1946,911],{},[52,1948,1949],{"encoding":54},"0.9\\times",[29,1951,1953],{"className":1952,"ariaHidden":60},[59],[29,1954,1956,1959,1962],{"className":1955},[64],[29,1957],{"className":1958,"style":956},[68],[29,1960,1944],{"className":1961},[73],[29,1963,911],{"className":1964},[73],[29,1966,1968,1985],{"className":1967},[32],[29,1969,1971],{"className":1970},[36],[38,1972,1973],{"xmlns":40},[42,1974,1975,1982],{},[45,1976,1977,1980],{},[231,1978,1979],{},"1.1",[227,1981,911],{},[52,1983,1984],{"encoding":54},"1.1\\times",[29,1986,1988],{"className":1987,"ariaHidden":60},[59],[29,1989,1991,1994,1997],{"className":1990},[64],[29,1992],{"className":1993,"style":956},[68],[29,1995,1979],{"className":1996},[73],[29,1998,911],{"className":1999},[73]," that value.",[1070,2002,2003,2007,2027],{},[1073,2004,2005],{},[651,2006,1077],{},[113,2008,2010],{"className":115,"code":2009,"language":117,"meta":118,"style":118},"X_mm = minmax_scaling(X_train)\nHa = (np.column_stack([X_mm, np.ones(m)]).T @ np.column_stack([X_mm, np.ones(m)])) \u002F m\na_crit = 2 \u002F np.linalg.eigvalsh(Ha)[-1]\n",[120,2011,2012,2017,2022],{"__ignoreMap":118},[29,2013,2014],{"class":124,"line":125},[29,2015,2016],{},"X_mm = minmax_scaling(X_train)\n",[29,2018,2019],{"class":124,"line":131},[29,2020,2021],{},"Ha = (np.column_stack([X_mm, np.ones(m)]).T @ np.column_stack([X_mm, np.ones(m)])) \u002F m\n",[29,2023,2024],{"class":124,"line":376},[29,2025,2026],{},"a_crit = 2 \u002F np.linalg.eigvalsh(Ha)[-1]\n",[11,2028,2029,2030,2058,2059,2087],{},"This works because ",[29,2031,2033,2046],{"className":2032},[32],[29,2034,2036],{"className":2035},[36],[38,2037,2038],{"xmlns":40},[42,2039,2040,2044],{},[45,2041,2042],{},[48,2043,1696],{},[52,2045,1696],{"encoding":54},[29,2047,2049],{"className":2048,"ariaHidden":60},[59],[29,2050,2052,2055],{"className":2051},[64],[29,2053],{"className":2054,"style":1107},[68],[29,2056,1696],{"className":2057,"style":1731},[73,74]," is exactly quadratic, which makes the Hessian constant at every point. For non-linear models (neural networks) the curvature ",[29,2060,2062,2075],{"className":2061},[32],[29,2063,2065],{"className":2064},[36],[38,2066,2067],{"xmlns":40},[42,2068,2069,2073],{},[45,2070,2071],{},[48,2072,1095],{},[52,2074,1095],{"encoding":54},[29,2076,2078],{"className":2077,"ariaHidden":60},[59],[29,2079,2081,2084],{"className":2080},[64],[29,2082],{"className":2083,"style":1107},[68],[29,2085,1095],{"className":2086},[73,74]," changes at every point, and this calculation only holds locally, which is exactly why adaptive optimizers like Adam exist.",[878,2089,2091],{"id":2090},"exercise-5-a-new-feature","Exercise 5: a new feature",[11,2093,2094,2095,2098],{},"Add the feature ",[120,2096,2097],{},"size_per_bedroom = size \u002F bedrooms",", normalize, and train. Does the error improve?",[1070,2100,2101,2105],{},[1073,2102,2103],{},[651,2104,1077],{},[11,2106,2107,2108,2112,2113,186,2116,2144,2145,2148,2149,2152],{},"Feature engineering is ",[15,2109,2111],{"href":2110},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab04-feature-engineering","the topic of the next lab",". The thing to watch for here: derived features tend to end up strongly correlated with the originals, which ",[141,2114,2115],{},"increases",[29,2117,2119,2132],{"className":2118},[32],[29,2120,2122],{"className":2121},[36],[38,2123,2124],{"xmlns":40},[42,2125,2126,2130],{},[45,2127,2128],{},[48,2129,1409],{},[52,2131,1412],{"encoding":54},[29,2133,2135],{"className":2134,"ariaHidden":60},[59],[29,2136,2138,2141],{"className":2137},[64],[29,2139],{"className":2140,"style":103},[68],[29,2142,1409],{"className":2143},[73,74]," and can slow convergence even after normalizing. Normalization fixes a difference in ",[141,2146,2147],{},"scale",", it doesn't fix ",[141,2150,2151],{},"collinearity",", that's what regularization (Ridge) and PCA are for.",[2154,2155,2156],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":118,"searchDepth":131,"depth":131,"links":2158},[2159,2160,2161,2162,2163],{"id":23,"depth":131,"text":24},{"id":201,"depth":131,"text":202},{"id":446,"depth":131,"text":447},{"id":556,"depth":131,"text":557},{"id":872,"depth":131,"text":873,"children":2164},[2165,2166,2167,2168,2170],{"id":880,"depth":376,"text":881},{"id":1339,"depth":376,"text":1340},{"id":1663,"depth":376,"text":1664},{"id":1868,"depth":376,"text":2169},"Exercise 4: predict the critical α\\alphaα",{"id":2090,"depth":376,"text":2091},null,"2026-08-19","Data leakage, constant features, when not to normalize, and how I checked my implementation against an exact solution. The traps that only show up once a model leaves the notebook and heads to production.","md",{},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls","machine-learning-specialization",{"title":6,"description":2173},"draft","en\u002Fplaylists\u002Fmachine-learning-specialization\u002Ffeature-scaling-pitfalls",[2182,2183,2184],"normalization","data-leakage","scikit-learn","62xf2T9qjczhnW43tS5GI-E_uzdOpJOUKrYEyResSvg",1787338985338]