[{"data":1,"prerenderedAt":1260},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation":3,"post-en-pattern-recognition-pipeline-cross-validation":4},"\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation",{"id":5,"title":6,"body":7,"cover":1245,"date":1246,"description":1247,"extension":1248,"meta":1249,"navigation":171,"order":124,"path":1250,"playlist":1251,"seo":1252,"status":1253,"stem":1254,"tags":1255,"__hash__":1259},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation.md","Pipeline, Cross-Validation, and GridSearch: the Real Workflow",{"type":8,"value":9,"toc":1234},"minimark",[10,20,25,37,60,70,81,89,95,128,139,143,157,194,206,222,226,229,312,357,375,389,594,598,630,683,702,705,709,720,745,755,773,783,789,793,813,874,900,915,918,922,928,937,944,1056,1060,1115,1119,1122,1170,1227,1230],[11,12,13,14,19],"p",{},"The rest of lecture 4, the \"real work\" hiding behind every accuracy number I've shown so far. Without this, every K I picked in ",[15,16,18],"a",{"href":17},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier","the previous two posts"," was, without exaggeration, an educated guess.",[21,22,24],"h2",{"id":23},"pipeline-what-i-already-knew-now-with-classification","Pipeline: what I already knew, now with classification",[11,26,27,31,32,36],{},[28,29,30],"code",{},"Pipeline"," ",[15,33,35],{"href":34},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fscikit-learn-pitfalls","already showed up in the other playlist",": it chains normalization and the model into a single object, so the scaler never sees data that should stay out of training. Here the professor confirms the same conclusion, just on a classification problem with 3 classes:",[38,39,44],"pre",{"className":40,"code":41,"language":42,"meta":43,"style":43},"language-python shiki shiki-themes github-light github-dark","model = KNeighborsClassifier()\nmodel.fit(X_train, y_train)\n","python","",[28,45,46,54],{"__ignoreMap":43},[47,48,51],"span",{"class":49,"line":50},"line",1,[47,52,53],{},"model = KNeighborsClassifier()\n",[47,55,57],{"class":49,"line":56},2,[47,58,59],{},"model.fit(X_train, y_train)\n",[61,62,63],"blockquote",{},[11,64,65,69],{},[66,67,68],"strong",{},"Output (no normalization):"," accuracy 0.67.",[11,71,72,73,76,77,80],{},"Then he normalizes four different ways (min-max by hand, ",[28,74,75],{},"MinMaxScaler",", z-score by hand, ",[28,78,79],{},"StandardScaler","), and they all match:",[61,82,83],{},[11,84,85,88],{},[66,86,87],{},"Output (any normalization):"," accuracy 0.92.",[11,90,91,92,94],{},"And ",[28,93,30],{}," lands on the same 0.92, chaining both steps automatically:",[38,96,98],{"className":40,"code":97,"language":42,"meta":43,"style":43},"pipeline = Pipeline([\n    ('scaler', StandardScaler()),\n    ('model', KNeighborsClassifier())\n])\npipeline.fit(X_train, y_train)\n",[28,99,100,105,110,116,122],{"__ignoreMap":43},[47,101,102],{"class":49,"line":50},[47,103,104],{},"pipeline = Pipeline([\n",[47,106,107],{"class":49,"line":56},[47,108,109],{},"    ('scaler', StandardScaler()),\n",[47,111,113],{"class":49,"line":112},3,[47,114,115],{},"    ('model', KNeighborsClassifier())\n",[47,117,119],{"class":49,"line":118},4,[47,120,121],{},"])\n",[47,123,125],{"class":49,"line":124},5,[47,126,127],{},"pipeline.fit(X_train, y_train)\n",[11,129,130,131,133,134,138],{},"Nothing new mechanically (I'd already seen ",[28,132,30],{}," from the outside), but the jump from 0.67 to 0.92 is the biggest I've seen so far just from normalizing, reinforcing the reason ",[15,135,137],{"href":136},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold","I already explained",": 13 variables at very different scales, and KNN decides everything by distance.",[21,140,142],{"id":141},"separating-validation-from-testing","Separating validation from testing",[11,144,145,148,149,152,153,156],{},[28,146,147],{},"aula04c"," starts slow: it splits off a slice of training just for validation (",[28,150,151],{},"X_tr","\u002F",[28,154,155],{},"X_val","), tries several K values, and picks the one that wins on validation:",[38,158,160],{"className":40,"code":159,"language":42,"meta":43,"style":43},"X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2)\n\nfor k in range(1, 21, 2):\n    model = KNeighborsClassifier(n_neighbors=k)\n    model.fit(X_tr, y_tr)\n    acc = accuracy_score(y_val, model.predict(X_val))\n",[28,161,162,167,173,178,183,188],{"__ignoreMap":43},[47,163,164],{"class":49,"line":50},[47,165,166],{},"X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2)\n",[47,168,169],{"class":49,"line":56},[47,170,172],{"emptyLinePlaceholder":171},true,"\n",[47,174,175],{"class":49,"line":112},[47,176,177],{},"for k in range(1, 21, 2):\n",[47,179,180],{"class":49,"line":118},[47,181,182],{},"    model = KNeighborsClassifier(n_neighbors=k)\n",[47,184,185],{"class":49,"line":124},[47,186,187],{},"    model.fit(X_tr, y_tr)\n",[47,189,191],{"class":49,"line":190},6,[47,192,193],{},"    acc = accuracy_score(y_val, model.predict(X_val))\n",[61,195,196],{},[11,197,198,201,202,205],{},[66,199,200],{},"Output:"," the best K found was ",[66,203,204],{},"k=9",", with 0.759 validation accuracy.",[11,207,208,209,212,213,216,217,152,219,221],{},"That's already much better than \"I tested on the test set and saw which K won\" (which would be committing exactly the mistake any stats course warns against: using the test set to pick a hyperparameter is a form of leakage, it biases the final estimate because it stops being about genuinely unseen data). But there's a visible problem here: neither ",[28,210,211],{},"train_test_split"," call, holdout or validation, fixes ",[28,214,215],{},"random_state",", so every time I rerun this cell, ",[28,218,151],{},[28,220,155],{}," change, and the winning K can change with it. A single validation split is just one sample, and it can get lucky or unlucky.",[21,223,225],{"id":224},"a-real-bug-hiding-inside-a-loop","A real bug, hiding inside a loop",[11,227,228],{},"The professor then moves on to cross-validation, implemented by hand:",[38,230,232],{"className":40,"code":231,"language":42,"meta":43,"style":43},"def cross_validation(model, X, y, k=3):\n    n = int(len(y)\u002Fk)\n    idx = np.random.permutation(len(y))\n    X = X[idx]\n    y = y[idx]\n    for i in range(k):\n        X_tr = np.concatenate([X[:i*n], X[(i+1)*n:]])\n        y_tr = np.concatenate([y[:i*n], y[(i+1)*n:]])\n        X_val = X[i*n:(i+1)*n]\n        y_val = y[i*n:(i+1)*n]\n        model.fit(X_tr, y_tr)\n        y_pred = model.predict(X_val)\n        acc = accuracy_score(y_val, y_pred)\n        return acc\n",[28,233,234,239,244,249,254,259,264,270,276,282,288,294,300,306],{"__ignoreMap":43},[47,235,236],{"class":49,"line":50},[47,237,238],{},"def cross_validation(model, X, y, k=3):\n",[47,240,241],{"class":49,"line":56},[47,242,243],{},"    n = int(len(y)\u002Fk)\n",[47,245,246],{"class":49,"line":112},[47,247,248],{},"    idx = np.random.permutation(len(y))\n",[47,250,251],{"class":49,"line":118},[47,252,253],{},"    X = X[idx]\n",[47,255,256],{"class":49,"line":124},[47,257,258],{},"    y = y[idx]\n",[47,260,261],{"class":49,"line":190},[47,262,263],{},"    for i in range(k):\n",[47,265,267],{"class":49,"line":266},7,[47,268,269],{},"        X_tr = np.concatenate([X[:i*n], X[(i+1)*n:]])\n",[47,271,273],{"class":49,"line":272},8,[47,274,275],{},"        y_tr = np.concatenate([y[:i*n], y[(i+1)*n:]])\n",[47,277,279],{"class":49,"line":278},9,[47,280,281],{},"        X_val = X[i*n:(i+1)*n]\n",[47,283,285],{"class":49,"line":284},10,[47,286,287],{},"        y_val = y[i*n:(i+1)*n]\n",[47,289,291],{"class":49,"line":290},11,[47,292,293],{},"        model.fit(X_tr, y_tr)\n",[47,295,297],{"class":49,"line":296},12,[47,298,299],{},"        y_pred = model.predict(X_val)\n",[47,301,303],{"class":49,"line":302},13,[47,304,305],{},"        acc = accuracy_score(y_val, y_pred)\n",[47,307,309],{"class":49,"line":308},14,[47,310,311],{},"        return acc\n",[11,313,314,315,325,326,329,330,333,334,337,338,341,342,345,346,348,349,352,353,356],{},"This function is worth reading closely, because it teaches a lesson that has nothing to do with machine learning: ",[66,316,317,318,321,322],{},"the ",[28,319,320],{},"return"," sits inside the ",[28,323,324],{},"for",". The loop runs ",[28,327,328],{},"i"," from 0 to ",[28,331,332],{},"k-1"," to build ",[28,335,336],{},"k"," different folds (the right idea behind cross-validation: every fold becomes validation once, the rest becomes training), but the function exits and returns as soon as the ",[66,339,340],{},"first"," iteration (",[28,343,344],{},"i=0",") finishes. The other ",[28,347,332],{}," folds never run. ",[28,350,351],{},"cross_validation()",", despite the name, computes just ",[66,354,355],{},"one"," train\u002Fvalidation split, the exact same limitation as the section before, just hiding behind a name that promises more than it delivers.",[61,358,359],{},[11,360,361,31,363,366,367,370,371,374],{},[66,362,200],{},[28,364,365],{},"cross_validation(model, X_train, y_train)"," returns ",[28,368,369],{},"0.723",", a single number from a single fold, shuffled differently on every call (because ",[28,372,373],{},"np.random.permutation"," runs again each time).",[11,376,377,378,381,382,384,385,388],{},"That explains something odd that shows up right after: ",[28,379,380],{},"repeated_cross_validation",", which calls ",[28,383,351],{}," ten times and averages, kind of works by accident. It isn't doing \"10 repeats of real cross-validation\" (which would be a full fold pass, repeated 10 times), it's doing ",[66,386,387],{},"repeated holdout",": 10 different random splits, each evaluated once. The average of those 10 (0.717, std 0.057, computed right after in the notebook) is still a more stable estimate than a single split, because it cuts down the \"I got lucky or unlucky on one split\" variance. It just isn't cross-validation in the technical sense of the term: no training point is guaranteed to become validation across those 10 rounds, since that only happens by chance, not by the full-coverage guarantee real k-fold gives you.",[11,390,391,392,438,439,467,468,528,529,557,558,586,587,589,590,593],{},"Bishop describes exactly this real k-fold (he calls it ",[47,393,396,418],{"className":394},[395],"katex",[47,397,400],{"className":398},[399],"katex-mathml",[401,402,404],"math",{"xmlns":403},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[405,406,407,414],"semantics",{},[408,409,410],"mrow",{},[411,412,413],"mi",{},"S",[415,416,413],"annotation",{"encoding":417},"application\u002Fx-tex",[47,419,423],{"className":420,"ariaHidden":422},[421],"katex-html","true",[47,424,427,432],{"className":425},[426],"base",[47,428],{"className":429,"style":431},[430],"strut","height:0.6833em;",[47,433,413],{"className":434,"style":437},[435,436],"mord","mathnormal","margin-right:0.0576em;","-fold): split the data into ",[47,440,442,455],{"className":441},[395],[47,443,445],{"className":444},[399],[401,446,447],{"xmlns":403},[405,448,449,453],{},[408,450,451],{},[411,452,413],{},[415,454,413],{"encoding":417},[47,456,458],{"className":457,"ariaHidden":422},[421],[47,459,461,464],{"className":460},[426],[47,462],{"className":463,"style":431},[430],[47,465,413],{"className":466,"style":437},[435,436]," blocks, use ",[47,469,471,493],{"className":470},[395],[47,472,474],{"className":473},[399],[401,475,476],{"xmlns":403},[405,477,478,490],{},[408,479,480,482,486],{},[411,481,413],{},[483,484,485],"mo",{},"−",[487,488,489],"mn",{},"1",[415,491,492],{"encoding":417},"S-1",[47,494,496,518],{"className":495,"ariaHidden":422},[421],[47,497,499,503,506,511,515],{"className":498},[426],[47,500],{"className":501,"style":502},[430],"height:0.7667em;vertical-align:-0.0833em;",[47,504,413],{"className":505,"style":437},[435,436],[47,507],{"className":508,"style":510},[509],"mspace","margin-right:0.2222em;",[47,512,485],{"className":513},[514],"mbin",[47,516],{"className":517,"style":510},[509],[47,519,521,525],{"className":520},[426],[47,522],{"className":523,"style":524},[430],"height:0.6444em;",[47,526,489],{"className":527},[435]," to train and 1 to validate, repeat ",[47,530,532,545],{"className":531},[395],[47,533,535],{"className":534},[399],[401,536,537],{"xmlns":403},[405,538,539,543],{},[408,540,541],{},[411,542,413],{},[415,544,413],{"encoding":417},[47,546,548],{"className":547,"ariaHidden":422},[421],[47,549,551,554],{"className":550},[426],[47,552],{"className":553,"style":431},[430],[47,555,413],{"className":556,"style":437},[435,436]," times swapping which block is held out, and average the ",[47,559,561,574],{"className":560},[395],[47,562,564],{"className":563},[399],[401,565,566],{"xmlns":403},[405,567,568,572],{},[408,569,570],{},[411,571,413],{},[415,573,413],{"encoding":417},[47,575,577],{"className":576,"ariaHidden":422},[421],[47,578,580,583],{"className":579},[426],[47,581],{"className":582,"style":431},[430],[47,584,413],{"className":585,"style":437},[435,436]," scores. The important guarantee the professor's ",[28,588,351],{}," loses to the bug: in real k-fold, ",[66,591,592],{},"every"," point becomes validation exactly once, covering the whole dataset with no overlap. With repeated holdout (even repeated many times), some points might never land in validation, and others might land there repeatedly, purely by the luck of random sampling.",[21,595,597],{"id":596},"doing-it-the-right-way","Doing it the right way",[11,599,600,601,629],{},"Scikit-learn already implements Bishop's ",[47,602,604,617],{"className":603},[395],[47,605,607],{"className":606},[399],[401,608,609],{"xmlns":403},[405,610,611,615],{},[408,612,613],{},[411,614,413],{},[415,616,413],{"encoding":417},[47,618,620],{"className":619,"ariaHidden":422},[421],[47,621,623,626],{"className":622},[426],[47,624],{"className":625,"style":431},[430],[47,627,413],{"className":628,"style":437},[435,436],"-fold correctly:",[38,631,633],{"className":40,"code":632,"language":42,"meta":43,"style":43},"from sklearn.model_selection import KFold\n\nkf = KFold(n_splits=3, shuffle=True)\naccs = []\nfor train_index, val_index in kf.split(X_train):\n    model = KNeighborsClassifier(n_neighbors=5)\n    model.fit(X_train[train_index], y_train[train_index])\n    accs.append(accuracy_score(y_train[val_index], model.predict(X_train[val_index])))\n\nprint(np.mean(accs))\n",[28,634,635,640,644,649,654,659,664,669,674,678],{"__ignoreMap":43},[47,636,637],{"class":49,"line":50},[47,638,639],{},"from sklearn.model_selection import KFold\n",[47,641,642],{"class":49,"line":56},[47,643,172],{"emptyLinePlaceholder":171},[47,645,646],{"class":49,"line":112},[47,647,648],{},"kf = KFold(n_splits=3, shuffle=True)\n",[47,650,651],{"class":49,"line":118},[47,652,653],{},"accs = []\n",[47,655,656],{"class":49,"line":124},[47,657,658],{},"for train_index, val_index in kf.split(X_train):\n",[47,660,661],{"class":49,"line":190},[47,662,663],{},"    model = KNeighborsClassifier(n_neighbors=5)\n",[47,665,666],{"class":49,"line":266},[47,667,668],{},"    model.fit(X_train[train_index], y_train[train_index])\n",[47,670,671],{"class":49,"line":272},[47,672,673],{},"    accs.append(accuracy_score(y_train[val_index], model.predict(X_train[val_index])))\n",[47,675,676],{"class":49,"line":278},[47,677,172],{"emptyLinePlaceholder":171},[47,679,680],{"class":49,"line":284},[47,681,682],{},"print(np.mean(accs))\n",[61,684,685],{},[11,686,687,689,690,693,694,697,698,701],{},[66,688,200],{}," 0.726 with ",[28,691,692],{},"KFold(n_splits=3)",". With ",[28,695,696],{},"RepeatedKFold(n_splits=3, n_repeats=10)"," (real k-fold, repeated 10 times with different shuffles, to shrink the estimate's variance even further): 0.676. And the one-line shortcut, ",[28,699,700],{},"cross_val_score",", matches both: 0.704 and 0.701 respectively.",[11,703,704],{},"Those numbers aren't wildly different from the \"buggy\" repeated holdout (0.717), and that's expected: even with the broken implementation, the general idea (test on chunks that weren't used for training) already captured most of the signal. The gain from real k-fold is robustness, not necessarily a dramatically different number on this specific dataset. But you don't know that without comparing, which is exactly why it's worth implementing (or using) the correct thing instead of trusting that \"it produced a plausible-looking number\" means \"the code is right.\"",[21,706,708],{"id":707},"gridsearchcv-automating-the-search","GridSearchCV: automating the search",[11,710,711,712,715,716,719],{},"Instead of writing a ",[28,713,714],{},"for k in range(1, 21, 2)"," every time, ",[28,717,718],{},"GridSearchCV"," runs the search (and the cross-validation behind it) automatically:",[38,721,723],{"className":40,"code":722,"language":42,"meta":43,"style":43},"params = {'n_neighbors': range(1, 21, 2)}\ngrid = GridSearchCV(KNeighborsClassifier(), params, scoring='accuracy')\ngrid.fit(X_train, y_train)\nprint(grid.best_params_, grid.best_score_)\n",[28,724,725,730,735,740],{"__ignoreMap":43},[47,726,727],{"class":49,"line":50},[47,728,729],{},"params = {'n_neighbors': range(1, 21, 2)}\n",[47,731,732],{"class":49,"line":56},[47,733,734],{},"grid = GridSearchCV(KNeighborsClassifier(), params, scoring='accuracy')\n",[47,736,737],{"class":49,"line":112},[47,738,739],{},"grid.fit(X_train, y_train)\n",[47,741,742],{"class":49,"line":118},[47,743,744],{},"print(grid.best_params_, grid.best_score_)\n",[61,746,747],{},[11,748,749,31,751,754],{},[66,750,200],{},[28,752,753],{},"{'n_neighbors': 1}",", with a cross-validation score of 0.761.",[11,756,757,758,761,762,761,765,768,769,772],{},"And expanding the search to three hyperparameters at once (",[28,759,760],{},"n_neighbors",", ",[28,763,764],{},"weights",[28,766,767],{},"metric","), with ",[28,770,771],{},"KFold(n_splits=5)",":",[61,774,775],{},[11,776,777,31,779,782],{},[66,778,200],{},[28,780,781],{},"{'metric': 'manhattan', 'n_neighbors': 11, 'weights': 'distance'}",", score 0.803.",[11,784,785,786,788],{},"Bishop already warns about exactly this situation in chapter 1.3: once you have more than one hyperparameter to tune, testing every combination by hand turns into a combinatorial explosion fast. ",[28,787,718],{}," is organized brute force: it tries every combination in the grid, cross-validates each one, and returns the best.",[21,790,792],{"id":791},"pipeline-gridsearch-this-is-where-it-takes-off","Pipeline + GridSearch: this is where it takes off",[11,794,795,796,31,798,801,802,804,805,808,809,812],{},"The turning point of the whole post: putting ",[28,797,718],{},[66,799,800],{},"inside"," a ",[28,803,30],{},", together with the scaler, and tuning even the ",[28,806,807],{},"KNeighborsClassifier","'s own hyperparameters (using the ",[28,810,811],{},"model__"," prefix to point at which pipeline step each parameter belongs to):",[38,814,816],{"className":40,"code":815,"language":42,"meta":43,"style":43},"pipeline = Pipeline([\n    ('scaler', StandardScaler()),\n    ('model', KNeighborsClassifier())\n])\nparams = {\n    'model__n_neighbors': range(1, 21, 2),\n    'model__weights': ['uniform', 'distance'],\n    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n}\ngrid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True))\nscores = cross_val_score(grid, X_train, y_train, cv=KFold(n_splits=5, shuffle=True))\nprint(np.mean(scores))\n",[28,817,818,822,826,830,834,839,844,849,854,859,864,869],{"__ignoreMap":43},[47,819,820],{"class":49,"line":50},[47,821,104],{},[47,823,824],{"class":49,"line":56},[47,825,109],{},[47,827,828],{"class":49,"line":112},[47,829,115],{},[47,831,832],{"class":49,"line":118},[47,833,121],{},[47,835,836],{"class":49,"line":124},[47,837,838],{},"params = {\n",[47,840,841],{"class":49,"line":190},[47,842,843],{},"    'model__n_neighbors': range(1, 21, 2),\n",[47,845,846],{"class":49,"line":266},[47,847,848],{},"    'model__weights': ['uniform', 'distance'],\n",[47,850,851],{"class":49,"line":272},[47,852,853],{},"    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n",[47,855,856],{"class":49,"line":278},[47,857,858],{},"}\n",[47,860,861],{"class":49,"line":284},[47,862,863],{},"grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True))\n",[47,865,866],{"class":49,"line":290},[47,867,868],{},"scores = cross_val_score(grid, X_train, y_train, cv=KFold(n_splits=5, shuffle=True))\n",[47,870,871],{"class":49,"line":296},[47,872,873],{},"print(np.mean(scores))\n",[11,875,876,877,31,879,882,883,885,886,889,890,892,893,895,896,899],{},"Notice the structure: there's a ",[28,878,700],{},[66,880,881],{},"wrapped around"," an entire ",[28,884,718],{},". That's ",[66,887,888],{},"nested"," cross-validation: the outer loop measures how well the whole process (normalize, search for the best hyperparameters, train) generalizes, and the inner loop (inside ",[28,891,718],{},") only picks the hyperparameters. Without that nesting, ",[28,894,718],{},"'s own cross-validation score (",[28,897,898],{},"best_score_",") runs slightly optimistic, because the same data that chose the hyperparameters also evaluated the final result.",[61,901,902],{},[11,903,904,906,907,910,911,914],{},[66,905,200],{}," average of 0.957 (versus 0.81 without normalizing inside the pipeline, and versus 0.68 for raw KNN with no grid at all). Adding ",[28,908,909],{},"scaler__with_mean"," and ",[28,912,913],{},"scaler__with_std"," to the search grid (letting even the normalization be part of what's optimized): 0.979.",[11,916,917],{},"From raw KNN (0.67-0.68) to the full pipeline with nested hyperparameter search (0.979): the entire distance between \"I ran the default model\" and \"I did this properly.\"",[21,919,921],{"id":920},"faster-than-a-full-grid-random-search","Faster than a full grid: random search",[11,923,924,927],{},[28,925,926],{},"RandomizedSearchCV"," only tries a random sample of combinations (here, 20) instead of all of them:",[38,929,931],{"className":40,"code":930,"language":42,"meta":43,"style":43},"grid = RandomizedSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True), n_iter=20)\n",[28,932,933],{"__ignoreMap":43},[47,934,935],{"class":49,"line":50},[47,936,930],{},[61,938,939],{},[11,940,941,943],{},[66,942,200],{}," 0.957, basically tied with the full grid, while testing far fewer combinations.",[11,945,946,947,1051,1052,1055],{},"That makes sense once the search grid gets too big to fully test (here it's already ",[47,948,950,982],{"className":949},[395],[47,951,953],{"className":952},[399],[401,954,955],{"xmlns":403},[405,956,957,979],{},[408,958,959,962,965,968,970,973,976],{},[487,960,961],{},"10",[483,963,964],{},"×",[487,966,967],{},"2",[483,969,964],{},[487,971,972],{},"3",[483,974,975],{},"=",[487,977,978],{},"60",[415,980,981],{"encoding":417},"10 \\times 2 \\times 3 = 60",[47,983,985,1004,1022,1042],{"className":984,"ariaHidden":422},[421],[47,986,988,992,995,998,1001],{"className":987},[426],[47,989],{"className":990,"style":991},[430],"height:0.7278em;vertical-align:-0.0833em;",[47,993,961],{"className":994},[435],[47,996],{"className":997,"style":510},[509],[47,999,964],{"className":1000},[514],[47,1002],{"className":1003,"style":510},[509],[47,1005,1007,1010,1013,1016,1019],{"className":1006},[426],[47,1008],{"className":1009,"style":991},[430],[47,1011,967],{"className":1012},[435],[47,1014],{"className":1015,"style":510},[509],[47,1017,964],{"className":1018},[514],[47,1020],{"className":1021,"style":510},[509],[47,1023,1025,1028,1031,1035,1039],{"className":1024},[426],[47,1026],{"className":1027,"style":524},[430],[47,1029,972],{"className":1030},[435],[47,1032],{"className":1033,"style":1034},[509],"margin-right:0.2778em;",[47,1036,975],{"className":1037},[1038],"mrel",[47,1040],{"className":1041,"style":1034},[509],[47,1043,1045,1048],{"className":1044},[426],[47,1046],{"className":1047,"style":524},[430],[47,1049,978],{"className":1050},[435]," combinations, each with 5 folds, 300 model fits. Add more hyperparameters and this explodes fast). The notebook takes a quick peek at ",[28,1053,1054],{},"Optuna",", a Bayesian search library that picks the next combination to try based on what's worked so far instead of sampling or testing blindly, but that's just a passing mention, not the lecture's focus.",[21,1057,1059],{"id":1058},"wrapping-up","Wrapping up",[1061,1062,1063,1077],"table",{},[1064,1065,1066],"thead",{},[1067,1068,1069,1074],"tr",{},[1070,1071,1073],"th",{"align":1072},"left","What I already knew",[1070,1075,1076],{"align":1072},"What these three lectures settled",[1078,1079,1080,1091,1102],"tbody",{},[1067,1081,1082,1088],{},[1083,1084,1085,1087],"td",{"align":1072},[28,1086,30],{}," prevents leakage between normalization and the model",[1083,1089,1090],{"align":1072},"The same holds for classification, not just regression, and the gain here was huge (0.67 → 0.92)",[1067,1092,1093,1096],{},[1083,1094,1095],{"align":1072},"Cross-validation exists to give a more stable estimate than a single train\u002Ftest split",[1083,1097,1098,1099,1101],{"align":1072},"A ",[28,1100,320],{}," in the wrong place can turn \"cross-validation\" into repeated holdout without me noticing, so it's worth reading the validation code itself, not just trusting the function's name",[1067,1103,1104,1107],{},[1083,1105,1106],{"align":1072},"A hyperparameter is my choice",[1083,1108,1109,1111,1112,1114],{"align":1072},[28,1110,718],{}," automates the search, and placed inside a ",[28,1113,30],{},", with nested cross-validation wrapped around it, gives the most honest generalization estimate I've produced in this playlist so far",[21,1116,1118],{"id":1117},"practical-application","Practical application",[11,1120,1121],{},"I reproduce the full pipeline (normalization + hyperparameter search + nested cross-validation) on the same wine dataset, with a fixed seed for a reproducible result, something none of the searches in the original notebook have.",[38,1123,1125],{"className":40,"code":1124,"language":42,"meta":43,"style":43},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\npipeline = Pipeline([('scaler', StandardScaler()), ('model', KNeighborsClassifier())])\nparams = {\n    'model__n_neighbors': range(1, 21, 2),\n    'model__weights': ['uniform', 'distance'],\n    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n}\ngrid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True, random_state=42))\ngrid.fit(X_train, y_train)\n",[28,1126,1127,1132,1136,1141,1145,1149,1153,1157,1161,1166],{"__ignoreMap":43},[47,1128,1129],{"class":49,"line":50},[47,1130,1131],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",[47,1133,1134],{"class":49,"line":56},[47,1135,172],{"emptyLinePlaceholder":171},[47,1137,1138],{"class":49,"line":112},[47,1139,1140],{},"pipeline = Pipeline([('scaler', StandardScaler()), ('model', KNeighborsClassifier())])\n",[47,1142,1143],{"class":49,"line":118},[47,1144,838],{},[47,1146,1147],{"class":49,"line":124},[47,1148,843],{},[47,1150,1151],{"class":49,"line":190},[47,1152,848],{},[47,1154,1155],{"class":49,"line":266},[47,1156,853],{},[47,1158,1159],{"class":49,"line":272},[47,1160,858],{},[47,1162,1163],{"class":49,"line":278},[47,1164,1165],{},"grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True, random_state=42))\n",[47,1167,1168],{"class":49,"line":284},[47,1169,739],{},[1061,1171,1172,1183],{},[1064,1173,1174],{},[1067,1175,1176,1179],{},[1070,1177,1178],{"align":1072},"Step",[1070,1180,1182],{"align":1181},"right","Result",[1078,1184,1185,1193,1203,1211,1219],{},[1067,1186,1187,1190],{},[1083,1188,1189],{"align":1072},"Raw KNN, no normalizing, no hyperparameter search",[1083,1191,1192],{"align":1181},"0.7222 (test)",[1067,1194,1195,1198],{},[1083,1196,1197],{"align":1072},"Best combination found by the grid",[1083,1199,1200],{"align":1181},[28,1201,1202],{},"metric=manhattan, n_neighbors=9, weights=uniform",[1067,1204,1205,1208],{},[1083,1206,1207],{"align":1072},"Grid's cross-validation score",[1083,1209,1210],{"align":1181},"0.9862",[1067,1212,1213,1216],{},[1083,1214,1215],{"align":1072},"Test accuracy, with the best pipeline",[1083,1217,1218],{"align":1181},"0.9722",[1067,1220,1221,1224],{},[1083,1222,1223],{"align":1072},"Nested cross-validation (honest estimate)",[1083,1225,1226],{"align":1181},"0.9791",[11,1228,1229],{},"The nested cross-validation score (0.9791) and the test accuracy (0.9722) land close to each other, and that's exactly what I want to see: it means the nested cross-validation wasn't running too optimistic, it genuinely predicted how the pipeline would do on data it had never seen. Against raw KNN (0.7222), the entire gap (25 percentage points) came just from normalizing and picking hyperparameters properly, without touching the algorithm itself.",[1231,1232,1233],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":43,"searchDepth":56,"depth":56,"links":1235},[1236,1237,1238,1239,1240,1241,1242,1243,1244],{"id":23,"depth":56,"text":24},{"id":141,"depth":56,"text":142},{"id":224,"depth":56,"text":225},{"id":596,"depth":56,"text":597},{"id":707,"depth":56,"text":708},{"id":791,"depth":56,"text":792},{"id":920,"depth":56,"text":921},{"id":1058,"depth":56,"text":1059},{"id":1117,"depth":56,"text":1118},null,"2026-08-19","Lectures 4b, 4c, and 4d: the professor builds the real professional workflow on top of KNN, and along the way I find an actual bug in his code, a return sitting inside the wrong loop.","md",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","pattern-recognition",{"title":6,"description":1247},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation",[1256,1257,1258],"pipeline","cross-validation","gridsearch","SXPBwoK0vI0bd6coxwMHRUwHReF5a1_ev03Nu69xb1I",1787338984343]