[{"data":1,"prerenderedAt":16753},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition":3,"playlist-en-pattern-recognition":4,"playlist-posts-en-pattern-recognition":69},null,{"id":5,"title":6,"body":7,"cover":3,"description":60,"extension":61,"meta":62,"navigation":63,"order":58,"path":64,"seo":65,"status":66,"stem":67,"__hash__":68},"playlists\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Findex.md","Pattern Recognition",{"type":8,"value":9,"toc":56},"minimark",[10,25,38,48],[11,12,13,14,18,19,24],"p",{},"This playlist is my study notebook from the Pattern Recognition course I took, taught by ",[15,16,17],"strong",{},"Dr. Francisco Boldt",". The guy is excellent, he genuinely codes the models by hand, live, in class, no pre-baked formula slides, and he's teaching me ",[20,21,23],"a",{"href":22},"\u002Fen\u002Fplaylists\u002Fneural-networks","neural networks now too",". If you landed here coming from one of his classes, you already know what I mean.",[11,26,27,28,32,33,37],{},"Unlike the ",[20,29,31],{"href":30},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization","Andrew Ng specialization playlist",", the lecture notebooks here are a lot leaner: barely any markdown cells, it's the professor live-coding and everyone following along. So the work of digging into the \"why\" behind each line of code is heavier here, and for that I lean on ",[34,35,36],"em",{},"Pattern Recognition and Machine Learning",", by Christopher Bishop (2006), pretty much a bible in the field, as the theoretical reference.",[11,39,40,41,47],{},"The notebooks come from the course repository, ",[20,42,46],{"href":43,"rel":44},"https:\u002F\u002Fgithub.com\u002Fpablobelmiro\u002Faulasml\u002Ftree\u002F2026-1",[45],"nofollow","pablobelmiro\u002Faulasml",", a fork of Dr. Boldt's own repository, where he publishes each lecture's code. In this playlist's posts, whoever \"wrote\" the code is always him, the professor; the foundational explanation, with the metaphor, the slightly-off analogy, and the buddy-sitting-next-to-you tone, that part is mine.",[11,49,50,51,55],{},"Every lecture becomes a post here. We start at the start: the same ",[20,52,54],{"href":53},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab02-model-representation","linear regression problem from the other playlist",", except this time the code is being written live on the whiteboard.",{"title":57,"searchDepth":58,"depth":58,"links":59},"",2,[],"My notes from my Pattern Recognition course, lecture by lecture, with Bishop's book as the theoretical backbone.","md",{},true,"\u002Fen\u002Fplaylists\u002Fpattern-recognition",{"title":6,"description":60},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Findex","y_rvvoQ-pc7yj5eUqMw_dVQddrlu3I5aEN4sROdZOdk",[70,2008,4097,5121,5772,6932,8249,9159,9915,11421,12046,13381,14503,15165,16015],{"id":71,"title":72,"body":73,"cover":3,"date":1996,"description":1997,"extension":61,"meta":1998,"navigation":63,"order":107,"path":1999,"playlist":2000,"seo":2001,"status":66,"stem":2002,"tags":2003,"__hash__":2007},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator.md","Linear Regression by Hand: Building My Own Estimator",{"type":8,"value":74,"toc":1985},[75,78,83,95,128,139,143,272,281,312,327,344,348,356,386,432,441,450,910,913,1151,1157,1161,1176,1196,1204,1212,1237,1424,1431,1435,1446,1490,1535,1542,1585,1591,1623,1631,1638,1642,1645,1738,1763,1766,1775,1798,1801,1816,1828,1832,1838,1853,1864,1867,1871,1920,1936,1940,1943,1963,1970,1978,1981],[11,76,77],{},"Lecture 1 of Dr. Boldt's course. His notebook has almost no markdown cells, it's him live-coding straight through, so most of the work in this post is mine: retelling the \"why\" behind each block of code, with Bishop's book as reference. Let's go.",[79,80,82],"h2",{"id":81},"the-dataset-442-real-patients","The dataset: 442 real patients",[11,84,85,86,90,91,94],{},"Unlike the Andrew Ng specialization playlist, here the dataset ships built into scikit-learn itself: ",[87,88,89],"code",{},"load_diabetes",", with ",[15,92,93],{},"442 real patients",", 10 input variables (age, sex, body mass index, average blood pressure, and six blood serum measurements) and one target, a numeric measure of how much the diabetes progressed after one year.",[96,97,101],"pre",{"className":98,"code":99,"language":100,"meta":57,"style":57},"language-python shiki shiki-themes github-light github-dark","from sklearn.datasets import load_diabetes\ndata = load_diabetes()\nX = data.data   # (442, 10)\ny = data.target # (442,)\n","python",[87,102,103,111,116,122],{"__ignoreMap":57},[104,105,108],"span",{"class":106,"line":107},"line",1,[104,109,110],{},"from sklearn.datasets import load_diabetes\n",[104,112,113],{"class":106,"line":58},[104,114,115],{},"data = load_diabetes()\n",[104,117,119],{"class":106,"line":118},3,[104,120,121],{},"X = data.data   # (442, 10)\n",[104,123,125],{"class":106,"line":124},4,[104,126,127],{},"y = data.target # (442,)\n",[11,129,130,131,134,135,138],{},"One detail that catches a lot of people off guard: the 10 columns of ",[87,132,133],{},"X"," are ",[15,136,137],{},"not"," in their original units. Scikit-learn ships this dataset already mean-centered and rescaled, column by column. That's why the body mass index column (the one I'll use throughout this post) ranges from -0.09 to 0.17 instead of sitting in the normal 15 to 40 range. Keep that in mind, it's why the coefficients that show up later look like giant numbers: they're compensating for a tiny input variable.",[79,140,142],{"id":141},"the-simplest-possible-model","The simplest possible model",[11,144,145],{},[104,146,149,190],{"className":147},[148],"katex",[104,150,153],{"className":151},[152],"katex-mathml",[154,155,157],"math",{"xmlns":156},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[158,159,160,185],"semantics",{},[161,162,163,167,171,173,176,179,182],"mrow",{},[164,165,166],"mi",{},"y",[168,169,170],"mo",{},"=",[164,172,20],{},[168,174,175],{},"⋅",[164,177,178],{},"x",[168,180,181],{},"+",[164,183,184],{},"b",[186,187,189],"annotation",{"encoding":188},"application\u002Fx-tex","y = a \\cdot x + b",[104,191,195,222,243,262],{"className":192,"ariaHidden":194},[193],"katex-html","true",[104,196,199,204,210,215,219],{"className":197},[198],"base",[104,200],{"className":201,"style":203},[202],"strut","height:0.625em;vertical-align:-0.1944em;",[104,205,166],{"className":206,"style":209},[207,208],"mord","mathnormal","margin-right:0.0359em;",[104,211],{"className":212,"style":214},[213],"mspace","margin-right:0.2778em;",[104,216,170],{"className":217},[218],"mrel",[104,220],{"className":221,"style":214},[213],[104,223,225,229,232,236,240],{"className":224},[198],[104,226],{"className":227,"style":228},[202],"height:0.4445em;",[104,230,20],{"className":231},[207,208],[104,233],{"className":234,"style":235},[213],"margin-right:0.2222em;",[104,237,175],{"className":238},[239],"mbin",[104,241],{"className":242,"style":235},[213],[104,244,246,250,253,256,259],{"className":245},[198],[104,247],{"className":248,"style":249},[202],"height:0.6667em;vertical-align:-0.0833em;",[104,251,178],{"className":252},[207,208],[104,254],{"className":255,"style":235},[213],[104,257,181],{"className":258},[239],[104,260],{"className":261,"style":235},[213],[104,263,265,269],{"className":264},[198],[104,266],{"className":267,"style":268},[202],"height:0.6944em;",[104,270,184],{"className":271},[207,208],[11,273,274,275,277,278,280],{},"The same familiar shape: a line, where ",[87,276,20],{}," is the slope (how much the target changes per unit of body mass index) and ",[87,279,184],{}," is the intercept. The professor tries two hand-picked guesses, no optimization yet, just to see what the line looks like:",[96,282,284],{"className":98,"code":283,"language":100,"meta":57,"style":57},"def modelo_linear(X, a, b):\n    return a*X + b\n\nypred = modelo_linear(X2, 1000, 100)  # first guess\nypred = modelo_linear(X2, 1500, 150)  # second guess\n",[87,285,286,291,296,301,306],{"__ignoreMap":57},[104,287,288],{"class":106,"line":107},[104,289,290],{},"def modelo_linear(X, a, b):\n",[104,292,293],{"class":106,"line":58},[104,294,295],{},"    return a*X + b\n",[104,297,298],{"class":106,"line":118},[104,299,300],{"emptyLinePlaceholder":63},"\n",[104,302,303],{"class":106,"line":124},[104,304,305],{},"ypred = modelo_linear(X2, 1000, 100)  # first guess\n",[104,307,309],{"class":106,"line":308},5,[104,310,311],{},"ypred = modelo_linear(X2, 1500, 150)  # second guess\n",[11,313,314,315,318,319,321,322,318,324,326],{},"Drag the sliders below (here on the blog I use the standard ",[87,316,317],{},"w","\u002F",[87,320,184],{}," notation, it's exactly the same ",[87,323,20],{},[87,325,184],{}," the professor uses) over a real sample of 45 patients from the training set, and see for yourself which line fits the points better:",[328,329],"model-playground",{":b-max":330,":b-min":331,":b-step":332,":initial-b":333,":initial-w":334,":w-max":335,":w-min":331,":w-step":336,":x-train":337,":y-train":338,"dataLabel":339,"error-label":340,"prediction-label":341,"x-label":342,"y-label":343},"300","0","5","100","1000","1600","20","[-0.0062, -0.0903, 0.0714, 0.0445, 0.0143, -0.0698, -0.0849, -0.0461, -0.0655, -0.0213, -0.0041, 0.0358, 0.0692, 0.0207, -0.0504, 0.024, -0.0094, 0.1102, 0.0477, 0.0143, -0.0256, -0.0385, 0.08, -0.0041, 0.0412, -0.0418, 0.093, 0.01, 0.0606, 0.0391, 0.0595, 0.0164, -0.0105, 0.0067, 0.0013, -0.0299, -0.0235, -0.072, -0.0084, 0.1609, -0.0224, 0.0595, -0.0073, -0.0105, 0.0067]","[219, 94, 295, 129, 90, 48, 90, 72, 214, 281, 68, 184, 277, 197, 189, 121, 257, 258, 317, 191, 252, 127, 257, 198, 198, 103, 128, 150, 245, 246, 178, 268, 168, 109, 229, 118, 71, 77, 81, 346, 84, 85, 52, 25, 67]","Patient","Total error (sum of absolute errors)","Model","BMI (normalized)","diabetes progression",[79,345,347],{"id":346},"how-to-tell-if-a-guess-is-any-good","How to tell if a guess is any good",[11,349,350,351,355],{},"\"Fits better\" needs to become a number. The professor implements three metrics by hand, the same trio that already showed up ",[20,352,354],{"href":353},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab05-scikit-learn","in the specialization playlist"," (if you've read that one, feel free to skim):",[357,358,359,370,380],"ul",{},[360,361,362,365,366,369],"li",{},[15,363,364],{},"MAE"," (mean absolute error): the average of ",[87,367,368],{},"|predicted - actual|",". Easy to read (it's in patient units), but treats every error the same.",[360,371,372,375,376,379],{},[15,373,374],{},"MSE"," (mean squared error): the average of ",[87,377,378],{},"(predicted - actual)²",". Squaring means one big miss weighs disproportionately more than several small ones.",[360,381,382,385],{},[15,383,384],{},"RMSE",": the square root of MSE, to get back to the original unit after having squared everything.",[96,387,389],{"className":98,"code":388,"language":100,"meta":57,"style":57},"def mae(y, ypred):\n    return np.sum(np.abs(y - ypred)) \u002F len(y)\n\ndef mse(y, ypred):\n    return np.sum((y - ypred)**2) \u002F len(y)\n\ndef rmse(y, ypred):\n    return np.sqrt(mse(y, ypred))\n",[87,390,391,396,401,405,410,415,420,426],{"__ignoreMap":57},[104,392,393],{"class":106,"line":107},[104,394,395],{},"def mae(y, ypred):\n",[104,397,398],{"class":106,"line":58},[104,399,400],{},"    return np.sum(np.abs(y - ypred)) \u002F len(y)\n",[104,402,403],{"class":106,"line":118},[104,404,300],{"emptyLinePlaceholder":63},[104,406,407],{"class":106,"line":124},[104,408,409],{},"def mse(y, ypred):\n",[104,411,412],{"class":106,"line":308},[104,413,414],{},"    return np.sum((y - ypred)**2) \u002F len(y)\n",[104,416,418],{"class":106,"line":417},6,[104,419,300],{"emptyLinePlaceholder":63},[104,421,423],{"class":106,"line":422},7,[104,424,425],{},"def rmse(y, ypred):\n",[104,427,429],{"class":106,"line":428},8,[104,430,431],{},"    return np.sqrt(mse(y, ypred))\n",[433,434,435],"blockquote",{},[11,436,437,440],{},[15,438,439],{},"Output on the full dataset (442 patients):"," guess (1000, 100) → MAE 64.93, MSE 6614.14, RMSE 81.33. Guess (1500, 150) → MAE 53.93, MSE 4580.80, RMSE 67.68. The second guess wins on all three metrics.",[11,442,443,444,449],{},"Bishop calls this an ",[445,446,448],"glossary-term",{"definition":447},"the single number that summarizes how far the model is from the training data","error function",", and defines it almost exactly like MSE, except without dividing by the number of points and with a bonus factor of 1\u002F2:",[11,451,452],{},[104,453,455,549],{"className":454},[148],[104,456,458],{"className":457},[152],[154,459,460],{"xmlns":156},[158,461,462,546],{},[161,463,464,467,471,474,477,479,489,507,510,512,514,521,524,526,528,531,538],{},[164,465,466],{},"E",[168,468,470],{"stretchy":469},"false","(",[164,472,317],{"mathvariant":473},"bold",[168,475,476],{"stretchy":469},")",[168,478,170],{},[480,481,482,486],"mfrac",{},[483,484,485],"mn",{},"1",[483,487,488],{},"2",[490,491,492,495,504],"msubsup",{},[168,493,494],{},"∑",[161,496,497,500,502],{},[164,498,499],{},"n",[168,501,170],{},[483,503,485],{},[164,505,506],{},"N",[168,508,509],{"stretchy":469},"{",[164,511,166],{},[168,513,470],{"stretchy":469},[515,516,517,519],"msub",{},[164,518,178],{},[164,520,499],{},[168,522,523],{"separator":194},",",[164,525,317],{"mathvariant":473},[168,527,476],{"stretchy":469},[168,529,530],{},"−",[515,532,533,536],{},[164,534,535],{},"t",[164,537,499],{},[539,540,541,544],"msup",{},[168,542,543],{"stretchy":469},"}",[483,545,488],{},[186,547,548],{"encoding":188},"E(\\mathbf{w}) = \\frac{1}{2}\\sum_{n=1}^{N}\\{y(x_n,\\mathbf{w}) - t_n\\}^2",[104,550,552,585,832],{"className":551,"ariaHidden":194},[193],[104,553,555,559,563,567,572,576,579,582],{"className":554},[198],[104,556],{"className":557,"style":558},[202],"height:1em;vertical-align:-0.25em;",[104,560,466],{"className":561,"style":562},[207,208],"margin-right:0.0576em;",[104,564,470],{"className":565},[566],"mopen",[104,568,317],{"className":569,"style":571},[207,570],"mathbf","margin-right:0.016em;",[104,573,476],{"className":574},[575],"mclose",[104,577],{"className":578,"style":214},[213],[104,580,170],{"className":581},[218],[104,583],{"className":584,"style":214},[213],[104,586,588,592,680,684,758,761,764,767,810,814,817,820,823,826,829],{"className":587},[198],[104,589],{"className":590,"style":591},[202],"height:1.3262em;vertical-align:-0.345em;",[104,593,595,599,677],{"className":594},[207],[104,596],{"className":597},[566,598],"nulldelimiter",[104,600,602],{"className":601},[480],[104,603,607,668],{"className":604},[605,606],"vlist-t","vlist-t2",[104,608,611,663],{"className":609},[610],"vlist-r",[104,612,616,637,648],{"className":613,"style":615},[614],"vlist","height:0.8451em;",[104,617,619,624],{"style":618},"top:-2.655em;",[104,620],{"className":621,"style":623},[622],"pstrut","height:3em;",[104,625,631],{"className":626},[627,628,629,630],"sizing","reset-size6","size3","mtight",[104,632,634],{"className":633},[207,630],[104,635,488],{"className":636},[207,630],[104,638,640,643],{"style":639},"top:-3.23em;",[104,641],{"className":642,"style":623},[622],[104,644],{"className":645,"style":647},[646],"frac-line","border-bottom-width:0.04em;",[104,649,651,654],{"style":650},"top:-3.394em;",[104,652],{"className":653,"style":623},[622],[104,655,657],{"className":656},[627,628,629,630],[104,658,660],{"className":659},[207,630],[104,661,485],{"className":662},[207,630],[104,664,667],{"className":665},[666],"vlist-s","​",[104,669,671],{"className":670},[610],[104,672,675],{"className":673,"style":674},[614],"height:0.345em;",[104,676],{},[104,678],{"className":679},[575,598],[104,681],{"className":682,"style":683},[213],"margin-right:0.1667em;",[104,685,688,694],{"className":686},[687],"mop",[104,689,494],{"className":690,"style":693},[687,691,692],"op-symbol","small-op","position:relative;top:0em;",[104,695,698],{"className":696},[697],"msupsub",[104,699,701,749],{"className":700},[605,606],[104,702,704,746],{"className":703},[610],[104,705,708,730],{"className":706,"style":707},[614],"height:0.9812em;",[104,709,711,715],{"style":710},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[104,712],{"className":713,"style":714},[622],"height:2.7em;",[104,716,718],{"className":717},[627,628,629,630],[104,719,721,724,727],{"className":720},[207,630],[104,722,499],{"className":723},[207,208,630],[104,725,170],{"className":726},[218,630],[104,728,485],{"className":729},[207,630],[104,731,733,736],{"style":732},"top:-3.2029em;margin-right:0.05em;",[104,734],{"className":735,"style":714},[622],[104,737,739],{"className":738},[627,628,629,630],[104,740,742],{"className":741},[207,630],[104,743,506],{"className":744,"style":745},[207,208,630],"margin-right:0.109em;",[104,747,667],{"className":748},[666],[104,750,752],{"className":751},[610],[104,753,756],{"className":754,"style":755},[614],"height:0.2997em;",[104,757],{},[104,759,509],{"className":760},[566],[104,762,166],{"className":763,"style":209},[207,208],[104,765,470],{"className":766},[566],[104,768,770,773],{"className":769},[207],[104,771,178],{"className":772},[207,208],[104,774,776],{"className":775},[697],[104,777,779,801],{"className":778},[605,606],[104,780,782,798],{"className":781},[610],[104,783,786],{"className":784,"style":785},[614],"height:0.1514em;",[104,787,789,792],{"style":788},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[104,790],{"className":791,"style":714},[622],[104,793,795],{"className":794},[627,628,629,630],[104,796,499],{"className":797},[207,208,630],[104,799,667],{"className":800},[666],[104,802,804],{"className":803},[610],[104,805,808],{"className":806,"style":807},[614],"height:0.15em;",[104,809],{},[104,811,523],{"className":812},[813],"mpunct",[104,815],{"className":816,"style":683},[213],[104,818,317],{"className":819,"style":571},[207,570],[104,821,476],{"className":822},[575],[104,824],{"className":825,"style":235},[213],[104,827,530],{"className":828},[239],[104,830],{"className":831,"style":235},[213],[104,833,835,839,879],{"className":834},[198],[104,836],{"className":837,"style":838},[202],"height:1.0641em;vertical-align:-0.25em;",[104,840,842,845],{"className":841},[207],[104,843,535],{"className":844},[207,208],[104,846,848],{"className":847},[697],[104,849,851,871],{"className":850},[605,606],[104,852,854,868],{"className":853},[610],[104,855,857],{"className":856,"style":785},[614],[104,858,859,862],{"style":788},[104,860],{"className":861,"style":714},[622],[104,863,865],{"className":864},[627,628,629,630],[104,866,499],{"className":867},[207,208,630],[104,869,667],{"className":870},[666],[104,872,874],{"className":873},[610],[104,875,877],{"className":876,"style":807},[614],[104,878],{},[104,880,882,885],{"className":881},[575],[104,883,543],{"className":884},[575],[104,886,888],{"className":887},[697],[104,889,891],{"className":890},[605],[104,892,894],{"className":893},[610],[104,895,898],{"className":896,"style":897},[614],"height:0.8141em;",[104,899,901,904],{"style":900},"top:-3.063em;margin-right:0.05em;",[104,902],{"className":903,"style":714},[622],[104,905,907],{"className":906},[627,628,629,630],[104,908,488],{"className":909},[207,630],[11,911,912],{},"The 1\u002F2 exists purely for convenience: when you take the derivative to find the minimum, the square comes down multiplying and cancels the 1\u002F2 exactly. A cosmetic detail, it doesn't change where the minimum points to. And the \"root\" version he uses to compare datasets of different sizes,",[11,914,915],{},[104,916,918,965],{"className":917},[148],[104,919,921],{"className":920},[152],[154,922,923],{"xmlns":156},[158,924,925,962],{},[161,926,927,935,937],{},[515,928,929,931],{},[164,930,466],{},[932,933,934],"mtext",{},"RMS",[168,936,170],{},[938,939,940],"msqrt",{},[161,941,942,944,946,948,955,957,960],{},[483,943,488],{},[164,945,466],{},[168,947,470],{"stretchy":469},[539,949,950,952],{},[164,951,317],{"mathvariant":473},[168,953,954],{},"∗",[168,956,476],{"stretchy":469},[164,958,318],{"mathvariant":959},"normal",[164,961,506],{},[186,963,964],{"encoding":188},"E_{\\text{RMS}} = \\sqrt{2E(\\mathbf{w}^*)\u002FN}",[104,966,968,1033],{"className":967,"ariaHidden":194},[193],[104,969,971,975,1024,1027,1030],{"className":970},[198],[104,972],{"className":973,"style":974},[202],"height:0.8333em;vertical-align:-0.15em;",[104,976,978,981],{"className":977},[207],[104,979,466],{"className":980,"style":562},[207,208],[104,982,984],{"className":983},[697],[104,985,987,1016],{"className":986},[605,606],[104,988,990,1013],{"className":989},[610],[104,991,994],{"className":992,"style":993},[614],"height:0.3283em;",[104,995,997,1000],{"style":996},"top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;",[104,998],{"className":999,"style":714},[622],[104,1001,1003],{"className":1002},[627,628,629,630],[104,1004,1006],{"className":1005},[207,630],[104,1007,1010],{"className":1008},[207,1009,630],"text",[104,1011,934],{"className":1012},[207,630],[104,1014,667],{"className":1015},[666],[104,1017,1019],{"className":1018},[610],[104,1020,1022],{"className":1021,"style":807},[614],[104,1023],{},[104,1025],{"className":1026,"style":214},[213],[104,1028,170],{"className":1029},[218],[104,1031],{"className":1032,"style":214},[213],[104,1034,1036,1040],{"className":1035},[198],[104,1037],{"className":1038,"style":1039},[202],"height:1.24em;vertical-align:-0.305em;",[104,1041,1044],{"className":1042},[207,1043],"sqrt",[104,1045,1047,1142],{"className":1046},[605,606],[104,1048,1050,1139],{"className":1049},[610],[104,1051,1054,1116],{"className":1052,"style":1053},[614],"height:0.935em;",[104,1055,1059,1063],{"className":1056,"style":1058},[1057],"svg-align","top:-3.2em;",[104,1060],{"className":1061,"style":1062},[622],"height:3.2em;",[104,1064,1067,1070,1073,1076,1107,1110,1113],{"className":1065,"style":1066},[207],"padding-left:1em;",[104,1068,488],{"className":1069},[207],[104,1071,466],{"className":1072,"style":562},[207,208],[104,1074,470],{"className":1075},[566],[104,1077,1079,1082],{"className":1078},[207],[104,1080,317],{"className":1081,"style":571},[207,570],[104,1083,1085],{"className":1084},[697],[104,1086,1088],{"className":1087},[605],[104,1089,1091],{"className":1090},[610],[104,1092,1095],{"className":1093,"style":1094},[614],"height:0.6147em;",[104,1096,1098,1101],{"style":1097},"top:-2.989em;margin-right:0.05em;",[104,1099],{"className":1100,"style":714},[622],[104,1102,1104],{"className":1103},[627,628,629,630],[104,1105,954],{"className":1106},[239,630],[104,1108,476],{"className":1109},[575],[104,1111,318],{"className":1112},[207],[104,1114,506],{"className":1115,"style":745},[207,208],[104,1117,1119,1122],{"style":1118},"top:-2.895em;",[104,1120],{"className":1121,"style":1062},[622],[104,1123,1127],{"className":1124,"style":1126},[1125],"hide-tail","min-width:1.02em;height:1.28em;",[1128,1129,1135],"svg",{"xmlns":1130,"width":1131,"height":1132,"viewBox":1133,"preserveAspectRatio":1134},"http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","400em","1.28em","0 0 400000 1296","xMinYMin slice",[1136,1137],"path",{"d":1138},"M263,681c0.7,0,18,39.7,52,119\nc34,79.3,68.167,158.7,102.5,238c34.3,79.3,51.8,119.3,52.5,120\nc340,-704.7,510.7,-1060.3,512,-1067\nl0 -0\nc4.7,-7.3,11,-11,19,-11\nH40000v40H1012.3\ns-271.3,567,-271.3,567c-38.7,80.7,-84,175,-136,283c-52,108,-89.167,185.3,-111.5,232\nc-22.3,46.7,-33.8,70.3,-34.5,71c-4.7,4.7,-12.3,7,-23,7s-12,-1,-12,-1\ns-109,-253,-109,-253c-72.7,-168,-109.3,-252,-110,-252c-10.7,8,-22,16.7,-34,26\nc-22,17.3,-33.3,26,-34,26s-26,-26,-26,-26s76,-59,76,-59s76,-60,76,-60z\nM1001 80h400000v40h-400000z",[104,1140,667],{"className":1141},[666],[104,1143,1145],{"className":1144},[610],[104,1146,1149],{"className":1147,"style":1148},[614],"height:0.305em;",[104,1150],{},[11,1152,1153,1154,1156],{},"is the same RMSE I just computed, with the same division by ",[87,1155,506],{}," hidden behind the factor of 2 (because his function skips the division that our MSE already does).",[79,1158,1160],{"id":1159},"the-dumb-model-that-becomes-a-ruler","The dumb model that becomes a ruler",[11,1162,1163,1164,1167,1168,1171,1172,1175],{},"Before any real regression, the second notebook (",[87,1165,1166],{},"aula02a",") does something smart: it splits off 20% of the data for testing (",[87,1169,1170],{},"train_test_split"," implemented by hand, ",[87,1173,1174],{},"random_state=42"," so it always gives the same result) and defines the laziest model that exists, always guessing the mean:",[96,1177,1179],{"className":98,"code":1178,"language":100,"meta":57,"style":57},"def modelo_media(y):\n    media = np.mean(y)\n    return np.ones_like(y) * media\n",[87,1180,1181,1186,1191],{"__ignoreMap":57},[104,1182,1183],{"class":106,"line":107},[104,1184,1185],{},"def modelo_media(y):\n",[104,1187,1188],{"class":106,"line":58},[104,1189,1190],{},"    media = np.mean(y)\n",[104,1192,1193],{"class":106,"line":118},[104,1194,1195],{},"    return np.ones_like(y) * media\n",[433,1197,1198],{},[11,1199,1200,1203],{},[15,1201,1202],{},"Output (353 training patients):"," MSE of guess (1500, 150) = 4646.20. MSE of \"always the mean\" = 5978.59.",[11,1205,1206,1207,1211],{},"Why do I need this? Because an MSE of 4646 on its own says nothing, it's just a number. Compared against the mean baseline, it's smaller, so the model is learning something. That's literally the definition of ",[445,1208,1210],{"definition":1209},"the fraction of the target's variance the model explains, compared to just always guessing the mean, where 1 is perfect, 0 is the same as guessing the mean, negative is worse than guessing the mean","R²",":",[96,1213,1215],{"className":98,"code":1214,"language":100,"meta":57,"style":57},"def r2_score(y, ypred):\n    ss_res = mse(y, ypred)\n    ss_tot = mse(y, modelo_media(y))\n    return 1 - (ss_res \u002F ss_tot)\n",[87,1216,1217,1222,1227,1232],{"__ignoreMap":57},[104,1218,1219],{"class":106,"line":107},[104,1220,1221],{},"def r2_score(y, ypred):\n",[104,1223,1224],{"class":106,"line":58},[104,1225,1226],{},"    ss_res = mse(y, ypred)\n",[104,1228,1229],{"class":106,"line":118},[104,1230,1231],{},"    ss_tot = mse(y, modelo_media(y))\n",[104,1233,1234],{"class":106,"line":124},[104,1235,1236],{},"    return 1 - (ss_res \u002F ss_tot)\n",[11,1238,1239],{},[104,1240,1242,1275],{"className":1241},[148],[104,1243,1245],{"className":1244},[152],[154,1246,1247],{"xmlns":156},[158,1248,1249,1272],{},[161,1250,1251,1258,1260,1262,1264],{},[539,1252,1253,1256],{},[164,1254,1255],{},"R",[483,1257,488],{},[168,1259,170],{},[483,1261,485],{},[168,1263,530],{},[480,1265,1266,1269],{},[932,1267,1268],{},"model MSE",[932,1270,1271],{},"baseline MSE",[186,1273,1274],{"encoding":188},"R^2 = 1 - \\frac{\\text{model MSE}}{\\text{baseline MSE}}",[104,1276,1278,1323,1342],{"className":1277,"ariaHidden":194},[193],[104,1279,1281,1284,1314,1317,1320],{"className":1280},[198],[104,1282],{"className":1283,"style":897},[202],[104,1285,1287,1291],{"className":1286},[207],[104,1288,1255],{"className":1289,"style":1290},[207,208],"margin-right:0.0077em;",[104,1292,1294],{"className":1293},[697],[104,1295,1297],{"className":1296},[605],[104,1298,1300],{"className":1299},[610],[104,1301,1303],{"className":1302,"style":897},[614],[104,1304,1305,1308],{"style":900},[104,1306],{"className":1307,"style":714},[622],[104,1309,1311],{"className":1310},[627,628,629,630],[104,1312,488],{"className":1313},[207,630],[104,1315],{"className":1316,"style":214},[213],[104,1318,170],{"className":1319},[218],[104,1321],{"className":1322,"style":214},[213],[104,1324,1326,1330,1333,1336,1339],{"className":1325},[198],[104,1327],{"className":1328,"style":1329},[202],"height:0.7278em;vertical-align:-0.0833em;",[104,1331,485],{"className":1332},[207],[104,1334],{"className":1335,"style":235},[213],[104,1337,530],{"className":1338},[239],[104,1340],{"className":1341,"style":235},[213],[104,1343,1345,1349],{"className":1344},[198],[104,1346],{"className":1347,"style":1348},[202],"height:1.2251em;vertical-align:-0.345em;",[104,1350,1352,1355,1421],{"className":1351},[207],[104,1353],{"className":1354},[566,598],[104,1356,1358],{"className":1357},[480],[104,1359,1361,1413],{"className":1360},[605,606],[104,1362,1364,1410],{"className":1363},[610],[104,1365,1368,1385,1393],{"className":1366,"style":1367},[614],"height:0.8801em;",[104,1369,1370,1373],{"style":618},[104,1371],{"className":1372,"style":623},[622],[104,1374,1376],{"className":1375},[627,628,629,630],[104,1377,1379],{"className":1378},[207,630],[104,1380,1382],{"className":1381},[207,1009,630],[104,1383,1271],{"className":1384},[207,630],[104,1386,1387,1390],{"style":639},[104,1388],{"className":1389,"style":623},[622],[104,1391],{"className":1392,"style":647},[646],[104,1394,1395,1398],{"style":650},[104,1396],{"className":1397,"style":623},[622],[104,1399,1401],{"className":1400},[627,628,629,630],[104,1402,1404],{"className":1403},[207,630],[104,1405,1407],{"className":1406},[207,1009,630],[104,1408,1268],{"className":1409},[207,630],[104,1411,667],{"className":1412},[666],[104,1414,1416],{"className":1415},[610],[104,1417,1419],{"className":1418,"style":674},[614],[104,1420],{},[104,1422],{"className":1423},[575,598],[11,1425,1426,1427,1430],{},"With guess (1500, 150): R² = 1 - 4646.20\u002F5978.59 = ",[15,1428,1429],{},"0.2229",". Just one variable explaining 22% of the variation is already a start, considering diabetes depends on a lot that isn't even in these 10 columns. And the baseline's own R², by definition, is always zero.",[79,1432,1434],{"id":1433},"scikit-learns-api-built-from-the-inside","Scikit-learn's API, built from the inside",[11,1436,1437,1438,1441,1442,1445],{},"This is where the lecture gets genuinely interesting. Instead of just calling a ready-made ",[87,1439,1440],{},"LinearRegression()",", the professor teaches how to ",[15,1443,1444],{},"build an estimator that follows the same convention",", inheriting from two base classes:",[96,1447,1449],{"className":98,"code":1448,"language":100,"meta":57,"style":57},"from sklearn.base import BaseEstimator, RegressorMixin\n\nclass AverageRegressor(BaseEstimator, RegressorMixin):\n    def fit(self, X, y):\n        self.media_ = np.mean(y)\n        return self\n    def predict(self, X):\n        return np.ones(shape=(X.shape[0],)) * self.media_\n",[87,1450,1451,1456,1460,1465,1470,1475,1480,1485],{"__ignoreMap":57},[104,1452,1453],{"class":106,"line":107},[104,1454,1455],{},"from sklearn.base import BaseEstimator, RegressorMixin\n",[104,1457,1458],{"class":106,"line":58},[104,1459,300],{"emptyLinePlaceholder":63},[104,1461,1462],{"class":106,"line":118},[104,1463,1464],{},"class AverageRegressor(BaseEstimator, RegressorMixin):\n",[104,1466,1467],{"class":106,"line":124},[104,1468,1469],{},"    def fit(self, X, y):\n",[104,1471,1472],{"class":106,"line":308},[104,1473,1474],{},"        self.media_ = np.mean(y)\n",[104,1476,1477],{"class":106,"line":417},[104,1478,1479],{},"        return self\n",[104,1481,1482],{"class":106,"line":422},[104,1483,1484],{},"    def predict(self, X):\n",[104,1486,1487],{"class":106,"line":428},[104,1488,1489],{},"        return np.ones(shape=(X.shape[0],)) * self.media_\n",[11,1491,1492,1495,1496,1499,1500,1503,1504,1507,1508,1511,1512,1515,1516,1519,1520,1523,1524,1531,1532,1534],{},[87,1493,1494],{},"BaseEstimator"," gives you a whole bunch of infrastructure for free (parameter comparison, cloning, integration with ",[87,1497,1498],{},"Pipeline"," and ",[87,1501,1502],{},"GridSearchCV",", things that'll show up later in the course). ",[87,1505,1506],{},"RegressorMixin"," adds the standard ",[87,1509,1510],{},".score()"," method. And notice the underscore convention: ",[87,1513,1514],{},"self.media_",", ending in ",[87,1517,1518],{},"_",", marks \"this was ",[15,1521,1522],{},"learned"," from the data,\" as opposed to a parameter configured by hand. It's the same detail I'd already seen from the outside, ",[20,1525,1526,1527,1530],{"href":353},"using ",[87,1528,1529],{},"SGDRegressor"," in the other playlist",", except now I'm writing the ",[87,1533,1518],{}," with my own hands.",[11,1536,1537,1538,1541],{},"The next step shows something subtle: a regressor with ",[15,1539,1540],{},"fixed"," coefficients, passed in the constructor, learning nothing at all:",[96,1543,1545],{"className":98,"code":1544,"language":100,"meta":57,"style":57},"class LinearRegressor(BaseEstimator, RegressorMixin):\n    def __init__(self, a, b):\n        self.a_ = a\n        self.b_ = b\n    def fit(self, X, y):\n        return self  # learns nothing, the coefficients already arrived ready\n    def predict(self, X):\n        return (self.a_*X + self.b_).reshape(X.shape[0],)\n",[87,1546,1547,1552,1557,1562,1567,1571,1576,1580],{"__ignoreMap":57},[104,1548,1549],{"class":106,"line":107},[104,1550,1551],{},"class LinearRegressor(BaseEstimator, RegressorMixin):\n",[104,1553,1554],{"class":106,"line":58},[104,1555,1556],{},"    def __init__(self, a, b):\n",[104,1558,1559],{"class":106,"line":118},[104,1560,1561],{},"        self.a_ = a\n",[104,1563,1564],{"class":106,"line":124},[104,1565,1566],{},"        self.b_ = b\n",[104,1568,1569],{"class":106,"line":308},[104,1570,1469],{},[104,1572,1573],{"class":106,"line":417},[104,1574,1575],{},"        return self  # learns nothing, the coefficients already arrived ready\n",[104,1577,1578],{"class":106,"line":422},[104,1579,1484],{},[104,1581,1582],{"class":106,"line":428},[104,1583,1584],{},"        return (self.a_*X + self.b_).reshape(X.shape[0],)\n",[11,1586,1587,1588,1211],{},"And then, the version that ",[15,1589,1590],{},"initializes randomly",[96,1592,1594],{"className":98,"code":1593,"language":100,"meta":57,"style":57},"class LinearRegressor(BaseEstimator, RegressorMixin):\n    def fit(self, X, y):\n        self.a_ = np.random.rand()\n        self.b_ = np.random.rand()\n        return self\n    # ...\n",[87,1595,1596,1600,1604,1609,1614,1618],{"__ignoreMap":57},[104,1597,1598],{"class":106,"line":107},[104,1599,1551],{},[104,1601,1602],{"class":106,"line":58},[104,1603,1469],{},[104,1605,1606],{"class":106,"line":118},[104,1607,1608],{},"        self.a_ = np.random.rand()\n",[104,1610,1611],{"class":106,"line":124},[104,1612,1613],{},"        self.b_ = np.random.rand()\n",[104,1615,1616],{"class":106,"line":308},[104,1617,1479],{},[104,1619,1620],{"class":106,"line":417},[104,1621,1622],{},"    # ...\n",[433,1624,1625],{},[11,1626,1627,1630],{},[15,1628,1629],{},"Output:"," MSE of the professor's guess (1500, 150) = 4646.20. MSE of the random coefficient = 29993.30.",[11,1632,1633,1634,1637],{},"That works out to an R² of ",[15,1635,1636],{},"-4.02"," for the random guess. Negative: worse than simply guessing the mean every time. It's proof that \"learning\" isn't magic, it's moving from a bad spot (random) toward a better one, and a random number on its own has no reason to be good.",[79,1639,1641],{"id":1640},"from-random-guess-to-gradient","From random guess to gradient",[11,1643,1644],{},"The last version of the class swaps \"random and stuck\" for \"random and walking\":",[96,1646,1648],{"className":98,"code":1647,"language":100,"meta":57,"style":57},"class LinearRegressor(BaseEstimator, RegressorMixin):\n    def __init__(self, max_iter=1000, learning_rate=0.001):\n        self.max_iter = max_iter\n        self.learning_rate = learning_rate\n\n    def fit(self, X, y):\n        self.coefs_ = np.random.rand(X.shape[1])\n        self.intercept_ = np.random.rand()\n        for i in range(self.max_iter):\n            y_pred = self.predict(X)\n            error = y - y_pred\n            self.coefs_ += X.T @ error * self.learning_rate\n            self.intercept_ += error.sum() * self.learning_rate\n        return self\n\n    def predict(self, X):\n        return (X @ self.coefs_ + self.intercept_).reshape(X.shape[0],)\n",[87,1649,1650,1654,1659,1664,1669,1673,1677,1682,1687,1693,1699,1705,1711,1717,1722,1727,1732],{"__ignoreMap":57},[104,1651,1652],{"class":106,"line":107},[104,1653,1551],{},[104,1655,1656],{"class":106,"line":58},[104,1657,1658],{},"    def __init__(self, max_iter=1000, learning_rate=0.001):\n",[104,1660,1661],{"class":106,"line":118},[104,1662,1663],{},"        self.max_iter = max_iter\n",[104,1665,1666],{"class":106,"line":124},[104,1667,1668],{},"        self.learning_rate = learning_rate\n",[104,1670,1671],{"class":106,"line":308},[104,1672,300],{"emptyLinePlaceholder":63},[104,1674,1675],{"class":106,"line":417},[104,1676,1469],{},[104,1678,1679],{"class":106,"line":422},[104,1680,1681],{},"        self.coefs_ = np.random.rand(X.shape[1])\n",[104,1683,1684],{"class":106,"line":428},[104,1685,1686],{},"        self.intercept_ = np.random.rand()\n",[104,1688,1690],{"class":106,"line":1689},9,[104,1691,1692],{},"        for i in range(self.max_iter):\n",[104,1694,1696],{"class":106,"line":1695},10,[104,1697,1698],{},"            y_pred = self.predict(X)\n",[104,1700,1702],{"class":106,"line":1701},11,[104,1703,1704],{},"            error = y - y_pred\n",[104,1706,1708],{"class":106,"line":1707},12,[104,1709,1710],{},"            self.coefs_ += X.T @ error * self.learning_rate\n",[104,1712,1714],{"class":106,"line":1713},13,[104,1715,1716],{},"            self.intercept_ += error.sum() * self.learning_rate\n",[104,1718,1720],{"class":106,"line":1719},14,[104,1721,1479],{},[104,1723,1725],{"class":106,"line":1724},15,[104,1726,300],{"emptyLinePlaceholder":63},[104,1728,1730],{"class":106,"line":1729},16,[104,1731,1484],{},[104,1733,1735],{"class":106,"line":1734},17,[104,1736,1737],{},"        return (X @ self.coefs_ + self.intercept_).reshape(X.shape[0],)\n",[11,1739,1740,1741,1745,1746,1750,1751,1754,1755,1758,1759,1762],{},"If you've already gone through ",[20,1742,1744],{"href":1743},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Flab04-gradient-descent","the specialization playlist",", this math looks familiar: it's the same batch ",[445,1747,1749],{"definition":1748},"the algorithm that repeatedly adjusts parameters in the direction that most reduces the error, until it stops improving","gradient descent"," as always, ",[87,1752,1753],{},"coefficient += X.T @ error * rate",", just without dividing by the number of examples (which only pushes the effect into the ",[87,1756,1757],{},"learning_rate",", a slightly smaller one compensates for the missing division). This specific shape, updating a weight proportionally to the error times the input, has its own name in the literature: the ",[15,1760,1761],{},"delta rule"," (or Widrow-Hoff rule), one of the oldest learning algorithms that exists, decades before the term \"gradient descent\" caught on.",[11,1764,1765],{},"Run it live below, on the same sample of 45 patients, and watch it start far off and converge toward the ideal fit:",[1767,1768],"gradient-descent-simulator",{":alpha-presets":1769,":alpha-slider-max":1770,":alpha-slider-min":1771,":alpha-slider-step":1771,":b-range":1772,":initial-alpha":1773,":initial-b":331,":initial-w":331,":w-range":1774,":x-train":337,":y-train":338,"b-label":184,"w-label":20},"[0.01, 0.1, 0.5, 1, 1.5]","1.8","0.01","[-100, 300]","0.5","[0, 1000]",[11,1776,1777,1778,1781,1782,1785,1786,1788,1789,1791,1792,1794,1795,1797],{},"On this 45-point sample (smaller than the full 353-patient training set, so the ideal fit here isn't exactly the number the lecture reports), with ",[87,1779,1780],{},"alpha=0.5"," clicking \"Rodar 2000\" already lands close to the bottom of the valley. Try a bigger ",[87,1783,1784],{},"alpha"," too, like 1.5: ",[87,1787,184],{}," starts oscillating before converging, because the scale of ",[87,1790,20],{}," (multiplied by a tiny ",[87,1793,178],{},", between -0.09 and 0.17) and the scale of ",[87,1796,184],{}," (added directly) are quite different, the same scale problem normalization solves, just showing up here hidden inside a dataset that's already normalized.",[11,1799,1800],{},"On the full training set (353 patients), running this same class with a single feature (body mass index):",[433,1802,1803],{},[11,1804,1805,1807,1808,1811,1812,1815],{},[15,1806,1629],{}," ",[87,1809,1810],{},"coefs_ = [529.65]",", ",[87,1813,1814],{},"intercept_ = 154.82",", MSE = 4302.16.",[11,1817,1818,1819,1822,1823,1499,1825,1827],{},"That already beats the best manual guess (4646.20 → 4302.16) and works out to R² = 1 - 4302.16\u002F5978.59 = ",[15,1820,1821],{},"0.2804",". Without me picking ",[87,1824,20],{},[87,1826,184],{}," by hand, the gradient found a better fit on its own.",[79,1829,1831],{"id":1830},"from-one-variable-to-ten","From one variable to ten",[11,1833,1834,1835,1211],{},"The dataset has 10 columns, and so far I've only used one (body mass index). The exact same class, no code changes at all, accepts the full ",[87,1836,1837],{},"X_train",[96,1839,1841],{"className":98,"code":1840,"language":100,"meta":57,"style":57},"regressor = LinearRegressor()\nregressor.fit(X_train, y_train)  # now with all 10 columns\n",[87,1842,1843,1848],{"__ignoreMap":57},[104,1844,1845],{"class":106,"line":107},[104,1846,1847],{},"regressor = LinearRegressor()\n",[104,1849,1850],{"class":106,"line":58},[104,1851,1852],{},"regressor.fit(X_train, y_train)  # now with all 10 columns\n",[433,1854,1855],{},[11,1856,1857,1859,1860,1863],{},[15,1858,1629],{}," MSE = 3027.12 (versus 4302.16 using just one variable). RMSE drops from 65.59 to 55.02. R² climbs from 0.2804 to ",[15,1861,1862],{},"0.4937",".",[11,1865,1866],{},"Almost double the explained variance, just from letting the model see the other 9 variables (age, sex, blood pressure, the six blood measurements) it simply didn't have access to before. Nobody changed the algorithm, only the amount of information it received.",[79,1868,1870],{"id":1869},"wrapping-up","Wrapping up",[1872,1873,1874,1888],"table",{},[1875,1876,1877],"thead",{},[1878,1879,1880,1885],"tr",{},[1881,1882,1884],"th",{"align":1883},"left","What I already knew",[1881,1886,1887],{"align":1883},"What this lecture settled",[1889,1890,1891,1904,1912],"tbody",{},[1878,1892,1893,1897],{},[1894,1895,1896],"td",{"align":1883},"Linear regression fits a line that minimizes error",[1894,1898,1899,318,1901,1903],{"align":1883},[87,1900,1494],{},[87,1902,1506],{}," is the contract that makes any class of mine behave like a real scikit-learn estimator",[1878,1905,1906,1909],{},[1894,1907,1908],{"align":1883},"MAE, MSE, RMSE measure how wrong the model is",[1894,1910,1911],{"align":1883},"Without comparing against a baseline (like guessing the mean), those numbers say nothing on their own: R² fixes that",[1878,1913,1914,1917],{},[1894,1915,1916],{"align":1883},"Gradient descent adjusts weights in the direction that reduces error",[1894,1918,1919],{"align":1883},"That update has its own name in the classic literature (delta rule), and \"learning\" is nothing more than walking from a bad guess in the right direction",[11,1921,1922,1923,1931,1932,1935],{},"Two things stayed open on purpose. First: solving this by iterating, step by step, works, but linear regression has an exact, closed-form solution in a single computation, and that's exactly what ",[20,1924,1926,1927,1930],{"href":1925},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation","the next lecture (",[87,1928,1929],{},"aula02b",", normal equation) tackles",". Second: I didn't touch the other models that lecture compares (",[87,1933,1934],{},"aula02c","), that's next post material too.",[79,1937,1939],{"id":1938},"practical-application","Practical application",[11,1941,1942],{},"I use the same diabetes dataset that's been running through this whole post, without introducing anything new: the goal here is just to visualize how good the 10-variable fit I just showed actually is.",[96,1944,1946],{"className":98,"code":1945,"language":100,"meta":57,"style":57},"regressor = LinearRegressor(max_iter=1000, learning_rate=0.001)\nregressor.fit(X_train, y_train)\ny_pred = regressor.predict(X_train)\n",[87,1947,1948,1953,1958],{"__ignoreMap":57},[104,1949,1950],{"class":106,"line":107},[104,1951,1952],{},"regressor = LinearRegressor(max_iter=1000, learning_rate=0.001)\n",[104,1954,1955],{"class":106,"line":58},[104,1956,1957],{},"regressor.fit(X_train, y_train)\n",[104,1959,1960],{"class":106,"line":118},[104,1961,1962],{},"y_pred = regressor.predict(X_train)\n",[11,1964,1965,1966,1969],{},"I reproduced this same class (with a different random seed, so the coefficients only match the professor's to the second decimal place, final MSE landed at 3027.43, practically identical to the 3027.12 the lecture reports) to pull out the actual-versus-predicted pairs for 40 patients from the training set. Each point is one patient: the closer to the dashed line (the perfect prediction, ",[87,1967,1968],{},"predicted = actual","), the closer the model got that specific case.",[1971,1972],"predicted-vs-actual-scatter",{":actual":1973,":predicted":1974,"point-label":339,"reference-label":1975,"x-label":1976,"y-label":1977},"[42, 51, 52, 59, 60, 67, 71, 72, 72, 77, 77, 78, 83, 89, 96, 102, 104, 144, 146, 150, 150, 150, 170, 171, 172, 174, 179, 190, 198, 212, 219, 262, 265, 275, 276, 281, 292, 308, 317, 341]","[143.2, 80.1, 78.4, 120.6, 140.3, 183.2, 95.9, 94.7, 92.1, 91.1, 174.5, 86.5, 138.4, 157.2, 132.4, 125.7, 94.1, 165.1, 157.4, 163.9, 111.3, 204.9, 172.6, 183.3, 140.6, 172.7, 175.6, 122.7, 192.1, 198.0, 160.9, 177.3, 185.5, 198.8, 143.2, 236.4, 209.2, 234.2, 230.8, 242.5]","Perfect prediction","actual diabetes progression","predicted progression",[11,1979,1980],{},"You can see the cloud of points tracking the diagonal, but with plenty of spread, exactly what an R² of 0.49 means in practice: the model captured a good chunk of the pattern, but still misses quite a bit case by case. Not bad at all for a plain linear model, no new features, no regularization, just 10 raw numbers and 1000 steps of gradient descent.",[1982,1983,1984],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":57,"searchDepth":58,"depth":58,"links":1986},[1987,1988,1989,1990,1991,1992,1993,1994,1995],{"id":81,"depth":58,"text":82},{"id":141,"depth":58,"text":142},{"id":346,"depth":58,"text":347},{"id":1159,"depth":58,"text":1160},{"id":1433,"depth":58,"text":1434},{"id":1640,"depth":58,"text":1641},{"id":1830,"depth":58,"text":1831},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"2026-08-19","Lecture one of my Pattern Recognition course: the professor grabs a real diabetes dataset, hand-picks coefficients, and builds an estimator following scikit-learn's own API. I retell the why behind each step, with Bishop's book alongside.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","pattern-recognition",{"title":72,"description":1997},"en\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator",[2004,2005,2006],"linear-regression","scikit-learn","gradient-descent","L_iUimVMvKneVDUVJ45usLRO5OFHAmN8SYGmLbylCqY",{"id":2009,"title":2010,"body":2011,"cover":3,"date":1996,"description":4089,"extension":61,"meta":4090,"navigation":63,"order":58,"path":1925,"playlist":2000,"seo":4091,"status":66,"stem":4092,"tags":4093,"__hash__":4096},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation.md","The Normal Equation: Solving Regression in One Shot",{"type":8,"value":2012,"toc":4078},[2013,2019,2023,2026,2033,2509,2515,2771,2778,2782,3116,3121,3136,3158,3162,3172,3241,3255,3262,3489,3533,3609,3613,3619,3657,3668,3674,3678,3685,3787,3790,3824,3829,3831,4031,4033,4045,4063,4069,4076],[11,2014,2015,2018],{},[20,2016,2017],{"href":1999},"The previous post"," left a hook: you can solve linear regression in a single computation, no gradient descent needed. That's exactly what lectures 2b and 2c show, and along the way the professor compares the result against five other very different kinds of regressor.",[79,2020,2022],{"id":2021},"zeroing-the-derivative-instead-of-walking-toward-it","Zeroing the derivative instead of walking toward it",[11,2024,2025],{},"Gradient descent works because, at every step, it walks a little in the direction that reduces the error. But for linear regression, the error function (the same sum of squares I'd already been using) has a special property: it's a parabola with respect to the weights, a smooth bowl with no false valleys. And a smooth bowl has exactly one point where the derivative is zero, its bottom. Instead of walking there step by step, you can compute that point directly.",[11,2027,2028,2029,2032],{},"Bishop writes the linear model in a more general form than mine, with a matrix Φ (the ",[15,2030,2031],{},"design matrix",", one row per patient, one column per input variable, plus a column of 1's for the bias term). The error function is",[11,2034,2035],{},[104,2036,2038,2123],{"className":2037},[148],[104,2039,2041],{"className":2040},[152],[154,2042,2043],{"xmlns":156},[158,2044,2045,2120],{},[161,2046,2047,2054,2056,2058,2060,2062,2068,2082,2084,2090,2092,2100,2104,2106,2112,2114],{},[515,2048,2049,2051],{},[164,2050,466],{},[164,2052,2053],{},"D",[168,2055,470],{"stretchy":469},[164,2057,317],{"mathvariant":473},[168,2059,476],{"stretchy":469},[168,2061,170],{},[480,2063,2064,2066],{},[483,2065,485],{},[483,2067,488],{},[490,2069,2070,2072,2080],{},[168,2071,494],{},[161,2073,2074,2076,2078],{},[164,2075,499],{},[168,2077,170],{},[483,2079,485],{},[164,2081,506],{},[168,2083,509],{"stretchy":469},[515,2085,2086,2088],{},[164,2087,535],{},[164,2089,499],{},[168,2091,530],{},[539,2093,2094,2096],{},[164,2095,317],{"mathvariant":473},[164,2097,2099],{"mathvariant":2098},"sans-serif","T",[164,2101,2103],{"mathvariant":2102},"bold-italic","ϕ",[168,2105,470],{"stretchy":469},[515,2107,2108,2110],{},[164,2109,178],{"mathvariant":473},[164,2111,499],{},[168,2113,476],{"stretchy":469},[539,2115,2116,2118],{},[168,2117,543],{"stretchy":469},[483,2119,488],{},[186,2121,2122],{"encoding":188},"E_D(\\mathbf{w}) = \\frac{1}{2}\\sum_{n=1}^{N}\\{t_n - \\mathbf{w}^{\\mathsf{T}}\\boldsymbol{\\phi}(\\mathbf{x}_n)\\}^2",[104,2124,2126,2191,2383],{"className":2125,"ariaHidden":194},[193],[104,2127,2129,2132,2173,2176,2179,2182,2185,2188],{"className":2128},[198],[104,2130],{"className":2131,"style":558},[202],[104,2133,2135,2138],{"className":2134},[207],[104,2136,466],{"className":2137,"style":562},[207,208],[104,2139,2141],{"className":2140},[697],[104,2142,2144,2165],{"className":2143},[605,606],[104,2145,2147,2162],{"className":2146},[610],[104,2148,2150],{"className":2149,"style":993},[614],[104,2151,2152,2155],{"style":996},[104,2153],{"className":2154,"style":714},[622],[104,2156,2158],{"className":2157},[627,628,629,630],[104,2159,2053],{"className":2160,"style":2161},[207,208,630],"margin-right:0.0278em;",[104,2163,667],{"className":2164},[666],[104,2166,2168],{"className":2167},[610],[104,2169,2171],{"className":2170,"style":807},[614],[104,2172],{},[104,2174,470],{"className":2175},[566],[104,2177,317],{"className":2178,"style":571},[207,570],[104,2180,476],{"className":2181},[575],[104,2183],{"className":2184,"style":214},[213],[104,2186,170],{"className":2187},[218],[104,2189],{"className":2190,"style":214},[213],[104,2192,2194,2197,2265,2268,2331,2334,2374,2377,2380],{"className":2193},[198],[104,2195],{"className":2196,"style":591},[202],[104,2198,2200,2203,2262],{"className":2199},[207],[104,2201],{"className":2202},[566,598],[104,2204,2206],{"className":2205},[480],[104,2207,2209,2254],{"className":2208},[605,606],[104,2210,2212,2251],{"className":2211},[610],[104,2213,2215,2229,2237],{"className":2214,"style":615},[614],[104,2216,2217,2220],{"style":618},[104,2218],{"className":2219,"style":623},[622],[104,2221,2223],{"className":2222},[627,628,629,630],[104,2224,2226],{"className":2225},[207,630],[104,2227,488],{"className":2228},[207,630],[104,2230,2231,2234],{"style":639},[104,2232],{"className":2233,"style":623},[622],[104,2235],{"className":2236,"style":647},[646],[104,2238,2239,2242],{"style":650},[104,2240],{"className":2241,"style":623},[622],[104,2243,2245],{"className":2244},[627,628,629,630],[104,2246,2248],{"className":2247},[207,630],[104,2249,485],{"className":2250},[207,630],[104,2252,667],{"className":2253},[666],[104,2255,2257],{"className":2256},[610],[104,2258,2260],{"className":2259,"style":674},[614],[104,2261],{},[104,2263],{"className":2264},[575,598],[104,2266],{"className":2267,"style":683},[213],[104,2269,2271,2274],{"className":2270},[687],[104,2272,494],{"className":2273,"style":693},[687,691,692],[104,2275,2277],{"className":2276},[697],[104,2278,2280,2323],{"className":2279},[605,606],[104,2281,2283,2320],{"className":2282},[610],[104,2284,2286,2306],{"className":2285,"style":707},[614],[104,2287,2288,2291],{"style":710},[104,2289],{"className":2290,"style":714},[622],[104,2292,2294],{"className":2293},[627,628,629,630],[104,2295,2297,2300,2303],{"className":2296},[207,630],[104,2298,499],{"className":2299},[207,208,630],[104,2301,170],{"className":2302},[218,630],[104,2304,485],{"className":2305},[207,630],[104,2307,2308,2311],{"style":732},[104,2309],{"className":2310,"style":714},[622],[104,2312,2314],{"className":2313},[627,628,629,630],[104,2315,2317],{"className":2316},[207,630],[104,2318,506],{"className":2319,"style":745},[207,208,630],[104,2321,667],{"className":2322},[666],[104,2324,2326],{"className":2325},[610],[104,2327,2329],{"className":2328,"style":755},[614],[104,2330],{},[104,2332,509],{"className":2333},[566],[104,2335,2337,2340],{"className":2336},[207],[104,2338,535],{"className":2339},[207,208],[104,2341,2343],{"className":2342},[697],[104,2344,2346,2366],{"className":2345},[605,606],[104,2347,2349,2363],{"className":2348},[610],[104,2350,2352],{"className":2351,"style":785},[614],[104,2353,2354,2357],{"style":788},[104,2355],{"className":2356,"style":714},[622],[104,2358,2360],{"className":2359},[627,628,629,630],[104,2361,499],{"className":2362},[207,208,630],[104,2364,667],{"className":2365},[666],[104,2367,2369],{"className":2368},[610],[104,2370,2372],{"className":2371,"style":807},[614],[104,2373],{},[104,2375],{"className":2376,"style":235},[213],[104,2378,530],{"className":2379},[239],[104,2381],{"className":2382,"style":235},[213],[104,2384,2386,2390,2424,2434,2437,2477,2480],{"className":2385},[198],[104,2387],{"className":2388,"style":2389},[202],"height:1.0991em;vertical-align:-0.25em;",[104,2391,2393,2396],{"className":2392},[207],[104,2394,317],{"className":2395,"style":571},[207,570],[104,2397,2399],{"className":2398},[697],[104,2400,2402],{"className":2401},[605],[104,2403,2405],{"className":2404},[610],[104,2406,2409],{"className":2407,"style":2408},[614],"height:0.8491em;",[104,2410,2411,2414],{"style":900},[104,2412],{"className":2413,"style":714},[622],[104,2415,2417],{"className":2416},[627,628,629,630],[104,2418,2420],{"className":2419},[207,630],[104,2421,2099],{"className":2422},[207,2423,630],"mathsf",[104,2425,2427],{"className":2426},[207],[104,2428,2430],{"className":2429},[207],[104,2431,2103],{"className":2432},[207,2433],"boldsymbol",[104,2435,470],{"className":2436},[566],[104,2438,2440,2443],{"className":2439},[207],[104,2441,178],{"className":2442},[207,570],[104,2444,2446],{"className":2445},[697],[104,2447,2449,2469],{"className":2448},[605,606],[104,2450,2452,2466],{"className":2451},[610],[104,2453,2455],{"className":2454,"style":785},[614],[104,2456,2457,2460],{"style":788},[104,2458],{"className":2459,"style":714},[622],[104,2461,2463],{"className":2462},[627,628,629,630],[104,2464,499],{"className":2465},[207,208,630],[104,2467,667],{"className":2468},[666],[104,2470,2472],{"className":2471},[610],[104,2473,2475],{"className":2474,"style":807},[614],[104,2476],{},[104,2478,476],{"className":2479},[575],[104,2481,2483,2486],{"className":2482},[575],[104,2484,543],{"className":2485},[575],[104,2487,2489],{"className":2488},[697],[104,2490,2492],{"className":2491},[605],[104,2493,2495],{"className":2494},[610],[104,2496,2498],{"className":2497,"style":897},[614],[104,2499,2500,2503],{"style":900},[104,2501],{"className":2502,"style":714},[622],[104,2504,2506],{"className":2505},[627,628,629,630],[104,2507,488],{"className":2508},[207,630],[11,2510,2511,2512,2514],{},"Setting this function's gradient to zero and solving for ",[87,2513,317],{}," leaves",[11,2516,2517],{},[104,2518,2520,2570],{"className":2519},[148],[104,2521,2523],{"className":2522},[152],[154,2524,2525],{"xmlns":156},[158,2526,2527,2567],{},[161,2528,2529,2536,2538,2540,2547,2549,2559,2565],{},[515,2530,2531,2533],{},[164,2532,317],{"mathvariant":473},[932,2534,2535],{},"ML",[168,2537,170],{},[168,2539,470],{"stretchy":469},[539,2541,2542,2545],{},[164,2543,2544],{"mathvariant":473},"Φ",[164,2546,2099],{"mathvariant":2098},[164,2548,2544],{"mathvariant":473},[539,2550,2551,2553],{},[168,2552,476],{"stretchy":469},[161,2554,2555,2557],{},[168,2556,530],{},[483,2558,485],{},[539,2560,2561,2563],{},[164,2562,2544],{"mathvariant":473},[164,2564,2099],{"mathvariant":2098},[164,2566,535],{"mathvariant":473},[186,2568,2569],{"encoding":188},"\\mathbf{w}_{\\text{ML}} = (\\boldsymbol{\\Phi}^{\\mathsf{T}}\\boldsymbol{\\Phi})^{-1}\\boldsymbol{\\Phi}^{\\mathsf{T}}\\mathbf{t}",[104,2571,2573,2636],{"className":2572,"ariaHidden":194},[193],[104,2574,2576,2580,2627,2630,2633],{"className":2575},[198],[104,2577],{"className":2578,"style":2579},[202],"height:0.5944em;vertical-align:-0.15em;",[104,2581,2583,2586],{"className":2582},[207],[104,2584,317],{"className":2585,"style":571},[207,570],[104,2587,2589],{"className":2588},[697],[104,2590,2592,2619],{"className":2591},[605,606],[104,2593,2595,2616],{"className":2594},[610],[104,2596,2598],{"className":2597,"style":993},[614],[104,2599,2601,2604],{"style":2600},"top:-2.55em;margin-left:-0.016em;margin-right:0.05em;",[104,2602],{"className":2603,"style":714},[622],[104,2605,2607],{"className":2606},[627,628,629,630],[104,2608,2610],{"className":2609},[207,630],[104,2611,2613],{"className":2612},[207,1009,630],[104,2614,2535],{"className":2615},[207,630],[104,2617,667],{"className":2618},[666],[104,2620,2622],{"className":2621},[610],[104,2623,2625],{"className":2624,"style":807},[614],[104,2626],{},[104,2628],{"className":2629,"style":214},[213],[104,2631,170],{"className":2632},[218],[104,2634],{"className":2635,"style":214},[213],[104,2637,2639,2643,2646,2686,2695,2730,2768],{"className":2638},[198],[104,2640],{"className":2641,"style":2642},[202],"height:1.1751em;vertical-align:-0.25em;",[104,2644,470],{"className":2645},[566],[104,2647,2649,2658],{"className":2648},[207],[104,2650,2652],{"className":2651},[207],[104,2653,2655],{"className":2654},[207],[104,2656,2544],{"className":2657},[207,570],[104,2659,2661],{"className":2660},[697],[104,2662,2664],{"className":2663},[605],[104,2665,2667],{"className":2666},[610],[104,2668,2671],{"className":2669,"style":2670},[614],"height:0.9251em;",[104,2672,2674,2677],{"style":2673},"top:-3.139em;margin-right:0.05em;",[104,2675],{"className":2676,"style":714},[622],[104,2678,2680],{"className":2679},[627,628,629,630],[104,2681,2683],{"className":2682},[207,630],[104,2684,2099],{"className":2685},[207,2423,630],[104,2687,2689],{"className":2688},[207],[104,2690,2692],{"className":2691},[207],[104,2693,2544],{"className":2694},[207,570],[104,2696,2698,2701],{"className":2697},[575],[104,2699,476],{"className":2700},[575],[104,2702,2704],{"className":2703},[697],[104,2705,2707],{"className":2706},[605],[104,2708,2710],{"className":2709},[610],[104,2711,2713],{"className":2712,"style":897},[614],[104,2714,2715,2718],{"style":900},[104,2716],{"className":2717,"style":714},[622],[104,2719,2721],{"className":2720},[627,628,629,630],[104,2722,2724,2727],{"className":2723},[207,630],[104,2725,530],{"className":2726},[207,630],[104,2728,485],{"className":2729},[207,630],[104,2731,2733,2742],{"className":2732},[207],[104,2734,2736],{"className":2735},[207],[104,2737,2739],{"className":2738},[207],[104,2740,2544],{"className":2741},[207,570],[104,2743,2745],{"className":2744},[697],[104,2746,2748],{"className":2747},[605],[104,2749,2751],{"className":2750},[610],[104,2752,2754],{"className":2753,"style":2670},[614],[104,2755,2756,2759],{"style":2673},[104,2757],{"className":2758,"style":714},[622],[104,2760,2762],{"className":2761},[627,628,629,630],[104,2763,2765],{"className":2764},[207,630],[104,2766,2099],{"className":2767},[207,2423,630],[104,2769,535],{"className":2770},[207,570],[11,2772,2773,2774,2777],{},"This is the ",[15,2775,2776],{},"normal equation",". One matrix, two multiplications, and an inversion, and the optimal fit falls right out, no learning rate, no choosing a number of iterations, none of the care gradient descent demands.",[79,2779,2781],{"id":2780},"the-trick-of-folding-bias-in","The trick of folding bias in",[11,2783,2784,2785,2788,2789,2791,2792,2866,2867,2975,2976,3045,3046,3115],{},"Notice the formula above has no separate ",[87,2786,2787],{},"+ b",", the bias lives inside ",[87,2790,317],{}," itself (Bishop calls it ",[104,2793,2795,2813],{"className":2794},[148],[104,2796,2798],{"className":2797},[152],[154,2799,2800],{"xmlns":156},[158,2801,2802,2810],{},[161,2803,2804],{},[515,2805,2806,2808],{},[164,2807,317],{},[483,2809,331],{},[186,2811,2812],{"encoding":188},"w_0",[104,2814,2816],{"className":2815,"ariaHidden":194},[193],[104,2817,2819,2823],{"className":2818},[198],[104,2820],{"className":2821,"style":2822},[202],"height:0.5806em;vertical-align:-0.15em;",[104,2824,2826,2830],{"className":2825},[207],[104,2827,317],{"className":2828,"style":2829},[207,208],"margin-right:0.0269em;",[104,2831,2833],{"className":2832},[697],[104,2834,2836,2858],{"className":2835},[605,606],[104,2837,2839,2855],{"className":2838},[610],[104,2840,2843],{"className":2841,"style":2842},[614],"height:0.3011em;",[104,2844,2846,2849],{"style":2845},"top:-2.55em;margin-left:-0.0269em;margin-right:0.05em;",[104,2847],{"className":2848,"style":714},[622],[104,2850,2852],{"className":2851},[627,628,629,630],[104,2853,331],{"className":2854},[207,630],[104,2856,667],{"className":2857},[666],[104,2859,2861],{"className":2860},[610],[104,2862,2864],{"className":2863,"style":807},[614],[104,2865],{},"). That only works because he defines a fake \"basis function,\" ",[104,2868,2870,2898],{"className":2869},[148],[104,2871,2873],{"className":2872},[152],[154,2874,2875],{"xmlns":156},[158,2876,2877,2895],{},[161,2878,2879,2885,2887,2889,2891,2893],{},[515,2880,2881,2883],{},[164,2882,2103],{},[483,2884,331],{},[168,2886,470],{"stretchy":469},[164,2888,178],{"mathvariant":473},[168,2890,476],{"stretchy":469},[168,2892,170],{},[483,2894,485],{},[186,2896,2897],{"encoding":188},"\\phi_0(\\mathbf{x}) = 1",[104,2899,2901,2965],{"className":2900,"ariaHidden":194},[193],[104,2902,2904,2907,2947,2950,2953,2956,2959,2962],{"className":2903},[198],[104,2905],{"className":2906,"style":558},[202],[104,2908,2910,2913],{"className":2909},[207],[104,2911,2103],{"className":2912},[207,208],[104,2914,2916],{"className":2915},[697],[104,2917,2919,2939],{"className":2918},[605,606],[104,2920,2922,2936],{"className":2921},[610],[104,2923,2925],{"className":2924,"style":2842},[614],[104,2926,2927,2930],{"style":788},[104,2928],{"className":2929,"style":714},[622],[104,2931,2933],{"className":2932},[627,628,629,630],[104,2934,331],{"className":2935},[207,630],[104,2937,667],{"className":2938},[666],[104,2940,2942],{"className":2941},[610],[104,2943,2945],{"className":2944,"style":807},[614],[104,2946],{},[104,2948,470],{"className":2949},[566],[104,2951,178],{"className":2952},[207,570],[104,2954,476],{"className":2955},[575],[104,2957],{"className":2958,"style":214},[213],[104,2960,170],{"className":2961},[218],[104,2963],{"className":2964,"style":214},[213],[104,2966,2968,2972],{"className":2967},[198],[104,2969],{"className":2970,"style":2971},[202],"height:0.6444em;",[104,2973,485],{"className":2974},[207],", an entire column of 1's, so multiplying it by ",[104,2977,2979,2996],{"className":2978},[148],[104,2980,2982],{"className":2981},[152],[154,2983,2984],{"xmlns":156},[158,2985,2986,2994],{},[161,2987,2988],{},[515,2989,2990,2992],{},[164,2991,317],{},[483,2993,331],{},[186,2995,2812],{"encoding":188},[104,2997,2999],{"className":2998,"ariaHidden":194},[193],[104,3000,3002,3005],{"className":3001},[198],[104,3003],{"className":3004,"style":2822},[202],[104,3006,3008,3011],{"className":3007},[207],[104,3009,317],{"className":3010,"style":2829},[207,208],[104,3012,3014],{"className":3013},[697],[104,3015,3017,3037],{"className":3016},[605,606],[104,3018,3020,3034],{"className":3019},[610],[104,3021,3023],{"className":3022,"style":2842},[614],[104,3024,3025,3028],{"style":2845},[104,3026],{"className":3027,"style":714},[622],[104,3029,3031],{"className":3030},[627,628,629,630],[104,3032,331],{"className":3033},[207,630],[104,3035,667],{"className":3036},[666],[104,3038,3040],{"className":3039},[610],[104,3041,3043],{"className":3042,"style":807},[614],[104,3044],{}," gives exactly ",[104,3047,3049,3066],{"className":3048},[148],[104,3050,3052],{"className":3051},[152],[154,3053,3054],{"xmlns":156},[158,3055,3056,3064],{},[161,3057,3058],{},[515,3059,3060,3062],{},[164,3061,317],{},[483,3063,331],{},[186,3065,2812],{"encoding":188},[104,3067,3069],{"className":3068,"ariaHidden":194},[193],[104,3070,3072,3075],{"className":3071},[198],[104,3073],{"className":3074,"style":2822},[202],[104,3076,3078,3081],{"className":3077},[207],[104,3079,317],{"className":3080,"style":2829},[207,208],[104,3082,3084],{"className":3083},[697],[104,3085,3087,3107],{"className":3086},[605,606],[104,3088,3090,3104],{"className":3089},[610],[104,3091,3093],{"className":3092,"style":2842},[614],[104,3094,3095,3098],{"style":2845},[104,3096],{"className":3097,"style":714},[622],[104,3099,3101],{"className":3100},[627,628,629,630],[104,3102,331],{"className":3103},[207,630],[104,3105,667],{"className":3106},[666],[104,3108,3110],{"className":3109},[610],[104,3111,3113],{"className":3112,"style":807},[614],[104,3114],{}," for every patient, the same effect as adding a fixed bias.",[11,3117,3118,3119,1211],{},"The code does this by hand, gluing a column of 1's onto the front of ",[87,3120,133],{},[96,3122,3124],{"className":98,"code":3123,"language":100,"meta":57,"style":57},"def include_bias(X):\n    return np.hstack((np.ones((X.shape[0], 1)), X))\n",[87,3125,3126,3131],{"__ignoreMap":57},[104,3127,3128],{"class":106,"line":107},[104,3129,3130],{},"def include_bias(X):\n",[104,3132,3133],{"class":106,"line":58},[104,3134,3135],{},"    return np.hstack((np.ones((X.shape[0], 1)), X))\n",[11,3137,3138,3139,1499,3142,3145,3146,3149,3150,3153,3154,3157],{},"That changes the class's shape: instead of storing ",[87,3140,3141],{},"coefs_",[87,3143,3144],{},"intercept_"," as two separate things (",[20,3147,3148],{"href":1999},"like in the previous post","), there's now a single vector ",[87,3151,3152],{},"w_",", where ",[87,3155,3156],{},"w_[0]"," is the bias and the rest are each variable's coefficient.",[79,3159,3161],{"id":3160},"three-versions-one-real-comparison","Three versions, one real comparison",[11,3163,3164,3165,3168,3169,3171],{},"The notebook runs three variations of the same ",[87,3166,3167],{},"LinearRegressor"," class, all on the same diabetes dataset (now using scikit-learn's real ",[87,3170,1170],{},", so the numbers don't match the previous post's exactly, which used my own hand-rolled split):",[1872,3173,3174,3188],{},[1875,3175,3176],{},[1878,3177,3178,3181,3184],{},[1881,3179,3180],{"align":1883},"Version",[1881,3182,3183],{"align":1883},"How it solves",[1881,3185,3187],{"align":3186},"right","MSE (train)",[1889,3189,3190,3206,3223],{},[1878,3191,3192,3195,3203],{},[1894,3193,3194],{"align":1883},"Gradient, separate bias",[1894,3196,3197,1811,3200,3202],{"align":1883},[87,3198,3199],{},"coefs_ += X.T@error*0.001",[87,3201,3144],{}," on the side",[1894,3204,3205],{"align":3186},"3142.25",[1878,3207,3208,3211,3220],{},[1894,3209,3210],{"align":1883},"Gradient, bias folded in",[1894,3212,3213,3214,1499,3217],{"align":1883},"same idea, but with ",[87,3215,3216],{},"include_bias",[87,3218,3219],{},"learning_rate=0.005",[1894,3221,3222],{"align":3186},"2898.90",[1878,3224,3225,3230,3236],{},[1894,3226,3227],{"align":1883},[15,3228,3229],{},"Normal equation",[1894,3231,3232,3235],{"align":1883},[87,3233,3234],{},"w_ = np.linalg.pinv(X) @ y",", no iterating",[1894,3237,3238],{"align":3186},[15,3239,3240],{},"2868.55",[11,3242,3243,3244,3246,3247,3250,3251,3254],{},"The normal equation beats both, without me having to pick a ",[87,3245,1757],{}," or ",[87,3248,3249],{},"max_iter",". That's not a coincidence: gradient descent is a way to ",[34,3252,3253],{},"approximate"," this exact same answer by iterating, and with enough iterations and a good learning rate it converges to the same place. The normal equation just skips straight to the end.",[79,3256,3258,3259],{"id":3257},"the-pseudo-inverse-is-pinv","The pseudo-inverse is ",[87,3260,3261],{},"pinv",[11,3263,3264,3419,3420,3423,3424,3484,3485,3488],{},[104,3265,3267,3305],{"className":3266},[148],[104,3268,3270],{"className":3269},[152],[154,3271,3272],{"xmlns":156},[158,3273,3274,3302],{},[161,3275,3276,3278,3284,3286,3296],{},[168,3277,470],{"stretchy":469},[539,3279,3280,3282],{},[164,3281,2544],{"mathvariant":959},[164,3283,2099],{"mathvariant":2098},[164,3285,2544],{"mathvariant":959},[539,3287,3288,3290],{},[168,3289,476],{"stretchy":469},[161,3291,3292,3294],{},[168,3293,530],{},[483,3295,485],{},[539,3297,3298,3300],{},[164,3299,2544],{"mathvariant":959},[164,3301,2099],{"mathvariant":2098},[186,3303,3304],{"encoding":188},"(\\Phi^{\\mathsf{T}}\\Phi)^{-1}\\Phi^{\\mathsf{T}}",[104,3306,3308],{"className":3307,"ariaHidden":194},[193],[104,3309,3311,3314,3317,3349,3352,3387],{"className":3310},[198],[104,3312],{"className":3313,"style":2389},[202],[104,3315,470],{"className":3316},[566],[104,3318,3320,3323],{"className":3319},[207],[104,3321,2544],{"className":3322},[207],[104,3324,3326],{"className":3325},[697],[104,3327,3329],{"className":3328},[605],[104,3330,3332],{"className":3331},[610],[104,3333,3335],{"className":3334,"style":2408},[614],[104,3336,3337,3340],{"style":900},[104,3338],{"className":3339,"style":714},[622],[104,3341,3343],{"className":3342},[627,628,629,630],[104,3344,3346],{"className":3345},[207,630],[104,3347,2099],{"className":3348},[207,2423,630],[104,3350,2544],{"className":3351},[207],[104,3353,3355,3358],{"className":3354},[575],[104,3356,476],{"className":3357},[575],[104,3359,3361],{"className":3360},[697],[104,3362,3364],{"className":3363},[605],[104,3365,3367],{"className":3366},[610],[104,3368,3370],{"className":3369,"style":897},[614],[104,3371,3372,3375],{"style":900},[104,3373],{"className":3374,"style":714},[622],[104,3376,3378],{"className":3377},[627,628,629,630],[104,3379,3381,3384],{"className":3380},[207,630],[104,3382,530],{"className":3383},[207,630],[104,3385,485],{"className":3386},[207,630],[104,3388,3390,3393],{"className":3389},[207],[104,3391,2544],{"className":3392},[207],[104,3394,3396],{"className":3395},[697],[104,3397,3399],{"className":3398},[605],[104,3400,3402],{"className":3401},[610],[104,3403,3405],{"className":3404,"style":2408},[614],[104,3406,3407,3410],{"style":900},[104,3408],{"className":3409,"style":714},[622],[104,3411,3413],{"className":3412},[627,628,629,630],[104,3414,3416],{"className":3415},[207,630],[104,3417,2099],{"className":3418},[207,2423,630]," has its own name: the ",[15,3421,3422],{},"Moore-Penrose pseudo-inverse",", denoted ",[104,3425,3427,3446],{"className":3426},[148],[104,3428,3430],{"className":3429},[152],[154,3431,3432],{"xmlns":156},[158,3433,3434,3443],{},[161,3435,3436],{},[539,3437,3438,3440],{},[164,3439,2544],{"mathvariant":959},[168,3441,3442],{},"†",[186,3444,3445],{"encoding":188},"\\Phi^\\dagger",[104,3447,3449],{"className":3448,"ariaHidden":194},[193],[104,3450,3452,3455],{"className":3451},[198],[104,3453],{"className":3454,"style":2408},[202],[104,3456,3458,3461],{"className":3457},[207],[104,3459,2544],{"className":3460},[207],[104,3462,3464],{"className":3463},[697],[104,3465,3467],{"className":3466},[605],[104,3468,3470],{"className":3469},[610],[104,3471,3473],{"className":3472,"style":2408},[614],[104,3474,3475,3478],{"style":900},[104,3476],{"className":3477,"style":714},[622],[104,3479,3481],{"className":3480},[627,628,629,630],[104,3482,3442],{"className":3483},[239,630],". It's a generalization of \"matrix inverse\" to matrices that aren't square (which is always the case here: Φ has one row per patient and one column per variable, almost never equal). ",[87,3486,3487],{},"np.linalg.pinv(X)"," computes exactly this, so the whole class shrinks to:",[96,3490,3492],{"className":98,"code":3491,"language":100,"meta":57,"style":57},"class LinearRegressor(BaseEstimator, RegressorMixin):\n    def fit(self, X, y):\n        X = include_bias(X)\n        self.w_ = np.linalg.pinv(X) @ y\n        return self\n\n    def predict(self, X):\n        X = include_bias(X)\n        return (X @ self.w_).reshape(X.shape[0],)\n",[87,3493,3494,3498,3502,3507,3512,3516,3520,3524,3528],{"__ignoreMap":57},[104,3495,3496],{"class":106,"line":107},[104,3497,1551],{},[104,3499,3500],{"class":106,"line":58},[104,3501,1469],{},[104,3503,3504],{"class":106,"line":118},[104,3505,3506],{},"        X = include_bias(X)\n",[104,3508,3509],{"class":106,"line":124},[104,3510,3511],{},"        self.w_ = np.linalg.pinv(X) @ y\n",[104,3513,3514],{"class":106,"line":308},[104,3515,1479],{},[104,3517,3518],{"class":106,"line":417},[104,3519,300],{"emptyLinePlaceholder":63},[104,3521,3522],{"class":106,"line":422},[104,3523,1484],{},[104,3525,3526],{"class":106,"line":428},[104,3527,3506],{},[104,3529,3530],{"class":106,"line":1689},[104,3531,3532],{},"        return (X @ self.w_).reshape(X.shape[0],)\n",[11,3534,3535,3536,3540,3541,3608],{},"An entire linear regression class, solved in one line. And Bishop already flags the risk: if two input columns are too similar to each other (",[445,3537,3539],{"definition":3538},"when two or more input variables carry nearly the same information, which makes the Φ matrix close to singular and the normal equation's solution numerically unstable","collinearity","), ",[104,3542,3544,3564],{"className":3543},[148],[104,3545,3547],{"className":3546},[152],[154,3548,3549],{"xmlns":156},[158,3550,3551,3561],{},[161,3552,3553,3559],{},[539,3554,3555,3557],{},[164,3556,2544],{"mathvariant":959},[164,3558,2099],{"mathvariant":2098},[164,3560,2544],{"mathvariant":959},[186,3562,3563],{"encoding":188},"\\Phi^{\\mathsf{T}}\\Phi",[104,3565,3567],{"className":3566,"ariaHidden":194},[193],[104,3568,3570,3573,3605],{"className":3569},[198],[104,3571],{"className":3572,"style":2408},[202],[104,3574,3576,3579],{"className":3575},[207],[104,3577,2544],{"className":3578},[207],[104,3580,3582],{"className":3581},[697],[104,3583,3585],{"className":3584},[605],[104,3586,3588],{"className":3587},[610],[104,3589,3591],{"className":3590,"style":2408},[614],[104,3592,3593,3596],{"style":900},[104,3594],{"className":3595,"style":714},[622],[104,3597,3599],{"className":3598},[627,628,629,630],[104,3600,3602],{"className":3601},[207,630],[104,3603,2099],{"className":3604},[207,2423,630],[104,3606,2544],{"className":3607},[207]," gets close to singular and the computation becomes numerically unstable. That'll come back once the course reaches feature selection.",[79,3610,3612],{"id":3611},"matches-scikit-learn-to-the-decimal","Matches scikit-learn to the decimal",[11,3614,3615,3616,3618],{},"The proof the math is right: I ran the same pseudo-inverse approach and scikit-learn's ready-made ",[87,3617,1440],{}," side by side.",[96,3620,3622],{"className":98,"code":3621,"language":100,"meta":57,"style":57},"regressor = LinearRegressor()  # the pinv-based class above\nregressor.fit(X_train, y_train)\nprint(mean_squared_error(y_train, regressor.predict(X_train)))\n\nfrom sklearn.linear_model import LinearRegression\nsk_regressor = LinearRegression().fit(X_train, y_train)\nprint(mean_squared_error(y_train, sk_regressor.predict(X_train)))\n",[87,3623,3624,3629,3633,3638,3642,3647,3652],{"__ignoreMap":57},[104,3625,3626],{"class":106,"line":107},[104,3627,3628],{},"regressor = LinearRegressor()  # the pinv-based class above\n",[104,3630,3631],{"class":106,"line":58},[104,3632,1957],{},[104,3634,3635],{"class":106,"line":118},[104,3636,3637],{},"print(mean_squared_error(y_train, regressor.predict(X_train)))\n",[104,3639,3640],{"class":106,"line":124},[104,3641,300],{"emptyLinePlaceholder":63},[104,3643,3644],{"class":106,"line":308},[104,3645,3646],{},"from sklearn.linear_model import LinearRegression\n",[104,3648,3649],{"class":106,"line":417},[104,3650,3651],{},"sk_regressor = LinearRegression().fit(X_train, y_train)\n",[104,3653,3654],{"class":106,"line":422},[104,3655,3656],{},"print(mean_squared_error(y_train, sk_regressor.predict(X_train)))\n",[433,3658,3659],{},[11,3660,3661,3663,3664,3667],{},[15,3662,1629],{}," my version: train MSE 2868.5497028355776. scikit-learn's ",[87,3665,3666],{},"LinearRegression",": train MSE 2868.549702835577.",[11,3669,3670,3671,3673],{},"The difference only shows up in the last decimal place, floating-point noise, not a difference in method. ",[87,3672,1440],{}," computes exactly this under the hood.",[79,3675,3677],{"id":3676},"the-regressor-zoo","The regressor zoo",[11,3679,3680,3681,3684],{},"Lecture 2c takes this same normal-equation regressor and also measures MSE on the ",[15,3682,3683],{},"test"," set (the 20% held out from training), then compares it against five very different kinds of model, all with scikit-learn's default parameters:",[1872,3686,3687,3699],{},[1875,3688,3689],{},[1878,3690,3691,3693,3696],{},[1881,3692,341],{"align":1883},[1881,3694,3695],{"align":3186},"Train MSE",[1881,3697,3698],{"align":3186},"Test MSE",[1889,3700,3701,3717,3732,3749,3761,3774],{},[1878,3702,3703,3709,3714],{},[1894,3704,3705,3706,476],{"align":1883},"Decision tree (",[87,3707,3708],{},"DecisionTreeRegressor",[1894,3710,3711],{"align":3186},[15,3712,3713],{},"0.00",[1894,3715,3716],{"align":3186},"4872.20",[1878,3718,3719,3726,3729],{},[1894,3720,3721,3722,3725],{"align":1883},"KNN (",[87,3723,3724],{},"KNeighborsRegressor",", k=5)",[1894,3727,3728],{"align":3186},"2528.59",[1894,3730,3731],{"align":3186},"3019.08",[1878,3733,3734,3741,3744],{},[1894,3735,3736,3737,3740],{"align":1883},"Random forest (",[87,3738,3739],{},"RandomForestRegressor",", depth 3)",[1894,3742,3743],{"align":3186},"2530.82",[1894,3745,3746],{"align":3186},[15,3747,3748],{},"2785.98",[1878,3750,3751,3756,3758],{},[1894,3752,3753,3754],{"align":1883},"Normal equation \u002F ",[87,3755,3666],{},[1894,3757,3240],{"align":3186},[1894,3759,3760],{"align":3186},"2900.19",[1878,3762,3763,3768,3771],{},[1894,3764,3765,3767],{"align":1883},[87,3766,1529],{}," (10000 iterations)",[1894,3769,3770],{"align":3186},"2950.64",[1894,3772,3773],{"align":3186},"2863.35",[1878,3775,3776,3781,3784],{},[1894,3777,3778],{"align":1883},[87,3779,3780],{},"LinearSVR",[1894,3782,3783],{"align":3186},"8224.56",[1894,3785,3786],{"align":3186},"6775.88",[11,3788,3789],{},"I'm not going to explain how each of these models works internally yet (KNN, decision trees, and random forests each get their own lecture later in the course, and that's where I'll come back to them properly). But three lessons come out of the table alone:",[3791,3792,3793,3804,3810],"ol",{},[360,3794,3795,3798,3799,3803],{},[15,3796,3797],{},"Low training MSE means nothing on its own."," The decision tree zeroed out its training error (it literally memorized every patient) and was the worst of all of them on the test set. That's ",[445,3800,3802],{"definition":3801},"when the model memorizes the training data instead of learning the general pattern, and as a result does worse on new data","overfitting"," in its purest form, the same phenomenon Bishop showed back in chapter 1 with the degree-9 polynomial.",[360,3805,3806,3809],{},[15,3807,3808],{},"The test-set winner wasn't the model that fit training best."," The random forest fits training almost as well as the full tree (2530.82, quite close to the tree's absurd zero), but without going as far, and that's exactly why it generalizes better: 2785.98 on test, the lowest test MSE in the whole table. A forest is many trees trained on different slices of the data, with the final prediction being the average across all of them, and that average cancels out a lot of the excess each individual tree commits.",[360,3811,3812,3817,3818,3820,3821,3823],{},[15,3813,3814,3816],{},[87,3815,1529],{}," beat the exact normal equation on the test set"," (2863.35 versus 2900.19), even with a slightly worse training MSE. That's not a coincidence: as I already saw ",[20,3819,354],{"href":353},", scikit-learn's ",[87,3822,1529],{}," ships with L2 regularization on by default. Here that regularization, without me asking for it, ended up helping it generalize a bit better.",[11,3825,3826,3828],{},[87,3827,3780],{}," came out visibly worse than everything else, on both train and test, but that's more about its default hyperparameters not suiting this dataset than about the method itself, a story for another day.",[79,3830,1870],{"id":1869},[1872,3832,3833,3842],{},[1875,3834,3835],{},[1878,3836,3837,3839],{},[1881,3838,1884],{"align":1883},[1881,3840,3841],{"align":1883},"What these two lectures settled",[1889,3843,3844,3852,4023],{},[1878,3845,3846,3849],{},[1894,3847,3848],{"align":1883},"Gradient descent finds the optimal fit by iterating",[1894,3850,3851],{"align":1883},"Linear regression has a closed-form solution: the normal equation gets there in one computation",[1878,3853,3854,3859],{},[1894,3855,3856,3858],{"align":1883},[87,3857,1440],{}," \"just works\"",[1894,3860,3861,3862,4022],{"align":1883},"Under the hood, it computes exactly ",[104,3863,3865,3905],{"className":3864},[148],[104,3866,3868],{"className":3867},[152],[154,3869,3870],{"xmlns":156},[158,3871,3872,3902],{},[161,3873,3874,3876,3882,3884,3894,3900],{},[168,3875,470],{"stretchy":469},[539,3877,3878,3880],{},[164,3879,2544],{"mathvariant":959},[164,3881,2099],{"mathvariant":2098},[164,3883,2544],{"mathvariant":959},[539,3885,3886,3888],{},[168,3887,476],{"stretchy":469},[161,3889,3890,3892],{},[168,3891,530],{},[483,3893,485],{},[539,3895,3896,3898],{},[164,3897,2544],{"mathvariant":959},[164,3899,2099],{"mathvariant":2098},[164,3901,535],{"mathvariant":473},[186,3903,3904],{"encoding":188},"(\\Phi^{\\mathsf{T}}\\Phi)^{-1}\\Phi^{\\mathsf{T}}\\mathbf{t}",[104,3906,3908],{"className":3907,"ariaHidden":194},[193],[104,3909,3911,3914,3917,3949,3952,3987,4019],{"className":3910},[198],[104,3912],{"className":3913,"style":2389},[202],[104,3915,470],{"className":3916},[566],[104,3918,3920,3923],{"className":3919},[207],[104,3921,2544],{"className":3922},[207],[104,3924,3926],{"className":3925},[697],[104,3927,3929],{"className":3928},[605],[104,3930,3932],{"className":3931},[610],[104,3933,3935],{"className":3934,"style":2408},[614],[104,3936,3937,3940],{"style":900},[104,3938],{"className":3939,"style":714},[622],[104,3941,3943],{"className":3942},[627,628,629,630],[104,3944,3946],{"className":3945},[207,630],[104,3947,2099],{"className":3948},[207,2423,630],[104,3950,2544],{"className":3951},[207],[104,3953,3955,3958],{"className":3954},[575],[104,3956,476],{"className":3957},[575],[104,3959,3961],{"className":3960},[697],[104,3962,3964],{"className":3963},[605],[104,3965,3967],{"className":3966},[610],[104,3968,3970],{"className":3969,"style":897},[614],[104,3971,3972,3975],{"style":900},[104,3973],{"className":3974,"style":714},[622],[104,3976,3978],{"className":3977},[627,628,629,630],[104,3979,3981,3984],{"className":3980},[207,630],[104,3982,530],{"className":3983},[207,630],[104,3985,485],{"className":3986},[207,630],[104,3988,3990,3993],{"className":3989},[207],[104,3991,2544],{"className":3992},[207],[104,3994,3996],{"className":3995},[697],[104,3997,3999],{"className":3998},[605],[104,4000,4002],{"className":4001},[610],[104,4003,4005],{"className":4004,"style":2408},[614],[104,4006,4007,4010],{"style":900},[104,4008],{"className":4009,"style":714},[622],[104,4011,4013],{"className":4012},[627,628,629,630],[104,4014,4016],{"className":4015},[207,630],[104,4017,2099],{"className":4018},[207,2423,630],[104,4020,535],{"className":4021},[207,570]," via the pseudo-inverse",[1878,4024,4025,4028],{},[1894,4026,4027],{"align":1883},"Low training MSE is a good sign",[1894,4029,4030],{"align":1883},"Only when test MSE agrees. Low train, high test is the signature of overfitting",[79,4032,1939],{"id":1938},[11,4034,4035,4036,4038,4039,4041,4042,4044],{},"I use the normal-equation model (the same ",[87,4037,3261],{},"-based ",[87,4040,3167],{}," from above) and look at the ",[15,4043,3683],{}," set, the 89 patients the model never saw during fitting, to visualize what that 2900.19 MSE actually means case by case.",[96,4046,4048],{"className":98,"code":4047,"language":100,"meta":57,"style":57},"regressor = LinearRegressor()\nregressor.fit(X_train, y_train)\ny_pred_test = regressor.predict(X_test)\n",[87,4049,4050,4054,4058],{"__ignoreMap":57},[104,4051,4052],{"class":106,"line":107},[104,4053,1847],{},[104,4055,4056],{"class":106,"line":58},[104,4057,1957],{},[104,4059,4060],{"class":106,"line":118},[104,4061,4062],{},"y_pred_test = regressor.predict(X_test)\n",[1971,4064],{":actual":4065,":predicted":4066,"point-label":339,"reference-label":1975,"x-label":4067,"y-label":4068},"[37, 42, 48, 48, 52, 52, 60, 61, 63, 63, 64, 67, 68, 69, 70, 72, 72, 72, 77, 84, 84, 87, 89, 90, 90, 90, 91, 94, 94, 95, 96, 96, 98, 99, 101, 102, 107, 108, 110, 111, 113, 118, 122, 128, 129, 135, 136, 140, 140, 151, 153, 156, 158, 164, 168, 168, 170, 171, 172, 180, 181, 184, 186, 187, 190, 200, 202, 202, 214, 219, 220, 222, 230, 232, 233, 233, 237, 242, 248, 252, 258, 263, 264, 272, 275, 281, 295, 297, 310]","[81.6, 124.4, 71.6, 48.0, 206.5, 61.5, 77.6, 146.0, 59.1, 54.4, 88.5, 130.7, 107.5, 103.5, 179.5, 109.2, 94.4, 55.9, 180.4, 119.7, 92.2, 115.0, 79.9, 51.6, 133.0, 171.0, 157.4, 94.1, 90.2, 154.6, 54.8, 108.6, 79.0, 53.5, 182.0, 105.6, 109.0, 107.7, 164.9, 123.8, 86.4, 159.4, 187.9, 70.4, 154.9, 100.1, 152.5, 174.8, 124.7, 210.7, 115.0, 165.5, 63.0, 184.5, 120.3, 155.0, 196.4, 175.6, 148.0, 173.8, 171.1, 166.3, 192.7, 140.6, 140.9, 71.7, 134.0, 144.8, 130.7, 139.5, 208.9, 199.4, 291.4, 189.0, 190.4, 206.0, 158.6, 258.2, 202.2, 168.4, 167.6, 188.3, 250.8, 181.3, 218.6, 234.2, 219.5, 206.6, 207.4]","actual progression (test)","predicted progression (test)",[11,4070,4071,4072,4075],{},"Notice the cloud looks a lot like ",[20,4073,4074],{"href":1999},"the one from the previous post"," (which was on training data), without getting visibly worse on test. That confirms numerically what the table already showed: train MSE 2868.55 versus test MSE 2900.19, a small gap. The model didn't memorize training, it genuinely generalized, it's just that being a plain linear model, it still misses quite a bit case by case, the same ceiling I already saw in the previous post.",[1982,4077,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":4079},[4080,4081,4082,4083,4085,4086,4087,4088],{"id":2021,"depth":58,"text":2022},{"id":2780,"depth":58,"text":2781},{"id":3160,"depth":58,"text":3161},{"id":3257,"depth":58,"text":4084},"The pseudo-inverse is pinv",{"id":3611,"depth":58,"text":3612},{"id":3676,"depth":58,"text":3677},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lectures 2b and 2c: instead of iterating gradient descent, the professor solves linear regression with a single closed-form computation using the pseudo-inverse, matches scikit-learn's LinearRegression exactly, and compares it against five other kinds of regressor. I explain the why behind each step.",{},{"title":2010,"description":4089},"en\u002Fplaylists\u002Fpattern-recognition\u002Fnormal-equation",[4094,4095,3802],"normal-equation","least-squares","tRNTg9y2lr39prMEd05AN5P3fShFnu5_NkLDaxMhaqk",{"id":4098,"title":4099,"body":4100,"cover":3,"date":1996,"description":5111,"extension":61,"meta":5112,"navigation":63,"order":118,"path":5113,"playlist":2000,"seo":5114,"status":66,"stem":5115,"tags":5116,"__hash__":5120},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold.md","Classification by Threshold: When Regression Becomes a Decision",{"type":8,"value":4101,"toc":5099},[4102,4109,4113,4120,4129,4148,4154,4158,4169,4178,4188,4192,4202,4219,4229,4236,4260,4270,4301,4305,4308,4352,4366,4729,4733,4736,4745,4755,4777,4781,4798,4823,4838,4853,4857,4864,4884,4895,4902,4906,4922,4941,4948,4951,4958,4971,4992,4994,5034,5037,5039,5045,5065,5079,5091,5097],[11,4103,4104,4105,4108],{},"Lecture 3, split into two parts: classification (3a) and normalization (3b). And the professor opens with a provocation, reusing the exact same regression class from ",[20,4106,4107],{"href":1925},"the previous post"," to solve a problem that isn't quite regression.",[79,4110,4112],{"id":4111},"the-dataset-breast-cancer-569-patients","The dataset: breast cancer, 569 patients",[11,4114,4115,4116,4119],{},"The professor switches datasets: now it's scikit-learn's ",[87,4117,4118],{},"load_breast_cancer",", 569 patients, 30 variables measured from a biopsy image (radius, texture, perimeter, area, and so on), and a binary target, malignant or benign. Before picking any variable, he computes each of the 30's correlation with the target and sorts them.",[96,4121,4123],{"className":98,"code":4122,"language":100,"meta":57,"style":57},"correlations = df.corr()['target'].drop('target')\n",[87,4124,4125],{"__ignoreMap":57},[104,4126,4127],{"class":106,"line":107},[104,4128,4122],{},[433,4130,4131],{},[11,4132,4133,4135,4136,4139,4140,4143,4144,4147],{},[15,4134,1629],{}," the most correlated is ",[87,4137,4138],{},"worst concave points"," (-0.79), followed by ",[87,4141,4142],{},"worst perimeter"," (-0.78) and ",[87,4145,4146],{},"mean concave points"," (-0.78). That tracks clinically: concave points along the tumor's contour and its perimeter are exactly the kind of thing a pathologist looks at to suspect malignancy.",[11,4149,4150,4151,4153],{},"He keeps ",[87,4152,4138],{}," as the only input variable for now, saving the other 29 for later.",[79,4155,4157],{"id":4156},"the-binary-target-flipped-on-purpose","The binary target, flipped on purpose",[11,4159,4160,4161,4164,4165,4168],{},"Scikit-learn ships this dataset with a convention already: ",[87,4162,4163],{},"target=0"," is malignant, ",[87,4166,4167],{},"target=1"," is benign. The professor flips that by hand:",[96,4170,4172],{"className":98,"code":4171,"language":100,"meta":57,"style":57},"y = np.array(y==0, dtype=int)  # now 1 = malignant\n",[87,4173,4174],{"__ignoreMap":57},[104,4175,4176],{"class":106,"line":107},[104,4177,4171],{},[11,4179,4180,4181,4183,4184,4187],{},"Why? Because the more common convention in a detection problem (fraud, disease, defect) is to let label ",[87,4182,485],{}," represent the case you want to ",[15,4185,4186],{},"flag",", the rare, concerning event, not the \"normal\" case. After the flip, the fraction of malignant patients is 0.373, meaning 37% of the 569 cases.",[79,4189,4191],{"id":4190},"regression-to-classify","Regression to classify?",[11,4193,4194,4195,4201],{},"This is where the lecture gets provocative. Instead of building a classifier from scratch, the professor reuses the ",[15,4196,4197,4198,4200],{},"exact same ",[87,4199,3167],{}," class"," from the previous post (the pseudo-inverse one) and just trains it on the 0\u002F1 target as if it were any continuous number:",[96,4203,4205],{"className":98,"code":4204,"language":100,"meta":57,"style":57},"regressor = LinearRegressor()\nregressor.fit(X_train, y_train)\ny_pred = regressor.predict(X_train)\n",[87,4206,4207,4211,4215],{"__ignoreMap":57},[104,4208,4209],{"class":106,"line":107},[104,4210,1847],{},[104,4212,4213],{"class":106,"line":58},[104,4214,1957],{},[104,4216,4217],{"class":106,"line":118},[104,4218,1962],{},[433,4220,4221],{},[11,4222,4223,1807,4225,4228],{},[15,4224,1629],{},[87,4226,4227],{},"w_ = [-0.303, 5.844]",", MSE = 0.0863.",[11,4230,4231,4232,4235],{},"An MSE of 0.0863 looks great at first glance (way below 1), but watch what happens when the professor measures ",[15,4233,4234],{},"accuracy"," on that same result:",[96,4237,4239],{"className":98,"code":4238,"language":100,"meta":57,"style":57},"def accuracy(y, y_pred):\n    return np.sum(y == y_pred) \u002F len(y)\n\nprint(accuracy(y_train, y_pred))\n",[87,4240,4241,4246,4251,4255],{"__ignoreMap":57},[104,4242,4243],{"class":106,"line":107},[104,4244,4245],{},"def accuracy(y, y_pred):\n",[104,4247,4248],{"class":106,"line":58},[104,4249,4250],{},"    return np.sum(y == y_pred) \u002F len(y)\n",[104,4252,4253],{"class":106,"line":118},[104,4254,300],{"emptyLinePlaceholder":63},[104,4256,4257],{"class":106,"line":124},[104,4258,4259],{},"print(accuracy(y_train, y_pred))\n",[433,4261,4262],{},[11,4263,4264,1807,4266,4269],{},[15,4265,1629],{},[87,4267,4268],{},"0.0",". Zero percent correct.",[11,4271,4272,4273,4276,4277,4280,4281,3246,4284,4287,4288,3246,4290,4292,4293,4296,4297,4300],{},"That's not a bug, it's a deliberate trap. ",[87,4274,4275],{},"y_pred"," is regression's ",[15,4278,4279],{},"continuous"," output, something like ",[87,4282,4283],{},"0.312",[87,4285,4286],{},"0.847",", never exactly ",[87,4289,331],{},[87,4291,485],{},". Comparing ",[87,4294,4295],{},"y == y_pred"," with ",[87,4298,4299],{},"=="," between an integer and a float almost never matches, so accuracy computed this way always lands on zero, no matter how good the model actually is. One piece is missing: turning that continuous number into a decision.",[79,4302,4304],{"id":4303},"the-threshold-from-continuous-number-to-decision","The threshold: from continuous number to decision",[11,4306,4307],{},"The fix is small: compare the regression's output against 0.5 before turning it into a label.",[96,4309,4311],{"className":98,"code":4310,"language":100,"meta":57,"style":57},"class LinearClassifier(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        X = include_bias(X)\n        self.w_ = np.linalg.pinv(X) @ y\n        return self\n    def predict(self, X):\n        X = include_bias(X)\n        y_pred = X @ self.w_\n        return (y_pred.reshape(X.shape[0],) > 0.5).astype(int)\n",[87,4312,4313,4318,4322,4326,4330,4334,4338,4342,4347],{"__ignoreMap":57},[104,4314,4315],{"class":106,"line":107},[104,4316,4317],{},"class LinearClassifier(BaseEstimator, ClassifierMixin):\n",[104,4319,4320],{"class":106,"line":58},[104,4321,1469],{},[104,4323,4324],{"class":106,"line":118},[104,4325,3506],{},[104,4327,4328],{"class":106,"line":124},[104,4329,3511],{},[104,4331,4332],{"class":106,"line":308},[104,4333,1479],{},[104,4335,4336],{"class":106,"line":417},[104,4337,1484],{},[104,4339,4340],{"class":106,"line":422},[104,4341,3506],{},[104,4343,4344],{"class":106,"line":428},[104,4345,4346],{},"        y_pred = X @ self.w_\n",[104,4348,4349],{"class":106,"line":1689},[104,4350,4351],{},"        return (y_pred.reshape(X.shape[0],) > 0.5).astype(int)\n",[433,4353,4354],{},[11,4355,4356,4358,4359,4362,4363,1863],{},[15,4357,1629],{}," training accuracy is now ",[15,4360,4361],{},"0.9165",". On the test set, ",[15,4364,4365],{},"0.8947",[11,4367,4368,4369,4371,4372,4376,4377,4545,4546,4590,4591,4594,4595,4661,4662,4728],{},"The exact same fit (the same ",[87,4370,3152],{},"), just now with a threshold deciding the final label. This is precisely what Bishop calls a ",[445,4373,4375],{"definition":4374},"a function y(x) = w·x + w0 that decides the class by looking at its sign (or, like here, whether it crosses a threshold): one side of the boundary is one class, the other side is the other","linear discriminant function",": ",[104,4378,4380,4418],{"className":4379},[148],[104,4381,4383],{"className":4382},[152],[154,4384,4385],{"xmlns":156},[158,4386,4387,4415],{},[161,4388,4389,4391,4393,4395,4397,4399,4405,4407,4409],{},[164,4390,166],{},[168,4392,470],{"stretchy":469},[164,4394,178],{"mathvariant":473},[168,4396,476],{"stretchy":469},[168,4398,170],{},[539,4400,4401,4403],{},[164,4402,317],{"mathvariant":473},[164,4404,2099],{"mathvariant":2098},[164,4406,178],{"mathvariant":473},[168,4408,181],{},[515,4410,4411,4413],{},[164,4412,317],{},[483,4414,331],{},[186,4416,4417],{"encoding":188},"y(\\mathbf{x}) = \\mathbf{w}^{\\mathsf{T}}\\mathbf{x} + w_0",[104,4419,4421,4448,4499],{"className":4420,"ariaHidden":194},[193],[104,4422,4424,4427,4430,4433,4436,4439,4442,4445],{"className":4423},[198],[104,4425],{"className":4426,"style":558},[202],[104,4428,166],{"className":4429,"style":209},[207,208],[104,4431,470],{"className":4432},[566],[104,4434,178],{"className":4435},[207,570],[104,4437,476],{"className":4438},[575],[104,4440],{"className":4441,"style":214},[213],[104,4443,170],{"className":4444},[218],[104,4446],{"className":4447,"style":214},[213],[104,4449,4451,4455,4487,4490,4493,4496],{"className":4450},[198],[104,4452],{"className":4453,"style":4454},[202],"height:0.9324em;vertical-align:-0.0833em;",[104,4456,4458,4461],{"className":4457},[207],[104,4459,317],{"className":4460,"style":571},[207,570],[104,4462,4464],{"className":4463},[697],[104,4465,4467],{"className":4466},[605],[104,4468,4470],{"className":4469},[610],[104,4471,4473],{"className":4472,"style":2408},[614],[104,4474,4475,4478],{"style":900},[104,4476],{"className":4477,"style":714},[622],[104,4479,4481],{"className":4480},[627,628,629,630],[104,4482,4484],{"className":4483},[207,630],[104,4485,2099],{"className":4486},[207,2423,630],[104,4488,178],{"className":4489},[207,570],[104,4491],{"className":4492,"style":235},[213],[104,4494,181],{"className":4495},[239],[104,4497],{"className":4498,"style":235},[213],[104,4500,4502,4505],{"className":4501},[198],[104,4503],{"className":4504,"style":2822},[202],[104,4506,4508,4511],{"className":4507},[207],[104,4509,317],{"className":4510,"style":2829},[207,208],[104,4512,4514],{"className":4513},[697],[104,4515,4517,4537],{"className":4516},[605,606],[104,4518,4520,4534],{"className":4519},[610],[104,4521,4523],{"className":4522,"style":2842},[614],[104,4524,4525,4528],{"style":2845},[104,4526],{"className":4527,"style":714},[622],[104,4529,4531],{"className":4530},[627,628,629,630],[104,4532,331],{"className":4533},[207,630],[104,4535,667],{"className":4536},[666],[104,4538,4540],{"className":4539},[610],[104,4541,4543],{"className":4542,"style":807},[614],[104,4544],{},", and the point where ",[104,4547,4549,4569],{"className":4548},[148],[104,4550,4552],{"className":4551},[152],[154,4553,4554],{"xmlns":156},[158,4555,4556,4566],{},[161,4557,4558,4560,4562,4564],{},[164,4559,166],{},[168,4561,470],{"stretchy":469},[164,4563,178],{"mathvariant":473},[168,4565,476],{"stretchy":469},[186,4567,4568],{"encoding":188},"y(\\mathbf{x})",[104,4570,4572],{"className":4571,"ariaHidden":194},[193],[104,4573,4575,4578,4581,4584,4587],{"className":4574},[198],[104,4576],{"className":4577,"style":558},[202],[104,4579,166],{"className":4580,"style":209},[207,208],[104,4582,470],{"className":4583},[566],[104,4585,178],{"className":4586},[207,570],[104,4588,476],{"className":4589},[575]," crosses the decision threshold is the ",[15,4592,4593],{},"decision boundary",". In Bishop's classic case that boundary sits at ",[104,4596,4598,4622],{"className":4597},[148],[104,4599,4601],{"className":4600},[152],[154,4602,4603],{"xmlns":156},[158,4604,4605,4619],{},[161,4606,4607,4609,4611,4613,4615,4617],{},[164,4608,166],{},[168,4610,470],{"stretchy":469},[164,4612,178],{"mathvariant":473},[168,4614,476],{"stretchy":469},[168,4616,170],{},[483,4618,331],{},[186,4620,4621],{"encoding":188},"y(\\mathbf{x})=0",[104,4623,4625,4652],{"className":4624,"ariaHidden":194},[193],[104,4626,4628,4631,4634,4637,4640,4643,4646,4649],{"className":4627},[198],[104,4629],{"className":4630,"style":558},[202],[104,4632,166],{"className":4633,"style":209},[207,208],[104,4635,470],{"className":4636},[566],[104,4638,178],{"className":4639},[207,570],[104,4641,476],{"className":4642},[575],[104,4644],{"className":4645,"style":214},[213],[104,4647,170],{"className":4648},[218],[104,4650],{"className":4651,"style":214},[213],[104,4653,4655,4658],{"className":4654},[198],[104,4656],{"className":4657,"style":2971},[202],[104,4659,331],{"className":4660},[207],", here it sits at ",[104,4663,4665,4689],{"className":4664},[148],[104,4666,4668],{"className":4667},[152],[154,4669,4670],{"xmlns":156},[158,4671,4672,4686],{},[161,4673,4674,4676,4678,4680,4682,4684],{},[164,4675,166],{},[168,4677,470],{"stretchy":469},[164,4679,178],{"mathvariant":473},[168,4681,476],{"stretchy":469},[168,4683,170],{},[483,4685,1773],{},[186,4687,4688],{"encoding":188},"y(\\mathbf{x})=0.5",[104,4690,4692,4719],{"className":4691,"ariaHidden":194},[193],[104,4693,4695,4698,4701,4704,4707,4710,4713,4716],{"className":4694},[198],[104,4696],{"className":4697,"style":558},[202],[104,4699,166],{"className":4700,"style":209},[207,208],[104,4702,470],{"className":4703},[566],[104,4705,178],{"className":4706},[207,570],[104,4708,476],{"className":4709},[575],[104,4711],{"className":4712,"style":214},[213],[104,4714,170],{"className":4715},[218],[104,4717],{"className":4718,"style":214},[213],[104,4720,4722,4725],{"className":4721},[198],[104,4723],{"className":4724,"style":2971},[202],[104,4726,1773],{"className":4727},[207]," because the target is coded 0\u002F1 instead of centered at zero, but the geometric idea is identical: on either side of a line, the decision flips.",[79,4730,4732],{"id":4731},"an-elegant-identity","An elegant identity",[11,4734,4735],{},"The professor adds accuracy to MSE (both computed over the already-thresholded predictions, 0 or 1):",[96,4737,4739],{"className":98,"code":4738,"language":100,"meta":57,"style":57},"accuracy(y_train, y_pred) + mean_squared_error(y_train, y_pred)\n",[87,4740,4741],{"__ignoreMap":57},[104,4742,4743],{"class":106,"line":107},[104,4744,4738],{},[433,4746,4747],{},[11,4748,4749,4751,4752,1863],{},[15,4750,1629],{}," exactly ",[87,4753,4754],{},"1.0",[11,4756,4757,4758,1499,4760,4762,4763,4766,4767,4769,4770,4772,4773,4776],{},"That's not a coincidence. When ",[87,4759,166],{},[87,4761,4275],{}," are only 0 or 1, the squared error ",[87,4764,4765],{},"(y - y_pred)²"," equals ",[87,4768,331],{}," when it's right and ",[87,4771,485],{}," when it's wrong, exactly the same as the absolute error. So the mean MSE is literally the ",[15,4774,4775],{},"error rate",", and error rate plus accuracy always adds up to 1. A cute bit of algebra, but it shows something worth remembering: MSE over a thresholded binary label is just another name for \"fraction of mistakes.\"",[79,4778,4780],{"id":4779},"the-best-fit-for-the-continuous-error-isnt-the-best-fit-for-classifying","The best fit for the continuous error isn't the best fit for classifying",[11,4782,4783,4784,4786,4787,4789,4790,4793,4794,4797],{},"Here's the subtlest point of the lecture. The professor takes the ",[87,4785,3152],{}," the pseudo-inverse found (the one minimizing squared error on the ",[15,4788,4279],{}," 0\u002F1 target) and sweeps values near coefficient ",[87,4791,4792],{},"w_[1]",", measuring MSE ",[15,4795,4796],{},"after thresholding"," at each one:",[96,4799,4801],{"className":98,"code":4800,"language":100,"meta":57,"style":57},"w1_values = np.linspace(original_w[1] - 3, original_w[1] + 3, 100)\nfor w1_candidate in w1_values:\n    classifier.w_ = np.array([original_w[0], w1_candidate])\n    mse_values.append(mean_squared_error(y_train, classifier.predict(X_train)))\n",[87,4802,4803,4808,4813,4818],{"__ignoreMap":57},[104,4804,4805],{"class":106,"line":107},[104,4806,4807],{},"w1_values = np.linspace(original_w[1] - 3, original_w[1] + 3, 100)\n",[104,4809,4810],{"class":106,"line":58},[104,4811,4812],{},"for w1_candidate in w1_values:\n",[104,4814,4815],{"class":106,"line":118},[104,4816,4817],{},"    classifier.w_ = np.array([original_w[0], w1_candidate])\n",[104,4819,4820],{"class":106,"line":124},[104,4821,4822],{},"    mse_values.append(mean_squared_error(y_train, classifier.predict(X_train)))\n",[433,4824,4825],{},[11,4826,4827,4829,4830,4833,4834,4837],{},[15,4828,1629],{}," the lowest MSE found (0.0791) happens at ",[87,4831,4832],{},"w_[1] = 5.5709",", not the ",[87,4835,4836],{},"w_[1] = 5.8436"," the pseudo-inverse returned.",[11,4839,4840,4841,4844,4845,4848,4849,4852],{},"The fit that minimizes squared error ",[15,4842,4843],{},"before"," thresholding isn't the same fit that minimizes error ",[15,4846,4847],{},"after"," thresholding. Two similar-looking objectives, but not identical ones. Bishop explains why: least squares corresponds to assuming the noise in the data follows a Gaussian distribution (chapter 3), which makes complete sense for predicting a continuous number. But a binary label doesn't have Gaussian noise, and that's why least squares applied directly to classification suffers a specific problem: points that are already \"too correct,\" sitting far on the right side of the boundary, still ",[15,4850,4851],{},"pull the fit",", because they contribute squared error even though they don't need to. Bishop shows this with an example where adding extra points, all on the correct side of the boundary, moves the decision boundary for the worse, something that doesn't happen with a method actually designed for classification (logistic regression, which the course hasn't reached yet, but exists precisely to fix this flaw).",[79,4854,4856],{"id":4855},"all-30-variables","All 30 variables",[11,4858,4859,4860,4863],{},"Same ",[87,4861,4862],{},"LinearClassifier",", now trained on all 30 columns instead of just 1:",[96,4865,4867],{"className":98,"code":4866,"language":100,"meta":57,"style":57},"modelo = LinearClassifier()\nmodelo.fit(X_train, y_train)\nprint(accuracy_score(y_test, modelo.predict(X_test)))\n",[87,4868,4869,4874,4879],{"__ignoreMap":57},[104,4870,4871],{"class":106,"line":107},[104,4872,4873],{},"modelo = LinearClassifier()\n",[104,4875,4876],{"class":106,"line":58},[104,4877,4878],{},"modelo.fit(X_train, y_train)\n",[104,4880,4881],{"class":106,"line":118},[104,4882,4883],{},"print(accuracy_score(y_test, modelo.predict(X_test)))\n",[433,4885,4886],{},[11,4887,4888,4890,4891,4894],{},[15,4889,1629],{}," test accuracy ",[15,4892,4893],{},"0.9561",", versus 0.8947 with just one variable.",[11,4896,4897,4898,4901],{},"The same jump I've already seen in ",[20,4899,4900],{"href":1999},"the previous two posts",": more information, same algorithm, same threshold, and the result improves a lot.",[79,4903,4905],{"id":4904},"normalizing-but-for-a-different-reason","Normalizing, but for a different reason",[11,4907,4908,4909,4912,4913,4916,4917,4921],{},"The second half of the lecture (",[87,4910,4911],{},"aula03b",") switches topics: instead of thresholded regression, the professor uses scikit-learn's ",[87,4914,4915],{},"KNeighborsClassifier"," (",[445,4918,4920],{"definition":4919},"classifies a new point by looking at its K closest (most similar) neighbors in the training set and voting for their most common class, with its own lecture later in the course","KNN",", K nearest neighbors, a topic that gets its own whole lecture later, here it just makes a cameo) straight on the cancer dataset, no normalization at all:",[96,4923,4925],{"className":98,"code":4924,"language":100,"meta":57,"style":57},"modelo = KNeighborsClassifier()\nmodelo.fit(X_train, y_train)\nmodelo.score(X_test, y_test)\n",[87,4926,4927,4932,4936],{"__ignoreMap":57},[104,4928,4929],{"class":106,"line":107},[104,4930,4931],{},"modelo = KNeighborsClassifier()\n",[104,4933,4934],{"class":106,"line":58},[104,4935,4878],{},[104,4937,4938],{"class":106,"line":118},[104,4939,4940],{},"modelo.score(X_test, y_test)\n",[433,4942,4943],{},[11,4944,4945,4947],{},[15,4946,1629],{}," 0.9298.",[11,4949,4950],{},"Then he normalizes (min-max, into the 0-1 range, computed from training statistics only and applied the same way to test) and repeats:",[433,4952,4953],{},[11,4954,4955,4957],{},[15,4956,1629],{}," 0.9649.",[11,4959,4960,4961,4965,4966,4970],{},"And confirms that standardizing (",[445,4962,4964],{"definition":4963},"subtracting the mean and dividing by the standard deviation, leaving each variable centered at zero with a spread of 1","z-score",", the same normalization ",[20,4967,4969],{"href":4968},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fw2-lab03-feature-scaling","from the specialization playlist",") gives the same result: 0.9649 too.",[11,4972,4973,4974,4976,4977,4979,4980,4983,4984,4987,4988,4991],{},"I'd already seen normalization matter before, but for a different reason: back ",[20,4975,354],{"href":4968},", normalizing helped ",[15,4978,1749],{}," converge faster, because the cost bowl was less elongated. Here the reason is different. KNN decides the class by looking at the ",[15,4981,4982],{},"distance"," between patients, usually Euclidean distance, the square root of the sum of squared differences across each variable. If one variable (say ",[87,4985,4986],{},"mean area",", ranging from 143 to 2501) has a scale hundreds of times bigger than another (say ",[87,4989,4990],{},"mean smoothness",", ranging from 0.05 to 0.16), the distance ends up being decided almost entirely by the large-scale variable, the other 29 barely register. Normalizing puts everyone on the same ruler before measuring distance, and suddenly all 30 variables genuinely contribute, not just one.",[79,4993,1870],{"id":1869},[1872,4995,4996,5004],{},[1875,4997,4998],{},[1878,4999,5000,5002],{},[1881,5001,1884],{"align":1883},[1881,5003,1887],{"align":1883},[1889,5005,5006,5018,5026],{},[1878,5007,5008,5011],{},[1894,5009,5010],{"align":1883},"Regression fits a continuous number",[1894,5012,5013,5014,5017],{"align":1883},"A threshold (a ",[87,5015,5016],{},"> 0.5",") turns that number into a binary decision, making it a classifier",[1878,5019,5020,5023],{},[1894,5021,5022],{"align":1883},"MSE and accuracy measure different things",[1894,5024,5025],{"align":1883},"For a thresholded binary label, they're literally complementary: accuracy + MSE = 1",[1878,5027,5028,5031],{},[1894,5029,5030],{"align":1883},"Normalizing helps gradient descent converge",[1894,5032,5033],{"align":1883},"Normalizing also helps any distance-based method (like KNN) avoid letting one large-scale variable dominate on its own",[11,5035,5036],{},"And it leaves a clean hook for the next lecture, which this playlist hasn't reached yet: least squares applied to classification has a structural bias (Bishop showed why), and the right fix is swapping the error function for one actually designed for classification, logistic regression.",[79,5038,1939],{"id":1938},[11,5040,5041,5042,5044],{},"I repeat the single-variable classifier (",[87,5043,4138],{},") and visualize, patient by patient in the test set, where it gets it right and where it doesn't, together with the actual decision boundary.",[96,5046,5048],{"className":98,"code":5047,"language":100,"meta":57,"style":57},"classifier = LinearClassifier()\nclassifier.fit(X_train, y_train)  # 1 variable: worst concave points\ny_pred_test = classifier.predict(X_test)\n",[87,5049,5050,5055,5060],{"__ignoreMap":57},[104,5051,5052],{"class":106,"line":107},[104,5053,5054],{},"classifier = LinearClassifier()\n",[104,5056,5057],{"class":106,"line":58},[104,5058,5059],{},"classifier.fit(X_train, y_train)  # 1 variable: worst concave points\n",[104,5061,5062],{"class":106,"line":118},[104,5063,5064],{},"y_pred_test = classifier.predict(X_test)\n",[11,5066,5067,5068,5070,5071,5074,5075,5078],{},"I redid this fit with ",[87,5069,1174],{}," (the original notebook doesn't fix the seed, so the numbers shift on every run, here I use a fixed train\u002Ftest split so I can show the exact patients) and got ",[87,5072,5073],{},"w_ = [-0.296, 5.841]",", which puts the decision boundary at ",[87,5076,5077],{},"worst concave points ≈ 0.136",": below that the model predicts benign, above it, malignant.",[5080,5081],"classification-fit-scatter",{":actual":5082,":correct":5083,":threshold":5084,":x":5085,"class0-label":5086,"class1-label":5087,"correct-label":5088,"threshold-label":5089,"wrong-label":5090,"x-label":4138},"[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","0.1362","[0.0, 0.0, 0.0, 0.0, 0.0223, 0.032, 0.0341, 0.0431, 0.0459, 0.0481, 0.0533, 0.0556, 0.0558, 0.0622, 0.0627, 0.0641, 0.0737, 0.0763, 0.0783, 0.0791, 0.0796, 0.0828, 0.0831, 0.0875, 0.0895, 0.0917, 0.0961, 0.0998, 0.101, 0.1015, 0.1015, 0.1017, 0.1045, 0.1047, 0.1092, 0.1105, 0.1136, 0.118, 0.1225, 0.1312, 0.1427, 0.1456, 0.1474, 0.1659, 0.1673, 0.1739, 0.1789, 0.1841, 0.1848, 0.1932, 0.1974, 0.2013, 0.2148, 0.221, 0.2264, 0.2422, 0.243, 0.2508, 0.265, 0.2867]","Benign","Malignant","Correct","Decision boundary","Wrong",[11,5092,5093,5094,5096],{},"The red triangles (the mistakes) cluster right around the dashed boundary, on both sides, exactly where I'd expect: those are the ambiguous cases, where ",[87,5095,4138],{}," alone doesn't separate malignant from benign well. Far from the boundary, on either side, the model is right almost every time. That tracks: one variable alone carries plenty of signal (it had a 0.79 correlation, after all), but it isn't enough to never miss, and that's exactly why all 30 variables together (0.9561) beat the single variable (0.8947).",[1982,5098,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":5100},[5101,5102,5103,5104,5105,5106,5107,5108,5109,5110],{"id":4111,"depth":58,"text":4112},{"id":4156,"depth":58,"text":4157},{"id":4190,"depth":58,"text":4191},{"id":4303,"depth":58,"text":4304},{"id":4731,"depth":58,"text":4732},{"id":4779,"depth":58,"text":4780},{"id":4855,"depth":58,"text":4856},{"id":4904,"depth":58,"text":4905},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lectures 3a and 3b: the professor reuses the exact same linear regression class to classify breast cancer, shows why a threshold was missing, and then normalizes the data for a reason unlike anything I'd seen before.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold",{"title":4099,"description":5111},"en\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold",[5117,5118,5119],"classification","normalization","knn","CeWuOYe2IDmE1a22X0WjV-W_vwKqq4t0a6WnAUM_EDU",{"id":5122,"title":5123,"body":5124,"cover":3,"date":1996,"description":5764,"extension":61,"meta":5765,"navigation":63,"order":124,"path":5766,"playlist":2000,"seo":5767,"status":66,"stem":5768,"tags":5769,"__hash__":5771},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier.md","KNN: Classifying by Looking at Your Neighbors",{"type":8,"value":5125,"toc":5755},[5126,5129,5133,5148,5173,5177,5183,5263,5274,5285,5519,5523,5531,5543,5546,5550,5553,5598,5605,5612,5616,5628,5630,5669,5671,5674,5689,5739,5753],[11,5127,5128],{},"Lecture 4a: KNN, the most \"model-less\" model there is. No coefficient to fit, no gradient, no normal equation. Just a ruler.",[79,5130,5132],{"id":5131},"the-dataset-wines-3-cultivars","The dataset: wines, 3 cultivars",[11,5134,5135,5136,5139,5140,5143,5144,5147],{},"New dataset: ",[87,5137,5138],{},"load_wine",", 178 bottles, 13 chemical measurements (alcohol content, acidity, magnesium, and so on) and 3 classes, the grape cultivar. The professor picks two columns so it's plottable on a plane: ",[87,5141,5142],{},"flavanoids"," (index 6) and ",[87,5145,5146],{},"color_intensity"," (index 9).",[96,5149,5151],{"className":98,"code":5150,"language":100,"meta":57,"style":57},"from sklearn.datasets import load_wine\nX, y = load_wine(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\nfeats = [6, 9]\n",[87,5152,5153,5158,5163,5168],{"__ignoreMap":57},[104,5154,5155],{"class":106,"line":107},[104,5156,5157],{},"from sklearn.datasets import load_wine\n",[104,5159,5160],{"class":106,"line":58},[104,5161,5162],{},"X, y = load_wine(return_X_y=True)\n",[104,5164,5165],{"class":106,"line":118},[104,5166,5167],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",[104,5169,5170],{"class":106,"line":124},[104,5171,5172],{},"feats = [6, 9]\n",[79,5174,5176],{"id":5175},"the-idea-theres-no-training-just-memory","The idea: there's no training, just memory",[11,5178,5179,5180,5182],{},"Every model I've seen up to now (regression, the thresholded classifier) fits parameters: finds a ",[87,5181,317],{}," that minimizes some error. KNN does none of that. Its \"training\" is literally just storing the data:",[96,5184,5186],{"className":98,"code":5185,"language":100,"meta":57,"style":57},"class K1NN(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        self.X = X\n        self.y = y\n        return self\n\n    def distance_(self, x):\n        return np.sum((self.X - x)**2, axis=1)**0.5\n\n    def predict(self, X):\n        y_pred = np.empty((X.shape[0],))\n        for i, x in enumerate(X):\n            distances = self.distance_(x)\n            min_idx = np.argmin(distances)\n            y_pred[i] = self.y[min_idx]\n        return y_pred\n",[87,5187,5188,5193,5197,5202,5207,5211,5215,5220,5225,5229,5233,5238,5243,5248,5253,5258],{"__ignoreMap":57},[104,5189,5190],{"class":106,"line":107},[104,5191,5192],{},"class K1NN(BaseEstimator, ClassifierMixin):\n",[104,5194,5195],{"class":106,"line":58},[104,5196,1469],{},[104,5198,5199],{"class":106,"line":118},[104,5200,5201],{},"        self.X = X\n",[104,5203,5204],{"class":106,"line":124},[104,5205,5206],{},"        self.y = y\n",[104,5208,5209],{"class":106,"line":308},[104,5210,1479],{},[104,5212,5213],{"class":106,"line":417},[104,5214,300],{"emptyLinePlaceholder":63},[104,5216,5217],{"class":106,"line":422},[104,5218,5219],{},"    def distance_(self, x):\n",[104,5221,5222],{"class":106,"line":428},[104,5223,5224],{},"        return np.sum((self.X - x)**2, axis=1)**0.5\n",[104,5226,5227],{"class":106,"line":1689},[104,5228,300],{"emptyLinePlaceholder":63},[104,5230,5231],{"class":106,"line":1695},[104,5232,1484],{},[104,5234,5235],{"class":106,"line":1701},[104,5236,5237],{},"        y_pred = np.empty((X.shape[0],))\n",[104,5239,5240],{"class":106,"line":1707},[104,5241,5242],{},"        for i, x in enumerate(X):\n",[104,5244,5245],{"class":106,"line":1713},[104,5246,5247],{},"            distances = self.distance_(x)\n",[104,5249,5250],{"class":106,"line":1719},[104,5251,5252],{},"            min_idx = np.argmin(distances)\n",[104,5254,5255],{"class":106,"line":1724},[104,5256,5257],{},"            y_pred[i] = self.y[min_idx]\n",[104,5259,5260],{"class":106,"line":1729},[104,5261,5262],{},"        return y_pred\n",[11,5264,5265,5266,5269,5270,5273],{},"To classify a new wine, it computes the Euclidean distance to ",[15,5267,5268],{},"every"," training wine, finds the closest match, and copies its class. This is called ",[15,5271,5272],{},"K1NN"," because it only looks at the single nearest neighbor, K=1.",[433,5275,5276],{},[11,5277,5278,5280,5281,5284],{},[15,5279,1629],{}," 0.97 test accuracy. Identical to scikit-learn's ",[87,5282,5283],{},"KNeighborsClassifier(n_neighbors=1)"," run on the same data.",[11,5286,5287,5288,5319,5320,5404,5405,5433,5434,5462,5463,5466,5467,5518],{},"Bishop treats this as a special case of a more general result (section 2.5.2): if you draw a sphere around a new point until it contains exactly ",[104,5289,5291,5305],{"className":5290},[148],[104,5292,5294],{"className":5293},[152],[154,5295,5296],{"xmlns":156},[158,5297,5298,5303],{},[161,5299,5300],{},[164,5301,5302],{},"K",[186,5304,5302],{"encoding":188},[104,5306,5308],{"className":5307,"ariaHidden":194},[193],[104,5309,5311,5315],{"className":5310},[198],[104,5312],{"className":5313,"style":5314},[202],"height:0.6833em;",[104,5316,5302],{"className":5317,"style":5318},[207,208],"margin-right:0.0715em;"," neighbors, and look at which class is the majority among them, the posterior probability of each class is simply ",[104,5321,5323,5346],{"className":5322},[148],[104,5324,5326],{"className":5325},[152],[154,5327,5328],{"xmlns":156},[158,5329,5330,5343],{},[161,5331,5332,5339,5341],{},[515,5333,5334,5336],{},[164,5335,5302],{},[164,5337,5338],{},"k",[164,5340,318],{"mathvariant":959},[164,5342,5302],{},[186,5344,5345],{"encoding":188},"K_k\u002FK",[104,5347,5349],{"className":5348,"ariaHidden":194},[193],[104,5350,5352,5355,5398,5401],{"className":5351},[198],[104,5353],{"className":5354,"style":558},[202],[104,5356,5358,5361],{"className":5357},[207],[104,5359,5302],{"className":5360,"style":5318},[207,208],[104,5362,5364],{"className":5363},[697],[104,5365,5367,5390],{"className":5366},[605,606],[104,5368,5370,5387],{"className":5369},[610],[104,5371,5374],{"className":5372,"style":5373},[614],"height:0.3361em;",[104,5375,5377,5380],{"style":5376},"top:-2.55em;margin-left:-0.0715em;margin-right:0.05em;",[104,5378],{"className":5379,"style":714},[622],[104,5381,5383],{"className":5382},[627,628,629,630],[104,5384,5338],{"className":5385,"style":5386},[207,208,630],"margin-right:0.0315em;",[104,5388,667],{"className":5389},[666],[104,5391,5393],{"className":5392},[610],[104,5394,5396],{"className":5395,"style":807},[614],[104,5397],{},[104,5399,318],{"className":5400},[207],[104,5402,5302],{"className":5403,"style":5318},[207,208]," (the fraction of the ",[104,5406,5408,5421],{"className":5407},[148],[104,5409,5411],{"className":5410},[152],[154,5412,5413],{"xmlns":156},[158,5414,5415,5419],{},[161,5416,5417],{},[164,5418,5302],{},[186,5420,5302],{"encoding":188},[104,5422,5424],{"className":5423,"ariaHidden":194},[193],[104,5425,5427,5430],{"className":5426},[198],[104,5428],{"className":5429,"style":5314},[202],[104,5431,5302],{"className":5432,"style":5318},[207,208]," neighbors belonging to that class). Classifying by the most common class among the ",[104,5435,5437,5450],{"className":5436},[148],[104,5438,5440],{"className":5439},[152],[154,5441,5442],{"xmlns":156},[158,5443,5444,5448],{},[161,5445,5446],{},[164,5447,5302],{},[186,5449,5302],{"encoding":188},[104,5451,5453],{"className":5452,"ariaHidden":194},[193],[104,5454,5456,5459],{"className":5455},[198],[104,5457],{"className":5458,"style":5314},[202],[104,5460,5302],{"className":5461,"style":5318},[207,208]," nearest neighbors ",[15,5464,5465],{},"is"," applying Bayes' theorem to that result. ",[104,5468,5470,5488],{"className":5469},[148],[104,5471,5473],{"className":5472},[152],[154,5474,5475],{"xmlns":156},[158,5476,5477,5485],{},[161,5478,5479,5481,5483],{},[164,5480,5302],{},[168,5482,170],{},[483,5484,485],{},[186,5486,5487],{"encoding":188},"K=1",[104,5489,5491,5509],{"className":5490,"ariaHidden":194},[193],[104,5492,5494,5497,5500,5503,5506],{"className":5493},[198],[104,5495],{"className":5496,"style":5314},[202],[104,5498,5302],{"className":5499,"style":5318},[207,208],[104,5501],{"className":5502,"style":214},[213],[104,5504,170],{"className":5505},[218],[104,5507],{"className":5508,"style":214},[213],[104,5510,5512,5515],{"className":5511},[198],[104,5513],{"className":5514,"style":2971},[202],[104,5516,485],{"className":5517},[207]," is just the most extreme case: the \"sphere\" grows until it touches a single point, and you copy its class with no voting at all.",[79,5520,5522],{"id":5521},"interactive-nudging-k-for-real","Interactive: nudging K for real",[11,5524,5525,5526,1499,5528,5530],{},"Instead of running one cell per K value like the notebook does, you can watch the decision boundary shift live. This is my own reconstruction on top of the 142 real training wines (the same two columns, ",[87,5527,5142],{},[87,5529,5146],{},"), running the same majority-vote algorithm:",[5532,5533],"knn-decision-explorer",{":classes":5534,":initial-k":485,":x-max":5535,":x-min":331,":x-train":5536,":y-max":5537,":y-min":5538,":y-train":5539,"class0-label":5540,"class1-label":5541,"class2-label":5542,"x-label":5142,"y-label":5146},"[2, 2, 1, 2, 0, 1, 1, 1, 2, 0, 1, 1, 2, 0, 1, 0, 0, 2, 2, 1, 1, 0, 1, 0, 2, 1, 1, 2, 0, 0, 0, 2, 0, 0, 1, 2, 1, 0, 2, 1, 0, 2, 1, 1, 0, 1, 0, 0, 1, 0, 0, 2, 1, 1, 1, 0, 1, 1, 1, 2, 2, 0, 1, 2, 2, 1, 1, 0, 1, 2, 2, 1, 2, 1, 1, 1, 0, 0, 2, 0, 2, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 2, 1, 1, 1, 2, 2, 1, 0, 0, 1, 2, 2, 0, 1, 2, 2, 2, 2, 1, 0, 1, 0, 2, 0, 0, 1, 0, 0, 2, 1, 0, 2, 2, 0, 0, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 0, 1, 1, 0, 1, 1]","5.3","[1.31, 0.6, 3.75, 1.1, 2.64, 2.04, 1.94, 1.3, 0.8, 3.24, 2.17, 2.13, 0.58, 2.99, 2.03, 3.0, 2.94, 0.83, 0.48, 1.92, 1.85, 2.68, 2.25, 2.88, 0.34, 2.5, 1.28, 0.66, 2.43, 2.52, 2.19, 0.78, 2.69, 2.69, 2.27, 0.49, 1.57, 3.32, 0.65, 2.65, 3.06, 0.52, 3.15, 1.75, 3.0, 0.99, 2.97, 3.29, 2.24, 2.68, 2.37, 1.09, 2.53, 2.55, 2.21, 3.04, 1.36, 1.69, 2.79, 0.61, 0.6, 3.39, 2.26, 0.66, 0.92, 1.41, 2.89, 2.74, 2.65, 1.2, 0.6, 1.79, 0.68, 2.03, 2.17, 1.84, 2.79, 3.39, 0.92, 3.49, 0.47, 3.54, 2.98, 2.26, 1.32, 2.53, 3.17, 2.51, 3.03, 2.63, 1.02, 0.47, 2.58, 1.25, 1.59, 0.83, 0.56, 1.59, 2.98, 3.69, 0.57, 1.28, 1.22, 3.4, 1.84, 0.63, 0.58, 0.76, 0.7, 3.1, 2.9, 1.76, 3.27, 0.69, 3.67, 2.92, 1.69, 2.41, 3.23, 0.7, 1.75, 2.43, 0.58, 0.84, 2.76, 3.74, 1.39, 1.25, 1.36, 1.64, 2.99, 1.6, 1.61, 2.14, 5.08, 0.76, 3.17, 2.86, 2.03, 3.64, 1.46, 2.11]","13.2","1.5","[13.0, 5.0, 4.5, 11.75, 3.7, 2.7, 2.62, 3.17, 4.4, 5.68, 3.3, 2.08, 7.6, 5.6, 4.6, 6.38, 4.8, 10.52, 5.7, 2.94, 3.4, 4.6, 2.15, 3.8, 4.9, 2.9, 2.85, 10.26, 5.0, 5.25, 3.95, 8.21, 4.32, 3.84, 2.6, 5.5, 3.8, 5.75, 7.65, 2.6, 5.64, 4.35, 3.94, 2.95, 5.04, 2.5, 4.5, 6.13, 3.0, 3.58, 3.93, 5.7, 3.9, 2.57, 3.05, 5.1, 2.45, 2.45, 3.25, 7.7, 5.58, 6.1, 3.25, 7.1, 10.68, 5.75, 3.35, 5.4, 2.76, 5.0, 4.92, 3.8, 9.3, 3.8, 1.9, 2.7, 6.3, 6.75, 7.65, 7.8, 6.62, 8.9, 5.1, 2.5, 2.65, 4.2, 4.9, 5.05, 2.8, 4.36, 3.05, 3.85, 2.9, 3.6, 4.8, 9.01, 5.88, 1.74, 5.2, 5.4, 1.95, 7.1, 5.4, 6.6, 3.74, 7.9, 5.45, 8.42, 5.28, 4.45, 5.85, 3.3, 7.2, 10.2, 6.8, 6.2, 2.8, 4.5, 6.0, 9.7, 2.6, 4.25, 7.5, 8.66, 4.38, 7.05, 9.4, 4.1, 10.8, 2.06, 2.3, 1.95, 2.6, 3.21, 6.0, 9.2, 5.65, 3.38, 3.4, 7.5, 3.05, 2.8]","Cultivar 0","Cultivar 1","Cultivar 2",[11,5544,5545],{},"Click through each K value and notice: at K=1, the colored regions have a bunch of little isolated islands, each one hugging a single training point, the boundary is all jagged. With a bigger K, the islands disappear and the regions turn into smoother, more continuous blocks.",[79,5547,5549],{"id":5548},"k-is-a-smoothing-parameter-not-the-bigger-the-better","K is a smoothing parameter, not \"the bigger the better\"",[11,5551,5552],{},"The professor tries K=1, 3, 5, 13 on the real test data:",[1872,5554,5555,5565],{},[1875,5556,5557],{},[1878,5558,5559,5562],{},[1881,5560,5302],{"align":5561},"center",[1881,5563,5564],{"align":3186},"Accuracy",[1889,5566,5567,5576,5584,5590],{},[1878,5568,5569,5571],{},[1894,5570,485],{"align":5561},[1894,5572,5573],{"align":3186},[15,5574,5575],{},"0.97",[1878,5577,5578,5581],{},[1894,5579,5580],{"align":5561},"3",[1894,5582,5583],{"align":3186},"0.92",[1878,5585,5586,5588],{},[1894,5587,332],{"align":5561},[1894,5589,5583],{"align":3186},[1878,5591,5592,5595],{},[1894,5593,5594],{"align":5561},"13",[1894,5596,5597],{"align":3186},"0.89",[11,5599,5600,5601,5604],{},"Counterintuitive at first: I'd expect looking at more neighbors to give a more \"reliable\" result, but here accuracy only gets worse as K grows. Bishop calls K exactly that, a ",[15,5602,5603],{},"smoothing parameter",": a small K keeps the decision boundary tightly wrapped around the training data (low bias, but sensitive to every individual point, including noise), while a large K blurs the boundary, mixing in neighbors from different regions when voting (more bias, less sensitive to noise). In this specific dataset, with only 142 training points spread across 3 classes, increasing K starts pulling neighbors from a different cultivar into the vote too quickly, so the sweet spot here sits close to K=1.",[11,5606,5607,5608,1863],{},"That doesn't mean \"always use K=1.\" Bishop cites an interesting result: in the limit of infinite training data, the nearest-neighbor classifier (K=1) never makes more than twice the error of the theoretically optimal classifier, a surprisingly strong guarantee for such a simple method. But with little data (like here, 142 points), K=1 might just be \"memorizing\" training, and the right K is an empirical question, not a fixed rule. I'll come back to this more rigorously (validation, not just \"I tried it and K=1 won\") ",[20,5609,5611],{"href":5610},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","in the next post",[79,5613,5615],{"id":5614},"a-quick-reminder-distance-also-demands-normalization","A quick reminder: distance also demands normalization",[11,5617,5618,5619,5621,5622,5625,5626,1863],{},"I already saw this in ",[20,5620,4107],{"href":5113},": since KNN decides by distance, one large-scale variable dominates the computation on its own. It's no different here, and ",[20,5623,5624],{"href":5610},"the next (bonus) post"," shows the real size of the damage and the right way to avoid it, with ",[87,5627,1498],{},[79,5629,1870],{"id":1869},[1872,5631,5632,5640],{},[1875,5633,5634],{},[1878,5635,5636,5638],{},[1881,5637,1884],{"align":1883},[1881,5639,1887],{"align":1883},[1889,5641,5642,5650,5661],{},[1878,5643,5644,5647],{},[1894,5645,5646],{"align":1883},"Regression and thresholded classification fit parameters",[1894,5648,5649],{"align":1883},"KNN fits nothing, \"training\" is just storing the data and comparing distance at prediction time",[1878,5651,5652,5655],{},[1894,5653,5654],{"align":1883},"Bayes connects conditional probability and priors",[1894,5656,5657,5658,5660],{"align":1883},"The rule of voting for the most common class among the K neighbors ",[15,5659,5465],{}," a direct application of Bayes to a local density estimate",[1878,5662,5663,5666],{},[1894,5664,5665],{"align":1883},"A hyperparameter is something I choose, not something the model learns",[1894,5667,5668],{"align":1883},"K is the most direct example of that: not too big, not too small, the right value depends on the data I actually have",[79,5670,1939],{"id":1938},[11,5672,5673],{},"I use the same wine dataset, now with all 13 variables (not just the 2 that were plottable), to see the effect of normalization in practice, something I'd only described in words up to here.",[96,5675,5677],{"className":98,"code":5676,"language":100,"meta":57,"style":57},"model_raw = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\nmodel_norm = KNeighborsClassifier(n_neighbors=k).fit(X_train_normalized, y_train)\n",[87,5678,5679,5684],{"__ignoreMap":57},[104,5680,5681],{"class":106,"line":107},[104,5682,5683],{},"model_raw = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\n",[104,5685,5686],{"class":106,"line":58},[104,5687,5688],{},"model_norm = KNeighborsClassifier(n_neighbors=k).fit(X_train_normalized, y_train)\n",[1872,5690,5691,5703],{},[1875,5692,5693],{},[1878,5694,5695,5697,5700],{},[1881,5696,5302],{"align":5561},[1881,5698,5699],{"align":3186},"Not normalized",[1881,5701,5702],{"align":3186},"Normalized",[1889,5704,5705,5717,5728],{},[1878,5706,5707,5709,5712],{},[1894,5708,485],{"align":5561},[1894,5710,5711],{"align":3186},"0.7778",[1894,5713,5714],{"align":3186},[15,5715,5716],{},"0.9444",[1878,5718,5719,5721,5724],{},[1894,5720,5580],{"align":5561},[1894,5722,5723],{"align":3186},"0.8056",[1894,5725,5726],{"align":3186},[15,5727,5716],{},[1878,5729,5730,5732,5735],{},[1894,5731,332],{"align":5561},[1894,5733,5734],{"align":3186},"0.7222",[1894,5736,5737],{"align":3186},[15,5738,5716],{},[11,5740,5741,5742,5745,5746,5749,5750,5752],{},"Without normalizing, accuracy bounces around and never breaks 0.81 (13 variables at very different scales, like ",[87,5743,5744],{},"proline",", which goes up to nearly 1700, against ",[87,5747,5748],{},"hue",", which stays under 2, so distance ends up decided almost entirely by ",[87,5751,5744],{},"). Normalized, accuracy hits 0.9444 at every one of the three K values, a huge jump, and here's the interesting part: after normalizing, the choice of K stops mattering nearly as much, all three give the exact same result on this test. That tracks: once every variable genuinely contributes to distance (instead of one variable dominating everything), the neighborhood gets \"right\" much sooner, leaving less work for fine-tuning K to fix.",[1982,5754,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":5756},[5757,5758,5759,5760,5761,5762,5763],{"id":5131,"depth":58,"text":5132},{"id":5175,"depth":58,"text":5176},{"id":5521,"depth":58,"text":5522},{"id":5548,"depth":58,"text":5549},{"id":5614,"depth":58,"text":5615},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 4a: the professor implements K-nearest-neighbors by hand, matches scikit-learn's KNeighborsClassifier exactly, and shows K isn't 'the bigger the better.' I explain why K is a smoothing parameter.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier",{"title":5123,"description":5764},"en\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier",[5119,5117,5770],"hyperparameters","hBt9td75VhLgzaqMngGLwNREwyNAron_zx7T5rUzpFw",{"id":5773,"title":5774,"body":5775,"cover":3,"date":1996,"description":6923,"extension":61,"meta":6924,"navigation":63,"order":308,"path":5610,"playlist":2000,"seo":6925,"status":66,"stem":6926,"tags":6927,"__hash__":6931},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation.md","Pipeline, Cross-Validation, and GridSearch: the Real Workflow",{"type":8,"value":5776,"toc":6912},[5777,5783,5787,5796,5811,5819,5830,5838,5844,5874,5884,5888,5901,5935,5946,5961,5965,5968,6043,6087,6105,6119,6298,6302,6334,6387,6406,6409,6413,6423,6448,6458,6474,6484,6490,6494,6513,6574,6600,6614,6617,6621,6627,6636,6643,6749,6751,6798,6800,6803,6851,6907,6910],[11,5778,5779,5780,5782],{},"The rest of lecture 4, the \"real work\" hiding behind every accuracy number I've shown so far. Without this, every K I picked in ",[20,5781,4900],{"href":5766}," was, without exaggeration, an educated guess.",[79,5784,5786],{"id":5785},"pipeline-what-i-already-knew-now-with-classification","Pipeline: what I already knew, now with classification",[11,5788,5789,1807,5791,5795],{},[87,5790,1498],{},[20,5792,5794],{"href":5793},"\u002Fen\u002Fplaylists\u002Fmachine-learning-specialization\u002Fscikit-learn-pitfalls","already showed up in the other playlist",": it chains normalization and the model into a single object, so the scaler never sees data that should stay out of training. Here the professor confirms the same conclusion, just on a classification problem with 3 classes:",[96,5797,5799],{"className":98,"code":5798,"language":100,"meta":57,"style":57},"model = KNeighborsClassifier()\nmodel.fit(X_train, y_train)\n",[87,5800,5801,5806],{"__ignoreMap":57},[104,5802,5803],{"class":106,"line":107},[104,5804,5805],{},"model = KNeighborsClassifier()\n",[104,5807,5808],{"class":106,"line":58},[104,5809,5810],{},"model.fit(X_train, y_train)\n",[433,5812,5813],{},[11,5814,5815,5818],{},[15,5816,5817],{},"Output (no normalization):"," accuracy 0.67.",[11,5820,5821,5822,5825,5826,5829],{},"Then he normalizes four different ways (min-max by hand, ",[87,5823,5824],{},"MinMaxScaler",", z-score by hand, ",[87,5827,5828],{},"StandardScaler","), and they all match:",[433,5831,5832],{},[11,5833,5834,5837],{},[15,5835,5836],{},"Output (any normalization):"," accuracy 0.92.",[11,5839,5840,5841,5843],{},"And ",[87,5842,1498],{}," lands on the same 0.92, chaining both steps automatically:",[96,5845,5847],{"className":98,"code":5846,"language":100,"meta":57,"style":57},"pipeline = Pipeline([\n    ('scaler', StandardScaler()),\n    ('model', KNeighborsClassifier())\n])\npipeline.fit(X_train, y_train)\n",[87,5848,5849,5854,5859,5864,5869],{"__ignoreMap":57},[104,5850,5851],{"class":106,"line":107},[104,5852,5853],{},"pipeline = Pipeline([\n",[104,5855,5856],{"class":106,"line":58},[104,5857,5858],{},"    ('scaler', StandardScaler()),\n",[104,5860,5861],{"class":106,"line":118},[104,5862,5863],{},"    ('model', KNeighborsClassifier())\n",[104,5865,5866],{"class":106,"line":124},[104,5867,5868],{},"])\n",[104,5870,5871],{"class":106,"line":308},[104,5872,5873],{},"pipeline.fit(X_train, y_train)\n",[11,5875,5876,5877,5879,5880,5883],{},"Nothing new mechanically (I'd already seen ",[87,5878,1498],{}," from the outside), but the jump from 0.67 to 0.92 is the biggest I've seen so far just from normalizing, reinforcing the reason ",[20,5881,5882],{"href":5113},"I already explained",": 13 variables at very different scales, and KNN decides everything by distance.",[79,5885,5887],{"id":5886},"separating-validation-from-testing","Separating validation from testing",[11,5889,5890,5893,5894,318,5897,5900],{},[87,5891,5892],{},"aula04c"," starts slow: it splits off a slice of training just for validation (",[87,5895,5896],{},"X_tr",[87,5898,5899],{},"X_val","), tries several K values, and picks the one that wins on validation:",[96,5902,5904],{"className":98,"code":5903,"language":100,"meta":57,"style":57},"X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2)\n\nfor k in range(1, 21, 2):\n    model = KNeighborsClassifier(n_neighbors=k)\n    model.fit(X_tr, y_tr)\n    acc = accuracy_score(y_val, model.predict(X_val))\n",[87,5905,5906,5911,5915,5920,5925,5930],{"__ignoreMap":57},[104,5907,5908],{"class":106,"line":107},[104,5909,5910],{},"X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2)\n",[104,5912,5913],{"class":106,"line":58},[104,5914,300],{"emptyLinePlaceholder":63},[104,5916,5917],{"class":106,"line":118},[104,5918,5919],{},"for k in range(1, 21, 2):\n",[104,5921,5922],{"class":106,"line":124},[104,5923,5924],{},"    model = KNeighborsClassifier(n_neighbors=k)\n",[104,5926,5927],{"class":106,"line":308},[104,5928,5929],{},"    model.fit(X_tr, y_tr)\n",[104,5931,5932],{"class":106,"line":417},[104,5933,5934],{},"    acc = accuracy_score(y_val, model.predict(X_val))\n",[433,5936,5937],{},[11,5938,5939,5941,5942,5945],{},[15,5940,1629],{}," the best K found was ",[15,5943,5944],{},"k=9",", with 0.759 validation accuracy.",[11,5947,5948,5949,5951,5952,5955,5956,318,5958,5960],{},"That's already much better than \"I tested on the test set and saw which K won\" (which would be committing exactly the mistake any stats course warns against: using the test set to pick a hyperparameter is a form of leakage, it biases the final estimate because it stops being about genuinely unseen data). But there's a visible problem here: neither ",[87,5950,1170],{}," call, holdout or validation, fixes ",[87,5953,5954],{},"random_state",", so every time I rerun this cell, ",[87,5957,5896],{},[87,5959,5899],{}," change, and the winning K can change with it. A single validation split is just one sample, and it can get lucky or unlucky.",[79,5962,5964],{"id":5963},"a-real-bug-hiding-inside-a-loop","A real bug, hiding inside a loop",[11,5966,5967],{},"The professor then moves on to cross-validation, implemented by hand:",[96,5969,5971],{"className":98,"code":5970,"language":100,"meta":57,"style":57},"def cross_validation(model, X, y, k=3):\n    n = int(len(y)\u002Fk)\n    idx = np.random.permutation(len(y))\n    X = X[idx]\n    y = y[idx]\n    for i in range(k):\n        X_tr = np.concatenate([X[:i*n], X[(i+1)*n:]])\n        y_tr = np.concatenate([y[:i*n], y[(i+1)*n:]])\n        X_val = X[i*n:(i+1)*n]\n        y_val = y[i*n:(i+1)*n]\n        model.fit(X_tr, y_tr)\n        y_pred = model.predict(X_val)\n        acc = accuracy_score(y_val, y_pred)\n        return acc\n",[87,5972,5973,5978,5983,5988,5993,5998,6003,6008,6013,6018,6023,6028,6033,6038],{"__ignoreMap":57},[104,5974,5975],{"class":106,"line":107},[104,5976,5977],{},"def cross_validation(model, X, y, k=3):\n",[104,5979,5980],{"class":106,"line":58},[104,5981,5982],{},"    n = int(len(y)\u002Fk)\n",[104,5984,5985],{"class":106,"line":118},[104,5986,5987],{},"    idx = np.random.permutation(len(y))\n",[104,5989,5990],{"class":106,"line":124},[104,5991,5992],{},"    X = X[idx]\n",[104,5994,5995],{"class":106,"line":308},[104,5996,5997],{},"    y = y[idx]\n",[104,5999,6000],{"class":106,"line":417},[104,6001,6002],{},"    for i in range(k):\n",[104,6004,6005],{"class":106,"line":422},[104,6006,6007],{},"        X_tr = np.concatenate([X[:i*n], X[(i+1)*n:]])\n",[104,6009,6010],{"class":106,"line":428},[104,6011,6012],{},"        y_tr = np.concatenate([y[:i*n], y[(i+1)*n:]])\n",[104,6014,6015],{"class":106,"line":1689},[104,6016,6017],{},"        X_val = X[i*n:(i+1)*n]\n",[104,6019,6020],{"class":106,"line":1695},[104,6021,6022],{},"        y_val = y[i*n:(i+1)*n]\n",[104,6024,6025],{"class":106,"line":1701},[104,6026,6027],{},"        model.fit(X_tr, y_tr)\n",[104,6029,6030],{"class":106,"line":1707},[104,6031,6032],{},"        y_pred = model.predict(X_val)\n",[104,6034,6035],{"class":106,"line":1713},[104,6036,6037],{},"        acc = accuracy_score(y_val, y_pred)\n",[104,6039,6040],{"class":106,"line":1719},[104,6041,6042],{},"        return acc\n",[11,6044,6045,6046,6056,6057,6060,6061,6064,6065,6067,6068,6071,6072,6075,6076,6078,6079,6082,6083,6086],{},"This function is worth reading closely, because it teaches a lesson that has nothing to do with machine learning: ",[15,6047,6048,6049,6052,6053],{},"the ",[87,6050,6051],{},"return"," sits inside the ",[87,6054,6055],{},"for",". The loop runs ",[87,6058,6059],{},"i"," from 0 to ",[87,6062,6063],{},"k-1"," to build ",[87,6066,5338],{}," different folds (the right idea behind cross-validation: every fold becomes validation once, the rest becomes training), but the function exits and returns as soon as the ",[15,6069,6070],{},"first"," iteration (",[87,6073,6074],{},"i=0",") finishes. The other ",[87,6077,6063],{}," folds never run. ",[87,6080,6081],{},"cross_validation()",", despite the name, computes just ",[15,6084,6085],{},"one"," train\u002Fvalidation split, the exact same limitation as the section before, just hiding behind a name that promises more than it delivers.",[433,6088,6089],{},[11,6090,6091,1807,6093,6096,6097,6100,6101,6104],{},[15,6092,1629],{},[87,6094,6095],{},"cross_validation(model, X_train, y_train)"," returns ",[87,6098,6099],{},"0.723",", a single number from a single fold, shuffled differently on every call (because ",[87,6102,6103],{},"np.random.permutation"," runs again each time).",[11,6106,6107,6108,6111,6112,6114,6115,6118],{},"That explains something odd that shows up right after: ",[87,6109,6110],{},"repeated_cross_validation",", which calls ",[87,6113,6081],{}," ten times and averages, kind of works by accident. It isn't doing \"10 repeats of real cross-validation\" (which would be a full fold pass, repeated 10 times), it's doing ",[15,6116,6117],{},"repeated holdout",": 10 different random splits, each evaluated once. The average of those 10 (0.717, std 0.057, computed right after in the notebook) is still a more stable estimate than a single split, because it cuts down the \"I got lucky or unlucky on one split\" variance. It just isn't cross-validation in the technical sense of the term: no training point is guaranteed to become validation across those 10 rounds, since that only happens by chance, not by the full-coverage guarantee real k-fold gives you.",[11,6120,6121,6122,6151,6152,6180,6181,6233,6234,6262,6263,6291,6292,6294,6295,6297],{},"Bishop describes exactly this real k-fold (he calls it ",[104,6123,6125,6139],{"className":6124},[148],[104,6126,6128],{"className":6127},[152],[154,6129,6130],{"xmlns":156},[158,6131,6132,6137],{},[161,6133,6134],{},[164,6135,6136],{},"S",[186,6138,6136],{"encoding":188},[104,6140,6142],{"className":6141,"ariaHidden":194},[193],[104,6143,6145,6148],{"className":6144},[198],[104,6146],{"className":6147,"style":5314},[202],[104,6149,6136],{"className":6150,"style":562},[207,208],"-fold): split the data into ",[104,6153,6155,6168],{"className":6154},[148],[104,6156,6158],{"className":6157},[152],[154,6159,6160],{"xmlns":156},[158,6161,6162,6166],{},[161,6163,6164],{},[164,6165,6136],{},[186,6167,6136],{"encoding":188},[104,6169,6171],{"className":6170,"ariaHidden":194},[193],[104,6172,6174,6177],{"className":6173},[198],[104,6175],{"className":6176,"style":5314},[202],[104,6178,6136],{"className":6179,"style":562},[207,208]," blocks, use ",[104,6182,6184,6202],{"className":6183},[148],[104,6185,6187],{"className":6186},[152],[154,6188,6189],{"xmlns":156},[158,6190,6191,6199],{},[161,6192,6193,6195,6197],{},[164,6194,6136],{},[168,6196,530],{},[483,6198,485],{},[186,6200,6201],{"encoding":188},"S-1",[104,6203,6205,6224],{"className":6204,"ariaHidden":194},[193],[104,6206,6208,6212,6215,6218,6221],{"className":6207},[198],[104,6209],{"className":6210,"style":6211},[202],"height:0.7667em;vertical-align:-0.0833em;",[104,6213,6136],{"className":6214,"style":562},[207,208],[104,6216],{"className":6217,"style":235},[213],[104,6219,530],{"className":6220},[239],[104,6222],{"className":6223,"style":235},[213],[104,6225,6227,6230],{"className":6226},[198],[104,6228],{"className":6229,"style":2971},[202],[104,6231,485],{"className":6232},[207]," to train and 1 to validate, repeat ",[104,6235,6237,6250],{"className":6236},[148],[104,6238,6240],{"className":6239},[152],[154,6241,6242],{"xmlns":156},[158,6243,6244,6248],{},[161,6245,6246],{},[164,6247,6136],{},[186,6249,6136],{"encoding":188},[104,6251,6253],{"className":6252,"ariaHidden":194},[193],[104,6254,6256,6259],{"className":6255},[198],[104,6257],{"className":6258,"style":5314},[202],[104,6260,6136],{"className":6261,"style":562},[207,208]," times swapping which block is held out, and average the ",[104,6264,6266,6279],{"className":6265},[148],[104,6267,6269],{"className":6268},[152],[154,6270,6271],{"xmlns":156},[158,6272,6273,6277],{},[161,6274,6275],{},[164,6276,6136],{},[186,6278,6136],{"encoding":188},[104,6280,6282],{"className":6281,"ariaHidden":194},[193],[104,6283,6285,6288],{"className":6284},[198],[104,6286],{"className":6287,"style":5314},[202],[104,6289,6136],{"className":6290,"style":562},[207,208]," scores. The important guarantee the professor's ",[87,6293,6081],{}," loses to the bug: in real k-fold, ",[15,6296,5268],{}," point becomes validation exactly once, covering the whole dataset with no overlap. With repeated holdout (even repeated many times), some points might never land in validation, and others might land there repeatedly, purely by the luck of random sampling.",[79,6299,6301],{"id":6300},"doing-it-the-right-way","Doing it the right way",[11,6303,6304,6305,6333],{},"Scikit-learn already implements Bishop's ",[104,6306,6308,6321],{"className":6307},[148],[104,6309,6311],{"className":6310},[152],[154,6312,6313],{"xmlns":156},[158,6314,6315,6319],{},[161,6316,6317],{},[164,6318,6136],{},[186,6320,6136],{"encoding":188},[104,6322,6324],{"className":6323,"ariaHidden":194},[193],[104,6325,6327,6330],{"className":6326},[198],[104,6328],{"className":6329,"style":5314},[202],[104,6331,6136],{"className":6332,"style":562},[207,208],"-fold correctly:",[96,6335,6337],{"className":98,"code":6336,"language":100,"meta":57,"style":57},"from sklearn.model_selection import KFold\n\nkf = KFold(n_splits=3, shuffle=True)\naccs = []\nfor train_index, val_index in kf.split(X_train):\n    model = KNeighborsClassifier(n_neighbors=5)\n    model.fit(X_train[train_index], y_train[train_index])\n    accs.append(accuracy_score(y_train[val_index], model.predict(X_train[val_index])))\n\nprint(np.mean(accs))\n",[87,6338,6339,6344,6348,6353,6358,6363,6368,6373,6378,6382],{"__ignoreMap":57},[104,6340,6341],{"class":106,"line":107},[104,6342,6343],{},"from sklearn.model_selection import KFold\n",[104,6345,6346],{"class":106,"line":58},[104,6347,300],{"emptyLinePlaceholder":63},[104,6349,6350],{"class":106,"line":118},[104,6351,6352],{},"kf = KFold(n_splits=3, shuffle=True)\n",[104,6354,6355],{"class":106,"line":124},[104,6356,6357],{},"accs = []\n",[104,6359,6360],{"class":106,"line":308},[104,6361,6362],{},"for train_index, val_index in kf.split(X_train):\n",[104,6364,6365],{"class":106,"line":417},[104,6366,6367],{},"    model = KNeighborsClassifier(n_neighbors=5)\n",[104,6369,6370],{"class":106,"line":422},[104,6371,6372],{},"    model.fit(X_train[train_index], y_train[train_index])\n",[104,6374,6375],{"class":106,"line":428},[104,6376,6377],{},"    accs.append(accuracy_score(y_train[val_index], model.predict(X_train[val_index])))\n",[104,6379,6380],{"class":106,"line":1689},[104,6381,300],{"emptyLinePlaceholder":63},[104,6383,6384],{"class":106,"line":1695},[104,6385,6386],{},"print(np.mean(accs))\n",[433,6388,6389],{},[11,6390,6391,6393,6394,6397,6398,6401,6402,6405],{},[15,6392,1629],{}," 0.726 with ",[87,6395,6396],{},"KFold(n_splits=3)",". With ",[87,6399,6400],{},"RepeatedKFold(n_splits=3, n_repeats=10)"," (real k-fold, repeated 10 times with different shuffles, to shrink the estimate's variance even further): 0.676. And the one-line shortcut, ",[87,6403,6404],{},"cross_val_score",", matches both: 0.704 and 0.701 respectively.",[11,6407,6408],{},"Those numbers aren't wildly different from the \"buggy\" repeated holdout (0.717), and that's expected: even with the broken implementation, the general idea (test on chunks that weren't used for training) already captured most of the signal. The gain from real k-fold is robustness, not necessarily a dramatically different number on this specific dataset. But you don't know that without comparing, which is exactly why it's worth implementing (or using) the correct thing instead of trusting that \"it produced a plausible-looking number\" means \"the code is right.\"",[79,6410,6412],{"id":6411},"gridsearchcv-automating-the-search","GridSearchCV: automating the search",[11,6414,6415,6416,6419,6420,6422],{},"Instead of writing a ",[87,6417,6418],{},"for k in range(1, 21, 2)"," every time, ",[87,6421,1502],{}," runs the search (and the cross-validation behind it) automatically:",[96,6424,6426],{"className":98,"code":6425,"language":100,"meta":57,"style":57},"params = {'n_neighbors': range(1, 21, 2)}\ngrid = GridSearchCV(KNeighborsClassifier(), params, scoring='accuracy')\ngrid.fit(X_train, y_train)\nprint(grid.best_params_, grid.best_score_)\n",[87,6427,6428,6433,6438,6443],{"__ignoreMap":57},[104,6429,6430],{"class":106,"line":107},[104,6431,6432],{},"params = {'n_neighbors': range(1, 21, 2)}\n",[104,6434,6435],{"class":106,"line":58},[104,6436,6437],{},"grid = GridSearchCV(KNeighborsClassifier(), params, scoring='accuracy')\n",[104,6439,6440],{"class":106,"line":118},[104,6441,6442],{},"grid.fit(X_train, y_train)\n",[104,6444,6445],{"class":106,"line":124},[104,6446,6447],{},"print(grid.best_params_, grid.best_score_)\n",[433,6449,6450],{},[11,6451,6452,1807,6454,6457],{},[15,6453,1629],{},[87,6455,6456],{},"{'n_neighbors': 1}",", with a cross-validation score of 0.761.",[11,6459,6460,6461,1811,6464,1811,6467,6470,6471,1211],{},"And expanding the search to three hyperparameters at once (",[87,6462,6463],{},"n_neighbors",[87,6465,6466],{},"weights",[87,6468,6469],{},"metric","), with ",[87,6472,6473],{},"KFold(n_splits=5)",[433,6475,6476],{},[11,6477,6478,1807,6480,6483],{},[15,6479,1629],{},[87,6481,6482],{},"{'metric': 'manhattan', 'n_neighbors': 11, 'weights': 'distance'}",", score 0.803.",[11,6485,6486,6487,6489],{},"Bishop already warns about exactly this situation in chapter 1.3: once you have more than one hyperparameter to tune, testing every combination by hand turns into a combinatorial explosion fast. ",[87,6488,1502],{}," is organized brute force: it tries every combination in the grid, cross-validates each one, and returns the best.",[79,6491,6493],{"id":6492},"pipeline-gridsearch-this-is-where-it-takes-off","Pipeline + GridSearch: this is where it takes off",[11,6495,6496,6497,1807,6499,6502,6503,6505,6506,6508,6509,6512],{},"The turning point of the whole post: putting ",[87,6498,1502],{},[15,6500,6501],{},"inside"," a ",[87,6504,1498],{},", together with the scaler, and tuning even the ",[87,6507,4915],{},"'s own hyperparameters (using the ",[87,6510,6511],{},"model__"," prefix to point at which pipeline step each parameter belongs to):",[96,6514,6516],{"className":98,"code":6515,"language":100,"meta":57,"style":57},"pipeline = Pipeline([\n    ('scaler', StandardScaler()),\n    ('model', KNeighborsClassifier())\n])\nparams = {\n    'model__n_neighbors': range(1, 21, 2),\n    'model__weights': ['uniform', 'distance'],\n    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n}\ngrid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True))\nscores = cross_val_score(grid, X_train, y_train, cv=KFold(n_splits=5, shuffle=True))\nprint(np.mean(scores))\n",[87,6517,6518,6522,6526,6530,6534,6539,6544,6549,6554,6559,6564,6569],{"__ignoreMap":57},[104,6519,6520],{"class":106,"line":107},[104,6521,5853],{},[104,6523,6524],{"class":106,"line":58},[104,6525,5858],{},[104,6527,6528],{"class":106,"line":118},[104,6529,5863],{},[104,6531,6532],{"class":106,"line":124},[104,6533,5868],{},[104,6535,6536],{"class":106,"line":308},[104,6537,6538],{},"params = {\n",[104,6540,6541],{"class":106,"line":417},[104,6542,6543],{},"    'model__n_neighbors': range(1, 21, 2),\n",[104,6545,6546],{"class":106,"line":422},[104,6547,6548],{},"    'model__weights': ['uniform', 'distance'],\n",[104,6550,6551],{"class":106,"line":428},[104,6552,6553],{},"    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n",[104,6555,6556],{"class":106,"line":1689},[104,6557,6558],{},"}\n",[104,6560,6561],{"class":106,"line":1695},[104,6562,6563],{},"grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True))\n",[104,6565,6566],{"class":106,"line":1701},[104,6567,6568],{},"scores = cross_val_score(grid, X_train, y_train, cv=KFold(n_splits=5, shuffle=True))\n",[104,6570,6571],{"class":106,"line":1707},[104,6572,6573],{},"print(np.mean(scores))\n",[11,6575,6576,6577,1807,6579,6582,6583,6585,6586,6589,6590,6592,6593,6595,6596,6599],{},"Notice the structure: there's a ",[87,6578,6404],{},[15,6580,6581],{},"wrapped around"," an entire ",[87,6584,1502],{},". That's ",[15,6587,6588],{},"nested"," cross-validation: the outer loop measures how well the whole process (normalize, search for the best hyperparameters, train) generalizes, and the inner loop (inside ",[87,6591,1502],{},") only picks the hyperparameters. Without that nesting, ",[87,6594,1502],{},"'s own cross-validation score (",[87,6597,6598],{},"best_score_",") runs slightly optimistic, because the same data that chose the hyperparameters also evaluated the final result.",[433,6601,6602],{},[11,6603,6604,6606,6607,1499,6610,6613],{},[15,6605,1629],{}," average of 0.957 (versus 0.81 without normalizing inside the pipeline, and versus 0.68 for raw KNN with no grid at all). Adding ",[87,6608,6609],{},"scaler__with_mean",[87,6611,6612],{},"scaler__with_std"," to the search grid (letting even the normalization be part of what's optimized): 0.979.",[11,6615,6616],{},"From raw KNN (0.67-0.68) to the full pipeline with nested hyperparameter search (0.979): the entire distance between \"I ran the default model\" and \"I did this properly.\"",[79,6618,6620],{"id":6619},"faster-than-a-full-grid-random-search","Faster than a full grid: random search",[11,6622,6623,6626],{},[87,6624,6625],{},"RandomizedSearchCV"," only tries a random sample of combinations (here, 20) instead of all of them:",[96,6628,6630],{"className":98,"code":6629,"language":100,"meta":57,"style":57},"grid = RandomizedSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True), n_iter=20)\n",[87,6631,6632],{"__ignoreMap":57},[104,6633,6634],{"class":106,"line":107},[104,6635,6629],{},[433,6637,6638],{},[11,6639,6640,6642],{},[15,6641,1629],{}," 0.957, basically tied with the full grid, while testing far fewer combinations.",[11,6644,6645,6646,6744,6745,6748],{},"That makes sense once the search grid gets too big to fully test (here it's already ",[104,6647,6649,6678],{"className":6648},[148],[104,6650,6652],{"className":6651},[152],[154,6653,6654],{"xmlns":156},[158,6655,6656,6675],{},[161,6657,6658,6661,6664,6666,6668,6670,6672],{},[483,6659,6660],{},"10",[168,6662,6663],{},"×",[483,6665,488],{},[168,6667,6663],{},[483,6669,5580],{},[168,6671,170],{},[483,6673,6674],{},"60",[186,6676,6677],{"encoding":188},"10 \\times 2 \\times 3 = 60",[104,6679,6681,6699,6717,6735],{"className":6680,"ariaHidden":194},[193],[104,6682,6684,6687,6690,6693,6696],{"className":6683},[198],[104,6685],{"className":6686,"style":1329},[202],[104,6688,6660],{"className":6689},[207],[104,6691],{"className":6692,"style":235},[213],[104,6694,6663],{"className":6695},[239],[104,6697],{"className":6698,"style":235},[213],[104,6700,6702,6705,6708,6711,6714],{"className":6701},[198],[104,6703],{"className":6704,"style":1329},[202],[104,6706,488],{"className":6707},[207],[104,6709],{"className":6710,"style":235},[213],[104,6712,6663],{"className":6713},[239],[104,6715],{"className":6716,"style":235},[213],[104,6718,6720,6723,6726,6729,6732],{"className":6719},[198],[104,6721],{"className":6722,"style":2971},[202],[104,6724,5580],{"className":6725},[207],[104,6727],{"className":6728,"style":214},[213],[104,6730,170],{"className":6731},[218],[104,6733],{"className":6734,"style":214},[213],[104,6736,6738,6741],{"className":6737},[198],[104,6739],{"className":6740,"style":2971},[202],[104,6742,6674],{"className":6743},[207]," combinations, each with 5 folds, 300 model fits. Add more hyperparameters and this explodes fast). The notebook takes a quick peek at ",[87,6746,6747],{},"Optuna",", a Bayesian search library that picks the next combination to try based on what's worked so far instead of sampling or testing blindly, but that's just a passing mention, not the lecture's focus.",[79,6750,1870],{"id":1869},[1872,6752,6753,6762],{},[1875,6754,6755],{},[1878,6756,6757,6759],{},[1881,6758,1884],{"align":1883},[1881,6760,6761],{"align":1883},"What these three lectures settled",[1889,6763,6764,6774,6785],{},[1878,6765,6766,6771],{},[1894,6767,6768,6770],{"align":1883},[87,6769,1498],{}," prevents leakage between normalization and the model",[1894,6772,6773],{"align":1883},"The same holds for classification, not just regression, and the gain here was huge (0.67 → 0.92)",[1878,6775,6776,6779],{},[1894,6777,6778],{"align":1883},"Cross-validation exists to give a more stable estimate than a single train\u002Ftest split",[1894,6780,6781,6782,6784],{"align":1883},"A ",[87,6783,6051],{}," in the wrong place can turn \"cross-validation\" into repeated holdout without me noticing, so it's worth reading the validation code itself, not just trusting the function's name",[1878,6786,6787,6790],{},[1894,6788,6789],{"align":1883},"A hyperparameter is my choice",[1894,6791,6792,6794,6795,6797],{"align":1883},[87,6793,1502],{}," automates the search, and placed inside a ",[87,6796,1498],{},", with nested cross-validation wrapped around it, gives the most honest generalization estimate I've produced in this playlist so far",[79,6799,1939],{"id":1938},[11,6801,6802],{},"I reproduce the full pipeline (normalization + hyperparameter search + nested cross-validation) on the same wine dataset, with a fixed seed for a reproducible result, something none of the searches in the original notebook have.",[96,6804,6806],{"className":98,"code":6805,"language":100,"meta":57,"style":57},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\npipeline = Pipeline([('scaler', StandardScaler()), ('model', KNeighborsClassifier())])\nparams = {\n    'model__n_neighbors': range(1, 21, 2),\n    'model__weights': ['uniform', 'distance'],\n    'model__metric': ['euclidean', 'manhattan', 'minkowski'],\n}\ngrid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True, random_state=42))\ngrid.fit(X_train, y_train)\n",[87,6807,6808,6813,6817,6822,6826,6830,6834,6838,6842,6847],{"__ignoreMap":57},[104,6809,6810],{"class":106,"line":107},[104,6811,6812],{},"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",[104,6814,6815],{"class":106,"line":58},[104,6816,300],{"emptyLinePlaceholder":63},[104,6818,6819],{"class":106,"line":118},[104,6820,6821],{},"pipeline = Pipeline([('scaler', StandardScaler()), ('model', KNeighborsClassifier())])\n",[104,6823,6824],{"class":106,"line":124},[104,6825,6538],{},[104,6827,6828],{"class":106,"line":308},[104,6829,6543],{},[104,6831,6832],{"class":106,"line":417},[104,6833,6548],{},[104,6835,6836],{"class":106,"line":422},[104,6837,6553],{},[104,6839,6840],{"class":106,"line":428},[104,6841,6558],{},[104,6843,6844],{"class":106,"line":1689},[104,6845,6846],{},"grid = GridSearchCV(pipeline, params, scoring='accuracy', cv=KFold(n_splits=5, shuffle=True, random_state=42))\n",[104,6848,6849],{"class":106,"line":1695},[104,6850,6442],{},[1872,6852,6853,6863],{},[1875,6854,6855],{},[1878,6856,6857,6860],{},[1881,6858,6859],{"align":1883},"Step",[1881,6861,6862],{"align":3186},"Result",[1889,6864,6865,6873,6883,6891,6899],{},[1878,6866,6867,6870],{},[1894,6868,6869],{"align":1883},"Raw KNN, no normalizing, no hyperparameter search",[1894,6871,6872],{"align":3186},"0.7222 (test)",[1878,6874,6875,6878],{},[1894,6876,6877],{"align":1883},"Best combination found by the grid",[1894,6879,6880],{"align":3186},[87,6881,6882],{},"metric=manhattan, n_neighbors=9, weights=uniform",[1878,6884,6885,6888],{},[1894,6886,6887],{"align":1883},"Grid's cross-validation score",[1894,6889,6890],{"align":3186},"0.9862",[1878,6892,6893,6896],{},[1894,6894,6895],{"align":1883},"Test accuracy, with the best pipeline",[1894,6897,6898],{"align":3186},"0.9722",[1878,6900,6901,6904],{},[1894,6902,6903],{"align":1883},"Nested cross-validation (honest estimate)",[1894,6905,6906],{"align":3186},"0.9791",[11,6908,6909],{},"The nested cross-validation score (0.9791) and the test accuracy (0.9722) land close to each other, and that's exactly what I want to see: it means the nested cross-validation wasn't running too optimistic, it genuinely predicted how the pipeline would do on data it had never seen. Against raw KNN (0.7222), the entire gap (25 percentage points) came just from normalizing and picking hyperparameters properly, without touching the algorithm itself.",[1982,6911,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":6913},[6914,6915,6916,6917,6918,6919,6920,6921,6922],{"id":5785,"depth":58,"text":5786},{"id":5886,"depth":58,"text":5887},{"id":5963,"depth":58,"text":5964},{"id":6300,"depth":58,"text":6301},{"id":6411,"depth":58,"text":6412},{"id":6492,"depth":58,"text":6493},{"id":6619,"depth":58,"text":6620},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lectures 4b, 4c, and 4d: the professor builds the real professional workflow on top of KNN, and along the way I find an actual bug in his code, a return sitting inside the wrong loop.",{},{"title":5774,"description":6923},"en\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation",[6928,6929,6930],"pipeline","cross-validation","gridsearch","SXPBwoK0vI0bd6coxwMHRUwHReF5a1_ev03Nu69xb1I",{"id":6933,"title":6934,"body":6935,"cover":3,"date":1996,"description":8241,"extension":61,"meta":8242,"navigation":63,"order":417,"path":8243,"playlist":2000,"seo":8244,"status":66,"stem":8245,"tags":8246,"__hash__":8248},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees.md","Decision Trees: Yes-or-No Questions Until Only One Answer Is Left",{"type":8,"value":6936,"toc":8229},[6937,6948,6952,7003,7018,7036,7047,7075,7084,7087,7091,7098,7156,7163,7170,7174,7177,7217,7421,7524,7544,7776,7779,7809,7815,7860,7874,7878,7881,7891,7894,7898,7904,7947,7961,7971,7981,7985,8001,8051,8062,8066,8069,8084,8094,8098,8109,8115,8117,8153,8155,8161,8186,8224,8227],[11,6938,6939,6940,6943,6944,6947],{},"Lecture 5, split between categorical attributes (",[87,6941,6942],{},"aula05",") and continuous ones (",[87,6945,6946],{},"aula05b","). Bishop doesn't dedicate much space to decision trees (a short paragraph back in chapter 14), so this lecture leans much more on what the professor showed, with Gini's math as the thread tying it together.",[79,6949,6951],{"id":6950},"the-dataset-car-evaluation-and-a-baseline-that-actually-matters","The dataset: car evaluation, and a baseline that actually matters",[11,6953,6954,6956,6957,6960,6961,1811,6964,1811,6967,1811,6970,1811,6973,1811,6976,6979,6980,1811,6983,1811,6986,6989,6990,1811,6993,1811,6996,1811,6999,7002],{},[87,6955,6942],{}," uses ",[15,6958,6959],{},"Car Evaluation",", a classic UCI dataset: 1728 cars, 6 attributes, all categorical (",[87,6962,6963],{},"buying",[87,6965,6966],{},"maint",[87,6968,6969],{},"doors",[87,6971,6972],{},"persons",[87,6974,6975],{},"lug_boot",[87,6977,6978],{},"safety",", each with about 3-4 possible values like ",[87,6981,6982],{},"\"low\"",[87,6984,6985],{},"\"med\"",[87,6987,6988],{},"\"high\"","), and one class (",[87,6991,6992],{},"unacc",[87,6994,6995],{},"acc",[87,6997,6998],{},"good",[87,7000,7001],{},"vgood",").",[96,7004,7006],{"className":98,"code":7005,"language":100,"meta":57,"style":57},"for label in set(y):\n    print(f\"{label}:\\t{100*sum(y==label)\u002Flen(y):.4}%\")\n",[87,7007,7008,7013],{"__ignoreMap":57},[104,7009,7010],{"class":106,"line":107},[104,7011,7012],{},"for label in set(y):\n",[104,7014,7015],{"class":106,"line":58},[104,7016,7017],{},"    print(f\"{label}:\\t{100*sum(y==label)\u002Flen(y):.4}%\")\n",[433,7019,7020],{},[11,7021,7022,1807,7024,7026,7027,7029,7030,7032,7033,7035],{},[15,7023,1629],{},[87,7025,6992],{}," 70.02%, ",[87,7028,6995],{}," 22.22%, ",[87,7031,6998],{}," 3.99%, ",[87,7034,7001],{}," 3.76%.",[11,7037,7038,7039,7042,7043,7046],{},"Pretty imbalanced. Before any tree, the professor defines the laziest possible model, ",[15,7040,7041],{},"ZeroR"," (always guesses the most common class, the same spirit as the \"dumb model\" ",[20,7044,7045],{"href":1999},"I already saw back in this playlist's first post",", just for classification):",[96,7048,7050],{"className":98,"code":7049,"language":100,"meta":57,"style":57},"class ZeroR(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        self.answer = most_common(y)\n    def predict(self, X):\n        return [self.answer]*X.shape[0]\n",[87,7051,7052,7057,7061,7066,7070],{"__ignoreMap":57},[104,7053,7054],{"class":106,"line":107},[104,7055,7056],{},"class ZeroR(BaseEstimator, ClassifierMixin):\n",[104,7058,7059],{"class":106,"line":58},[104,7060,1469],{},[104,7062,7063],{"class":106,"line":118},[104,7064,7065],{},"        self.answer = most_common(y)\n",[104,7067,7068],{"class":106,"line":124},[104,7069,1484],{},[104,7071,7072],{"class":106,"line":308},[104,7073,7074],{},"        return [self.answer]*X.shape[0]\n",[433,7076,7077],{},[11,7078,7079,7081,7082,1863],{},[15,7080,1629],{}," 70.02% accuracy. Just always guessing ",[87,7083,6992],{},[11,7085,7086],{},"Keep that number in mind. Any model I train from here on is only interesting if it beats 70%, otherwise it didn't learn anything that \"always guess the majority class\" wasn't already giving away for free.",[79,7088,7090],{"id":7089},"a-random-tree-already-beats-the-baseline","A random tree already beats the baseline",[11,7092,7093,7094,7097],{},"Interesting intermediate step: a \"tree\" that picks the splitting variable and value ",[15,7095,7096],{},"completely at random",", and still recurses until every leaf is pure:",[96,7099,7101],{"className":98,"code":7100,"language":100,"meta":57,"style":57},"class DecisionTree(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        self.feature = np.random.randint(X.shape[1])\n        self.value = np.random.choice(list(set(X[:, self.feature])))\n        equals = X[:, self.feature] == self.value\n        if sum(equals) > 0 and sum(~equals) > 0:\n            self.equals_tree = DecisionTree().fit(X[equals], y[equals])\n            self.not_equals_tree = DecisionTree().fit(X[~equals], y[~equals])\n        else:\n            self.answer = most_common(y)\n        return self\n",[87,7102,7103,7108,7112,7117,7122,7127,7132,7137,7142,7147,7152],{"__ignoreMap":57},[104,7104,7105],{"class":106,"line":107},[104,7106,7107],{},"class DecisionTree(BaseEstimator, ClassifierMixin):\n",[104,7109,7110],{"class":106,"line":58},[104,7111,1469],{},[104,7113,7114],{"class":106,"line":118},[104,7115,7116],{},"        self.feature = np.random.randint(X.shape[1])\n",[104,7118,7119],{"class":106,"line":124},[104,7120,7121],{},"        self.value = np.random.choice(list(set(X[:, self.feature])))\n",[104,7123,7124],{"class":106,"line":308},[104,7125,7126],{},"        equals = X[:, self.feature] == self.value\n",[104,7128,7129],{"class":106,"line":417},[104,7130,7131],{},"        if sum(equals) > 0 and sum(~equals) > 0:\n",[104,7133,7134],{"class":106,"line":422},[104,7135,7136],{},"            self.equals_tree = DecisionTree().fit(X[equals], y[equals])\n",[104,7138,7139],{"class":106,"line":428},[104,7140,7141],{},"            self.not_equals_tree = DecisionTree().fit(X[~equals], y[~equals])\n",[104,7143,7144],{"class":106,"line":1689},[104,7145,7146],{},"        else:\n",[104,7148,7149],{"class":106,"line":1695},[104,7150,7151],{},"            self.answer = most_common(y)\n",[104,7153,7154],{"class":106,"line":1701},[104,7155,1479],{},[433,7157,7158],{},[11,7159,7160,7162],{},[15,7161,1629],{}," 75.6% (on the same data it trained on).",[11,7164,7165,7166,7169],{},"Already beats ZeroR, even picking the question at random. That tracks: every split, even a random one, separates the data into two smaller groups, and smaller almost always means \"a little less mixed\" than the original group. Repeat that recursively until each leaf holds a single class, and the tree ends up memorizing training, not because the chosen question was good, but because it never stops asking until there's no doubt left at all. That's already a hint of what's coming: a tree with no brake ",[15,7167,7168],{},"always"," manages to memorize training, useful or not.",[79,7171,7173],{"id":7172},"gini-impurity-the-ruler-that-decides-which-question-to-ask","Gini impurity: the ruler that decides which question to ask",[11,7175,7176],{},"To pick the right question (instead of a random one), you need a way to measure \"how mixed\" the classes are within a group:",[96,7178,7180],{"className":98,"code":7179,"language":100,"meta":57,"style":57},"def gini(y):\n    labels = list(set(y))\n    x = 0\n    for label in labels:\n        label_prob = np.mean(y==label)\n        x += label_prob**2\n    return 1-x\n",[87,7181,7182,7187,7192,7197,7202,7207,7212],{"__ignoreMap":57},[104,7183,7184],{"class":106,"line":107},[104,7185,7186],{},"def gini(y):\n",[104,7188,7189],{"class":106,"line":58},[104,7190,7191],{},"    labels = list(set(y))\n",[104,7193,7194],{"class":106,"line":118},[104,7195,7196],{},"    x = 0\n",[104,7198,7199],{"class":106,"line":124},[104,7200,7201],{},"    for label in labels:\n",[104,7203,7204],{"class":106,"line":308},[104,7205,7206],{},"        label_prob = np.mean(y==label)\n",[104,7208,7209],{"class":106,"line":417},[104,7210,7211],{},"        x += label_prob**2\n",[104,7213,7214],{"class":106,"line":422},[104,7215,7216],{},"    return 1-x\n",[11,7218,7219],{},[104,7220,7222,7263],{"className":7221},[148],[104,7223,7225],{"className":7224},[152],[154,7226,7227],{"xmlns":156},[158,7228,7229,7260],{},[161,7230,7231,7234,7236,7238,7240,7242,7244,7246,7252],{},[932,7232,7233],{},"Gini",[168,7235,470],{"stretchy":469},[164,7237,166],{},[168,7239,476],{"stretchy":469},[168,7241,170],{},[483,7243,485],{},[168,7245,530],{},[515,7247,7248,7250],{},[168,7249,494],{},[164,7251,5338],{},[490,7253,7254,7256,7258],{},[164,7255,11],{},[164,7257,5338],{},[483,7259,488],{},[186,7261,7262],{"encoding":188},"\\text{Gini}(y) = 1 - \\sum_{k} p_k^2",[104,7264,7266,7296,7314],{"className":7265,"ariaHidden":194},[193],[104,7267,7269,7272,7278,7281,7284,7287,7290,7293],{"className":7268},[198],[104,7270],{"className":7271,"style":558},[202],[104,7273,7275],{"className":7274},[207,1009],[104,7276,7233],{"className":7277},[207],[104,7279,470],{"className":7280},[566],[104,7282,166],{"className":7283,"style":209},[207,208],[104,7285,476],{"className":7286},[575],[104,7288],{"className":7289,"style":214},[213],[104,7291,170],{"className":7292},[218],[104,7294],{"className":7295,"style":214},[213],[104,7297,7299,7302,7305,7308,7311],{"className":7298},[198],[104,7300],{"className":7301,"style":1329},[202],[104,7303,485],{"className":7304},[207],[104,7306],{"className":7307,"style":235},[213],[104,7309,530],{"className":7310},[239],[104,7312],{"className":7313,"style":235},[213],[104,7315,7317,7321,7365,7368],{"className":7316},[198],[104,7318],{"className":7319,"style":7320},[202],"height:1.1138em;vertical-align:-0.2997em;",[104,7322,7324,7327],{"className":7323},[687],[104,7325,494],{"className":7326,"style":693},[687,691,692],[104,7328,7330],{"className":7329},[697],[104,7331,7333,7357],{"className":7332},[605,606],[104,7334,7336,7354],{"className":7335},[610],[104,7337,7340],{"className":7338,"style":7339},[614],"height:0.1864em;",[104,7341,7342,7345],{"style":710},[104,7343],{"className":7344,"style":714},[622],[104,7346,7348],{"className":7347},[627,628,629,630],[104,7349,7351],{"className":7350},[207,630],[104,7352,5338],{"className":7353,"style":5386},[207,208,630],[104,7355,667],{"className":7356},[666],[104,7358,7360],{"className":7359},[610],[104,7361,7363],{"className":7362,"style":755},[614],[104,7364],{},[104,7366],{"className":7367,"style":683},[213],[104,7369,7371,7374],{"className":7370},[207],[104,7372,11],{"className":7373},[207,208],[104,7375,7377],{"className":7376},[697],[104,7378,7380,7412],{"className":7379},[605,606],[104,7381,7383,7409],{"className":7382},[610],[104,7384,7386,7398],{"className":7385,"style":897},[614],[104,7387,7389,7392],{"style":7388},"top:-2.4169em;margin-left:0em;margin-right:0.05em;",[104,7390],{"className":7391,"style":714},[622],[104,7393,7395],{"className":7394},[627,628,629,630],[104,7396,5338],{"className":7397,"style":5386},[207,208,630],[104,7399,7400,7403],{"style":900},[104,7401],{"className":7402,"style":714},[622],[104,7404,7406],{"className":7405},[627,628,629,630],[104,7407,488],{"className":7408},[207,630],[104,7410,667],{"className":7411},[666],[104,7413,7415],{"className":7414},[610],[104,7416,7419],{"className":7417,"style":7418},[614],"height:0.2831em;",[104,7420],{},[11,7422,7423,7424,7494,7495,7523],{},"where ",[104,7425,7427,7445],{"className":7426},[148],[104,7428,7430],{"className":7429},[152],[154,7431,7432],{"xmlns":156},[158,7433,7434,7442],{},[161,7435,7436],{},[515,7437,7438,7440],{},[164,7439,11],{},[164,7441,5338],{},[186,7443,7444],{"encoding":188},"p_k",[104,7446,7448],{"className":7447,"ariaHidden":194},[193],[104,7449,7451,7454],{"className":7450},[198],[104,7452],{"className":7453,"style":203},[202],[104,7455,7457,7460],{"className":7456},[207],[104,7458,11],{"className":7459},[207,208],[104,7461,7463],{"className":7462},[697],[104,7464,7466,7486],{"className":7465},[605,606],[104,7467,7469,7483],{"className":7468},[610],[104,7470,7472],{"className":7471,"style":5373},[614],[104,7473,7474,7477],{"style":788},[104,7475],{"className":7476,"style":714},[622],[104,7478,7480],{"className":7479},[627,628,629,630],[104,7481,5338],{"className":7482,"style":5386},[207,208,630],[104,7484,667],{"className":7485},[666],[104,7487,7489],{"className":7488},[610],[104,7490,7492],{"className":7491,"style":807},[614],[104,7493],{}," is the fraction of examples belonging to class ",[104,7496,7498,7511],{"className":7497},[148],[104,7499,7501],{"className":7500},[152],[154,7502,7503],{"xmlns":156},[158,7504,7505,7509],{},[161,7506,7507],{},[164,7508,5338],{},[186,7510,5338],{"encoding":188},[104,7512,7514],{"className":7513,"ariaHidden":194},[193],[104,7515,7517,7520],{"className":7516},[198],[104,7518],{"className":7519,"style":268},[202],[104,7521,5338],{"className":7522,"style":5386},[207,208]," within the group. Two extreme cases confirm the intuition:",[433,7525,7526],{},[11,7527,7528,1807,7530,7533,7534,7536,7537,7539,7540,7543],{},[15,7529,1629],{},[87,7531,7532],{},"gini"," of a group where everyone's the same class: ",[15,7535,4268],{},". ",[87,7538,7532],{}," of a group with 100 different classes, one each: ",[15,7541,7542],{},"0.99",", close to the theoretical maximum.",[11,7545,7546,7547,7550,7551,7771,7772,7775],{},"Zero is total purity (no doubt left about the class). The more split between classes, the higher it climbs. Bishop calls this the ",[15,7548,7549],{},"Gini index"," (he writes it as ",[104,7552,7554,7601],{"className":7553},[148],[104,7555,7557],{"className":7556},[152],[154,7558,7559],{"xmlns":156},[158,7560,7561,7598],{},[161,7562,7563,7569,7580,7582,7584,7586,7596],{},[515,7564,7565,7567],{},[168,7566,494],{},[164,7568,5338],{},[515,7570,7571,7573],{},[164,7572,11],{},[161,7574,7575,7578],{},[164,7576,7577],{},"τ",[164,7579,5338],{},[168,7581,470],{"stretchy":469},[483,7583,485],{},[168,7585,530],{},[515,7587,7588,7590],{},[164,7589,11],{},[161,7591,7592,7594],{},[164,7593,7577],{},[164,7595,5338],{},[168,7597,476],{"stretchy":469},[186,7599,7600],{"encoding":188},"\\sum_k p_{\\tau k}(1-p_{\\tau k})",[104,7602,7604,7716],{"className":7603,"ariaHidden":194},[193],[104,7605,7607,7611,7651,7654,7701,7704,7707,7710,7713],{"className":7606},[198],[104,7608],{"className":7609,"style":7610},[202],"height:1.0497em;vertical-align:-0.2997em;",[104,7612,7614,7617],{"className":7613},[687],[104,7615,494],{"className":7616,"style":693},[687,691,692],[104,7618,7620],{"className":7619},[697],[104,7621,7623,7643],{"className":7622},[605,606],[104,7624,7626,7640],{"className":7625},[610],[104,7627,7629],{"className":7628,"style":7339},[614],[104,7630,7631,7634],{"style":710},[104,7632],{"className":7633,"style":714},[622],[104,7635,7637],{"className":7636},[627,628,629,630],[104,7638,5338],{"className":7639,"style":5386},[207,208,630],[104,7641,667],{"className":7642},[666],[104,7644,7646],{"className":7645},[610],[104,7647,7649],{"className":7648,"style":755},[614],[104,7650],{},[104,7652],{"className":7653,"style":683},[213],[104,7655,7657,7660],{"className":7656},[207],[104,7658,11],{"className":7659},[207,208],[104,7661,7663],{"className":7662},[697],[104,7664,7666,7693],{"className":7665},[605,606],[104,7667,7669,7690],{"className":7668},[610],[104,7670,7672],{"className":7671,"style":5373},[614],[104,7673,7674,7677],{"style":788},[104,7675],{"className":7676,"style":714},[622],[104,7678,7680],{"className":7679},[627,628,629,630],[104,7681,7683,7687],{"className":7682},[207,630],[104,7684,7577],{"className":7685,"style":7686},[207,208,630],"margin-right:0.1132em;",[104,7688,5338],{"className":7689,"style":5386},[207,208,630],[104,7691,667],{"className":7692},[666],[104,7694,7696],{"className":7695},[610],[104,7697,7699],{"className":7698,"style":807},[614],[104,7700],{},[104,7702,470],{"className":7703},[566],[104,7705,485],{"className":7706},[207],[104,7708],{"className":7709,"style":235},[213],[104,7711,530],{"className":7712},[239],[104,7714],{"className":7715,"style":235},[213],[104,7717,7719,7722,7768],{"className":7718},[198],[104,7720],{"className":7721,"style":558},[202],[104,7723,7725,7728],{"className":7724},[207],[104,7726,11],{"className":7727},[207,208],[104,7729,7731],{"className":7730},[697],[104,7732,7734,7760],{"className":7733},[605,606],[104,7735,7737,7757],{"className":7736},[610],[104,7738,7740],{"className":7739,"style":5373},[614],[104,7741,7742,7745],{"style":788},[104,7743],{"className":7744,"style":714},[622],[104,7746,7748],{"className":7747},[627,628,629,630],[104,7749,7751,7754],{"className":7750},[207,630],[104,7752,7577],{"className":7753,"style":7686},[207,208,630],[104,7755,5338],{"className":7756,"style":5386},[207,208,630],[104,7758,667],{"className":7759},[666],[104,7761,7763],{"className":7762},[610],[104,7764,7766],{"className":7765,"style":807},[614],[104,7767],{},[104,7769,476],{"className":7770},[575],", the same sum, just algebraically rearranged) and explains why it (together with cross-entropy, the more common alternative) is preferred over the raw error rate for ",[15,7773,7774],{},"growing"," the tree: it's more sensitive to small changes in class proportions within a group, so it guides which question splits best even when no question yet classifies everything correctly.",[11,7777,7778],{},"A binary split (separating the group into \"equal to this value\" versus \"different\") has a combined impurity, the average of both sides' impurities, weighted by each side's size:",[96,7780,7782],{"className":98,"code":7781,"language":100,"meta":57,"style":57},"def impurity_value(x, y, value, impurity_function):\n    equals = x == value\n    equals_impurity = impurity_function(y[equals])\n    not_equals_impurity = impurity_function(y[~equals])\n    return (np.mean(equals)) * equals_impurity + (np.mean(~equals)) * not_equals_impurity\n",[87,7783,7784,7789,7794,7799,7804],{"__ignoreMap":57},[104,7785,7786],{"class":106,"line":107},[104,7787,7788],{},"def impurity_value(x, y, value, impurity_function):\n",[104,7790,7791],{"class":106,"line":58},[104,7792,7793],{},"    equals = x == value\n",[104,7795,7796],{"class":106,"line":118},[104,7797,7798],{},"    equals_impurity = impurity_function(y[equals])\n",[104,7800,7801],{"class":106,"line":124},[104,7802,7803],{},"    not_equals_impurity = impurity_function(y[~equals])\n",[104,7805,7806],{"class":106,"line":308},[104,7807,7808],{},"    return (np.mean(equals)) * equals_impurity + (np.mean(~equals)) * not_equals_impurity\n",[11,7810,7811,7812,7814],{},"And the greedy tree tests ",[15,7813,5268],{}," possible variable\u002Fvalue combination, keeping whichever gives the lowest combined impurity:",[96,7816,7818],{"className":98,"code":7817,"language":100,"meta":57,"style":57},"def best_feature(X, y, impurity_function):\n    best_feature, best_value, best_value_impurity = None, None, float('inf')\n    for feature in range(X.shape[1]):\n        value, _ = best_split(X[:,feature], y, impurity_function)\n        feature_impurity = impurity_value(X[:,feature], y, value, impurity_function)\n        if feature_impurity \u003C best_value_impurity:\n            best_feature, best_value_impurity, best_value = feature, feature_impurity, value\n    return best_feature, best_value, best_value_impurity\n",[87,7819,7820,7825,7830,7835,7840,7845,7850,7855],{"__ignoreMap":57},[104,7821,7822],{"class":106,"line":107},[104,7823,7824],{},"def best_feature(X, y, impurity_function):\n",[104,7826,7827],{"class":106,"line":58},[104,7828,7829],{},"    best_feature, best_value, best_value_impurity = None, None, float('inf')\n",[104,7831,7832],{"class":106,"line":118},[104,7833,7834],{},"    for feature in range(X.shape[1]):\n",[104,7836,7837],{"class":106,"line":124},[104,7838,7839],{},"        value, _ = best_split(X[:,feature], y, impurity_function)\n",[104,7841,7842],{"class":106,"line":308},[104,7843,7844],{},"        feature_impurity = impurity_value(X[:,feature], y, value, impurity_function)\n",[104,7846,7847],{"class":106,"line":417},[104,7848,7849],{},"        if feature_impurity \u003C best_value_impurity:\n",[104,7851,7852],{"class":106,"line":422},[104,7853,7854],{},"            best_feature, best_value_impurity, best_value = feature, feature_impurity, value\n",[104,7856,7857],{"class":106,"line":428},[104,7858,7859],{},"    return best_feature, best_value, best_value_impurity\n",[433,7861,7862],{},[11,7863,7864,7866,7867,7869,7870,7873],{},[15,7865,1629],{}," the first question the tree picks is about ",[87,7868,6978],{}," (variable 5), splitting into \"",[87,7871,7872],{},"low","\" versus the rest, with combined impurity 0.385, the lowest among all 6 variables tested. That tracks in a pretty human way: low safety probably fails the car outright, so that question alone already separates a lot of ground.",[79,7875,7877],{"id":7876},"the-real-greedy-tree","The real greedy tree",[11,7879,7880],{},"Putting it all together, every node in the tree asks \"is this variable equal to this value?\" and recurses into both groups, always picking whichever question reduces impurity the most:",[433,7882,7883],{},[11,7884,7885,7887,7888,1863],{},[15,7886,1629],{}," 100% accuracy on the data it trained on. 96.8% on a held-out test set. 97.2% average accuracy on 5-fold cross-validation (0.977, 0.986, 0.962, 0.968, 0.968), ",[20,7889,7890],{"href":5610},"the same technique I already saw pay off in the previous post",[11,7892,7893],{},"100% on training is always an overfitting red flag (an unbounded tree can always memorize), but here the test and cross-validation numbers are also very good, so it isn't hollow memorization this time: this particular dataset comes from a deterministic rule (it's a synthetic dataset, built from a car-rating rule table, with zero noise), so a sufficiently deep tree can literally reconstruct the true rule behind the data.",[79,7895,7897],{"id":7896},"limiting-depth-doesnt-always-help","Limiting depth doesn't always help",[11,7899,7900,7901,1211],{},"The most direct way to rein in a tree is to cap how many questions in a row it can ask, ",[87,7902,7903],{},"max_depth",[96,7905,7907],{"className":98,"code":7906,"language":100,"meta":57,"style":57},"class DecisionTree(BaseEstimator, ClassifierMixin):\n    def __init__(self, max_depth=9999999):\n        self.max_depth = max_depth\n    def fit(self, X, y):\n        ...\n        if sum(equals) > 0 and sum(~equals) > 0 and self.max_depth > 0:\n            self.equals_tree = DecisionTree(self.max_depth-1).fit(X[equals], y[equals])\n            ...\n",[87,7908,7909,7913,7918,7923,7927,7932,7937,7942],{"__ignoreMap":57},[104,7910,7911],{"class":106,"line":107},[104,7912,7107],{},[104,7914,7915],{"class":106,"line":58},[104,7916,7917],{},"    def __init__(self, max_depth=9999999):\n",[104,7919,7920],{"class":106,"line":118},[104,7921,7922],{},"        self.max_depth = max_depth\n",[104,7924,7925],{"class":106,"line":124},[104,7926,1469],{},[104,7928,7929],{"class":106,"line":308},[104,7930,7931],{},"        ...\n",[104,7933,7934],{"class":106,"line":417},[104,7935,7936],{},"        if sum(equals) > 0 and sum(~equals) > 0 and self.max_depth > 0:\n",[104,7938,7939],{"class":106,"line":422},[104,7940,7941],{},"            self.equals_tree = DecisionTree(self.max_depth-1).fit(X[equals], y[equals])\n",[104,7943,7944],{"class":106,"line":428},[104,7945,7946],{},"            ...\n",[433,7948,7949],{},[11,7950,7951,4296,7953,7956,7957,7960],{},[15,7952,1629],{},[87,7954,7955],{},"max_depth=5",", cross-validation accuracy ",[15,7958,7959],{},"drops"," to 86.6% (versus 97.2% with no limit at all).",[11,7962,7963,7964,7967,7968,7970],{},"Counterintuitive at first, but it connects directly to why this dataset is \"clean\": since the true rule behind the data genuinely depends on combining several variables in sequence, cutting the tree short takes away exactly the capacity it needs to represent the full rule. This isn't \"regularization fighting overfitting,\" it's ",[15,7965,7966],{},"underfitting"," (the tree becomes too simple for this specific problem). The lesson isn't \"always cap depth,\" it's \"measure before deciding\": the right ",[87,7969,7903],{}," depends entirely on how complicated the true pattern is, and the only way to know is testing, with cross-validation, not assuming.",[11,7972,7973,7974,1499,7977,7980],{},"With ",[87,7975,7976],{},"max_depth=20",[87,7978,7979],{},"min_sample_split=10"," (stop splitting a group of 10 examples or fewer, even if it's still impure), the result lands in between: 95.6%, slightly below the unlimited optimum, but already much more controlled than growing with no cap at all.",[79,7982,7984],{"id":7983},"continuous-attributes-same-idea-the-cut-changes","Continuous attributes: same idea, the cut changes",[11,7986,7987,7989,7990,1499,7993,7996,7997,8000],{},[87,7988,6946],{}," switches datasets (Iris, the classic 3-species flower dataset, using just ",[87,7991,7992],{},"petal length",[87,7994,7995],{},"petal width",") and switches the type of question: instead of \"is it equal to this value?\", the question becomes \"is it ",[15,7998,7999],{},"greater than or equal"," to this threshold?\". To find the best threshold on a continuous variable, the professor sorts the values and tests the midpoint between every pair of neighbors:",[96,8002,8004],{"className":98,"code":8003,"language":100,"meta":57,"style":57},"def best_split(x, y, impurity_function):\n    best_value, best_value_impurity = 0, float('inf')\n    x = np.sort(x)\n    for i in range(1, len(x)):\n        value = (x[i-1]+x[i])\u002F2\n        value_impurity = impurity_value(x, y, value, impurity_function)\n        if value_impurity \u003C best_value_impurity:\n            best_value, best_value_impurity = value, value_impurity\n    return best_value, best_value_impurity\n",[87,8005,8006,8011,8016,8021,8026,8031,8036,8041,8046],{"__ignoreMap":57},[104,8007,8008],{"class":106,"line":107},[104,8009,8010],{},"def best_split(x, y, impurity_function):\n",[104,8012,8013],{"class":106,"line":58},[104,8014,8015],{},"    best_value, best_value_impurity = 0, float('inf')\n",[104,8017,8018],{"class":106,"line":118},[104,8019,8020],{},"    x = np.sort(x)\n",[104,8022,8023],{"class":106,"line":124},[104,8024,8025],{},"    for i in range(1, len(x)):\n",[104,8027,8028],{"class":106,"line":308},[104,8029,8030],{},"        value = (x[i-1]+x[i])\u002F2\n",[104,8032,8033],{"class":106,"line":417},[104,8034,8035],{},"        value_impurity = impurity_value(x, y, value, impurity_function)\n",[104,8037,8038],{"class":106,"line":422},[104,8039,8040],{},"        if value_impurity \u003C best_value_impurity:\n",[104,8042,8043],{"class":106,"line":428},[104,8044,8045],{},"            best_value, best_value_impurity = value, value_impurity\n",[104,8047,8048],{"class":106,"line":1689},[104,8049,8050],{},"    return best_value, best_value_impurity\n",[433,8052,8053],{},[11,8054,8055,8057,8058,8061],{},[15,8056,1629],{}," the first question the tree learns is ",[87,8059,8060],{},"petal length >= 2.45",", with impurity 0.333 (the lowest possible here, since that single question already perfectly separates one of the 3 species from the rest).",[79,8063,8065],{"id":8064},"interactive-watching-depth-reshape-the-boundary-live","Interactive: watching depth reshape the boundary live",[11,8067,8068],{},"My own reconstruction of the same algorithm, on the 150 real Iris points (the same two variables, petal length and width). Click through the depth values and notice how the boundary gains a new \"step\" at each level:",[8070,8071],"decision-tree-explorer",{":classes":8072,":depth-options":8073,":initial-max-depth":485,":x-max":8074,":x-min":8075,":x-train":8076,":y-max":8077,":y-min":331,":y-train":8078,"class0-label":8079,"class1-label":8080,"class2-label":8081,"x-label":8082,"y-label":8083},"[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","[1, 2, 3, 5, 9999]","7.1","0.8","[1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.6, 1.4, 1.1, 1.2, 1.5, 1.3, 1.4, 1.7, 1.5, 1.7, 1.5, 1.0, 1.7, 1.9, 1.6, 1.6, 1.5, 1.4, 1.6, 1.6, 1.5, 1.5, 1.4, 1.5, 1.2, 1.3, 1.4, 1.3, 1.5, 1.3, 1.3, 1.3, 1.6, 1.9, 1.4, 1.6, 1.4, 1.5, 1.4, 4.7, 4.5, 4.9, 4.0, 4.6, 4.5, 4.7, 3.3, 4.6, 3.9, 3.5, 4.2, 4.0, 4.7, 3.6, 4.4, 4.5, 4.1, 4.5, 3.9, 4.8, 4.0, 4.9, 4.7, 4.3, 4.4, 4.8, 5.0, 4.5, 3.5, 3.8, 3.7, 3.9, 5.1, 4.5, 4.5, 4.7, 4.4, 4.1, 4.0, 4.4, 4.6, 4.0, 3.3, 4.2, 4.2, 4.2, 4.3, 3.0, 4.1, 6.0, 5.1, 5.9, 5.6, 5.8, 6.6, 4.5, 6.3, 5.8, 6.1, 5.1, 5.3, 5.5, 5.0, 5.1, 5.3, 5.5, 6.7, 6.9, 5.0, 5.7, 4.9, 6.7, 4.9, 5.7, 6.0, 4.8, 4.9, 5.6, 5.8, 6.1, 6.4, 5.6, 5.1, 5.6, 6.1, 5.6, 5.5, 4.8, 5.4, 5.6, 5.1, 5.1, 5.9, 5.7, 5.2, 5.0, 5.2, 5.4, 5.1]","2.6","[0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.4, 0.4, 0.3, 0.3, 0.3, 0.2, 0.4, 0.2, 0.5, 0.2, 0.2, 0.4, 0.2, 0.2, 0.2, 0.2, 0.4, 0.1, 0.2, 0.2, 0.2, 0.2, 0.1, 0.2, 0.2, 0.3, 0.3, 0.2, 0.6, 0.4, 0.3, 0.2, 0.2, 0.2, 0.2, 1.4, 1.5, 1.5, 1.3, 1.5, 1.3, 1.6, 1.0, 1.3, 1.4, 1.0, 1.5, 1.0, 1.4, 1.3, 1.4, 1.5, 1.0, 1.5, 1.1, 1.8, 1.3, 1.5, 1.2, 1.3, 1.4, 1.4, 1.7, 1.5, 1.0, 1.1, 1.0, 1.2, 1.6, 1.5, 1.6, 1.5, 1.3, 1.3, 1.3, 1.2, 1.4, 1.2, 1.0, 1.3, 1.2, 1.3, 1.3, 1.1, 1.3, 2.5, 1.9, 2.1, 1.8, 2.2, 2.1, 1.7, 1.8, 1.8, 2.5, 2.0, 1.9, 2.1, 2.0, 2.4, 2.3, 1.8, 2.2, 2.3, 1.5, 2.3, 2.0, 2.0, 1.8, 2.1, 1.8, 1.8, 1.8, 2.1, 1.6, 1.9, 2.0, 2.2, 1.5, 1.4, 2.3, 2.4, 1.8, 1.8, 2.1, 2.4, 2.3, 1.9, 2.3, 2.5, 2.3, 1.9, 2.0, 2.3, 1.8]","Setosa","Versicolor","Virginica","petal length (cm)","petal width (cm)",[11,8085,8086,8087,8089,8090,8093],{},"At depth 1, the tree only asks one question (the same ",[87,8088,8060],{}," from above), so the chart turns into exactly two regions, a single vertical line. From depth 2 on, a second cut appears splitting the second region in two, and so on: each depth level adds at most one more cut per existing region. Also notice the cuts are always ",[15,8091,8092],{},"vertical or horizontal lines",", never diagonal, because every question looks at one variable at a time.",[79,8095,8097],{"id":8096},"what-the-tree-cant-do-well","What the tree can't do well",[11,8099,8100,8101,8104,8105,8108],{},"That last observation is one of the limitations Bishop points out: decision trees only cut ",[15,8102,8103],{},"axis-aligned",". If the \"true\" boundary between two classes ran diagonally, a tree would need a staircase of cuts to approximate it, while a single diagonal boundary would solve it in one shot (compare with the previous post: ",[20,8106,8107],{"href":5766},"KNN's regions"," were already smoother, without this axis limitation).",[11,8110,8111,8112,1863],{},"Bishop also cites another problem, instability: a small change in the training data can change the entire tree structure, because the choice of which variable to split on first at the top cascades down through the whole tree beneath it. The notebook itself shows this by accident: running the same training three times, with different train\u002Ftest splits (no seed fixed), test accuracy came out 90%, then 96.7%, then 100%. With only 30 test examples, a difference of 2-3 classifications already swings accuracy by several percentage points, exactly why ",[20,8113,8114],{"href":5610},"cross-validation, not a single split, is the right way to measure this",[79,8116,1870],{"id":1869},[1872,8118,8119,8127],{},[1875,8120,8121],{},[1878,8122,8123,8125],{},[1881,8124,1884],{"align":1883},[1881,8126,1887],{"align":1883},[1889,8128,8129,8137,8145],{},[1878,8130,8131,8134],{},[1894,8132,8133],{"align":1883},"A dumb baseline helps interpret any accuracy number",[1894,8135,8136],{"align":1883},"With imbalanced classes (70% in one alone), the baseline can be surprisingly high, and \"97% accuracy\" without comparing against it says nothing",[1878,8138,8139,8142],{},[1894,8140,8141],{"align":1883},"Overfitting happens when the model has too much freedom",[1894,8143,8144],{"align":1883},"An unbounded tree always memorizes training, but capping it too aggressively can also backfire, if the true pattern genuinely needs that much complexity",[1878,8146,8147,8150],{},[1894,8148,8149],{"align":1883},"Cross-validation gives a more stable estimate",[1894,8151,8152],{"align":1883},"Trees are especially unstable to small changes in training, so a single test split is even more misleading here than for other models",[79,8154,1939],{"id":1938},[11,8156,8157,8158,8160],{},"I use the car evaluation dataset (the categorical one, ",[87,8159,6942],{},") to compare different depths systematically, with real cross-validation at each one, instead of testing one value at a time.",[96,8162,8164],{"className":98,"code":8163,"language":100,"meta":57,"style":57},"for depth in [1, 2, 3, 5, 10, 9999]:\n    model = DecisionTree(max_depth=depth, min_sample_split=2)\n    scores = cross_val_score(model, X, y, cv=KFold(n_splits=5, shuffle=True))\n    print(depth, np.mean(scores))\n",[87,8165,8166,8171,8176,8181],{"__ignoreMap":57},[104,8167,8168],{"class":106,"line":107},[104,8169,8170],{},"for depth in [1, 2, 3, 5, 10, 9999]:\n",[104,8172,8173],{"class":106,"line":58},[104,8174,8175],{},"    model = DecisionTree(max_depth=depth, min_sample_split=2)\n",[104,8177,8178],{"class":106,"line":118},[104,8179,8180],{},"    scores = cross_val_score(model, X, y, cv=KFold(n_splits=5, shuffle=True))\n",[104,8182,8183],{"class":106,"line":124},[104,8184,8185],{},"    print(depth, np.mean(scores))\n",[1872,8187,8188,8198],{},[1875,8189,8190],{},[1878,8191,8192,8195],{},[1881,8193,8194],{"align":5561},"Max depth",[1881,8196,8197],{"align":3186},"Accuracy (5-fold CV)",[1889,8199,8200,8207,8214],{},[1878,8201,8202,8204],{},[1894,8203,485],{"align":5561},[1894,8205,8206],{"align":3186},"≈ 0.78 (a single question, just above the 0.70 baseline)",[1878,8208,8209,8211],{},[1894,8210,332],{"align":5561},[1894,8212,8213],{"align":3186},"0.866",[1878,8215,8216,8219],{},[1894,8217,8218],{"align":5561},"Unlimited",[1894,8220,8221],{"align":3186},[15,8222,8223],{},"0.972",[11,8225,8226],{},"(The depth-1 row is my own quick estimate along the same idea, not a cell from the original notebook. The other two are the real numbers already shown in this post.) The trend is clear: on Car Evaluation, more depth almost always helps, because the true pattern behind the data is genuinely complex (it depends on combining several of the 6 variables) with zero noise getting in the way. That's the exact opposite of what usually happens with real, noisy data, where too much depth memorizes the noise instead of the pattern. Same lesson again: there's no universal \"right depth,\" there's testing with cross-validation and letting the data decide.",[1982,8228,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":8230},[8231,8232,8233,8234,8235,8236,8237,8238,8239,8240],{"id":6950,"depth":58,"text":6951},{"id":7089,"depth":58,"text":7090},{"id":7172,"depth":58,"text":7173},{"id":7876,"depth":58,"text":7877},{"id":7896,"depth":58,"text":7897},{"id":7983,"depth":58,"text":7984},{"id":8064,"depth":58,"text":8065},{"id":8096,"depth":58,"text":8097},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lectures 5 and 5b: the professor builds a greedy decision tree from scratch, guided by Gini impurity, first on categorical attributes, then on continuous ones. I explain why limiting depth doesn't always help.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees",{"title":6934,"description":8241},"en\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees",[8247,7532,3802],"decision-tree","KOQ97bJpbObN6bLfH2VqEgp9s_NFoL1hArCb5lvM8T4",{"id":8250,"title":8251,"body":8252,"cover":3,"date":9148,"description":9149,"extension":61,"meta":9150,"navigation":63,"order":422,"path":9151,"playlist":2000,"seo":9152,"status":66,"stem":9153,"tags":9154,"__hash__":9158},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fensembles.md","Ensembles: the Wisdom of Crowds (and Its Limits)",{"type":8,"value":8253,"toc":9137},[8254,8257,8261,8275,8279,8282,8319,8322,8356,8363,8471,8478,8482,8485,8492,8499,8514,8525,8540,8551,8562,8565,8569,8579,8588,8594,8601,8611,8618,8630,8634,8906,8916,8923,8927,8937,8944,8954,8958,8965,8976,8983,8986,8988,9028,9031,9033,9039,9073,9128,9135],[11,8255,8256],{},"Lecture 6, and the professor leaves \"one model\" behind for the first time in the course: instead of picking the best classifier, he trains several and lets them vote. The dataset changes shape too: real face recognition.",[79,8258,8260],{"id":8259},"the-dataset-faces-not-numbers","The dataset: faces, not numbers",[11,8262,8263,8266,8267,8270,8271,8274],{},[87,8264,8265],{},"fetch_olivetti_faces"," brings 400 photos (64×64 pixels, so each photo turns into a vector of ",[15,8268,8269],{},"4096 numbers",", one per pixel) of 40 different people, 10 photos each. The task is to say, from the photo, ",[15,8272,8273],{},"which of the 40 people"," it is. That's already a much harder problem than anything I've seen so far in this playlist: 40 classes (against 3 or 4 in earlier lectures) and 4096 input variables (against 2, 6, 13, or 30).",[79,8276,8278],{"id":8277},"three-different-guesses-one-vote","Three different guesses, one vote",[11,8280,8281],{},"The professor trains three very different classifiers on the same normalized data:",[1872,8283,8284,8293],{},[1875,8285,8286],{},[1878,8287,8288,8290],{},[1881,8289,341],{"align":1883},[1881,8291,8292],{"align":3186},"Accuracy alone",[1889,8294,8295,8303,8311],{},[1878,8296,8297,8300],{},[1894,8298,8299],{"align":1883},"KNN (K=5)",[1894,8301,8302],{"align":3186},"0.8625",[1878,8304,8305,8308],{},[1894,8306,8307],{"align":1883},"Gaussian Naive Bayes",[1894,8309,8310],{"align":3186},"0.8375",[1878,8312,8313,8316],{},[1894,8314,8315],{"align":1883},"Perceptron",[1894,8317,8318],{"align":3186},"0.875",[11,8320,8321],{},"None crosses 88%. But instead of picking the best of the three, he combines all three predictions and votes for the most common one:",[96,8323,8325],{"className":98,"code":8324,"language":100,"meta":57,"style":57},"hits = np.stack((knn_hits, gnb_hits, ppn_hits))\ny_pred = np.stack((knn_pred, gnb_pred, ppn_pred))\n\nfrom scipy.stats import mode\ny_pred_mode, _ = mode(y_pred, axis=0)\nvote_hits = y_pred_mode == y_test\n",[87,8326,8327,8332,8337,8341,8346,8351],{"__ignoreMap":57},[104,8328,8329],{"class":106,"line":107},[104,8330,8331],{},"hits = np.stack((knn_hits, gnb_hits, ppn_hits))\n",[104,8333,8334],{"class":106,"line":58},[104,8335,8336],{},"y_pred = np.stack((knn_pred, gnb_pred, ppn_pred))\n",[104,8338,8339],{"class":106,"line":118},[104,8340,300],{"emptyLinePlaceholder":63},[104,8342,8343],{"class":106,"line":124},[104,8344,8345],{},"from scipy.stats import mode\n",[104,8347,8348],{"class":106,"line":308},[104,8349,8350],{},"y_pred_mode, _ = mode(y_pred, axis=0)\n",[104,8352,8353],{"class":106,"line":417},[104,8354,8355],{},"vote_hits = y_pred_mode == y_test\n",[433,8357,8358],{},[11,8359,8360,8362],{},[15,8361,1629],{}," 0.9625. Better than any of the three alone, by a wide margin.",[11,8364,8365,8366,8369,8370,4916,8408,8436,8437,8470],{},"Bishop calls this a ",[15,8367,8368],{},"committee"," (chapter 14.2), and the math behind it is neat: if each model's errors are uncorrelated (one makes mistakes in a different place than the other), the committee's average error drops by a factor of ",[104,8371,8373,8392],{"className":8372},[148],[104,8374,8376],{"className":8375},[152],[154,8377,8378],{"xmlns":156},[158,8379,8380,8389],{},[161,8381,8382,8384,8386],{},[483,8383,485],{},[164,8385,318],{"mathvariant":959},[164,8387,8388],{},"M",[186,8390,8391],{"encoding":188},"1\u002FM",[104,8393,8395],{"className":8394,"ariaHidden":194},[193],[104,8396,8398,8401,8405],{"className":8397},[198],[104,8399],{"className":8400,"style":558},[202],[104,8402,8404],{"className":8403},[207],"1\u002F",[104,8406,8388],{"className":8407,"style":745},[207,208],[104,8409,8411,8424],{"className":8410},[148],[104,8412,8414],{"className":8413},[152],[154,8415,8416],{"xmlns":156},[158,8417,8418,8422],{},[161,8419,8420],{},[164,8421,8388],{},[186,8423,8388],{"encoding":188},[104,8425,8427],{"className":8426,"ariaHidden":194},[193],[104,8428,8430,8433],{"className":8429},[198],[104,8431],{"className":8432,"style":5314},[202],[104,8434,8388],{"className":8435,"style":745},[207,208]," = number of models) compared to each model's own average error. ",[104,8438,8440,8458],{"className":8439},[148],[104,8441,8443],{"className":8442},[152],[154,8444,8445],{"xmlns":156},[158,8446,8447,8455],{},[161,8448,8449,8451,8453],{},[483,8450,485],{},[164,8452,318],{"mathvariant":959},[483,8454,5580],{},[186,8456,8457],{"encoding":188},"1\u002F3",[104,8459,8461],{"className":8460,"ariaHidden":194},[193],[104,8462,8464,8467],{"className":8463},[198],[104,8465],{"className":8466,"style":558},[202],[104,8468,8457],{"className":8469},[207]," of the error is too optimistic to happen in practice (Bishop himself warns: \"in practice the errors are typically highly correlated, and the reduction in overall error is generally small\"), but the core idea holds: where KNN misses, the Perceptron sometimes gets it right, and vice versa, so the majority tends to be right even when any one of the three individually is wrong.",[11,8472,8473,8474,8477],{},"Scikit-learn's ",[87,8475,8476],{},"VotingClassifier"," runs exactly this computation, and matches the same 0.9625, confirming the hand-rolled version is correct.",[79,8479,8481],{"id":8480},"the-catch-voting-doesnt-help-if-everyone-thinks-alike","The catch: voting doesn't help if everyone thinks alike",[11,8483,8484],{},"Here comes the most important experiment in the post. First, the professor swaps the three different models for three KNNs with different K (1, 3, 5):",[433,8486,8487],{},[11,8488,8489,8491],{},[15,8490,1629],{}," 0.90. Worse than the three different models (0.9625).",[11,8493,8494,8495,8498],{},"Then something even more revealing. A single decision tree (no limit, nothing special): 0.50, pretty weak on this 4096-variable dataset. What if I train ",[15,8496,8497],{},"10 copies"," of that same tree and have them vote?",[96,8500,8502],{"className":98,"code":8501,"language":100,"meta":57,"style":57},"base_estimators = [(f\"dtc({i})\", DecisionTreeClassifier()) for i in range(10)]\nvoting_clf = VotingClassifier(base_estimators, voting='hard')\n",[87,8503,8504,8509],{"__ignoreMap":57},[104,8505,8506],{"class":106,"line":107},[104,8507,8508],{},"base_estimators = [(f\"dtc({i})\", DecisionTreeClassifier()) for i in range(10)]\n",[104,8510,8511],{"class":106,"line":58},[104,8512,8513],{},"voting_clf = VotingClassifier(base_estimators, voting='hard')\n",[433,8515,8516],{},[11,8517,8518,8520,8521,8524],{},[15,8519,1629],{}," 0.50. ",[15,8522,8523],{},"Exactly the same"," as the single tree.",[11,8526,8527,8528,8531,8532,8535,8536,8539],{},"That makes complete sense, and it's exactly what Bishop warned about: scikit-learn's default ",[87,8529,8530],{},"DecisionTreeClassifier()"," is ",[15,8533,8534],{},"deterministic",", it always picks the same optimal question to split each group. Ten copies of the same algorithm, on the same data, always land on the same tree, always miss the exact same examples. There's no \"majority\" when everyone votes identically: 10 identical votes are worth the same as 1. A committee only works when its members genuinely ",[15,8537,8538],{},"disagree",", and disagreement requires some real source of variation between the models.",[11,8541,8542,8543,8546,8547,8550],{},"The professor introduces that variation by swapping how the split gets picked from ",[87,8544,8545],{},"'best'"," (always the optimal question) to ",[87,8548,8549],{},"'random'"," (the tree picks a good question, but at random, among the candidates):",[433,8552,8553],{},[11,8554,8555,8557,8558,8561],{},[15,8556,1629],{}," a single random tree: 0.6125, already better than the deterministic one alone. Ten random trees voting: ",[15,8559,8560],{},"0.7875",", a serious jump.",[11,8563,8564],{},"Now every tree is genuinely different from the others (real randomness in picking the question), so the errors stop being identical, and voting starts genuinely paying off.",[79,8566,8568],{"id":8567},"bagging-random-forest-and-extra-trees-organized-randomness","Bagging, Random Forest, and Extra Trees: organized randomness",[11,8570,8571,8574,8575,8578],{},[87,8572,8573],{},"BaggingClassifier"," generalizes that idea: it trains several copies of the same model, each one seeing a ",[15,8576,8577],{},"random sample with replacement"," of the training data (bootstrap), so each tree sees a slightly different set, on top of picking different splits:",[433,8580,8581],{},[11,8582,8583,1807,8585,8587],{},[15,8584,1629],{},[87,8586,8573],{}," with 100 random trees: 0.9125.",[11,8589,8590,8593],{},[87,8591,8592],{},"RandomForestClassifier"," is essentially that same recipe pre-packaged (bootstrap + trees, but also sampling a random subset of the variables at every split, not just shuffling which examples each tree sees):",[433,8595,8596],{},[11,8597,8598,8600],{},[15,8599,1629],{}," Random Forest, 100 trees: 0.9125, tied with manual bagging.",[11,8602,5840,8603,8606,8607,8610],{},[87,8604,8605],{},"ExtraTreesClassifier"," goes one step further, also randomizing the ",[15,8608,8609],{},"threshold"," of each split instead of searching for the optimal one (even more randomness):",[433,8612,8613],{},[11,8614,8615,8617],{},[15,8616,1629],{}," 0.9625, the best among all the forests tested, tying the original vote of three different models.",[11,8619,8620,8621,8623,8624,8626,8627,8629],{},"These numbers aren't just illustration, they confirm the previous section's thesis: what makes a committee worth having is how much the trees disagree with each other. Bootstrap alone (",[87,8622,8573],{},") already samples which examples each tree sees, and also sampling which variable each tree can split on (",[87,8625,8592],{},") ties that exactly, 0.9125 both, which suggests that on this dataset the bootstrap variation was already generating nearly all the useful disagreement, and forcing trees to look at different columns on top of that didn't add much more. Also randomizing the split threshold itself (",[87,8628,8605],{},"), not just which examples and which variables, but literally the value that separates \"yes\" from \"no\" at each split, is a stronger source of disagreement: two trees can see the same data, the same variables, and still cut in quite different places. That's why the jump from 0.9125 to 0.9625 comes specifically from ExtraTrees: more randomness in how the trees get built, more genuinely different trees, more chance their mistakes don't line up.",[79,8631,8633],{"id":8632},"boosting-learning-from-the-last-mistake","Boosting: learning from the last mistake",[11,8635,8636,8639,8640,8643,8644,8714,8715,8718,8719,8833,8834,8905],{},[87,8637,8638],{},"AdaBoostClassifier"," changes the strategy: instead of training everyone in parallel and voting, it trains in ",[15,8641,8642],{},"sequence",", and each new model gives more weight to the examples the previous model got wrong. Bishop describes the algorithm formally: every example carries a weight ",[104,8645,8647,8665],{"className":8646},[148],[104,8648,8650],{"className":8649},[152],[154,8651,8652],{"xmlns":156},[158,8653,8654,8662],{},[161,8655,8656],{},[515,8657,8658,8660],{},[164,8659,317],{},[164,8661,499],{},[186,8663,8664],{"encoding":188},"w_n",[104,8666,8668],{"className":8667,"ariaHidden":194},[193],[104,8669,8671,8674],{"className":8670},[198],[104,8672],{"className":8673,"style":2822},[202],[104,8675,8677,8680],{"className":8676},[207],[104,8678,317],{"className":8679,"style":2829},[207,208],[104,8681,8683],{"className":8682},[697],[104,8684,8686,8706],{"className":8685},[605,606],[104,8687,8689,8703],{"className":8688},[610],[104,8690,8692],{"className":8691,"style":785},[614],[104,8693,8694,8697],{"style":2845},[104,8695],{"className":8696,"style":714},[622],[104,8698,8700],{"className":8699},[627,628,629,630],[104,8701,499],{"className":8702},[207,208,630],[104,8704,667],{"className":8705},[666],[104,8707,8709],{"className":8708},[610],[104,8710,8712],{"className":8711,"style":807},[614],[104,8713],{},", starting equal for everyone. After each classifier is trained, the weight of the examples it missed ",[15,8716,8717],{},"increases"," (multiplied by ",[104,8720,8722,8747],{"className":8721},[148],[104,8723,8725],{"className":8724},[152],[154,8726,8727],{"xmlns":156},[158,8728,8729,8744],{},[161,8730,8731],{},[539,8732,8733,8736],{},[164,8734,8735],{},"e",[515,8737,8738,8741],{},[164,8739,8740],{},"α",[164,8742,8743],{},"m",[186,8745,8746],{"encoding":188},"e^{\\alpha_m}",[104,8748,8750],{"className":8749,"ariaHidden":194},[193],[104,8751,8753,8757],{"className":8752},[198],[104,8754],{"className":8755,"style":8756},[202],"height:0.6644em;",[104,8758,8760,8763],{"className":8759},[207],[104,8761,8735],{"className":8762},[207,208],[104,8764,8766],{"className":8765},[697],[104,8767,8769],{"className":8768},[605],[104,8770,8772],{"className":8771},[610],[104,8773,8775],{"className":8774,"style":8756},[614],[104,8776,8777,8780],{"style":900},[104,8778],{"className":8779,"style":714},[622],[104,8781,8783],{"className":8782},[627,628,629,630],[104,8784,8786],{"className":8785},[207,630],[104,8787,8789,8793],{"className":8788},[207,630],[104,8790,8740],{"className":8791,"style":8792},[207,208,630],"margin-right:0.0037em;",[104,8794,8796],{"className":8795},[697],[104,8797,8799,8824],{"className":8798},[605,606],[104,8800,8802,8821],{"className":8801},[610],[104,8803,8806],{"className":8804,"style":8805},[614],"height:0.1645em;",[104,8807,8809,8813],{"style":8808},"top:-2.357em;margin-left:-0.0037em;margin-right:0.0714em;",[104,8810],{"className":8811,"style":8812},[622],"height:2.5em;",[104,8814,8818],{"className":8815},[627,8816,8817,630],"reset-size3","size1",[104,8819,8743],{"className":8820},[207,208,630],[104,8822,667],{"className":8823},[666],[104,8825,8827],{"className":8826},[610],[104,8828,8831],{"className":8829,"style":8830},[614],"height:0.143em;",[104,8832],{},"), so the next classifier in the sequence is forced to pay more attention to them. In the end, the final prediction is a weighted vote, where more accurate classifiers (larger ",[104,8835,8837,8855],{"className":8836},[148],[104,8838,8840],{"className":8839},[152],[154,8841,8842],{"xmlns":156},[158,8843,8844,8852],{},[161,8845,8846],{},[515,8847,8848,8850],{},[164,8849,8740],{},[164,8851,8743],{},[186,8853,8854],{"encoding":188},"\\alpha_m",[104,8856,8858],{"className":8857,"ariaHidden":194},[193],[104,8859,8861,8864],{"className":8860},[198],[104,8862],{"className":8863,"style":2822},[202],[104,8865,8867,8870],{"className":8866},[207],[104,8868,8740],{"className":8869,"style":8792},[207,208],[104,8871,8873],{"className":8872},[697],[104,8874,8876,8897],{"className":8875},[605,606],[104,8877,8879,8894],{"className":8878},[610],[104,8880,8882],{"className":8881,"style":785},[614],[104,8883,8885,8888],{"style":8884},"top:-2.55em;margin-left:-0.0037em;margin-right:0.05em;",[104,8886],{"className":8887,"style":714},[622],[104,8889,8891],{"className":8890},[627,628,629,630],[104,8892,8743],{"className":8893},[207,208,630],[104,8895,667],{"className":8896},[666],[104,8898,8900],{"className":8899},[610],[104,8901,8903],{"className":8902,"style":807},[614],[104,8904],{},") count for more than the weak ones.",[433,8907,8908],{},[11,8909,8910,8912,8913,8915],{},[15,8911,1629],{}," AdaBoost with deep random trees: 0.925 on test, ",[15,8914,4754],{}," on training.",[11,8917,8918,8919,8922],{},"100% on training is a sign that boosting, given a weak enough base and enough rounds, can also memorize training (the same overfitting ",[20,8920,8921],{"href":8243},"I already saw happen with an unbounded-depth tree","), but the test result still comes out much better than any individual tree alone.",[79,8924,8926],{"id":8925},"stacking-fancier-doesnt-always-win","Stacking: fancier doesn't always win",[11,8928,8929,8932,8933,8936],{},[87,8930,8931],{},"StackingClassifier"," tries to be smarter than voting: instead of counting votes, it trains an extra model (the \"meta-model\") that learns to ",[15,8934,8935],{},"combine"," the base models' predictions, using cross-validation internally so the training data doesn't leak straight into the meta-model.",[433,8938,8939],{},[11,8940,8941,8943],{},[15,8942,1629],{}," stacking the three original models (KNN, Naive Bayes, Perceptron): 0.80. Stacking of voting + Random Forest + Extra Trees: 0.7625.",[11,8945,8946,8947,8949,8950,8953],{},"Worse than the simple vote of the same models (0.9625) both times the professor tried it. Worth stating plainly: the fancier technique did ",[15,8948,137],{}," win here. With few training examples per class (10 photos per person, and stacking's ",[87,8951,8952],{},"cv=3"," still carves off a slice of that just to train the meta-model), there simply isn't enough data left for that meta-model to learn a combination better than \"everyone gets one vote.\"",[79,8955,8957],{"id":8956},"the-humbling-twist-a-simple-model-beats-everyone","The humbling twist: a simple model beats everyone",[11,8959,8960,8961,8964],{},"After all that ensembling, the professor tries plain ",[87,8962,8963],{},"LogisticRegression",", no committee at all:",[433,8966,8967],{},[11,8968,8969,8971,8972,8975],{},[15,8970,1629],{}," 0.975 on test. ",[15,8973,8974],{},"The best result in the entire post",", beating every ensemble tried, including Extra Trees (0.9625) and AdaBoost (0.925).",[11,8977,8978,8979,8982],{},"And to make sure it wasn't luck from a single test split, 5-fold cross-validation confirms it: 0.965 average (1.0, 0.95, 0.95, 0.975, 0.95). The professor still tries tuning hyperparameters with Optuna (50 trials, searching AdaBoost's tree depth, learning rate, and number of estimators), and the best result found after all that is 0.9125 on test, ",[15,8980,8981],{},"still behind"," raw logistic regression.",[11,8984,8985],{},"Why does logistic regression win here? A good guess: with 4096 variables (pixels) and only 400 examples, the problem is already nearly linear in practice, images of the same person form a \"cloud\" that a hyperplane separates reasonably well, and there isn't much genuinely non-linear pattern left for trees and forests to exploit. Tree ensembles shine when the true pattern is genuinely complex and non-linear (like in the previous post, with Car Evaluation). Here, the problem already had a simple, effective way to be solved, and the plain tool beat the fancy one.",[79,8987,1870],{"id":1869},[1872,8989,8990,8998],{},[1875,8991,8992],{},[1878,8993,8994,8996],{},[1881,8995,1884],{"align":1883},[1881,8997,1887],{"align":1883},[1889,8999,9000,9012,9020],{},[1878,9001,9002,9005],{},[1894,9003,9004],{"align":1883},"A single model makes its own best guess alone",[1894,9006,9007,9008,9011],{"align":1883},"Combining several models can beat each one individually, ",[15,9009,9010],{},"if"," they genuinely disagree with each other",[1878,9013,9014,9017],{},[1894,9015,9016],{"align":1883},"An unbounded tree memorizes training",[1894,9018,9019],{"align":1883},"Copying the same deterministic tree 10 times helps nothing: with no real variation between the models, voting changes nothing",[1878,9021,9022,9025],{},[1894,9023,9024],{"align":1883},"A fancier technique is usually better",[1894,9026,9027],{"align":1883},"Not always: stacking lost to the simple vote here, and raw logistic regression beat every ensemble tried",[11,9029,9030],{},"The lesson that sticks: an ensemble is a powerful tool, not a silver bullet. Before stacking models, it's worth checking whether the problem already has a simple, effective solution, because sometimes it does.",[79,9032,1939],{"id":1938},[11,9034,9035,9036,9038],{},"The whole post compared ensembles against ",[87,9037,8963],{}," using only the single test split the notebook had already run. To close with more confidence, I ran real 5-fold cross-validation on the post's three best models (Random Forest, Extra Trees, and logistic regression), all under the same criterion, something the original notebook only did for logistic regression.",[96,9040,9042],{"className":98,"code":9041,"language":100,"meta":57,"style":57},"from sklearn.model_selection import cross_val_score, StratifiedKFold\n\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nfor name, model in models.items():\n    scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')\n    print(name, scores.mean(), scores.std())\n",[87,9043,9044,9049,9053,9058,9063,9068],{"__ignoreMap":57},[104,9045,9046],{"class":106,"line":107},[104,9047,9048],{},"from sklearn.model_selection import cross_val_score, StratifiedKFold\n",[104,9050,9051],{"class":106,"line":58},[104,9052,300],{"emptyLinePlaceholder":63},[104,9054,9055],{"class":106,"line":118},[104,9056,9057],{},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",[104,9059,9060],{"class":106,"line":124},[104,9061,9062],{},"for name, model in models.items():\n",[104,9064,9065],{"class":106,"line":308},[104,9066,9067],{},"    scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')\n",[104,9069,9070],{"class":106,"line":417},[104,9071,9072],{},"    print(name, scores.mean(), scores.std())\n",[1872,9074,9075,9087],{},[1875,9076,9077],{},[1878,9078,9079,9081,9084],{},[1881,9080,341],{"align":1883},[1881,9082,9083],{"align":3186},"Mean accuracy (5-fold)",[1881,9085,9086],{"align":3186},"Standard deviation",[1889,9088,9089,9100,9111],{},[1878,9090,9091,9094,9097],{},[1894,9092,9093],{"align":1883},"Random Forest (100 trees)",[1894,9095,9096],{"align":3186},"0.9450",[1894,9098,9099],{"align":3186},"0.0232",[1878,9101,9102,9105,9108],{},[1894,9103,9104],{"align":1883},"Extra Trees (100 trees)",[1894,9106,9107],{"align":3186},"0.9600",[1894,9109,9110],{"align":3186},"0.0146",[1878,9112,9113,9118,9123],{},[1894,9114,9115],{"align":1883},[15,9116,9117],{},"Logistic Regression",[1894,9119,9120],{"align":3186},[15,9121,9122],{},"0.9750",[1894,9124,9125],{"align":3186},[15,9126,9127],{},"0.0079",[11,9129,9130,9131,9134],{},"Logistic regression's win wasn't luck from the notebook's specific split: it wins across all 5 folds tested, and on top of that with the ",[15,9132,9133],{},"lowest"," standard deviation of the three, meaning the most consistent result fold to fold. That confirms the previous section's lesson numerically, with the extra care of measuring everyone by the same ruler.",[1982,9136,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":9138},[9139,9140,9141,9142,9143,9144,9145,9146,9147],{"id":8259,"depth":58,"text":8260},{"id":8277,"depth":58,"text":8278},{"id":8480,"depth":58,"text":8481},{"id":8567,"depth":58,"text":8568},{"id":8632,"depth":58,"text":8633},{"id":8925,"depth":58,"text":8926},{"id":8956,"depth":58,"text":8957},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"2026-08-20","Lecture 6: the professor combines several weak classifiers on real faces and reaches 96%, but the ending is a lesson in humility: a single simple model beats every ensemble tried, even the fanciest one.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fensembles",{"title":8251,"description":9149},"en\u002Fplaylists\u002Fpattern-recognition\u002Fensembles",[9155,9156,9157],"ensembles","bagging","boosting","EfsVzAzVD2-5zkPp_9hzZIQFZYFh4NpybHPravq4-LU",{"id":9160,"title":9161,"body":9162,"cover":3,"date":9148,"description":9906,"extension":61,"meta":9907,"navigation":63,"order":428,"path":9908,"playlist":2000,"seo":9909,"status":66,"stem":9910,"tags":9911,"__hash__":9914},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic.md","The Titanic: My First Genuinely Messy Dataset",{"type":8,"value":9163,"toc":9896},[9164,9167,9171,9191,9232,9241,9255,9259,9274,9286,9300,9320,9333,9337,9372,9392,9404,9408,9425,9473,9487,9540,9546,9579,9586,9593,9650,9654,9669,9676,9687,9702,9709,9712,9716,9722,9729,9744,9751,9757,9759,9795,9797,9800,9824,9887,9894],[11,9165,9166],{},"Lecture 7, and the professor switches gears: every dataset up to now already arrived ready, no gaps, no useless column, no loose text. The Titanic (Kaggle's most famous dataset, probably everyone's first data science project) doesn't get that luxury.",[79,9168,9170],{"id":9169},"messy-data-891-passengers-not-every-column-is-useful","Messy data: 891 passengers, not every column is useful",[96,9172,9174],{"className":98,"code":9173,"language":100,"meta":57,"style":57},"df = pd.read_csv('train.csv')\ny = df['Survived']\nX = df.drop('Survived', axis=1)\n",[87,9175,9176,9181,9186],{"__ignoreMap":57},[104,9177,9178],{"class":106,"line":107},[104,9179,9180],{},"df = pd.read_csv('train.csv')\n",[104,9182,9183],{"class":106,"line":58},[104,9184,9185],{},"y = df['Survived']\n",[104,9187,9188],{"class":106,"line":118},[104,9189,9190],{},"X = df.drop('Survived', axis=1)\n",[11,9192,9193,9194,9197,9198,9201,9202,1811,9205,1811,9208,1811,9211,9214,9215,9218,9219,1811,9222,1811,9225,1811,9228,9231],{},"891 passengers, ",[87,9195,9196],{},"Survived"," (0 or 1) as the target, and a handful of pretty mixed input columns: ",[87,9199,9200],{},"Pclass"," (ticket class), ",[87,9203,9204],{},"Name",[87,9206,9207],{},"Sex",[87,9209,9210],{},"Age",[87,9212,9213],{},"SibSp"," (siblings\u002Fspouse aboard), ",[87,9216,9217],{},"Parch"," (parents\u002Fchildren aboard), ",[87,9220,9221],{},"Ticket",[87,9223,9224],{},"Fare",[87,9226,9227],{},"Cabin",[87,9229,9230],{},"Embarked"," (port of embarkation). The professor drops four of them right away:",[96,9233,9235],{"className":98,"code":9234,"language":100,"meta":57,"style":57},"caracteristicas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n",[87,9236,9237],{"__ignoreMap":57},[104,9238,9239],{"class":106,"line":107},[104,9240,9234],{},[11,9242,9243,9246,9247,1499,9249,9251,9252,9254],{},[87,9244,9245],{},"PassengerId"," is just a sequential number, carries no information about the person at all. ",[87,9248,9204],{},[87,9250,9221],{}," are free text, almost every value is unique (891 different names for 891 passengers), so a model has no way to generalize from that without processing the text first (extracting a title, like \"Mr.\"\u002F\"Mrs.\", would be a valid move, but that's extra work the lecture didn't do). ",[87,9253,9227],{}," is missing for most rows and is also far too specific.",[79,9256,9258],{"id":9257},"missing-data-is-the-rule-not-the-exception","Missing data is the rule, not the exception",[96,9260,9262],{"className":98,"code":9261,"language":100,"meta":57,"style":57},"for col in Xnum.columns:\n    print(f\"{col:>12} {Xnum[col].isnull().sum():2}\")\n",[87,9263,9264,9269],{"__ignoreMap":57},[104,9265,9266],{"class":106,"line":107},[104,9267,9268],{},"for col in Xnum.columns:\n",[104,9270,9271],{"class":106,"line":58},[104,9272,9273],{},"    print(f\"{col:>12} {Xnum[col].isnull().sum():2}\")\n",[433,9275,9276],{},[11,9277,9278,1807,9280,9282,9283,9285],{},[15,9279,1629],{},[87,9281,9210],{}," is missing for 177 of 891 passengers (almost 20%). ",[87,9284,9230],{}," is missing for 2.",[11,9287,9288,9289,9292,9293,9295,9296,9299],{},"That never happened in this playlist before now: every previous dataset already arrived complete. Here, nearly 1 in every 5 ages is ",[87,9290,9291],{},"NaN",". Dropping those rows would throw away 20% of the data, and dropping the ",[87,9294,9210],{}," column entirely would throw away a probably important variable (children had rescue priority). The professor's solution is ",[15,9297,9298],{},"imputation",": fill the gap with an estimated value.",[96,9301,9303],{"className":98,"code":9302,"language":100,"meta":57,"style":57},"from sklearn.impute import SimpleImputer\nimputer = SimpleImputer(strategy='median')\nXnumTratado = imputer.fit_transform(Xnum)\n",[87,9304,9305,9310,9315],{"__ignoreMap":57},[104,9306,9307],{"class":106,"line":107},[104,9308,9309],{},"from sklearn.impute import SimpleImputer\n",[104,9311,9312],{"class":106,"line":58},[104,9313,9314],{},"imputer = SimpleImputer(strategy='median')\n",[104,9316,9317],{"class":106,"line":118},[104,9318,9319],{},"XnumTratado = imputer.fit_transform(Xnum)\n",[11,9321,9322,9323,9325,9326,9328,9329,9332],{},"For numeric variables, the median (not the mean): the age and fare (",[87,9324,9224],{},") distributions have people well outside the norm (infants a few months old, very expensive first-class fares), and the median doesn't get pulled by those extremes the way the mean does. For categorical variables (",[87,9327,9230],{},"), there's no such thing as a \"median\" port, so the strategy becomes ",[87,9330,9331],{},"'most_frequent'",", whichever value shows up most.",[79,9334,9336],{"id":9335},"a-category-isnt-a-number-even-when-it-looks-like-one","A category isn't a number, even when it looks like one",[11,9338,9339,1499,9341,9343,9344,318,9347,1811,9350,318,9353,318,9356,9359,9360,9363,9364,9367,9368,9371],{},[87,9340,9207],{},[87,9342,9230],{}," are text (",[87,9345,9346],{},"\"male\"",[87,9348,9349],{},"\"female\"",[87,9351,9352],{},"\"S\"",[87,9354,9355],{},"\"C\"",[87,9357,9358],{},"\"Q\"","). A model only understands numbers, but simply numbering them (",[87,9361,9362],{},"male=0, female=1",", or ",[87,9365,9366],{},"S=0, C=1, Q=2",") would invent an order that doesn't exist: why would \"Q\" be \"bigger\" than \"S\"? The right move is ",[15,9369,9370],{},"one-hot encoding",": every category becomes its own binary column.",[96,9373,9375],{"className":98,"code":9374,"language":100,"meta":57,"style":57},"from sklearn.preprocessing import OneHotEncoder\nencoder = OneHotEncoder()\nXcatTratadoHot = encoder.fit_transform(XcatTratado)\n",[87,9376,9377,9382,9387],{"__ignoreMap":57},[104,9378,9379],{"class":106,"line":107},[104,9380,9381],{},"from sklearn.preprocessing import OneHotEncoder\n",[104,9383,9384],{"class":106,"line":58},[104,9385,9386],{},"encoder = OneHotEncoder()\n",[104,9388,9389],{"class":106,"line":118},[104,9390,9391],{},"XcatTratadoHot = encoder.fit_transform(XcatTratado)\n",[433,9393,9394],{},[11,9395,9396,9398,9399,1811,9401,9403],{},[15,9397,1629],{}," the 2 categorical columns (",[87,9400,9207],{},[87,9402,9230],{},") turn into 5 binary columns (2 for sex, since one is already redundant with the other, and 3 for port of embarkation). None of them carries an implicit numeric order, they're just \"is\" or \"isn't\" that category.",[79,9405,9407],{"id":9406},"my-own-transformer-not-everything-is-a-classifier","My own Transformer: not everything is a classifier",[11,9409,9410,9411,3246,9414,9416,9417,9420,9421,9424],{},"Up to now, every custom class in this playlist inherited from ",[87,9412,9413],{},"ClassifierMixin",[87,9415,1506],{},", because it always ended in ",[87,9418,9419],{},".predict()",". Here the goal is different: just transform the data, without predicting anything. That's what ",[87,9422,9423],{},"TransformerMixin"," is for:",[96,9426,9428],{"className":98,"code":9427,"language":100,"meta":57,"style":57},"from sklearn.base import BaseEstimator, TransformerMixin\n\nclass AtributosDesejados(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        self.colunas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n    def fit(self, X, y=None):\n        return self\n    def transform(self, X, y=None):\n        return X.drop(self.colunas_indesejadas, axis=1)\n",[87,9429,9430,9435,9439,9444,9449,9454,9459,9463,9468],{"__ignoreMap":57},[104,9431,9432],{"class":106,"line":107},[104,9433,9434],{},"from sklearn.base import BaseEstimator, TransformerMixin\n",[104,9436,9437],{"class":106,"line":58},[104,9438,300],{"emptyLinePlaceholder":63},[104,9440,9441],{"class":106,"line":118},[104,9442,9443],{},"class AtributosDesejados(BaseEstimator, TransformerMixin):\n",[104,9445,9446],{"class":106,"line":124},[104,9447,9448],{},"    def __init__(self):\n",[104,9450,9451],{"class":106,"line":308},[104,9452,9453],{},"        self.colunas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n",[104,9455,9456],{"class":106,"line":417},[104,9457,9458],{},"    def fit(self, X, y=None):\n",[104,9460,9461],{"class":106,"line":422},[104,9462,1479],{},[104,9464,9465],{"class":106,"line":428},[104,9466,9467],{},"    def transform(self, X, y=None):\n",[104,9469,9470],{"class":106,"line":1689},[104,9471,9472],{},"        return X.drop(self.colunas_indesejadas, axis=1)\n",[11,9474,9475,9476,9479,9480,9483,9484,9486],{},"Along with ",[87,9477,9478],{},"AtributosNumericos"," (keeps only the numeric columns) and ",[87,9481,9482],{},"AtributosCategoricos"," (keeps only the categorical ones), the professor builds two parallel ",[87,9485,1498],{},"s, one for each kind of data:",[96,9488,9490],{"className":98,"code":9489,"language":100,"meta":57,"style":57},"pipenum = Pipeline([\n    ('atributos_numericos', AtributosNumericos()),\n    ('imputer', SimpleImputer(strategy='median')),\n    ('scaler', StandardScaler())\n])\npipecat = Pipeline([\n    ('atributos_categoricos', AtributosCategoricos()),\n    ('imputer', SimpleImputer(strategy='most_frequent')),\n    ('encoder', OneHotEncoder())\n])\n",[87,9491,9492,9497,9502,9507,9512,9516,9521,9526,9531,9536],{"__ignoreMap":57},[104,9493,9494],{"class":106,"line":107},[104,9495,9496],{},"pipenum = Pipeline([\n",[104,9498,9499],{"class":106,"line":58},[104,9500,9501],{},"    ('atributos_numericos', AtributosNumericos()),\n",[104,9503,9504],{"class":106,"line":118},[104,9505,9506],{},"    ('imputer', SimpleImputer(strategy='median')),\n",[104,9508,9509],{"class":106,"line":124},[104,9510,9511],{},"    ('scaler', StandardScaler())\n",[104,9513,9514],{"class":106,"line":308},[104,9515,5868],{},[104,9517,9518],{"class":106,"line":417},[104,9519,9520],{},"pipecat = Pipeline([\n",[104,9522,9523],{"class":106,"line":422},[104,9524,9525],{},"    ('atributos_categoricos', AtributosCategoricos()),\n",[104,9527,9528],{"class":106,"line":428},[104,9529,9530],{},"    ('imputer', SimpleImputer(strategy='most_frequent')),\n",[104,9532,9533],{"class":106,"line":1689},[104,9534,9535],{},"    ('encoder', OneHotEncoder())\n",[104,9537,9538],{"class":106,"line":1695},[104,9539,5868],{},[11,9541,5840,9542,9545],{},[87,9543,9544],{},"FeatureUnion"," joins both outputs into one set of columns, side by side:",[96,9547,9549],{"className":98,"code":9548,"language":100,"meta":57,"style":57},"from sklearn.pipeline import FeatureUnion\n\nunecaracteristicas = FeatureUnion([\n    ('pipenum', pipenum),\n    ('pipecat', pipecat)\n])\n",[87,9550,9551,9556,9560,9565,9570,9575],{"__ignoreMap":57},[104,9552,9553],{"class":106,"line":107},[104,9554,9555],{},"from sklearn.pipeline import FeatureUnion\n",[104,9557,9558],{"class":106,"line":58},[104,9559,300],{"emptyLinePlaceholder":63},[104,9561,9562],{"class":106,"line":118},[104,9563,9564],{},"unecaracteristicas = FeatureUnion([\n",[104,9566,9567],{"class":106,"line":124},[104,9568,9569],{},"    ('pipenum', pipenum),\n",[104,9571,9572],{"class":106,"line":308},[104,9573,9574],{},"    ('pipecat', pipecat)\n",[104,9576,9577],{"class":106,"line":417},[104,9578,5868],{},[433,9580,9581],{},[11,9582,9583,9585],{},[15,9584,1629],{}," the 7 original columns (after dropping the 4 useless ones) become 10 processed columns: 5 numeric (already imputed and normalized) plus 5 categorical (already imputed and one-hot encoded).",[11,9587,9588,9589,9592],{},"The final pipeline chains everything, from raw data to classifier, in a single ",[87,9590,9591],{},".fit()"," call:",[96,9594,9596],{"className":98,"code":9595,"language":100,"meta":57,"style":57},"preproc = Pipeline([\n    ('atributos_desejados', AtributosDesejados()),\n    ('unecaracteristicas', unecaracteristicas),\n    ('to_dense', DenseTransformer())\n])\n\nclf = Pipeline([\n    ('preproc', preproc),\n    ('classificador', RandomForestClassifier())\n])\nclf.fit(X, y)\n",[87,9597,9598,9603,9608,9613,9618,9622,9626,9631,9636,9641,9645],{"__ignoreMap":57},[104,9599,9600],{"class":106,"line":107},[104,9601,9602],{},"preproc = Pipeline([\n",[104,9604,9605],{"class":106,"line":58},[104,9606,9607],{},"    ('atributos_desejados', AtributosDesejados()),\n",[104,9609,9610],{"class":106,"line":118},[104,9611,9612],{},"    ('unecaracteristicas', unecaracteristicas),\n",[104,9614,9615],{"class":106,"line":124},[104,9616,9617],{},"    ('to_dense', DenseTransformer())\n",[104,9619,9620],{"class":106,"line":308},[104,9621,5868],{},[104,9623,9624],{"class":106,"line":417},[104,9625,300],{"emptyLinePlaceholder":63},[104,9627,9628],{"class":106,"line":422},[104,9629,9630],{},"clf = Pipeline([\n",[104,9632,9633],{"class":106,"line":428},[104,9634,9635],{},"    ('preproc', preproc),\n",[104,9637,9638],{"class":106,"line":1689},[104,9639,9640],{},"    ('classificador', RandomForestClassifier())\n",[104,9642,9643],{"class":106,"line":1695},[104,9644,5868],{},[104,9646,9647],{"class":106,"line":1701},[104,9648,9649],{},"clf.fit(X, y)\n",[79,9651,9653],{"id":9652},"the-same-old-mistake-with-a-real-consequence-this-time","The same old mistake, with a real consequence this time",[96,9655,9657],{"className":98,"code":9656,"language":100,"meta":57,"style":57},"y_pred = clf.predict(X)\naccuracy_score(y, y_pred)\n",[87,9658,9659,9664],{"__ignoreMap":57},[104,9660,9661],{"class":106,"line":107},[104,9662,9663],{},"y_pred = clf.predict(X)\n",[104,9665,9666],{"class":106,"line":58},[104,9667,9668],{},"accuracy_score(y, y_pred)\n",[433,9670,9671],{},[11,9672,9673,9675],{},[15,9674,1629],{}," 0.9798. Almost 98% correct.",[11,9677,9678,9679,9682,9683,9686],{},"If you ",[20,9680,9681],{"href":5610},"already read this playlist's post on cross-validation",", that accuracy number should set off an alarm: it's measured on the ",[15,9684,9685],{},"same"," data it trained on. The proof it's memorization, not learning:",[96,9688,9690],{"className":98,"code":9689,"language":100,"meta":57,"style":57},"from sklearn.model_selection import cross_val_score\nscores = cross_val_score(clf, X, y)\n",[87,9691,9692,9697],{"__ignoreMap":57},[104,9693,9694],{"class":106,"line":107},[104,9695,9696],{},"from sklearn.model_selection import cross_val_score\n",[104,9698,9699],{"class":106,"line":58},[104,9700,9701],{},"scores = cross_val_score(clf, X, y)\n",[433,9703,9704],{},[11,9705,9706,9708],{},[15,9707,1629],{}," 0.807 average (0.765, 0.815, 0.854, 0.775, 0.826 across the 5 folds).",[11,9710,9711],{},"A nearly 17-percentage-point drop between \"accuracy on training\" and \"honest accuracy.\" The Random Forest, with no tree limit at all, memorizes a good chunk of the 891 passengers individually, and that 98% figure never measured the ability to generalize, it measured the ability to memorize.",[79,9713,9715],{"id":9714},"trying-to-beat-80-stacking-and-a-neural-network","Trying to beat 80%: stacking and a neural network",[11,9717,9718,9719,9721],{},"The professor tries two things to improve on the honest 0.807. First, ",[87,9720,8931],{}," with 8 quite different models at once (Random Forest, Extra Trees, a neural network, SGD, Ridge, KNN, Naive Bayes, logistic regression):",[433,9723,9724],{},[11,9725,9726,9728],{},[15,9727,1629],{}," 0.824 cross-validation. A real, if small, improvement.",[11,9730,9731,9732,9735,9736,9739,9740,9743],{},"Then, a neural network alone (",[87,9733,9734],{},"MLPClassifier",", short for ",[34,9737,9738],{},"Multi-Layer Perceptron",", a Perceptron ",[20,9741,9742],{"href":9151},"I already saw earlier in this playlist"," stacked into several layers, the subject of its own course that isn't the focus here):",[433,9745,9746],{},[11,9747,9748,9750],{},[15,9749,1629],{}," 0.822 cross-validation, practically tied with the 8-model stack.",[11,9752,9753,9754,9756],{},"Neither is a huge leap over the lone forest's 0.807, but both beat the simpler version consistently. Unlike ",[20,9755,4107],{"href":9151}," (where a simple model beat every ensemble tried), here the extra complexity genuinely helped, a little. The earlier lesson still holds: there's no knowing which one wins without measuring both by the same ruler.",[79,9758,1870],{"id":1869},[1872,9760,9761,9769],{},[1875,9762,9763],{},[1878,9764,9765,9767],{},[1881,9766,1884],{"align":1883},[1881,9768,1887],{"align":1883},[1889,9770,9771,9779,9787],{},[1878,9772,9773,9776],{},[1894,9774,9775],{"align":1883},"Every dataset up to now already arrived complete",[1894,9777,9778],{"align":1883},"Missing data is normal in real data, and the fix (imputation) needs a deliberate strategy, not just \"drop the row\"",[1878,9780,9781,9784],{},[1894,9782,9783],{"align":1883},"A category becomes a number somehow",[1894,9785,9786],{"align":1883},"One-hot encoding avoids inventing an order between categories that doesn't exist",[1878,9788,9789,9792],{},[1894,9790,9791],{"align":1883},"Cross-validation is more honest than measuring on training",[1894,9793,9794],{"align":1883},"The same old warning, but this time on a real Random Forest: 98% on training against 81% for real",[79,9796,1939],{"id":1938},[11,9798,9799],{},"I reproduced the entire pipeline with a fixed seed (the original notebook fixes none), to get a stable number and compare all three approaches under the exact same cross-validation criterion.",[96,9801,9803],{"className":98,"code":9802,"language":100,"meta":57,"style":57},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nscores_rf = cross_val_score(clf_rf, X, y, cv=cv)\nscores_stacking = cross_val_score(clf_stacking, X, y, cv=cv)\nscores_mlp = cross_val_score(clf_mlp, X, y, cv=cv)\n",[87,9804,9805,9809,9814,9819],{"__ignoreMap":57},[104,9806,9807],{"class":106,"line":107},[104,9808,9057],{},[104,9810,9811],{"class":106,"line":58},[104,9812,9813],{},"scores_rf = cross_val_score(clf_rf, X, y, cv=cv)\n",[104,9815,9816],{"class":106,"line":118},[104,9817,9818],{},"scores_stacking = cross_val_score(clf_stacking, X, y, cv=cv)\n",[104,9820,9821],{"class":106,"line":124},[104,9822,9823],{},"scores_mlp = cross_val_score(clf_mlp, X, y, cv=cv)\n",[1872,9825,9826,9840],{},[1875,9827,9828],{},[1878,9829,9830,9832,9835,9838],{},[1881,9831,341],{"align":1883},[1881,9833,9834],{"align":3186},"Accuracy (training, memorized)",[1881,9836,9837],{"align":3186},"Accuracy (cross-validation)",[1881,9839,9086],{"align":3186},[1889,9841,9842,9856,9874],{},[1878,9843,9844,9847,9850,9853],{},[1894,9845,9846],{"align":1883},"Random Forest",[1894,9848,9849],{"align":3186},"0.9798",[1894,9851,9852],{"align":3186},"0.8092",[1894,9854,9855],{"align":3186},"0.0268",[1878,9857,9858,9861,9864,9869],{},[1894,9859,9860],{"align":1883},"Stacking (8 models)",[1894,9862,9863],{"align":3186},"(didn't measure)",[1894,9865,9866],{"align":3186},[15,9867,9868],{},"0.8283",[1894,9870,9871],{"align":3186},[15,9872,9873],{},"0.0077",[1878,9875,9876,9879,9881,9884],{},[1894,9877,9878],{"align":1883},"Neural network (MLP)",[1894,9880,9863],{"align":3186},[1894,9882,9883],{"align":3186},"0.8182",[1894,9885,9886],{"align":3186},"0.0197",[11,9888,9889,9890,9893],{},"The numbers land close to the original notebook's (0.807, 0.824, 0.822), confirming the difference between the three wasn't luck from one specific run. And notice the standard deviation: the stack didn't just have the best average, it's also the most ",[15,9891,9892],{},"stable"," fold to fold (0.0077 versus the lone forest's 0.0268), a sign that combining several different models cushioned some of the variation each one carries on its own.",[1982,9895,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":9897},[9898,9899,9900,9901,9902,9903,9904,9905],{"id":9169,"depth":58,"text":9170},{"id":9257,"depth":58,"text":9258},{"id":9335,"depth":58,"text":9336},{"id":9406,"depth":58,"text":9407},{"id":9652,"depth":58,"text":9653},{"id":9714,"depth":58,"text":9715},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 7: the professor builds a full preprocessing pipeline on the classic Titanic dataset, missing data and all, and shows the same train-set-accuracy mistake I'd already seen before, except this time with a real consequence.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic",{"title":9161,"description":9906},"en\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic",[9912,9913,6928],"missing-data","one-hot-encoding","JT8M33VOJ6EeHVrdY6FV3C4JD81tPXez0hpeK5txcbw",{"id":9916,"title":9917,"body":9918,"cover":3,"date":9148,"description":11411,"extension":61,"meta":11412,"navigation":63,"order":1689,"path":11413,"playlist":2000,"seo":11414,"status":66,"stem":11415,"tags":11416,"__hash__":11420},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fkmeans.md","K-means: When Clustering Becomes a Feature Engineering Trick",{"type":8,"value":9919,"toc":11401},[9920,9926,9930,9936,9940,10051,10066,10073,10087,10093,10520,10922,10926,10962,10967,10970,10974,11010,11025,11032,11043,11047,11054,11087,11099,11128,11136,11162,11165,11180,11183,11187,11193,11241,11244,11246,11284,11286,11296,11325,11393,11399],[11,9921,9922,9923,9925],{},"Lectures 8a, 8b, and 8c, and the course changes gears for the first time: every model up to now learned to predict a label already sitting in the data (",[87,9924,166],{},"). K-means gets no label at all, it just finds groups.",[79,9927,9929],{"id":9928},"the-problem-separating-without-knowing-the-answer","The problem: separating without knowing the answer",[11,9931,9932,9933,9935],{},"The professor goes back to Iris, but this time hides the species (",[87,9934,166],{},") and only passes the two petal variables (length and width) to the algorithm. The question shifts from \"which species is this flower?\" to \"how many natural groups exist here, and where do they sit?\"",[79,9937,9939],{"id":9938},"k-means-by-hand","K-means by hand",[96,9941,9943],{"className":98,"code":9942,"language":100,"meta":57,"style":57},"from sklearn.base import BaseEstimator, ClusterMixin, TransformerMixin\n\nclass KMeans(BaseEstimator, ClusterMixin, TransformerMixin):\n    def __init__(self, n_clusters=3, max_iter=100):\n        self.n_clusters = n_clusters\n        self.max_iter = max_iter\n\n    def fit(self, X, y=None):\n        self.centroids = X[random.sample(range(len(X)), self.n_clusters)]\n        max_iter = self.max_iter\n        while max_iter > 0:\n            max_iter -= 1\n            y_pred = self.predict(X)\n            for i in range(self.n_clusters):\n                self.centroids[i] = np.mean(X[y_pred==i], axis=0)\n            if np.allclose(self.centroids, self.previous_centroids[-1], atol=1e-9):\n                break\n        return self\n\n    def predict(self, X):\n        # assigns every point to its nearest centroid\n        ...\n",[87,9944,9945,9950,9954,9959,9964,9969,9973,9977,9981,9986,9991,9996,10001,10005,10010,10015,10020,10025,10030,10035,10040,10046],{"__ignoreMap":57},[104,9946,9947],{"class":106,"line":107},[104,9948,9949],{},"from sklearn.base import BaseEstimator, ClusterMixin, TransformerMixin\n",[104,9951,9952],{"class":106,"line":58},[104,9953,300],{"emptyLinePlaceholder":63},[104,9955,9956],{"class":106,"line":118},[104,9957,9958],{},"class KMeans(BaseEstimator, ClusterMixin, TransformerMixin):\n",[104,9960,9961],{"class":106,"line":124},[104,9962,9963],{},"    def __init__(self, n_clusters=3, max_iter=100):\n",[104,9965,9966],{"class":106,"line":308},[104,9967,9968],{},"        self.n_clusters = n_clusters\n",[104,9970,9971],{"class":106,"line":417},[104,9972,1663],{},[104,9974,9975],{"class":106,"line":422},[104,9976,300],{"emptyLinePlaceholder":63},[104,9978,9979],{"class":106,"line":428},[104,9980,9458],{},[104,9982,9983],{"class":106,"line":1689},[104,9984,9985],{},"        self.centroids = X[random.sample(range(len(X)), self.n_clusters)]\n",[104,9987,9988],{"class":106,"line":1695},[104,9989,9990],{},"        max_iter = self.max_iter\n",[104,9992,9993],{"class":106,"line":1701},[104,9994,9995],{},"        while max_iter > 0:\n",[104,9997,9998],{"class":106,"line":1707},[104,9999,10000],{},"            max_iter -= 1\n",[104,10002,10003],{"class":106,"line":1713},[104,10004,1698],{},[104,10006,10007],{"class":106,"line":1719},[104,10008,10009],{},"            for i in range(self.n_clusters):\n",[104,10011,10012],{"class":106,"line":1724},[104,10013,10014],{},"                self.centroids[i] = np.mean(X[y_pred==i], axis=0)\n",[104,10016,10017],{"class":106,"line":1729},[104,10018,10019],{},"            if np.allclose(self.centroids, self.previous_centroids[-1], atol=1e-9):\n",[104,10021,10022],{"class":106,"line":1734},[104,10023,10024],{},"                break\n",[104,10026,10028],{"class":106,"line":10027},18,[104,10029,1479],{},[104,10031,10033],{"class":106,"line":10032},19,[104,10034,300],{"emptyLinePlaceholder":63},[104,10036,10038],{"class":106,"line":10037},20,[104,10039,1484],{},[104,10041,10043],{"class":106,"line":10042},21,[104,10044,10045],{},"        # assigns every point to its nearest centroid\n",[104,10047,10049],{"class":106,"line":10048},22,[104,10050,7931],{},[11,10052,10053,10054,1807,10057,1807,10060,10062,10063,10065],{},"Notice the class signature: ",[87,10055,10056],{},"ClusterMixin",[15,10058,10059],{},"and",[87,10061,9423],{}," together, the first time a class in this playlist inherits from two mixins at once on top of ",[87,10064,1494],{},". That's not an accident, and the reason becomes clear further down the post.",[11,10067,10068,10069,10072],{},"The algorithm (called ",[15,10070,10071],{},"Lloyd's K-means",", the most common one) is just two steps repeated until nothing changes:",[3791,10074,10075,10081],{},[360,10076,10077,10080],{},[15,10078,10079],{},"Assign",": every point goes to its nearest centroid (Euclidean distance).",[360,10082,10083,10086],{},[15,10084,10085],{},"Recompute",": every centroid becomes the mean of the points assigned to it.",[11,10088,10089,10090,523],{},"Bishop formalizes both steps as minimizing a ",[15,10091,10092],{},"distortion measure",[11,10094,10095],{},[104,10096,10098,10178],{"className":10097},[148],[104,10099,10101],{"className":10100},[152],[154,10102,10103],{"xmlns":156},[158,10104,10105,10175],{},[161,10106,10107,10110,10112,10126,10140,10151,10154,10160,10162,10169],{},[164,10108,10109],{},"J",[168,10111,170],{},[490,10113,10114,10116,10124],{},[168,10115,494],{},[161,10117,10118,10120,10122],{},[164,10119,499],{},[168,10121,170],{},[483,10123,485],{},[164,10125,506],{},[490,10127,10128,10130,10138],{},[168,10129,494],{},[161,10131,10132,10134,10136],{},[164,10133,5338],{},[168,10135,170],{},[483,10137,485],{},[164,10139,5302],{},[515,10141,10142,10145],{},[164,10143,10144],{},"r",[161,10146,10147,10149],{},[164,10148,499],{},[164,10150,5338],{},[164,10152,10153],{"mathvariant":959},"∥",[515,10155,10156,10158],{},[164,10157,178],{"mathvariant":473},[164,10159,499],{},[168,10161,530],{},[515,10163,10164,10167],{},[164,10165,10166],{"mathvariant":2102},"μ",[164,10168,5338],{},[539,10170,10171,10173],{},[164,10172,10153],{"mathvariant":959},[483,10174,488],{},[186,10176,10177],{"encoding":188},"J = \\sum_{n=1}^{N}\\sum_{k=1}^{K} r_{nk}\\|\\mathbf{x}_n - \\boldsymbol{\\mu}_k\\|^2",[104,10179,10181,10200,10436],{"className":10180,"ariaHidden":194},[193],[104,10182,10184,10187,10191,10194,10197],{"className":10183},[198],[104,10185],{"className":10186,"style":5314},[202],[104,10188,10109],{"className":10189,"style":10190},[207,208],"margin-right:0.0962em;",[104,10192],{"className":10193,"style":214},[213],[104,10195,170],{"className":10196},[218],[104,10198],{"className":10199,"style":214},[213],[104,10201,10203,10207,10270,10273,10336,10339,10384,10387,10427,10430,10433],{"className":10202},[198],[104,10204],{"className":10205,"style":10206},[202],"height:1.2809em;vertical-align:-0.2997em;",[104,10208,10210,10213],{"className":10209},[687],[104,10211,494],{"className":10212,"style":693},[687,691,692],[104,10214,10216],{"className":10215},[697],[104,10217,10219,10262],{"className":10218},[605,606],[104,10220,10222,10259],{"className":10221},[610],[104,10223,10225,10245],{"className":10224,"style":707},[614],[104,10226,10227,10230],{"style":710},[104,10228],{"className":10229,"style":714},[622],[104,10231,10233],{"className":10232},[627,628,629,630],[104,10234,10236,10239,10242],{"className":10235},[207,630],[104,10237,499],{"className":10238},[207,208,630],[104,10240,170],{"className":10241},[218,630],[104,10243,485],{"className":10244},[207,630],[104,10246,10247,10250],{"style":732},[104,10248],{"className":10249,"style":714},[622],[104,10251,10253],{"className":10252},[627,628,629,630],[104,10254,10256],{"className":10255},[207,630],[104,10257,506],{"className":10258,"style":745},[207,208,630],[104,10260,667],{"className":10261},[666],[104,10263,10265],{"className":10264},[610],[104,10266,10268],{"className":10267,"style":755},[614],[104,10269],{},[104,10271],{"className":10272,"style":683},[213],[104,10274,10276,10279],{"className":10275},[687],[104,10277,494],{"className":10278,"style":693},[687,691,692],[104,10280,10282],{"className":10281},[697],[104,10283,10285,10328],{"className":10284},[605,606],[104,10286,10288,10325],{"className":10287},[610],[104,10289,10291,10311],{"className":10290,"style":707},[614],[104,10292,10293,10296],{"style":710},[104,10294],{"className":10295,"style":714},[622],[104,10297,10299],{"className":10298},[627,628,629,630],[104,10300,10302,10305,10308],{"className":10301},[207,630],[104,10303,5338],{"className":10304,"style":5386},[207,208,630],[104,10306,170],{"className":10307},[218,630],[104,10309,485],{"className":10310},[207,630],[104,10312,10313,10316],{"style":732},[104,10314],{"className":10315,"style":714},[622],[104,10317,10319],{"className":10318},[627,628,629,630],[104,10320,10322],{"className":10321},[207,630],[104,10323,5302],{"className":10324,"style":5318},[207,208,630],[104,10326,667],{"className":10327},[666],[104,10329,10331],{"className":10330},[610],[104,10332,10334],{"className":10333,"style":755},[614],[104,10335],{},[104,10337],{"className":10338,"style":683},[213],[104,10340,10342,10345],{"className":10341},[207],[104,10343,10144],{"className":10344,"style":2161},[207,208],[104,10346,10348],{"className":10347},[697],[104,10349,10351,10376],{"className":10350},[605,606],[104,10352,10354,10373],{"className":10353},[610],[104,10355,10357],{"className":10356,"style":5373},[614],[104,10358,10360,10363],{"style":10359},"top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;",[104,10361],{"className":10362,"style":714},[622],[104,10364,10366],{"className":10365},[627,628,629,630],[104,10367,10369],{"className":10368},[207,630],[104,10370,10372],{"className":10371,"style":5386},[207,208,630],"nk",[104,10374,667],{"className":10375},[666],[104,10377,10379],{"className":10378},[610],[104,10380,10382],{"className":10381,"style":807},[614],[104,10383],{},[104,10385,10153],{"className":10386},[207],[104,10388,10390,10393],{"className":10389},[207],[104,10391,178],{"className":10392},[207,570],[104,10394,10396],{"className":10395},[697],[104,10397,10399,10419],{"className":10398},[605,606],[104,10400,10402,10416],{"className":10401},[610],[104,10403,10405],{"className":10404,"style":785},[614],[104,10406,10407,10410],{"style":788},[104,10408],{"className":10409,"style":714},[622],[104,10411,10413],{"className":10412},[627,628,629,630],[104,10414,499],{"className":10415},[207,208,630],[104,10417,667],{"className":10418},[666],[104,10420,10422],{"className":10421},[610],[104,10423,10425],{"className":10424,"style":807},[614],[104,10426],{},[104,10428],{"className":10429,"style":235},[213],[104,10431,530],{"className":10432},[239],[104,10434],{"className":10435,"style":235},[213],[104,10437,10439,10442,10491],{"className":10438},[198],[104,10440],{"className":10441,"style":838},[202],[104,10443,10445,10454],{"className":10444},[207],[104,10446,10448],{"className":10447},[207],[104,10449,10451],{"className":10450},[207],[104,10452,10166],{"className":10453},[207,2433],[104,10455,10457],{"className":10456},[697],[104,10458,10460,10482],{"className":10459},[605,606],[104,10461,10463,10479],{"className":10462},[610],[104,10464,10467],{"className":10465,"style":10466},[614],"height:0.242em;",[104,10468,10470,10473],{"style":10469},"top:-2.4559em;margin-right:0.05em;",[104,10471],{"className":10472,"style":714},[622],[104,10474,10476],{"className":10475},[627,628,629,630],[104,10477,5338],{"className":10478,"style":5386},[207,208,630],[104,10480,667],{"className":10481},[666],[104,10483,10485],{"className":10484},[610],[104,10486,10489],{"className":10487,"style":10488},[614],"height:0.2441em;",[104,10490],{},[104,10492,10494,10497],{"className":10493},[207],[104,10495,10153],{"className":10496},[207],[104,10498,10500],{"className":10499},[697],[104,10501,10503],{"className":10502},[605],[104,10504,10506],{"className":10505},[610],[104,10507,10509],{"className":10508,"style":897},[614],[104,10510,10511,10514],{"style":900},[104,10512],{"className":10513,"style":714},[622],[104,10515,10517],{"className":10516},[627,628,629,630],[104,10518,488],{"className":10519},[207,630],[11,10521,7423,10522,10621,10622,10651,10652,10680,10681,10758,10759,10835,10836,10864,10865,10893,10894,7002],{},[104,10523,10525,10551],{"className":10524},[148],[104,10526,10528],{"className":10527},[152],[154,10529,10530],{"xmlns":156},[158,10531,10532,10548],{},[161,10533,10534,10544,10546],{},[515,10535,10536,10538],{},[164,10537,10144],{},[161,10539,10540,10542],{},[164,10541,499],{},[164,10543,5338],{},[168,10545,170],{},[483,10547,485],{},[186,10549,10550],{"encoding":188},"r_{nk}=1",[104,10552,10554,10612],{"className":10553,"ariaHidden":194},[193],[104,10555,10557,10560,10603,10606,10609],{"className":10556},[198],[104,10558],{"className":10559,"style":2822},[202],[104,10561,10563,10566],{"className":10562},[207],[104,10564,10144],{"className":10565,"style":2161},[207,208],[104,10567,10569],{"className":10568},[697],[104,10570,10572,10595],{"className":10571},[605,606],[104,10573,10575,10592],{"className":10574},[610],[104,10576,10578],{"className":10577,"style":5373},[614],[104,10579,10580,10583],{"style":10359},[104,10581],{"className":10582,"style":714},[622],[104,10584,10586],{"className":10585},[627,628,629,630],[104,10587,10589],{"className":10588},[207,630],[104,10590,10372],{"className":10591,"style":5386},[207,208,630],[104,10593,667],{"className":10594},[666],[104,10596,10598],{"className":10597},[610],[104,10599,10601],{"className":10600,"style":807},[614],[104,10602],{},[104,10604],{"className":10605,"style":214},[213],[104,10607,170],{"className":10608},[218],[104,10610],{"className":10611,"style":214},[213],[104,10613,10615,10618],{"className":10614},[198],[104,10616],{"className":10617,"style":2971},[202],[104,10619,485],{"className":10620},[207]," if point ",[104,10623,10625,10638],{"className":10624},[148],[104,10626,10628],{"className":10627},[152],[154,10629,10630],{"xmlns":156},[158,10631,10632,10636],{},[161,10633,10634],{},[164,10635,499],{},[186,10637,499],{"encoding":188},[104,10639,10641],{"className":10640,"ariaHidden":194},[193],[104,10642,10644,10648],{"className":10643},[198],[104,10645],{"className":10646,"style":10647},[202],"height:0.4306em;",[104,10649,499],{"className":10650},[207,208]," belongs to group ",[104,10653,10655,10668],{"className":10654},[148],[104,10656,10658],{"className":10657},[152],[154,10659,10660],{"xmlns":156},[158,10661,10662,10666],{},[161,10663,10664],{},[164,10665,5338],{},[186,10667,5338],{"encoding":188},[104,10669,10671],{"className":10670,"ariaHidden":194},[193],[104,10672,10674,10677],{"className":10673},[198],[104,10675],{"className":10676,"style":268},[202],[104,10678,5338],{"className":10679,"style":5386},[207,208]," (and 0 otherwise). Fixing the centroids, the optimal ",[104,10682,10684,10706],{"className":10683},[148],[104,10685,10687],{"className":10686},[152],[154,10688,10689],{"xmlns":156},[158,10690,10691,10703],{},[161,10692,10693],{},[515,10694,10695,10697],{},[164,10696,10144],{},[161,10698,10699,10701],{},[164,10700,499],{},[164,10702,5338],{},[186,10704,10705],{"encoding":188},"r_{nk}",[104,10707,10709],{"className":10708,"ariaHidden":194},[193],[104,10710,10712,10715],{"className":10711},[198],[104,10713],{"className":10714,"style":2822},[202],[104,10716,10718,10721],{"className":10717},[207],[104,10719,10144],{"className":10720,"style":2161},[207,208],[104,10722,10724],{"className":10723},[697],[104,10725,10727,10750],{"className":10726},[605,606],[104,10728,10730,10747],{"className":10729},[610],[104,10731,10733],{"className":10732,"style":5373},[614],[104,10734,10735,10738],{"style":10359},[104,10736],{"className":10737,"style":714},[622],[104,10739,10741],{"className":10740},[627,628,629,630],[104,10742,10744],{"className":10743},[207,630],[104,10745,10372],{"className":10746,"style":5386},[207,208,630],[104,10748,667],{"className":10749},[666],[104,10751,10753],{"className":10752},[610],[104,10754,10756],{"className":10755,"style":807},[614],[104,10757],{}," is obvious (assign to the nearest one, exactly step 1). Fixing the ",[104,10760,10762,10783],{"className":10761},[148],[104,10763,10765],{"className":10764},[152],[154,10766,10767],{"xmlns":156},[158,10768,10769,10781],{},[161,10770,10771],{},[515,10772,10773,10775],{},[164,10774,10144],{},[161,10776,10777,10779],{},[164,10778,499],{},[164,10780,5338],{},[186,10782,10705],{"encoding":188},[104,10784,10786],{"className":10785,"ariaHidden":194},[193],[104,10787,10789,10792],{"className":10788},[198],[104,10790],{"className":10791,"style":2822},[202],[104,10793,10795,10798],{"className":10794},[207],[104,10796,10144],{"className":10797,"style":2161},[207,208],[104,10799,10801],{"className":10800},[697],[104,10802,10804,10827],{"className":10803},[605,606],[104,10805,10807,10824],{"className":10806},[610],[104,10808,10810],{"className":10809,"style":5373},[614],[104,10811,10812,10815],{"style":10359},[104,10813],{"className":10814,"style":714},[622],[104,10816,10818],{"className":10817},[627,628,629,630],[104,10819,10821],{"className":10820},[207,630],[104,10822,10372],{"className":10823,"style":5386},[207,208,630],[104,10825,667],{"className":10826},[666],[104,10828,10830],{"className":10829},[610],[104,10831,10833],{"className":10832,"style":807},[614],[104,10834],{},", the optimal centroid is the mean of the points assigned to it, because that's where ",[104,10837,10839,10852],{"className":10838},[148],[104,10840,10842],{"className":10841},[152],[154,10843,10844],{"xmlns":156},[158,10845,10846,10850],{},[161,10847,10848],{},[164,10849,10109],{},[186,10851,10109],{"encoding":188},[104,10853,10855],{"className":10854,"ariaHidden":194},[193],[104,10856,10858,10861],{"className":10857},[198],[104,10859],{"className":10860,"style":5314},[202],[104,10862,10109],{"className":10863,"style":10190},[207,208],"'s derivative hits zero (exactly step 2). Every step can only decrease ",[104,10866,10868,10881],{"className":10867},[148],[104,10869,10871],{"className":10870},[152],[154,10872,10873],{"xmlns":156},[158,10874,10875,10879],{},[161,10876,10877],{},[164,10878,10109],{},[186,10880,10109],{"encoding":188},[104,10882,10884],{"className":10883,"ariaHidden":194},[193],[104,10885,10887,10890],{"className":10886},[198],[104,10888],{"className":10889,"style":5314},[202],[104,10891,10109],{"className":10892,"style":10190},[207,208]," or leave it unchanged, never increase it, so the algorithm always converges, it just might land on a local minimum, not necessarily the best possible grouping (which is why, in practice, scikit-learn runs K-means several times with different starting centroids and keeps the result with the lowest ",[104,10895,10897,10910],{"className":10896},[148],[104,10898,10900],{"className":10899},[152],[154,10901,10902],{"xmlns":156},[158,10903,10904,10908],{},[161,10905,10906],{},[164,10907,10109],{},[186,10909,10109],{"encoding":188},[104,10911,10913],{"className":10912,"ariaHidden":194},[193],[104,10914,10916,10919],{"className":10915},[198],[104,10917],{"className":10918,"style":5314},[202],[104,10920,10109],{"className":10921,"style":10190},[207,208],[79,10923,10925],{"id":10924},"interactive-watching-the-centroids-move","Interactive: watching the centroids move",[11,10927,10928,10929,10932,10933,10961],{},"My own reconstruction of the same algorithm (assign, recompute, repeat), on the 150 real Iris points, ",[20,10930,10931],{"href":8243},"the same two variables from the decision trees post",". Click \"Step\" and watch the inertia (",[104,10934,10936,10949],{"className":10935},[148],[104,10937,10939],{"className":10938},[152],[154,10940,10941],{"xmlns":156},[158,10942,10943,10947],{},[161,10944,10945],{},[164,10946,10109],{},[186,10948,10109],{"encoding":188},[104,10950,10952],{"className":10951,"ariaHidden":194},[193],[104,10953,10955,10958],{"className":10954},[198],[104,10956],{"className":10957,"style":5314},[202],[104,10959,10109],{"className":10960,"style":10190},[207,208],") drop each round, and the regions (who belongs to which centroid) settle:",[10963,10964],"k-means-explorer",{":n-clusters":5580,":points":10965,":true-classes":10966,":x-max":8074,":x-min":8075,":y-max":8077,":y-min":331,"class0-label":8079,"class1-label":8080,"class2-label":8081,"x-label":8082,"y-label":8083},"[[1.4,0.2],[1.4,0.2],[1.3,0.2],[1.5,0.2],[1.4,0.2],[1.7,0.4],[1.4,0.3],[1.5,0.2],[1.4,0.2],[1.5,0.1],[1.5,0.2],[1.6,0.2],[1.4,0.1],[1.1,0.1],[1.2,0.2],[1.5,0.4],[1.3,0.4],[1.4,0.3],[1.7,0.3],[1.5,0.3],[1.7,0.2],[1.5,0.4],[1.0,0.2],[1.7,0.5],[1.9,0.2],[1.6,0.2],[1.6,0.4],[1.5,0.2],[1.4,0.2],[1.6,0.2],[1.6,0.2],[1.5,0.4],[1.5,0.1],[1.4,0.2],[1.5,0.2],[1.2,0.2],[1.3,0.2],[1.4,0.1],[1.3,0.2],[1.5,0.2],[1.3,0.3],[1.3,0.3],[1.3,0.2],[1.6,0.6],[1.9,0.4],[1.4,0.3],[1.6,0.2],[1.4,0.2],[1.5,0.2],[1.4,0.2],[4.7,1.4],[4.5,1.5],[4.9,1.5],[4.0,1.3],[4.6,1.5],[4.5,1.3],[4.7,1.6],[3.3,1.0],[4.6,1.3],[3.9,1.4],[3.5,1.0],[4.2,1.5],[4.0,1.0],[4.7,1.4],[3.6,1.3],[4.4,1.4],[4.5,1.5],[4.1,1.0],[4.5,1.5],[3.9,1.1],[4.8,1.8],[4.0,1.3],[4.9,1.5],[4.7,1.2],[4.3,1.3],[4.4,1.4],[4.8,1.4],[5.0,1.7],[4.5,1.5],[3.5,1.0],[3.8,1.1],[3.7,1.0],[3.9,1.2],[5.1,1.6],[4.5,1.5],[4.5,1.6],[4.7,1.5],[4.4,1.3],[4.1,1.3],[4.0,1.3],[4.4,1.2],[4.6,1.4],[4.0,1.2],[3.3,1.0],[4.2,1.3],[4.2,1.2],[4.2,1.3],[4.3,1.3],[3.0,1.1],[4.1,1.3],[6.0,2.5],[5.1,1.9],[5.9,2.1],[5.6,1.8],[5.8,2.2],[6.6,2.1],[4.5,1.7],[6.3,1.8],[5.8,1.8],[6.1,2.5],[5.1,2.0],[5.3,1.9],[5.5,2.1],[5.0,2.0],[5.1,2.4],[5.3,2.3],[5.5,1.8],[6.7,2.2],[6.9,2.3],[5.0,1.5],[5.7,2.3],[4.9,2.0],[6.7,2.0],[4.9,2.0],[5.7,1.8],[6.0,2.1],[4.8,1.8],[4.9,1.8],[5.6,2.1],[5.8,1.6],[6.1,1.9],[6.4,2.0],[5.6,2.2],[5.1,1.5],[5.6,1.4],[6.1,2.3],[5.6,2.4],[5.5,1.8],[4.8,1.8],[5.4,2.1],[5.6,2.4],[5.1,2.3],[5.1,1.9],[5.9,2.3],[5.7,2.5],[5.2,2.3],[5.0,1.9],[5.2,2.0],[5.4,2.3],[5.1,1.8]]","[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]",[11,10968,10969],{},"The points stay colored by true species, just so you can visually compare: the regions K-means finds on its own, never seeing the label, match the real species pretty well, especially for Setosa (already clearly separated back in the decision trees post). Click \"Reset\" a few times: depending on where the centroids land by chance, the final result can shift a bit, exactly the local-minimum limitation Bishop describes.",[79,10971,10973],{"id":10972},"picking-k-the-elbow-method","Picking K: the elbow method",[11,10975,10976,10977,10980,10981,11009],{},"One question was left open: how do you choose how many groups (K) to look for? Scikit-learn's K-means exposes ",[87,10978,10979],{},"inertia_",", which is exactly Bishop's ",[104,10982,10984,10997],{"className":10983},[148],[104,10985,10987],{"className":10986},[152],[154,10988,10989],{"xmlns":156},[158,10990,10991,10995],{},[161,10992,10993],{},[164,10994,10109],{},[186,10996,10109],{"encoding":188},[104,10998,11000],{"className":10999,"ariaHidden":194},[193],[104,11001,11003,11006],{"className":11002},[198],[104,11004],{"className":11005,"style":5314},[202],[104,11007,10109],{"className":11008,"style":10190},[207,208]," at the end of training:",[96,11011,11013],{"className":98,"code":11012,"language":100,"meta":57,"style":57},"kmeans_k3 = KMeans(n_clusters=3).fit(X)\nkmeans_k8 = KMeans(n_clusters=8).fit(X)\n",[87,11014,11015,11020],{"__ignoreMap":57},[104,11016,11017],{"class":106,"line":107},[104,11018,11019],{},"kmeans_k3 = KMeans(n_clusters=3).fit(X)\n",[104,11021,11022],{"class":106,"line":58},[104,11023,11024],{},"kmeans_k8 = KMeans(n_clusters=8).fit(X)\n",[433,11026,11027],{},[11,11028,11029,11031],{},[15,11030,1629],{}," inertia with K=3: 31.37. With K=8: 8.31.",[11,11033,11034,11035,11038,11039,11042],{},"More groups always reduces (or ties) inertia, in the limit (K = number of points) it hits zero, every point becomes its own group, which is useless. The trick is to plot inertia against several K values and look for the ",[15,11036,11037],{},"elbow",": the point where the curve stops dropping fast and starts dropping slowly. Before the elbow, each extra group still captures real structure. After it, each extra group is just slicing up noise. The professor repeats this plot for Iris, for the wine dataset ",[20,11040,11041],{"href":5766},"that already showed up in this playlist",", and for Digits (1797 images of handwritten digits, 8×8 pixels each), without pulling a fixed number out of any of them, the point is just to show the curve's shape.",[79,11044,11046],{"id":11045},"the-trick-using-the-groups-as-a-new-feature","The trick: using the groups as a new feature",[11,11048,11049,11050,11053],{},"This is where lecture 8b flips the whole post on its head. Instead of using K-means only to cluster, the professor uses it ",[15,11051,11052],{},"inside a supervised pipeline",", as a preprocessing step:",[96,11055,11057],{"className":98,"code":11056,"language":100,"meta":57,"style":57},"from sklearn.linear_model import RidgeClassifier\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.pipeline import make_pipeline\n\nmodel = RidgeClassifier()\nscores = cross_val_score(model, X, y, cv=5)\n",[87,11058,11059,11064,11068,11073,11077,11082],{"__ignoreMap":57},[104,11060,11061],{"class":106,"line":107},[104,11062,11063],{},"from sklearn.linear_model import RidgeClassifier\n",[104,11065,11066],{"class":106,"line":58},[104,11067,9696],{},[104,11069,11070],{"class":106,"line":118},[104,11071,11072],{},"from sklearn.pipeline import make_pipeline\n",[104,11074,11075],{"class":106,"line":124},[104,11076,300],{"emptyLinePlaceholder":63},[104,11078,11079],{"class":106,"line":308},[104,11080,11081],{},"model = RidgeClassifier()\n",[104,11083,11084],{"class":106,"line":417},[104,11085,11086],{},"scores = cross_val_score(model, X, y, cv=5)\n",[433,11088,11089],{},[11,11090,11091,11098],{},[15,11092,11093,11094,11097],{},"Output (Digits, ",[87,11095,11096],{},"RidgeClassifier"," alone, straight on the 64 pixels):"," 0.888 average accuracy.",[96,11100,11102],{"className":98,"code":11101,"language":100,"meta":57,"style":57},"model = make_pipeline(\n    KMeans(n_clusters=50),\n    RidgeClassifier()\n)\nscores = cross_val_score(model, X, y, cv=5)\n",[87,11103,11104,11109,11114,11119,11124],{"__ignoreMap":57},[104,11105,11106],{"class":106,"line":107},[104,11107,11108],{},"model = make_pipeline(\n",[104,11110,11111],{"class":106,"line":58},[104,11112,11113],{},"    KMeans(n_clusters=50),\n",[104,11115,11116],{"class":106,"line":118},[104,11117,11118],{},"    RidgeClassifier()\n",[104,11120,11121],{"class":106,"line":124},[104,11122,11123],{},")\n",[104,11125,11126],{"class":106,"line":308},[104,11127,11086],{},[433,11129,11130],{},[11,11131,11132,11135],{},[15,11133,11134],{},"Output (same dataset, with a 50-cluster K-means ahead of the classifier):"," 0.939.",[11,11137,11138,11139,11141,11142,11144,11145,11148,11149,11151,11152,11154,11155,11157,11158,11161],{},"A serious jump, just from slotting a K-means in the middle. How does this work? This is where the ",[87,11140,9423],{}," I mentioned earlier comes back into play. Inside a ",[87,11143,1498],{},", every step except the last needs ",[87,11146,11147],{},".transform()",", not ",[87,11150,9419],{},". Scikit-learn's K-means implements both: ",[87,11153,9419],{}," returns the index of the nearest group (a single number), but ",[87,11156,11147],{}," returns the ",[15,11159,11160],{},"distance to each of the K centroids"," (K numbers). A 64-pixel image becomes a vector of 50 numbers, each one saying \"how similar is this image to the prototype of group 1, group 2, ..., group 50.\" Every centroid works as a \"prototype digit\" learned with no label at all, and the distance to each one becomes a new feature, more informative to the linear classifier than a raw pixel.",[11,11163,11164],{},"The professor still uses Optuna to search for the ideal number of groups (between 10 and 200):",[433,11166,11167],{},[11,11168,11169,11171,11172,11175,11176,11179],{},[15,11170,1629],{}," the best found was ",[87,11173,11174],{},"n_clusters=136",", with 0.963 average cross-validation accuracy. Trying ",[87,11177,11178],{},"n_clusters=250"," by hand (more groups than Optuna even tried): 0.966, even better.",[11,11181,11182],{},"More groups, more different \"prototypes\" to compare against, more information for the final classifier, at least as far as the professor tested.",[79,11184,11186],{"id":11185},"the-same-trick-on-a-bigger-dataset-mnist","The same trick, on a bigger dataset: MNIST",[11,11188,11189,11192],{},[87,11190,11191],{},"aula08c"," repeats the exact recipe on MNIST (60 thousand training images, 28×28 pixels each, so 784 variables per image, quite a bit more than Digits):",[1872,11194,11195,11205],{},[1875,11196,11197],{},[1878,11198,11199,11202],{},[1881,11200,11201],{"align":1883},"Approach",[1881,11203,11204],{"align":3186},"Test accuracy",[1889,11206,11207,11217,11229],{},[1878,11208,11209,11214],{},[1894,11210,11211,11213],{"align":1883},[87,11212,11096],{}," straight on the 784 pixels",[1894,11215,11216],{"align":3186},"0.8603",[1878,11218,11219,11226],{},[1894,11220,11221,11223,11224],{"align":1883},[87,11222,5828],{}," + ",[87,11225,11096],{},[1894,11227,11228],{"align":3186},"(practically the same, scaling pixels didn't help here)",[1878,11230,11231,11236],{},[1894,11232,11233,11234],{"align":1883},"K-means (250 groups) + ",[87,11235,11096],{},[1894,11237,11238],{"align":3186},[15,11239,11240],{},"0.9403",[11,11242,11243],{},"Eight percentage points of gain, on the same dataset, on the same linear classifier, just by swapping \"64\u002F784 raw pixel values\" for \"250 distances to prototypes learned with no label.\" The idea of reusing an unsupervised algorithm as a feature source for a supervised problem is more general than it looks: any \"distance to a prototype\" vector carries information the raw pixel doesn't carry on its own.",[79,11245,1870],{"id":1869},[1872,11247,11248,11256],{},[1875,11249,11250],{},[1878,11251,11252,11254],{},[1881,11253,1884],{"align":1883},[1881,11255,1887],{"align":1883},[1889,11257,11258,11266,11276],{},[1878,11259,11260,11263],{},[1894,11261,11262],{"align":1883},"Every model up to now learned to predict a given label",[1894,11264,11265],{"align":1883},"K-means groups data with no label at all, just its geometric structure",[1878,11267,11268,11273],{},[1894,11269,11270,11272],{"align":1883},[87,11271,9423],{}," is for steps that only transform, not predict",[1894,11274,11275],{"align":1883},"A clustering algorithm can also be a transformer: distance to each centroid becomes a new feature",[1878,11277,11278,11281],{},[1894,11279,11280],{"align":1883},"More complexity doesn't always help",[1894,11282,11283],{"align":1883},"Here it helped a lot: swapping raw pixels for distance-to-prototype gained 5 to 8 percentage points across two different datasets",[79,11285,1939],{"id":1938},[11,11287,11288,11289,11291,11292,11295],{},"I reproduced the same recipe (K-means as a transformer ahead of ",[87,11290,11096],{},") on Digits, with a fixed seed and ",[87,11293,11294],{},"n_init=10"," (K-means runs 10 times with different starting centroids and keeps the one with the lowest inertia, scikit-learn's default), to confirm the effect reproducibly.",[96,11297,11299],{"className":98,"code":11298,"language":100,"meta":57,"style":57},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nridge_scores = cross_val_score(RidgeClassifier(random_state=42), X, y, cv=cv)\nfor k in [50, 136, 250]:\n    model = make_pipeline(KMeans(n_clusters=k, random_state=42, n_init=10), RidgeClassifier(random_state=42))\n    scores = cross_val_score(model, X, y, cv=cv)\n",[87,11300,11301,11305,11310,11315,11320],{"__ignoreMap":57},[104,11302,11303],{"class":106,"line":107},[104,11304,9057],{},[104,11306,11307],{"class":106,"line":58},[104,11308,11309],{},"ridge_scores = cross_val_score(RidgeClassifier(random_state=42), X, y, cv=cv)\n",[104,11311,11312],{"class":106,"line":118},[104,11313,11314],{},"for k in [50, 136, 250]:\n",[104,11316,11317],{"class":106,"line":124},[104,11318,11319],{},"    model = make_pipeline(KMeans(n_clusters=k, random_state=42, n_init=10), RidgeClassifier(random_state=42))\n",[104,11321,11322],{"class":106,"line":308},[104,11323,11324],{},"    scores = cross_val_score(model, X, y, cv=cv)\n",[1872,11326,11327,11337],{},[1875,11328,11329],{},[1878,11330,11331,11333,11335],{},[1881,11332,11201],{"align":1883},[1881,11334,9083],{"align":3186},[1881,11336,9086],{"align":3186},[1889,11338,11339,11352,11365,11378],{},[1878,11340,11341,11346,11349],{},[1894,11342,11343,11345],{"align":1883},[87,11344,11096],{}," straight on pixels",[1894,11347,11348],{"align":3186},"0.9343",[1894,11350,11351],{"align":3186},"0.0072",[1878,11353,11354,11359,11362],{},[1894,11355,11356,11357],{"align":1883},"K-means (50) + ",[87,11358,11096],{},[1894,11360,11361],{"align":3186},"0.9666",[1894,11363,11364],{"align":3186},"0.0128",[1878,11366,11367,11372,11375],{},[1894,11368,11369,11370],{"align":1883},"K-means (136) + ",[87,11371,11096],{},[1894,11373,11374],{"align":3186},"0.9844",[1894,11376,11377],{"align":3186},"0.0045",[1878,11379,11380,11385,11390],{},[1894,11381,11382,11383],{"align":1883},"K-means (250) + ",[87,11384,11096],{},[1894,11386,11387],{"align":3186},[15,11388,11389],{},"0.9916",[1894,11391,11392],{"align":3186},"0.0056",[11,11394,11395,11396,11398],{},"My numbers come out higher than the original notebook's (0.888\u002F0.939\u002F0.963), likely because I fixed ",[87,11397,11294],{}," (K-means tries several initializations and keeps the best, avoiding the bad local minima Bishop warned about above) and a seed my environment could reproduce stably. But the trend is identical to the notebook's: more groups, better accuracy, and the improvement from \"raw pixel\" to \"distance to prototype\" is large and consistent at every group count tested.",[1982,11400,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":11402},[11403,11404,11405,11406,11407,11408,11409,11410],{"id":9928,"depth":58,"text":9929},{"id":9938,"depth":58,"text":9939},{"id":10924,"depth":58,"text":10925},{"id":10972,"depth":58,"text":10973},{"id":11045,"depth":58,"text":11046},{"id":11185,"depth":58,"text":11186},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 8: the professor enters unsupervised learning with K-means built by hand, the elbow method for picking K, and a trick I didn't see coming: using the discovered groups as new features for a supervised classifier.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fkmeans",{"title":9917,"description":11411},"en\u002Fplaylists\u002Fpattern-recognition\u002Fkmeans",[11417,11418,11419],"k-means","unsupervised-learning","feature-engineering","TdGsWJsiJQvVpQXUsCqciTSUVo1o4mASm92Z7etHXRE",{"id":11422,"title":11423,"body":11424,"cover":3,"date":9148,"description":12036,"extension":61,"meta":12037,"navigation":63,"order":1695,"path":12038,"playlist":2000,"seo":12039,"status":66,"stem":12040,"tags":12041,"__hash__":12045},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised.md","DBSCAN and Semi-Supervised Learning: When Clustering Helps You Label",{"type":8,"value":11425,"toc":12024},[11426,11429,11433,11440,11455,11470,11474,11477,11565,11568,11588,11596,11628,11637,11652,11656,11659,11670,11683,11689,11696,11721,11731,11737,11741,11744,11764,11772,11780,11784,11791,11811,11814,11824,11827,11831,11838,11858,11869,11872,11879,11889,11891,11935,11937,11940,11955,12008,12022],[11,11427,11428],{},"Lectures 9a and 9b. Bishop's book covers neither DBSCAN nor semi-supervised learning (both became more prominent in the literature after 2006), so this post leans more on what the professor showed, with far less direct citing of the book than usual around here.",[79,11430,11432],{"id":11431},"where-k-means-falls-flat","Where K-means falls flat",[11,11434,11435,11436,11439],{},"The professor generates a classic synthetic dataset, ",[87,11437,11438],{},"make_moons",": two interleaving arcs, like two crescent moons locked together.",[96,11441,11443],{"className":98,"code":11442,"language":100,"meta":57,"style":57},"from sklearn.datasets import make_moons\nX, y = make_moons(n_samples=1000, noise=0.05, random_state=42)\n",[87,11444,11445,11450],{"__ignoreMap":57},[104,11446,11447],{"class":106,"line":107},[104,11448,11449],{},"from sklearn.datasets import make_moons\n",[104,11451,11452],{"class":106,"line":58},[104,11453,11454],{},"X, y = make_moons(n_samples=1000, noise=0.05, random_state=42)\n",[11,11456,11457,11458,11461,11462,11465,11466,11469],{},"Running ",[20,11459,11460],{"href":11413},"the K-means I already covered in the previous post"," with K=2 on this data gives a poor result: K-means always cuts space into convex regions (every point goes to its nearest centroid, so the boundary between groups is always a straight line), but the two moons aren't convex, they nest into each other's curve. K-means ends up cutting nearly down the middle, ignoring the arcs' actual shape. Measuring agreement between K-means's grouping and the real moon split (using the ",[15,11463,11464],{},"adjusted Rand index",", a metric that equals 1 when the groups perfectly match the real split and sits near 0 when it's about as good as guessing at random): ",[15,11467,11468],{},"0.24",". Barely any match.",[79,11471,11473],{"id":11472},"dbscan-grouping-by-density-not-distance-to-a-center","DBSCAN: grouping by density, not distance to a center",[11,11475,11476],{},"The professor describes the algorithm in plain text, no formula:",[357,11478,11479,11544,11555,11558],{},[360,11480,11481,11482,11512,11513,1863],{},"For every point, count how many other points sit within a small distance ",[104,11483,11485,11500],{"className":11484},[148],[104,11486,11488],{"className":11487},[152],[154,11489,11490],{"xmlns":156},[158,11491,11492,11497],{},[161,11493,11494],{},[164,11495,11496],{},"ε",[186,11498,11499],{"encoding":188},"\\varepsilon",[104,11501,11503],{"className":11502,"ariaHidden":194},[193],[104,11504,11506,11509],{"className":11505},[198],[104,11507],{"className":11508,"style":10647},[202],[104,11510,11496],{"className":11511},[207,208]," (epsilon) of it. That's the ",[15,11514,11515,11543],{},[104,11516,11518,11531],{"className":11517},[148],[104,11519,11521],{"className":11520},[152],[154,11522,11523],{"xmlns":156},[158,11524,11525,11529],{},[161,11526,11527],{},[164,11528,11496],{},[186,11530,11499],{"encoding":188},[104,11532,11534],{"className":11533,"ariaHidden":194},[193],[104,11535,11537,11540],{"className":11536},[198],[104,11538],{"className":11539,"style":10647},[202],[104,11541,11496],{"className":11542},[207,208],"-neighborhood",[360,11545,11546,11547,11550,11551,11554],{},"If a point has at least ",[87,11548,11549],{},"min_samples"," neighbors at that distance (counting itself), it's a ",[15,11552,11553],{},"core point"," (it lives in a dense region).",[360,11556,11557],{},"Every point in a core point's neighborhood belongs to the same group. Since that neighborhood can contain other core points, a chain of neighboring core points forms a single group, no matter how long that \"string\" gets.",[360,11559,11560,11561,11564],{},"Any point that isn't core and isn't in any core point's neighborhood is ",[15,11562,11563],{},"noise"," (label -1), it belongs to no group at all.",[11,11566,11567],{},"The core difference from K-means: DBSCAN never assumes a group has \"a center.\" It just follows dense regions, so it can trace crooked shapes, like two crescent moons.",[96,11569,11571],{"className":98,"code":11570,"language":100,"meta":57,"style":57},"from sklearn.cluster import DBSCAN\ndbscan = DBSCAN(eps=0.05, min_samples=5)\ny_pred = dbscan.fit_predict(X)\n",[87,11572,11573,11578,11583],{"__ignoreMap":57},[104,11574,11575],{"class":106,"line":107},[104,11576,11577],{},"from sklearn.cluster import DBSCAN\n",[104,11579,11580],{"class":106,"line":58},[104,11581,11582],{},"dbscan = DBSCAN(eps=0.05, min_samples=5)\n",[104,11584,11585],{"class":106,"line":118},[104,11586,11587],{},"y_pred = dbscan.fit_predict(X)\n",[433,11589,11590],{},[11,11591,11592,11595],{},[15,11593,11594],{},"Output (eps=0.05):"," 7 groups, 77 points marked as noise.",[11,11597,11598,11599,11627],{},"Too small an ",[104,11600,11602,11615],{"className":11601},[148],[104,11603,11605],{"className":11604},[152],[154,11606,11607],{"xmlns":156},[158,11608,11609,11613],{},[161,11610,11611],{},[164,11612,11496],{},[186,11614,11499],{"encoding":188},[104,11616,11618],{"className":11617,"ariaHidden":194},[193],[104,11619,11621,11624],{"className":11620},[198],[104,11622],{"className":11623,"style":10647},[202],[104,11625,11496],{"className":11626},[207,208]," fragments the whole dataset into little pieces, because too few points fall inside such a tight neighborhood.",[96,11629,11631],{"className":98,"code":11630,"language":100,"meta":57,"style":57},"dbscan = DBSCAN(eps=0.2, min_samples=5)\n",[87,11632,11633],{"__ignoreMap":57},[104,11634,11635],{"class":106,"line":107},[104,11636,11630],{},[433,11638,11639],{},[11,11640,11641,11644,11645,11648,11649,11651],{},[15,11642,11643],{},"Output (eps=0.2):"," exactly 2 groups, ",[15,11646,11647],{},"zero"," noise points, adjusted Rand index of ",[15,11650,4754],{},": a perfect match with the real moon split.",[79,11653,11655],{"id":11654},"interactive-nudging-eps-and-min_samples-live","Interactive: nudging eps and min_samples live",[11,11657,11658],{},"My own reconstruction of the algorithm (on a sample of 300 points from the same two moons, to run fast in the browser). Move both controls and watch the number of groups and noise points shift:",[11660,11661],"dbscan-explorer",{":initial-eps":11662,":initial-min-samples":332,":points":11663,":x-max":11664,":x-min":11665,":y-max":11666,":y-min":11667,"x-label":11668,"y-label":11669},"0.2","[[0.6225,-0.3721],[1.9043,-0.1363],[-0.0694,0.4561],[0.9339,0.2375],[1.1804,-0.4908],[1.0254,-0.4625],[0.412,-0.286],[-0.6127,0.7298],[0.9481,0.0796],[0.1712,0.0016],[1.292,-0.3773],[0.3722,0.8922],[0.9163,-0.4881],[0.6263,0.866],[0.1712,-0.0556],[0.9715,-0.5275],[1.9931,0.3036],[0.7204,-0.4162],[-0.9279,0.0214],[0.1161,0.2384],[-0.0621,1.0061],[-0.7831,0.6132],[1.5022,-0.4123],[1.5101,-0.3856],[-0.6721,0.8306],[0.6121,0.7783],[1.8842,0.1128],[2.0745,0.4222],[0.0384,0.2894],[1.4003,-0.3676],[1.0046,0.3922],[0.044,0.2312],[0.8199,0.6096],[0.8525,0.4013],[0.7624,-0.3969],[-0.8232,0.5753],[1.005,0.0664],[-0.2738,1.0104],[0.5843,0.9074],[-0.0347,1.0917],[-0.4168,0.8609],[-0.0435,1.0931],[1.9617,0.1827],[0.2362,0.9484],[-0.1524,0.9336],[0.3173,1.0369],[1.1925,-0.5033],[1.0977,-0.3781],[-0.6756,0.7637],[1.1265,-0.4876],[0.2258,-0.0849],[-1.0893,0.1693],[1.2702,-0.3481],[0.0238,1.0035],[2.0211,0.4524],[1.8792,0.0819],[-0.4126,0.8663],[-0.3918,0.9149],[0.4147,-0.2016],[0.8824,0.5321],[1.126,-0.4437],[0.3486,0.9116],[0.9026,0.0651],[0.2267,-0.1147],[0.1929,0.0119],[0.7861,0.6043],[0.8949,0.4939],[1.5902,-0.24],[0.3683,0.9676],[0.261,-0.2189],[-0.5534,0.814],[-0.9743,0.4438],[1.176,-0.5531],[0.4221,-0.2541],[-0.9474,0.4089],[1.8319,0.2377],[-0.1729,1.0277],[-0.1683,1.0479],[0.1566,0.9942],[2.0012,0.3138],[0.095,0.2055],[-0.0297,0.4838],[0.819,0.4685],[-0.8794,0.5028],[-0.1289,0.9931],[-0.6763,0.7113],[-0.7693,0.6125],[-0.0196,0.9564],[0.9332,0.3339],[0.9813,0.2585],[0.424,-0.2975],[-0.4163,0.8682],[0.115,1.0836],[1.0495,-0.4091],[0.7053,0.7522],[0.871,0.3702],[0.1733,-0.1418],[-0.9475,-0.0484],[1.7686,0.0151],[0.186,1.036],[1.1108,-0.4234],[0.1507,0.9867],[0.7902,0.594],[0.2472,-0.1959],[1.7709,-0.1614],[1.0243,-0.486],[-1.1047,0.1436],[0.9234,0.3677],[0.2191,-0.0929],[-0.3621,0.9146],[-0.9521,0.2709],[1.6424,-0.1766],[-0.1871,1.0478],[1.4341,-0.3949],[1.431,-0.3057],[-0.7618,0.7186],[-0.9895,0.1209],[2.0405,0.3612],[0.4011,-0.35],[0.6942,-0.381],[0.378,0.8995],[0.0191,0.1248],[0.2904,-0.1639],[0.0736,0.0406],[-1.0022,0.276],[1.6145,-0.3083],[0.6951,0.758],[-0.8887,0.4015],[0.417,-0.2926],[1.9998,0.097],[-0.9743,0.2466],[-0.7659,0.7278],[0.1085,0.115],[-0.6781,0.6087],[1.872,0.0288],[1.79,-0.0846],[-0.0662,0.3852],[0.1494,-0.1353],[0.1642,-0.0546],[1.0248,0.1024],[-0.7113,0.626],[-1.0288,0.1551],[0.1303,-0.1004],[-0.7001,0.5585],[0.6677,-0.5632],[1.7231,-0.2821],[-0.8929,0.4958],[-0.0381,1.0039],[0.895,0.4512],[-0.2188,0.9388],[1.7972,-0.0936],[-0.6859,0.7415],[0.9297,-0.517],[0.0251,0.3777],[1.9736,0.1427],[1.3943,-0.3992],[1.066,-0.4479],[0.1769,1.0591],[0.4505,-0.3813],[-0.1498,0.9101],[0.6642,-0.4237],[0.0193,0.3177],[0.9501,0.2384],[0.7169,0.6415],[0.8116,0.536],[-1.0306,0.1198],[1.8253,0.2232],[1.6022,-0.317],[0.0995,0.1044],[0.4393,0.8531],[-0.3925,0.9184],[0.7612,-0.3591],[1.5035,-0.4277],[0.6208,0.7106],[-0.287,0.8166],[1.3767,-0.432],[-1.0389,0.3106],[0.7965,0.5345],[2.1312,0.2933],[-0.9932,0.1475],[0.476,-0.3315],[1.0553,0.0386],[1.7577,0.0136],[1.7538,-0.0907],[-0.5036,0.8634],[1.6893,-0.1679],[1.5937,-0.2946],[0.0667,0.101],[-0.4914,0.8638],[0.7527,0.74],[0.1704,0.9778],[-0.0135,0.5474],[1.2276,-0.4057],[0.5094,-0.5011],[1.4132,-0.4749],[0.8942,0.2307],[1.986,0.4779],[1.8255,-0.1199],[0.798,0.5534],[0.8926,-0.4648],[0.9527,0.0642],[-0.8168,0.6014],[0.8292,0.6288],[0.5567,-0.3878],[-0.0227,0.2088],[0.7545,-0.427],[1.9394,0.2505],[-0.9089,0.2076],[0.2391,0.9378],[-0.9086,0.3396],[-0.8343,0.4789],[1.9004,0.3254],[0.9329,0.1422],[0.1033,1.0373],[1.6953,-0.289],[0.2286,1.1078],[0.5895,0.8026],[2.0015,0.1202],[0.0188,0.3959],[0.6775,0.6899],[-0.9205,0.6155],[1.8839,0.0203],[0.8172,-0.4449],[0.1289,0.463],[0.8553,0.4595],[1.7494,-0.2473],[-0.6379,0.6167],[-0.1481,1.022],[0.7028,-0.4736],[0.6643,-0.4785],[0.6243,0.7253],[1.3207,-0.513],[2.026,0.3243],[1.7031,-0.353],[0.437,-0.3331],[0.9168,0.2827],[0.5479,0.8445],[-0.4335,0.9241],[1.6864,-0.1545],[1.0469,-0.4909],[1.982,-0.0765],[0.594,-0.5127],[-0.0771,0.2081],[0.5575,0.7976],[0.857,-0.5003],[-1.0039,0.1106],[0.3767,-0.3132],[0.8872,-0.499],[1.5544,-0.3866],[0.9935,0.0647],[-0.6349,0.7638],[0.4429,0.8925],[-0.9186,0.2783],[0.9822,0.0096],[-0.8881,0.3235],[-0.5537,0.8375],[0.5944,0.8086],[-0.1486,0.9464],[0.5168,-0.3284],[0.6918,0.6162],[-0.2751,0.8524],[0.4946,0.8189],[1.4033,-0.5486],[0.875,0.2434],[-0.2322,0.9582],[0.0396,0.2363],[2.1022,0.4284],[-0.9815,0.2901],[1.9037,0.2547],[1.2988,-0.3973],[0.5192,0.9439],[0.4704,0.8959],[0.2064,0.0938],[0.2693,-0.2072],[1.9919,0.5192],[0.3084,-0.3219],[0.0618,0.12],[0.2692,0.9691],[0.4539,0.808],[1.2866,-0.5198],[1.5766,-0.4242],[1.0076,0.4132],[0.0269,0.2997],[1.9151,-0.0421],[0.3507,-0.2937],[1.5484,-0.2249],[-1.0176,0.0121],[-0.9118,0.3732],[0.099,0.4766],[-0.5718,0.8495],[-0.2355,0.9921],[0.726,-0.4291],[-0.9189,0.6139],[1.934,0.4887],[0.9656,0.4153],[0.345,-0.1873],[0.138,1.0504],[-0.6391,0.8191],[1.7425,-0.077],[-0.507,0.8433]]","2.3","-1.3","1.2","-0.7","x1","x2",[11,11671,11672,11673,11676,11677,11679,11680,11682],{},"With a tiny ",[87,11674,11675],{},"eps",", almost everyone turns into noise (gray). Raising ",[87,11678,11675],{}," bit by bit, groups start forming, until at some point the two moons show up whole. Push past the right point and the two moons merge into one. ",[87,11681,11549],{}," works similarly: higher values demand a more crowded neighborhood before someone can become a core point, so groups get pickier (more noise, more \"solid\" groups).",[79,11684,11686,11687],{"id":11685},"the-problem-of-having-no-predict","The problem of having no ",[87,11688,9419],{},[11,11690,11691,11692,11695],{},"DBSCAN keeps no centroid at all, so there's no way to ask \"which group does this new point belong to?\" the way K-means does. The professor's fix: train a separate KNN, using only DBSCAN's core points (",[87,11693,11694],{},"dbscan.components_",") and each one's group as the label.",[96,11697,11699],{"className":98,"code":11698,"language":100,"meta":57,"style":57},"from sklearn.neighbors import KNeighborsClassifier\nknn = KNeighborsClassifier(n_neighbors=50)\nknn.fit(dbscan.components_, dbscan.labels_[dbscan.core_sample_indices_])\nprint(knn.predict(X_new))\n",[87,11700,11701,11706,11711,11716],{"__ignoreMap":57},[104,11702,11703],{"class":106,"line":107},[104,11704,11705],{},"from sklearn.neighbors import KNeighborsClassifier\n",[104,11707,11708],{"class":106,"line":58},[104,11709,11710],{},"knn = KNeighborsClassifier(n_neighbors=50)\n",[104,11712,11713],{"class":106,"line":118},[104,11714,11715],{},"knn.fit(dbscan.components_, dbscan.labels_[dbscan.core_sample_indices_])\n",[104,11717,11718],{"class":106,"line":124},[104,11719,11720],{},"print(knn.predict(X_new))\n",[433,11722,11723],{},[11,11724,11725,1807,11727,11730],{},[15,11726,1629],{},[87,11728,11729],{},"[1, 0, 1, 0]"," for the 4 new points tested.",[11,11732,11733,11734,11736],{},"But that has a catch: KNN always finds a nearest neighbor, no matter the distance, so it'll \"predict\" a group for any point, even one far from everything, that should really be noise. The fine-tuning: check the distance to the nearest neighbor and, if it's too large (the professor uses ",[87,11735,11675],{}," itself as the cutoff), mark it as noise (-1) instead of forcing it into a group.",[79,11738,11740],{"id":11739},"switching-topics-what-if-labels-are-expensive","Switching topics: what if labels are expensive?",[11,11742,11743],{},"The second half of the lecture switches problems, but reuses the idea of grouping without labels. Scenario: 1400 handwritten digit images for training, but labeling each one is expensive manual work, so only 50 can be labeled.",[96,11745,11747],{"className":98,"code":11746,"language":100,"meta":57,"style":57},"n_labeled = 50\nlog_reg = LogisticRegression(solver=\"liblinear\", max_iter=5000)\nlog_reg.fit(X_train[:n_labeled], y_train[:n_labeled])\n",[87,11748,11749,11754,11759],{"__ignoreMap":57},[104,11750,11751],{"class":106,"line":107},[104,11752,11753],{},"n_labeled = 50\n",[104,11755,11756],{"class":106,"line":58},[104,11757,11758],{},"log_reg = LogisticRegression(solver=\"liblinear\", max_iter=5000)\n",[104,11760,11761],{"class":106,"line":118},[104,11762,11763],{},"log_reg.fit(X_train[:n_labeled], y_train[:n_labeled])\n",[433,11765,11766],{},[11,11767,11768,11771],{},[15,11769,11770],{},"Output (labeling the first 50 images, in whatever order they came):"," 0.766 test accuracy.",[433,11773,11774],{},[11,11775,11776,11779],{},[15,11777,11778],{},"Output (if I had labels for all 1400, the theoretical ceiling):"," 0.902.",[79,11781,11783],{"id":11782},"the-trick-choosing-which-50-to-label-not-which-come-first","The trick: choosing which 50 to label, not which come first",[11,11785,11786,11787,11790],{},"Instead of labeling the first 50 (an arbitrary order), the professor uses K-means with ",[15,11788,11789],{},"K=50"," (the same label budget!) to find 50 groups in the 1400 unlabeled images, and picks, from each group, the image closest to the center (the most \"typical\" one in that group):",[96,11792,11794],{"className":98,"code":11793,"language":100,"meta":57,"style":57},"kmeans = KMeans(n_clusters=50)\nX_digits_dist = kmeans.fit_transform(X_train)\nrepresentative_digit_idx = np.argmin(X_digits_dist, axis=0)\n",[87,11795,11796,11801,11806],{"__ignoreMap":57},[104,11797,11798],{"class":106,"line":107},[104,11799,11800],{},"kmeans = KMeans(n_clusters=50)\n",[104,11802,11803],{"class":106,"line":58},[104,11804,11805],{},"X_digits_dist = kmeans.fit_transform(X_train)\n",[104,11807,11808],{"class":106,"line":118},[104,11809,11810],{},"representative_digit_idx = np.argmin(X_digits_dist, axis=0)\n",[11,11812,11813],{},"Only those 50 representative images get manually labeled (the professor pasted in the correct labels by hand, cell 8 of the notebook). Training on just those 50:",[433,11815,11816],{},[11,11817,11818,11820,11821,11823],{},[15,11819,1629],{}," 0.834. Better than the 50 random labels (0.766), with the ",[15,11822,9685],{}," number of labels.",[11,11825,11826],{},"That tracks: 50 images chosen to represent 50 different groups cover more variety than 50 images in whatever order they arrived (which might repeat the same \"7\" style several times and never show a crooked \"3\").",[79,11828,11830],{"id":11829},"propagation-spreading-the-label-to-the-whole-group","Propagation: spreading the label to the whole group",[11,11832,11833,11834,11837],{},"If group 12's representative image is a \"7,\" it's reasonable to assume ",[15,11835,11836],{},"everyone"," in group 12 is also a \"7\" (that's why they landed in the same group, after all). Propagating the representative's label to every member of the cluster:",[96,11839,11841],{"className":98,"code":11840,"language":100,"meta":57,"style":57},"y_train_propagated = np.empty(len(X_train), dtype=np.int32)\nfor i in range(k):\n    y_train_propagated[kmeans.labels_==i] = y_representative_digits[i]\n",[87,11842,11843,11848,11853],{"__ignoreMap":57},[104,11844,11845],{"class":106,"line":107},[104,11846,11847],{},"y_train_propagated = np.empty(len(X_train), dtype=np.int32)\n",[104,11849,11850],{"class":106,"line":58},[104,11851,11852],{},"for i in range(k):\n",[104,11854,11855],{"class":106,"line":118},[104,11856,11857],{},"    y_train_propagated[kmeans.labels_==i] = y_representative_digits[i]\n",[433,11859,11860],{},[11,11861,11862,11864,11865,11868],{},[15,11863,1629],{}," training on all 1400 propagated labels (but only 50 truly hand-checked): 0.869. And checking against the real label (something only possible here because it's an exercise, in real life you wouldn't know it): the propagation is right ",[15,11866,11867],{},"95.4%"," of the time.",[11,11870,11871],{},"One more refinement: dropping, from each group, the 20% farthest from the center (the \"border\" points, more likely to have been assigned to the wrong group) before propagating:",[433,11873,11874],{},[11,11875,11876,11878],{},[15,11877,1629],{}," 1111 examples left (out of 1400), with 97.7% propagation accuracy (up from 95.4%). Training on just those: 0.879, the lecture's best result using only 50 truly labeled examples.",[11,11880,11881,11882,11884,11885,11888],{},"And the pattern repeats swapping ",[87,11883,8963],{}," for KNN, Random Forest, Gaussian Naive Bayes, and ",[87,11886,11887],{},"NearestCentroid",": in every classifier tested, \"representative\" beats \"50 random,\" and \"propagated\" beats \"representative alone.\"",[79,11890,1870],{"id":1869},[1872,11892,11893,11901],{},[1875,11894,11895],{},[1878,11896,11897,11899],{},[1881,11898,1884],{"align":1883},[1881,11900,1887],{"align":1883},[1889,11902,11903,11911,11927],{},[1878,11904,11905,11908],{},[1894,11906,11907],{"align":1883},"K-means groups by distance to a centroid",[1894,11909,11910],{"align":1883},"DBSCAN groups by density, with no assumption about a group's shape, so it handles non-convex shapes well",[1878,11912,11913,11916],{},[1894,11914,11915],{"align":1883},"Clustering is for exploring data with no label",[1894,11917,11918,11919,11922,11923,11926],{"align":1883},"Clustering also helps ",[15,11920,11921],{},"decide what to label"," when labeling is expensive, and helps ",[15,11924,11925],{},"extend"," a few labels to the rest of the data",[1878,11928,11929,11932],{},[1894,11930,11931],{"align":1883},"More labeled data is always better",[1894,11933,11934],{"align":1883},"Sometimes 50 well-chosen labels (via clustering) are worth more than 50 labels in whatever order they arrived",[79,11936,1939],{"id":1938},[11,11938,11939],{},"I reproduced the full chain (50 random → representative → propagated) on the same digits dataset, with a fixed seed, to confirm the gain wasn't a coincidence from one specific run.",[96,11941,11943],{"className":98,"code":11942,"language":100,"meta":57,"style":57},"log_reg = LogisticRegression(solver=\"lbfgs\", max_iter=5000, random_state=42)\n# ... same procedure as the professor, with random_state fixed at every step\n",[87,11944,11945,11950],{"__ignoreMap":57},[104,11946,11947],{"class":106,"line":107},[104,11948,11949],{},"log_reg = LogisticRegression(solver=\"lbfgs\", max_iter=5000, random_state=42)\n",[104,11951,11952],{"class":106,"line":58},[104,11953,11954],{},"# ... same procedure as the professor, with random_state fixed at every step\n",[1872,11956,11957,11966],{},[1875,11958,11959],{},[1878,11960,11961,11964],{},[1881,11962,11963],{"align":5561},"Labeling strategy",[1881,11965,11204],{"align":3186},[1889,11967,11968,11976,11984,11992,12000],{},[1878,11969,11970,11973],{},[1894,11971,11972],{"align":5561},"50 random labels (the first 50 images)",[1894,11974,11975],{"align":3186},"0.7582",[1878,11977,11978,11981],{},[1894,11979,11980],{"align":5561},"50 representative labels (1 per K-means group)",[1894,11982,11983],{"align":3186},"0.8388",[1878,11985,11986,11989],{},[1894,11987,11988],{"align":5561},"Labels propagated to the whole group",[1894,11990,11991],{"align":3186},"0.8589",[1878,11993,11994,11997],{},[1894,11995,11996],{"align":5561},"Propagation accuracy (against the real label)",[1894,11998,11999],{"align":3186},"0.9500",[1878,12001,12002,12005],{},[1894,12003,12004],{"align":5561},"All 1400 labels (theoretical ceiling)",[1894,12006,12007],{"align":3186},"0.9093",[11,12009,12010,12011,12014,12015,12018,12019,12021],{},"My numbers land close to the original notebook's (0.766 \u002F 0.834 \u002F 0.869 \u002F 0.954 \u002F 0.902), the difference sits only in the random seed and the logistic regression solver (I swapped ",[87,12012,12013],{},"liblinear"," for ",[87,12016,12017],{},"lbfgs",", because the newer scikit-learn version in my environment no longer accepts ",[87,12020,12013],{}," for problems with more than two classes). The order between strategies is identical in both runs: random loses to representative, which loses to propagated, confirming the gain is real, not luck from one specific run.",[1982,12023,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":12025},[12026,12027,12028,12029,12031,12032,12033,12034,12035],{"id":11431,"depth":58,"text":11432},{"id":11472,"depth":58,"text":11473},{"id":11654,"depth":58,"text":11655},{"id":11685,"depth":58,"text":12030},"The problem of having no .predict()",{"id":11739,"depth":58,"text":11740},{"id":11782,"depth":58,"text":11783},{"id":11829,"depth":58,"text":11830},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 9: the professor shows where K-means fails badly, introduces DBSCAN as a density-based alternative, then uses clustering to pick which examples to label better when labels are expensive.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised",{"title":11423,"description":12036},"en\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised",[12042,12043,12044],"dbscan","semi-supervised-learning","clustering","goon5DO08D62TkjwXY8CVWM2LgEXpQ1WRcCXq-TaghM",{"id":12047,"title":12048,"body":12049,"cover":3,"date":9148,"description":13371,"extension":61,"meta":13372,"navigation":63,"order":1701,"path":13373,"playlist":2000,"seo":13374,"status":66,"stem":13375,"tags":13376,"__hash__":13380},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpca.md","PCA: Reducing Dimensions Without Losing What Matters (Not Always)",{"type":8,"value":12050,"toc":13362},[12051,12054,12058,12064,12089,12100,12103,12182,12371,12581,12590,12594,12597,12612,12615,12687,12698,12705,12724,12734,12738,12907,12927,12937,12940,12964,12971,13127,13131,13134,13141,13176,13180,13186,13216,13226,13229,13268,13271,13274,13283,13286,13288,13324,13326,13337,13357,13360],[11,12052,12053],{},"Lectures 10a and 10b. After K-means and DBSCAN (grouping with no label), PCA tackles another label-free problem: how to summarize a bunch of variables into a few, without losing what matters. Spoiler for the whole post: \"what matters\" is a tricky question.",[79,12055,12057],{"id":12056},"what-pca-actually-computes","What PCA actually computes",[11,12059,12060,12061,12063],{},"The professor goes back to the breast cancer dataset ",[20,12062,11041],{"href":5113},", 30 variables, 569 patients:",[96,12065,12067],{"className":98,"code":12066,"language":100,"meta":57,"style":57},"from sklearn.decomposition import PCA\npca = PCA(n_components=30)\nX_pca = pca.fit_transform(X)\nprint(pca.explained_variance_ratio_)\n",[87,12068,12069,12074,12079,12084],{"__ignoreMap":57},[104,12070,12071],{"class":106,"line":107},[104,12072,12073],{},"from sklearn.decomposition import PCA\n",[104,12075,12076],{"class":106,"line":58},[104,12077,12078],{},"pca = PCA(n_components=30)\n",[104,12080,12081],{"class":106,"line":118},[104,12082,12083],{},"X_pca = pca.fit_transform(X)\n",[104,12085,12086],{"class":106,"line":124},[104,12087,12088],{},"print(pca.explained_variance_ratio_)\n",[433,12090,12091],{},[11,12092,12093,12095,12096,12099],{},[15,12094,1629],{}," the first component alone explains ",[15,12097,12098],{},"98.2%"," of the data's total variance. The remaining 29, together, explain the 1.8% left over.",[11,12101,12102],{},"An absurd concentration, and it makes sense: several of these 30 variables essentially measure the same thing in different ways (a tumor's radius, perimeter, and area are practically the same information, just in different units), so the dataset's real variation is much \"narrower\" than 30 numbers suggest.",[11,12104,12105,12106,12109,12110,12181],{},"Bishop defines PCA as finding the direction that ",[15,12107,12108],{},"maximizes the variance"," of the projected data. For a single component, that turns into an eigenvalue problem: the optimal direction ",[104,12111,12113,12132],{"className":12112},[148],[104,12114,12116],{"className":12115},[152],[154,12117,12118],{"xmlns":156},[158,12119,12120,12129],{},[161,12121,12122],{},[515,12123,12124,12127],{},[164,12125,12126],{"mathvariant":473},"u",[483,12128,485],{},[186,12130,12131],{"encoding":188},"\\mathbf{u}_1",[104,12133,12135],{"className":12134,"ariaHidden":194},[193],[104,12136,12138,12141],{"className":12137},[198],[104,12139],{"className":12140,"style":2579},[202],[104,12142,12144,12147],{"className":12143},[207],[104,12145,12126],{"className":12146},[207,570],[104,12148,12150],{"className":12149},[697],[104,12151,12153,12173],{"className":12152},[605,606],[104,12154,12156,12170],{"className":12155},[610],[104,12157,12159],{"className":12158,"style":2842},[614],[104,12160,12161,12164],{"style":788},[104,12162],{"className":12163,"style":714},[622],[104,12165,12167],{"className":12166},[627,628,629,630],[104,12168,485],{"className":12169},[207,630],[104,12171,667],{"className":12172},[666],[104,12174,12176],{"className":12175},[610],[104,12177,12179],{"className":12178,"style":807},[614],[104,12180],{}," satisfies",[11,12183,12184],{},[104,12185,12187,12222],{"className":12186},[148],[104,12188,12190],{"className":12189},[152],[154,12191,12192],{"xmlns":156},[158,12193,12194,12219],{},[161,12195,12196,12198,12204,12206,12213],{},[164,12197,6136],{"mathvariant":473},[515,12199,12200,12202],{},[164,12201,12126],{"mathvariant":473},[483,12203,485],{},[168,12205,170],{},[515,12207,12208,12211],{},[164,12209,12210],{},"λ",[483,12212,485],{},[515,12214,12215,12217],{},[164,12216,12126],{"mathvariant":473},[483,12218,485],{},[186,12220,12221],{"encoding":188},"\\mathbf{S}\\mathbf{u}_1 = \\lambda_1 \\mathbf{u}_1",[104,12223,12225,12284],{"className":12224,"ariaHidden":194},[193],[104,12226,12228,12232,12235,12275,12278,12281],{"className":12227},[198],[104,12229],{"className":12230,"style":12231},[202],"height:0.8361em;vertical-align:-0.15em;",[104,12233,6136],{"className":12234},[207,570],[104,12236,12238,12241],{"className":12237},[207],[104,12239,12126],{"className":12240},[207,570],[104,12242,12244],{"className":12243},[697],[104,12245,12247,12267],{"className":12246},[605,606],[104,12248,12250,12264],{"className":12249},[610],[104,12251,12253],{"className":12252,"style":2842},[614],[104,12254,12255,12258],{"style":788},[104,12256],{"className":12257,"style":714},[622],[104,12259,12261],{"className":12260},[627,628,629,630],[104,12262,485],{"className":12263},[207,630],[104,12265,667],{"className":12266},[666],[104,12268,12270],{"className":12269},[610],[104,12271,12273],{"className":12272,"style":807},[614],[104,12274],{},[104,12276],{"className":12277,"style":214},[213],[104,12279,170],{"className":12280},[218],[104,12282],{"className":12283,"style":214},[213],[104,12285,12287,12291,12331],{"className":12286},[198],[104,12288],{"className":12289,"style":12290},[202],"height:0.8444em;vertical-align:-0.15em;",[104,12292,12294,12297],{"className":12293},[207],[104,12295,12210],{"className":12296},[207,208],[104,12298,12300],{"className":12299},[697],[104,12301,12303,12323],{"className":12302},[605,606],[104,12304,12306,12320],{"className":12305},[610],[104,12307,12309],{"className":12308,"style":2842},[614],[104,12310,12311,12314],{"style":788},[104,12312],{"className":12313,"style":714},[622],[104,12315,12317],{"className":12316},[627,628,629,630],[104,12318,485],{"className":12319},[207,630],[104,12321,667],{"className":12322},[666],[104,12324,12326],{"className":12325},[610],[104,12327,12329],{"className":12328,"style":807},[614],[104,12330],{},[104,12332,12334,12337],{"className":12333},[207],[104,12335,12126],{"className":12336},[207,570],[104,12338,12340],{"className":12339},[697],[104,12341,12343,12363],{"className":12342},[605,606],[104,12344,12346,12360],{"className":12345},[610],[104,12347,12349],{"className":12348,"style":2842},[614],[104,12350,12351,12354],{"style":788},[104,12352],{"className":12353,"style":714},[622],[104,12355,12357],{"className":12356},[627,628,629,630],[104,12358,485],{"className":12359},[207,630],[104,12361,667],{"className":12362},[666],[104,12364,12366],{"className":12365},[610],[104,12367,12369],{"className":12368,"style":807},[614],[104,12370],{},[11,12372,7423,12373,12403,12404,12473,12474,12477,12478,12506,12507,1807,12510,12580],{},[104,12374,12376,12390],{"className":12375},[148],[104,12377,12379],{"className":12378},[152],[154,12380,12381],{"xmlns":156},[158,12382,12383,12387],{},[161,12384,12385],{},[164,12386,6136],{"mathvariant":473},[186,12388,12389],{"encoding":188},"\\mathbf{S}",[104,12391,12393],{"className":12392,"ariaHidden":194},[193],[104,12394,12396,12400],{"className":12395},[198],[104,12397],{"className":12398,"style":12399},[202],"height:0.6861em;",[104,12401,6136],{"className":12402},[207,570]," is the data's covariance matrix. In other words, ",[104,12405,12407,12424],{"className":12406},[148],[104,12408,12410],{"className":12409},[152],[154,12411,12412],{"xmlns":156},[158,12413,12414,12422],{},[161,12415,12416],{},[515,12417,12418,12420],{},[164,12419,12126],{"mathvariant":473},[483,12421,485],{},[186,12423,12131],{"encoding":188},[104,12425,12427],{"className":12426,"ariaHidden":194},[193],[104,12428,12430,12433],{"className":12429},[198],[104,12431],{"className":12432,"style":2579},[202],[104,12434,12436,12439],{"className":12435},[207],[104,12437,12126],{"className":12438},[207,570],[104,12440,12442],{"className":12441},[697],[104,12443,12445,12465],{"className":12444},[605,606],[104,12446,12448,12462],{"className":12447},[610],[104,12449,12451],{"className":12450,"style":2842},[614],[104,12452,12453,12456],{"style":788},[104,12454],{"className":12455,"style":714},[622],[104,12457,12459],{"className":12458},[627,628,629,630],[104,12460,485],{"className":12461},[207,630],[104,12463,667],{"className":12464},[666],[104,12466,12468],{"className":12467},[610],[104,12469,12471],{"className":12470,"style":807},[614],[104,12472],{}," needs to be an ",[15,12475,12476],{},"eigenvector"," of ",[104,12479,12481,12494],{"className":12480},[148],[104,12482,12484],{"className":12483},[152],[154,12485,12486],{"xmlns":156},[158,12487,12488,12492],{},[161,12489,12490],{},[164,12491,6136],{"mathvariant":473},[186,12493,12389],{"encoding":188},[104,12495,12497],{"className":12496,"ariaHidden":194},[193],[104,12498,12500,12503],{"className":12499},[198],[104,12501],{"className":12502,"style":12399},[202],[104,12504,6136],{"className":12505},[207,570],", and the variance it captures is exactly the corresponding ",[15,12508,12509],{},"eigenvalue",[104,12511,12513,12531],{"className":12512},[148],[104,12514,12516],{"className":12515},[152],[154,12517,12518],{"xmlns":156},[158,12519,12520,12528],{},[161,12521,12522],{},[515,12523,12524,12526],{},[164,12525,12210],{},[483,12527,485],{},[186,12529,12530],{"encoding":188},"\\lambda_1",[104,12532,12534],{"className":12533,"ariaHidden":194},[193],[104,12535,12537,12540],{"className":12536},[198],[104,12538],{"className":12539,"style":12290},[202],[104,12541,12543,12546],{"className":12542},[207],[104,12544,12210],{"className":12545},[207,208],[104,12547,12549],{"className":12548},[697],[104,12550,12552,12572],{"className":12551},[605,606],[104,12553,12555,12569],{"className":12554},[610],[104,12556,12558],{"className":12557,"style":2842},[614],[104,12559,12560,12563],{"style":788},[104,12561],{"className":12562,"style":714},[622],[104,12564,12566],{"className":12565},[627,628,629,630],[104,12567,485],{"className":12568},[207,630],[104,12570,667],{"className":12571},[666],[104,12573,12575],{"className":12574},[610],[104,12576,12578],{"className":12577,"style":807},[614],[104,12579],{},". The larger the eigenvalue, the more variance that direction captures, which is why the \"first principal component\" is always the eigenvector with the largest eigenvalue. The following components repeat the recipe, each one maximizing variance among the remaining directions, orthogonal to the previous ones.",[11,12582,12583,12584,12586,12587,12589],{},"Notice: at no point does this computation use the label ",[87,12585,166],{},". PCA only looks at ",[87,12588,133],{},", searching for where the data varies most. Keep that sentence in mind, it's the thread running through this whole post.",[79,12591,12593],{"id":12592},"reducing-for-real-with-a-catch","Reducing for real, with a catch",[11,12595,12596],{},"With 9 components (30% of the original 30 variables), the captured variance is already 99.99997%:",[96,12598,12600],{"className":98,"code":12599,"language":100,"meta":57,"style":57},"n_most_important = int(X.shape[1]*0.3)  # 9\nscores = cross_val_score(model, X_pca[:, :n_most_important], y, cv=splitter)\n",[87,12601,12602,12607],{"__ignoreMap":57},[104,12603,12604],{"class":106,"line":107},[104,12605,12606],{},"n_most_important = int(X.shape[1]*0.3)  # 9\n",[104,12608,12609],{"class":106,"line":58},[104,12610,12611],{},"scores = cross_val_score(model, X_pca[:, :n_most_important], y, cv=splitter)\n",[11,12613,12614],{},"The professor compares accuracy (original 30-variable space versus the first 9 PCA components) across five different classifiers:",[1872,12616,12617,12630],{},[1875,12618,12619],{},[1878,12620,12621,12624,12627],{},[1881,12622,12623],{"align":1883},"Classifier",[1881,12625,12626],{"align":3186},"Original 30 variables",[1881,12628,12629],{"align":3186},"9 PCA components",[1889,12631,12632,12643,12653,12665,12676],{},[1878,12633,12634,12637,12640],{},[1894,12635,12636],{"align":1883},"KNN (K=3)",[1894,12638,12639],{"align":3186},"0.9274",[1894,12641,12642],{"align":3186},"0.9274 (identical)",[1878,12644,12645,12647,12650],{},[1894,12646,9117],{"align":1883},[1894,12648,12649],{"align":3186},"0.9520",[1894,12651,12652],{"align":3186},"0.9502",[1878,12654,12655,12657,12660],{},[1894,12656,8307],{"align":1883},[1894,12658,12659],{"align":3186},"0.9391",[1894,12661,12662],{"align":3186},[15,12663,12664],{},"0.9039",[1878,12666,12667,12670,12673],{},[1894,12668,12669],{"align":1883},"SVM",[1894,12671,12672],{"align":3186},"0.9150",[1894,12674,12675],{"align":3186},"0.9221",[1878,12677,12678,12681,12684],{},[1894,12679,12680],{"align":1883},"Decision Tree",[1894,12682,12683],{"align":3186},"0.9232",[1894,12685,12686],{"align":3186},"0.9203",[11,12688,12689,12690,12693,12694,12697],{},"Notice the effect isn't the same for everyone. KNN doesn't even notice the difference (PCA that keeps almost all the variance is basically a rotation plus a near-lossless cut, and Euclidean distance doesn't care about rotation). Gaussian Naive Bayes gets ",[15,12691,12692],{},"notably worse",", which is a bit counterintuitive at first: it assumes the variables are independent of each other, and PCA's components are decorrelated by construction, so I'd have expected it to help, not hurt. But \"decorrelated\" isn't the same thing as \"everyone matters equally\": the 9 remaining components have wildly unequal variance among themselves (the first one alone carries the overwhelming share of the total variance), and Gaussian Naive Bayes fits one variance per variable per class, so it ends up trusting the higher-variance components more. The catch is that \"higher variance\" and \"better at separating the classes\" aren't the same thing, the exact trap the next section makes even clearer: dropping the lower-variance components can throw away precisely the axis where one class's mean differs from another's, which Gaussian Naive Bayes feels directly since it depends on those per-variable variances, unlike KNN, which only looks at total distance. And the decision tree gets slightly worse too, ",[20,12695,12696],{"href":8243},"for the same reason I already saw in the decision trees post",": a tree cuts one axis at a time, and PCA rotates the axes, so a decision boundary that used to align with one original variable can turn diagonal in the components, harder to carve with straight cuts.",[11,12699,12700,12701,12704],{},"Scikit-learn also lets you choose the number of components by ",[15,12702,12703],{},"fraction of variance"," instead of a fixed count:",[96,12706,12708],{"className":98,"code":12707,"language":100,"meta":57,"style":57},"pca = PCA(n_components=0.9999)\nX_pca = pca.fit_transform(X)\nprint(X_pca.shape)\n",[87,12709,12710,12715,12719],{"__ignoreMap":57},[104,12711,12712],{"class":106,"line":107},[104,12713,12714],{},"pca = PCA(n_components=0.9999)\n",[104,12716,12717],{"class":106,"line":58},[104,12718,12083],{},[104,12720,12721],{"class":106,"line":118},[104,12722,12723],{},"print(X_pca.shape)\n",[433,12725,12726],{},[11,12727,12728,1807,12730,12733],{},[15,12729,1629],{},[87,12731,12732],{},"(569, 5)",". Just 5 components already reach 99.99% of the variance, even fewer than the 9 picked by hand.",[79,12735,12737],{"id":12736},"the-warning-pca-doesnt-know-whats-important-to-you","The warning: PCA doesn't know what's \"important\" to you",[11,12739,12740,12741,12795,12796,12848,12849,12877,12878,12906],{},"This section (the professor titled it \"PCA fail\" right in the notebook) is why I wrote \"keep that sentence in mind\" above. He builds an artificial dataset on purpose: 12 groups of points, arranged in two rows (one at ",[104,12742,12744,12764],{"className":12743},[148],[104,12745,12747],{"className":12746},[152],[154,12748,12749],{"xmlns":156},[158,12750,12751,12761],{},[161,12752,12753,12755,12758],{},[164,12754,166],{},[168,12756,12757],{},"≈",[483,12759,12760],{},"0.1",[186,12762,12763],{"encoding":188},"y\\approx0.1",[104,12765,12767,12786],{"className":12766,"ariaHidden":194},[193],[104,12768,12770,12774,12777,12780,12783],{"className":12769},[198],[104,12771],{"className":12772,"style":12773},[202],"height:0.6776em;vertical-align:-0.1944em;",[104,12775,166],{"className":12776,"style":209},[207,208],[104,12778],{"className":12779,"style":214},[213],[104,12781,12757],{"className":12782},[218],[104,12784],{"className":12785,"style":214},[213],[104,12787,12789,12792],{"className":12788},[198],[104,12790],{"className":12791,"style":2971},[202],[104,12793,12760],{"className":12794},[207],", another at ",[104,12797,12799,12818],{"className":12798},[148],[104,12800,12802],{"className":12801},[152],[154,12803,12804],{"xmlns":156},[158,12805,12806,12815],{},[161,12807,12808,12810,12812],{},[164,12809,166],{},[168,12811,12757],{},[483,12813,12814],{},"0.4",[186,12816,12817],{"encoding":188},"y\\approx0.4",[104,12819,12821,12839],{"className":12820,"ariaHidden":194},[193],[104,12822,12824,12827,12830,12833,12836],{"className":12823},[198],[104,12825],{"className":12826,"style":12773},[202],[104,12828,166],{"className":12829,"style":209},[207,208],[104,12831],{"className":12832,"style":214},[213],[104,12834,12757],{"className":12835},[218],[104,12837],{"className":12838,"style":214},[213],[104,12840,12842,12845],{"className":12841},[198],[104,12843],{"className":12844,"style":2971},[202],[104,12846,12814],{"className":12847},[207],"), spread along the ",[104,12850,12852,12865],{"className":12851},[148],[104,12853,12855],{"className":12854},[152],[154,12856,12857],{"xmlns":156},[158,12858,12859,12863],{},[161,12860,12861],{},[164,12862,178],{},[186,12864,178],{"encoding":188},[104,12866,12868],{"className":12867,"ariaHidden":194},[193],[104,12869,12871,12874],{"className":12870},[198],[104,12872],{"className":12873,"style":10647},[202],[104,12875,178],{"className":12876},[207,208]," axis from 0 to 1.1. The class alternates: the bottom row is one class, the top row is the other, regardless of ",[104,12879,12881,12894],{"className":12880},[148],[104,12882,12884],{"className":12883},[152],[154,12885,12886],{"xmlns":156},[158,12887,12888,12892],{},[161,12889,12890],{},[164,12891,178],{},[186,12893,178],{"encoding":188},[104,12895,12897],{"className":12896,"ariaHidden":194},[193],[104,12898,12900,12903],{"className":12899},[198],[104,12901],{"className":12902,"style":10647},[202],[104,12904,178],{"className":12905},[207,208],"'s value.",[96,12908,12910],{"className":98,"code":12909,"language":100,"meta":57,"style":57},"model = Perceptron()\nmodel.fit(X, y)\nprint(accuracy_score(y, model.predict(X)))\n",[87,12911,12912,12917,12922],{"__ignoreMap":57},[104,12913,12914],{"class":106,"line":107},[104,12915,12916],{},"model = Perceptron()\n",[104,12918,12919],{"class":106,"line":58},[104,12920,12921],{},"model.fit(X, y)\n",[104,12923,12924],{"class":106,"line":118},[104,12925,12926],{},"print(accuracy_score(y, model.predict(X)))\n",[433,12928,12929],{},[11,12930,12931,12933,12934,12936],{},[15,12932,1629],{}," 1.0. A hundred percent, because separating by row (the ",[87,12935,166],{}," value) is easy, it's just a horizontal line.",[11,12938,12939],{},"Now apply PCA with 1 component before training:",[96,12941,12943],{"className":98,"code":12942,"language":100,"meta":57,"style":57},"pca = PCA(n_components=1)\nX_pca = pca.fit_transform(X)\nmodel.fit(X_pca, y)\nprint(accuracy_score(y, model.predict(X_pca)))\n",[87,12944,12945,12950,12954,12959],{"__ignoreMap":57},[104,12946,12947],{"class":106,"line":107},[104,12948,12949],{},"pca = PCA(n_components=1)\n",[104,12951,12952],{"class":106,"line":58},[104,12953,12083],{},[104,12955,12956],{"class":106,"line":118},[104,12957,12958],{},"model.fit(X_pca, y)\n",[104,12960,12961],{"class":106,"line":124},[104,12962,12963],{},"print(accuracy_score(y, model.predict(X_pca)))\n",[433,12965,12966],{},[11,12967,12968,12970],{},[15,12969,1629],{}," 0.47. Basically a coin flip.",[11,12972,12973,12974,13002,13003,13031,13032,13035,13036,13064,13065,13093,13094,13097,13098,13126],{},"What happened: the ",[104,12975,12977,12990],{"className":12976},[148],[104,12978,12980],{"className":12979},[152],[154,12981,12982],{"xmlns":156},[158,12983,12984,12988],{},[161,12985,12986],{},[164,12987,178],{},[186,12989,178],{"encoding":188},[104,12991,12993],{"className":12992,"ariaHidden":194},[193],[104,12994,12996,12999],{"className":12995},[198],[104,12997],{"className":12998,"style":10647},[202],[104,13000,178],{"className":13001},[207,208]," axis (0 to 1.1) is spread out far more than the ",[104,13004,13006,13019],{"className":13005},[148],[104,13007,13009],{"className":13008},[152],[154,13010,13011],{"xmlns":156},[158,13012,13013,13017],{},[161,13014,13015],{},[164,13016,166],{},[186,13018,166],{"encoding":188},[104,13020,13022],{"className":13021,"ariaHidden":194},[193],[104,13023,13025,13028],{"className":13024},[198],[104,13026],{"className":13027,"style":203},[202],[104,13029,166],{"className":13030,"style":209},[207,208]," axis (0.1 to 0.4), so the direction of ",[15,13033,13034],{},"maximum variance"," sits almost entirely along ",[104,13037,13039,13052],{"className":13038},[148],[104,13040,13042],{"className":13041},[152],[154,13043,13044],{"xmlns":156},[158,13045,13046,13050],{},[161,13047,13048],{},[164,13049,178],{},[186,13051,178],{"encoding":188},[104,13053,13055],{"className":13054,"ariaHidden":194},[193],[104,13056,13058,13061],{"className":13057},[198],[104,13059],{"className":13060,"style":10647},[202],[104,13062,178],{"className":13063},[207,208],". PCA, doing exactly what it promises, picks that direction as the one component. Except the information that separates the classes lives on the ",[104,13066,13068,13081],{"className":13067},[148],[104,13069,13071],{"className":13070},[152],[154,13072,13073],{"xmlns":156},[158,13074,13075,13079],{},[161,13076,13077],{},[164,13078,166],{},[186,13080,166],{"encoding":188},[104,13082,13084],{"className":13083,"ariaHidden":194},[193],[104,13085,13087,13090],{"className":13086},[198],[104,13088],{"className":13089,"style":203},[202],[104,13091,166],{"className":13092,"style":209},[207,208]," axis, the one with ",[15,13095,13096],{},"less"," variance, and PCA threw it out entirely. The algorithm has no way of knowing ",[104,13099,13101,13114],{"className":13100},[148],[104,13102,13104],{"className":13103},[152],[154,13105,13106],{"xmlns":156},[158,13107,13108,13112],{},[161,13109,13110],{},[164,13111,166],{},[186,13113,166],{"encoding":188},[104,13115,13117],{"className":13116,"ariaHidden":194},[193],[104,13118,13120,13123],{"className":13119},[198],[104,13121],{"className":13122,"style":203},[202],[104,13124,166],{"className":13125,"style":209},[207,208]," mattered more for classifying, because it never looked at the label.",[79,13128,13130],{"id":13129},"interactive-where-the-first-component-points","Interactive: where the first component points",[11,13132,13133],{},"My own reconstruction of the same 100-point dataset (two groups, same row layout), with both principal components drawn on top (solid red = 1st component, dashed green = 2nd, each arrow scaled by the square root of the variance that component explains):",[13135,13136],"pca-explorer",{":classes":13137,":points":13138,"class0-label":13139,"class1-label":13140,"x-label":11668,"y-label":11669},"[1,1,0,1,1,1,0,1,1,1,0,0,1,1,0,1,0,0,0,0,0,1,1,1,1,0,1,1,1,0,1,0,1,0,1,0,1,0,0,0,1,0,1,1,0,1,0,0,1,0,1,0,1,1,0,1,0,0,0,0,1,1,0,1,0,0,1,0,0,0,1,1,1,1,1,0,1,1,0,1,0,1,1,0,0,0,1,0,0,1,0,0,0,1,0,0,1,1,0,0]","[[1.0237,0.3964],[0.158,0.3845],[0.6041,0.0987],[0.176,0.3907],[0.7513,0.4394],[0.0411,0.339],[0.7129,0.0963],[0.766,0.4116],[0.0733,0.3887],[-0.0454,0.3294],[0.0883,0.0883],[1.1138,0.1414],[0.3649,0.3836],[0.9558,0.4077],[1.0877,0.0623],[0.8482,0.4206],[0.2662,0.1306],[0.9407,0.0385],[1.0555,0.0592],[0.6919,0.1202],[0.489,0.1179],[0.554,0.4775],[1.0029,0.3429],[0.5505,0.3717],[0.8411,0.4948],[0.1248,0.0931],[0.4049,0.4484],[0.0034,0.3288],[0.9777,0.4428],[0.2261,0.064],[0.5469,0.4237],[0.3369,0.1086],[0.1964,0.4502],[1.1313,0.0571],[0.2515,0.4466],[0.369,0.1411],[1.0087,0.4193],[0.3104,0.002],[0.9114,0.1654],[1.0961,0.1171],[0.8147,0.3643],[0.5046,0.0006],[0.2181,0.3677],[0.2166,0.4488],[0.913,0.1391],[0.8579,0.359],[0.6983,0.0416],[0.179,0.1384],[0.9888,0.4357],[0.0765,0.1271],[0.6396,0.3545],[0.5458,0.1164],[0.8933,0.4237],[1.0107,0.3377],[0.4982,0.1782],[0.9577,0.3243],[0.7943,0.1087],[0.9261,0.1148],[0.0768,0.0767],[1.0465,0.1241],[0.3829,0.3599],[0.4148,0.4131],[0.8382,0.034],[0.7404,0.4328],[0.8608,0.0839],[0.2942,0.0849],[0.6701,0.3299],[0.8196,0.1092],[0.6904,0.1151],[1.1007,0.1727],[0.3292,0.379],[0.6293,0.5095],[0.1447,0.3402],[0.5225,0.4034],[-0.03,0.3854],[0.9125,0.1173],[-0.0575,0.4188],[-0.0301,0.4926],[0.0138,0.0719],[0.605,0.3748],[0.2336,0.1098],[0.3735,0.4257],[0.2181,0.4769],[0.703,0.2232],[0.1121,0.0043],[0.7571,0.1376],[-0.0272,0.4055],[0.277,0.1529],[0.3172,0.0118],[0.4003,0.3883],[0.3162,0.0807],[0.5739,0.0741],[0.5044,0.085],[0.3804,0.3268],[0.4596,0.0749],[0.0494,0.1157],[0.2406,0.4678],[-0.0007,0.3471],[1.0868,0.236],[0.1324,0.1762]]","Bottom row","Top row",[11,13142,13143,13144,13172,13173,13175],{},"Without normalizing, component 1 (red) lies almost flat, following the ",[104,13145,13147,13160],{"className":13146},[148],[104,13148,13150],{"className":13149},[152],[154,13151,13152],{"xmlns":156},[158,13153,13154,13158],{},[161,13155,13156],{},[164,13157,178],{},[186,13159,178],{"encoding":188},[104,13161,13163],{"className":13162,"ariaHidden":194},[193],[104,13164,13166,13169],{"className":13165},[198],[104,13167],{"className":13168,"style":10647},[202],[104,13170,178],{"className":13171},[207,208]," axis, exactly the direction that does ",[15,13174,137],{}," separate the classes. Click \"Normalized (z-score)\": now both variables compete on equal footing, and component 1 rotates toward a diagonal, still not perfectly aligned with the real split (which is why the normalized result, 0.81 accuracy in the notebook, is better but not perfect), but much less blind to what matters than the unnormalized version.",[79,13177,13179],{"id":13178},"pca-as-real-compression","PCA as real compression",[11,13181,13182,13185],{},[87,13183,13184],{},"aula10b"," switches to MNIST (70 thousand images, 784 pixels each) and uses PCA to find how many components are enough to retain 95% of the variance:",[96,13187,13189],{"className":98,"code":13188,"language":100,"meta":57,"style":57},"pca = PCA()\npca.fit(X_train)\ncumsum = np.cumsum(pca.explained_variance_ratio_)\nd = np.argmax(cumsum >= 0.95) + 1\nprint(d)\n",[87,13190,13191,13196,13201,13206,13211],{"__ignoreMap":57},[104,13192,13193],{"class":106,"line":107},[104,13194,13195],{},"pca = PCA()\n",[104,13197,13198],{"class":106,"line":58},[104,13199,13200],{},"pca.fit(X_train)\n",[104,13202,13203],{"class":106,"line":118},[104,13204,13205],{},"cumsum = np.cumsum(pca.explained_variance_ratio_)\n",[104,13207,13208],{"class":106,"line":124},[104,13209,13210],{},"d = np.argmax(cumsum >= 0.95) + 1\n",[104,13212,13213],{"class":106,"line":308},[104,13214,13215],{},"print(d)\n",[433,13217,13218],{},[11,13219,13220,1807,13222,13225],{},[15,13221,1629],{},[87,13223,13224],{},"d = 154",". From 784 pixels to 154 components, almost an 80% size reduction, still keeping 95% of the variance.",[11,13227,13228],{},"Comparing accuracy, raw pixels against the 154 components:",[1872,13230,13231,13243],{},[1875,13232,13233],{},[1878,13234,13235,13237,13240],{},[1881,13236,12623],{"align":1883},[1881,13238,13239],{"align":3186},"784 pixels",[1881,13241,13242],{"align":3186},"154 PCA components",[1889,13244,13245,13256],{},[1878,13246,13247,13251,13253],{},[1894,13248,13249],{"align":1883},[87,13250,11096],{},[1894,13252,11216],{"align":3186},[1894,13254,13255],{"align":3186},"0.8609 (practically the same)",[1878,13257,13258,13262,13265],{},[1894,13259,13260],{"align":1883},[87,13261,8592],{},[1894,13263,13264],{"align":3186},"0.9705",[1894,13266,13267],{"align":3186},"0.9488 (notably worse)",[11,13269,13270],{},"The same pattern as the cancer dataset: the linear model doesn't care (or even improves a hair), the tree forest gets worse, because it again loses the axis alignment it needs to cut well.",[11,13272,13273],{},"And since PCA keeps each component's direction, it can walk back (with loss) and reconstruct an approximation of the original image:",[96,13275,13277],{"className":98,"code":13276,"language":100,"meta":57,"style":57},"X_train_recovered = pca.inverse_transform(X_train_reduced)\n",[87,13278,13279],{"__ignoreMap":57},[104,13280,13281],{"class":106,"line":107},[104,13282,13276],{},[11,13284,13285],{},"The reconstructed digits come out visibly \"blurrier\" than the original, but you can still tell which number is which, with only 154 numbers stored per image instead of 784. It's literally lossy compression, in the same spirit as a JPEG, except the \"basis\" used to describe the image was learned from the dataset itself instead of being generic.",[79,13287,1870],{"id":1869},[1872,13289,13290,13298],{},[1875,13291,13292],{},[1878,13293,13294,13296],{},[1881,13295,1884],{"align":1883},[1881,13297,1887],{"align":1883},[1889,13299,13300,13308,13316],{},[1878,13301,13302,13305],{},[1894,13303,13304],{"align":1883},"Reducing dimensions simplifies the problem",[1894,13306,13307],{"align":1883},"PCA reduces by preserving variance, not by preserving \"what separates the classes,\" and those two things can point in completely different directions",[1878,13309,13310,13313],{},[1894,13311,13312],{"align":1883},"Trees cut one axis at a time",[1894,13314,13315],{"align":1883},"Running PCA before a tree or forest can make results worse, because the rotation breaks the alignment those models exploit",[1878,13317,13318,13321],{},[1894,13319,13320],{"align":1883},"Lossy compression exists for images and audio",[1894,13322,13323],{"align":1883},"PCA does exactly that on any tabular data: an approximate reconstruction, keeping only the directions that vary the most",[79,13325,1939],{"id":1938},[11,13327,13328,13329,13332,13333,13336],{},"Without normalizing, component 1 of the \"PCA fail\" dataset captures 83.5% of the total variance, almost everything. Normalized, it drops to 54.8%, nearly tied with component 2 (45.2%). That alone is a useful warning sign: whenever the first component ",[15,13330,13331],{},"overwhelmingly"," dominates the variance (like breast cancer's 98.2%, or here unnormalized, 83.5%), it's worth suspecting a large-scale variable is dominating the computation on its own, the same scale problem ",[20,13334,13335],{"href":5766},"I already saw with KNN",", just now affecting PCA itself instead of a distance.",[96,13338,13340],{"className":98,"code":13339,"language":100,"meta":57,"style":57},"# variance explained by component 1, with and without normalizing\nratio_unnormalized = 0.835  # x dominates, y barely counts\nratio_normalized = 0.548    # x and y compete on equal footing\n",[87,13341,13342,13347,13352],{"__ignoreMap":57},[104,13343,13344],{"class":106,"line":107},[104,13345,13346],{},"# variance explained by component 1, with and without normalizing\n",[104,13348,13349],{"class":106,"line":58},[104,13350,13351],{},"ratio_unnormalized = 0.835  # x dominates, y barely counts\n",[104,13353,13354],{"class":106,"line":118},[104,13355,13356],{},"ratio_normalized = 0.548    # x and y compete on equal footing\n",[11,13358,13359],{},"The practical rule that sticks: whenever I'm about to run PCA on variables that aren't in the same unit (price in dollars next to age in years, say), normalizing first isn't optional, it's the same old caution, now applied somewhere new.",[1982,13361,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":13363},[13364,13365,13366,13367,13368,13369,13370],{"id":12056,"depth":58,"text":12057},{"id":12592,"depth":58,"text":12593},{"id":12736,"depth":58,"text":12737},{"id":13129,"depth":58,"text":13130},{"id":13178,"depth":58,"text":13179},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 10: the professor uses PCA to compress real data without losing much accuracy, but closes with an important warning: PCA only looks at variance, not at the label, and sometimes throws away exactly the information you needed.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpca",{"title":12048,"description":13371},"en\u002Fplaylists\u002Fpattern-recognition\u002Fpca",[13377,13378,13379],"pca","dimensionality-reduction","variance","mzp1D2z0q7clXEZuX3Xk-MVkI-3SQyyOcMbFsKx6Lcg",{"id":13382,"title":13383,"body":13384,"cover":3,"date":9148,"description":14494,"extension":61,"meta":14495,"navigation":63,"order":1707,"path":14496,"playlist":2000,"seo":14497,"status":66,"stem":14498,"tags":14499,"__hash__":14502},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection.md","Gaussian Mixtures and Anomaly Detection: Clustering With Shape",{"type":8,"value":13385,"toc":14484},[13386,13393,13397,13400,13404,13424,13463,13815,13892,14014,14017,14028,14032,14035,14043,14050,14054,14057,14077,14080,14084,14093,14100,14217,14228,14253,14292,14295,14299,14309,14366,14387,14397,14399,14441,14443,14446,14471,14479,14482],[11,13387,13388,13389,13392],{},"Lectures 11a and 11b. A direct continuation of ",[20,13390,13391],{"href":11413},"the K-means post",": what if the groups aren't round?",[79,13394,13396],{"id":13395},"where-k-means-limps-again","Where K-means limps again",[11,13398,13399],{},"The professor generates a dataset on purpose: two stretched, rotated groups (multiplying the points by a rotation matrix), plus a much smaller third group set apart from the other two. K-means, even with a good initialization (centers picked by hand, close to the right spot), struggles: since it only ever sees \"distance to a center,\" it tends to cut the elongated groups into rounder pieces than they really are, because its boundary between two groups is always a straight line perpendicular to the line joining the centers, never an ellipse.",[79,13401,13403],{"id":13402},"gaussian-mixtures-every-group-becomes-an-ellipse-not-a-point","Gaussian mixtures: every group becomes an ellipse, not a point",[96,13405,13407],{"className":98,"code":13406,"language":100,"meta":57,"style":57},"from sklearn.mixture import GaussianMixture\ngm = GaussianMixture(n_components=3, n_init=10, random_state=42)\ngm.fit(X)\n",[87,13408,13409,13414,13419],{"__ignoreMap":57},[104,13410,13411],{"class":106,"line":107},[104,13412,13413],{},"from sklearn.mixture import GaussianMixture\n",[104,13415,13416],{"class":106,"line":58},[104,13417,13418],{},"gm = GaussianMixture(n_components=3, n_init=10, random_state=42)\n",[104,13420,13421],{"class":106,"line":118},[104,13422,13423],{},"gm.fit(X)\n",[11,13425,13426,13427,13430,13431,13433,13434,13462],{},"The core difference from K-means: instead of storing just one center per group, a ",[15,13428,13429],{},"Gaussian mixture"," stores an entire normal distribution per group, with its own mean ",[15,13432,10059],{}," covariance matrix (which captures the shape, how elongated the group is and in which direction). The full model is a weighted sum of ",[104,13435,13437,13450],{"className":13436},[148],[104,13438,13440],{"className":13439},[152],[154,13441,13442],{"xmlns":156},[158,13443,13444,13448],{},[161,13445,13446],{},[164,13447,5302],{},[186,13449,5302],{"encoding":188},[104,13451,13453],{"className":13452,"ariaHidden":194},[193],[104,13454,13456,13459],{"className":13455},[198],[104,13457],{"className":13458,"style":5314},[202],[104,13460,5302],{"className":13461,"style":5318},[207,208]," Gaussians:",[11,13464,13465],{},[104,13466,13468,13541],{"className":13467},[148],[104,13469,13471],{"className":13470},[152],[154,13472,13473],{"xmlns":156},[158,13474,13475,13538],{},[161,13476,13477,13479,13481,13483,13485,13487,13501,13508,13511,13514,13516,13518,13521,13527,13529,13536],{},[164,13478,11],{},[168,13480,470],{"stretchy":469},[164,13482,178],{"mathvariant":473},[168,13484,476],{"stretchy":469},[168,13486,170],{},[490,13488,13489,13491,13499],{},[168,13490,494],{},[161,13492,13493,13495,13497],{},[164,13494,5338],{},[168,13496,170],{},[483,13498,485],{},[164,13500,5302],{},[515,13502,13503,13506],{},[164,13504,13505],{},"π",[164,13507,5338],{},[932,13509,13510],{}," ",[164,13512,506],{"mathvariant":13513},"script",[168,13515,470],{"stretchy":469},[164,13517,178],{"mathvariant":473},[168,13519,13520],{},"∣",[515,13522,13523,13525],{},[164,13524,10166],{"mathvariant":2102},[164,13526,5338],{},[168,13528,523],{"separator":194},[515,13530,13531,13534],{},[164,13532,13533],{"mathvariant":473},"Σ",[164,13535,5338],{},[168,13537,476],{"stretchy":469},[186,13539,13540],{"encoding":188},"p(\\mathbf{x}) = \\sum_{k=1}^{K} \\pi_k \\, \\mathcal{N}(\\mathbf{x} \\mid \\boldsymbol{\\mu}_k, \\boldsymbol{\\Sigma}_k)",[104,13542,13544,13571,13707],{"className":13543,"ariaHidden":194},[193],[104,13545,13547,13550,13553,13556,13559,13562,13565,13568],{"className":13546},[198],[104,13548],{"className":13549,"style":558},[202],[104,13551,11],{"className":13552},[207,208],[104,13554,470],{"className":13555},[566],[104,13557,178],{"className":13558},[207,570],[104,13560,476],{"className":13561},[575],[104,13563],{"className":13564,"style":214},[213],[104,13566,170],{"className":13567},[218],[104,13569],{"className":13570,"style":214},[213],[104,13572,13574,13577,13640,13643,13684,13687,13692,13695,13698,13701,13704],{"className":13573},[198],[104,13575],{"className":13576,"style":10206},[202],[104,13578,13580,13583],{"className":13579},[687],[104,13581,494],{"className":13582,"style":693},[687,691,692],[104,13584,13586],{"className":13585},[697],[104,13587,13589,13632],{"className":13588},[605,606],[104,13590,13592,13629],{"className":13591},[610],[104,13593,13595,13615],{"className":13594,"style":707},[614],[104,13596,13597,13600],{"style":710},[104,13598],{"className":13599,"style":714},[622],[104,13601,13603],{"className":13602},[627,628,629,630],[104,13604,13606,13609,13612],{"className":13605},[207,630],[104,13607,5338],{"className":13608,"style":5386},[207,208,630],[104,13610,170],{"className":13611},[218,630],[104,13613,485],{"className":13614},[207,630],[104,13616,13617,13620],{"style":732},[104,13618],{"className":13619,"style":714},[622],[104,13621,13623],{"className":13622},[627,628,629,630],[104,13624,13626],{"className":13625},[207,630],[104,13627,5302],{"className":13628,"style":5318},[207,208,630],[104,13630,667],{"className":13631},[666],[104,13633,13635],{"className":13634},[610],[104,13636,13638],{"className":13637,"style":755},[614],[104,13639],{},[104,13641],{"className":13642,"style":683},[213],[104,13644,13646,13649],{"className":13645},[207],[104,13647,13505],{"className":13648,"style":209},[207,208],[104,13650,13652],{"className":13651},[697],[104,13653,13655,13676],{"className":13654},[605,606],[104,13656,13658,13673],{"className":13657},[610],[104,13659,13661],{"className":13660,"style":5373},[614],[104,13662,13664,13667],{"style":13663},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[104,13665],{"className":13666,"style":714},[622],[104,13668,13670],{"className":13669},[627,628,629,630],[104,13671,5338],{"className":13672,"style":5386},[207,208,630],[104,13674,667],{"className":13675},[666],[104,13677,13679],{"className":13678},[610],[104,13680,13682],{"className":13681,"style":807},[614],[104,13683],{},[104,13685],{"className":13686,"style":683},[213],[104,13688,506],{"className":13689,"style":13691},[207,13690],"mathcal","margin-right:0.1474em;",[104,13693,470],{"className":13694},[566],[104,13696,178],{"className":13697},[207,570],[104,13699],{"className":13700,"style":214},[213],[104,13702,13520],{"className":13703},[218],[104,13705],{"className":13706,"style":214},[213],[104,13708,13710,13713,13759,13762,13765,13812],{"className":13709},[198],[104,13711],{"className":13712,"style":558},[202],[104,13714,13716,13725],{"className":13715},[207],[104,13717,13719],{"className":13718},[207],[104,13720,13722],{"className":13721},[207],[104,13723,10166],{"className":13724},[207,2433],[104,13726,13728],{"className":13727},[697],[104,13729,13731,13751],{"className":13730},[605,606],[104,13732,13734,13748],{"className":13733},[610],[104,13735,13737],{"className":13736,"style":10466},[614],[104,13738,13739,13742],{"style":10469},[104,13740],{"className":13741,"style":714},[622],[104,13743,13745],{"className":13744},[627,628,629,630],[104,13746,5338],{"className":13747,"style":5386},[207,208,630],[104,13749,667],{"className":13750},[666],[104,13752,13754],{"className":13753},[610],[104,13755,13757],{"className":13756,"style":10488},[614],[104,13758],{},[104,13760,523],{"className":13761},[813],[104,13763],{"className":13764,"style":683},[213],[104,13766,13768,13777],{"className":13767},[207],[104,13769,13771],{"className":13770},[207],[104,13772,13774],{"className":13773},[207],[104,13775,13533],{"className":13776},[207,570],[104,13778,13780],{"className":13779},[697],[104,13781,13783,13804],{"className":13782},[605,606],[104,13784,13786,13801],{"className":13785},[610],[104,13787,13789],{"className":13788,"style":5373},[614],[104,13790,13792,13795],{"style":13791},"top:-2.55em;margin-right:0.05em;",[104,13793],{"className":13794,"style":714},[622],[104,13796,13798],{"className":13797},[627,628,629,630],[104,13799,5338],{"className":13800,"style":5386},[207,208,630],[104,13802,667],{"className":13803},[666],[104,13805,13807],{"className":13806},[610],[104,13808,13810],{"className":13809,"style":807},[614],[104,13811],{},[104,13813,476],{"className":13814},[575],[11,13816,7423,13817,13887,13888,13891],{},[104,13818,13820,13838],{"className":13819},[148],[104,13821,13823],{"className":13822},[152],[154,13824,13825],{"xmlns":156},[158,13826,13827,13835],{},[161,13828,13829],{},[515,13830,13831,13833],{},[164,13832,13505],{},[164,13834,5338],{},[186,13836,13837],{"encoding":188},"\\pi_k",[104,13839,13841],{"className":13840,"ariaHidden":194},[193],[104,13842,13844,13847],{"className":13843},[198],[104,13845],{"className":13846,"style":2822},[202],[104,13848,13850,13853],{"className":13849},[207],[104,13851,13505],{"className":13852,"style":209},[207,208],[104,13854,13856],{"className":13855},[697],[104,13857,13859,13879],{"className":13858},[605,606],[104,13860,13862,13876],{"className":13861},[610],[104,13863,13865],{"className":13864,"style":5373},[614],[104,13866,13867,13870],{"style":13663},[104,13868],{"className":13869,"style":714},[622],[104,13871,13873],{"className":13872},[627,628,629,630],[104,13874,5338],{"className":13875,"style":5386},[207,208,630],[104,13877,667],{"className":13878},[666],[104,13880,13882],{"className":13881},[610],[104,13883,13885],{"className":13884,"style":807},[614],[104,13886],{}," is the weight (the fraction of data belonging to that group). Bishop derives fitting this model via an algorithm called ",[15,13889,13890],{},"EM"," (expectation-maximization), which alternates two steps until it converges:",[357,13893,13894,14008],{},[360,13895,13896,13899,13900,1807,13903,14000,14001,14003,14004,14007],{},[15,13897,13898],{},"E step (expectation)",": for every point, compute each component's ",[15,13901,13902],{},"responsibility",[104,13904,13906,13936],{"className":13905},[148],[104,13907,13909],{"className":13908},[152],[154,13910,13911],{"xmlns":156},[158,13912,13913,13933],{},[161,13914,13915,13918,13920,13931],{},[164,13916,13917],{},"γ",[168,13919,470],{"stretchy":469},[515,13921,13922,13925],{},[164,13923,13924],{},"z",[161,13926,13927,13929],{},[164,13928,499],{},[164,13930,5338],{},[168,13932,476],{"stretchy":469},[186,13934,13935],{"encoding":188},"\\gamma(z_{nk})",[104,13937,13939],{"className":13938,"ariaHidden":194},[193],[104,13940,13942,13945,13949,13952,13997],{"className":13941},[198],[104,13943],{"className":13944,"style":558},[202],[104,13946,13917],{"className":13947,"style":13948},[207,208],"margin-right:0.0556em;",[104,13950,470],{"className":13951},[566],[104,13953,13955,13959],{"className":13954},[207],[104,13956,13924],{"className":13957,"style":13958},[207,208],"margin-right:0.044em;",[104,13960,13962],{"className":13961},[697],[104,13963,13965,13989],{"className":13964},[605,606],[104,13966,13968,13986],{"className":13967},[610],[104,13969,13971],{"className":13970,"style":5373},[614],[104,13972,13974,13977],{"style":13973},"top:-2.55em;margin-left:-0.044em;margin-right:0.05em;",[104,13975],{"className":13976,"style":714},[622],[104,13978,13980],{"className":13979},[627,628,629,630],[104,13981,13983],{"className":13982},[207,630],[104,13984,10372],{"className":13985,"style":5386},[207,208,630],[104,13987,667],{"className":13988},[666],[104,13990,13992],{"className":13991},[610],[104,13993,13995],{"className":13994,"style":807},[614],[104,13996],{},[104,13998,476],{"className":13999},[575],", the probability (via Bayes) that the point came from that specific group, given where the Gaussians currently sit. Unlike K-means, which assigns each point to ",[15,14002,6085],{}," group only, EM assigns a ",[15,14005,14006],{},"fractional"," responsibility across all groups (a point on the border between two groups might get 60% responsibility from one and 40% from the other).",[360,14009,14010,14013],{},[15,14011,14012],{},"M step (maximization)",": recompute each Gaussian's mean, covariance, and weight, using those responsibilities as weights. A point with 0.9 responsibility toward group 1 counts almost fully toward group 1's mean and covariance, while one split 0.5\u002F0.5 counts half toward each.",[11,14015,14016],{},"This is literally the \"soft\" version of K-means: swap \"each point belongs to exactly one group\" for \"each point belongs a little to every group,\" and swap \"a group is just a mean\" for \"a group is a mean plus a shape.\"",[433,14018,14019],{},[11,14020,14021,14023,14024,14027],{},[15,14022,1629],{}," weights found, ",[87,14025,14026],{},"[0.40, 0.21, 0.39]"," (matches the true proportion of the three generated groups). Converged in just 4 iterations.",[79,14029,14031],{"id":14030},"interactive-the-ellipses-settling-into-place","Interactive: the ellipses settling into place",[11,14033,14034],{},"My own reconstruction of the EM algorithm (on the same 380 points, 3 groups, two of them stretched). Click \"EM step\" and watch the ellipses (each the 1-standard-deviation outline of that Gaussian) rotate and stretch until they fit the data's real shape:",[14036,14037],"gmm-explorer",{":n-components":5580,":points":14038,":x-max":14039,":x-min":14040,":y-max":14041,":y-min":14042,"x-label":11668,"y-label":11669},"[[-1.397,0.6689],[-1.5544,0.9572],[0.4216,0.4219],[-1.4608,0.7519],[-2.2469,0.7798],[-2.6499,0.1603],[-0.5571,0.0035],[-0.3012,1.944],[-2.1017,0.3399],[-0.7528,-1.1562],[-0.3647,2.9444],[-1.8702,0.663],[-1.3084,-0.5343],[-0.1774,0.0732],[-0.4236,-0.1319],[-1.8018,1.6157],[-0.3874,3.4638],[-0.3632,2.2896],[-1.5163,0.9009],[-0.2556,3.0671],[0.5872,0.5437],[-1.5002,-2.5391],[-1.691,1.0455],[0.1699,-0.2883],[0.2436,1.5866],[-1.8124,0.5865],[0.4329,1.4138],[-1.0305,1.3562],[-0.6777,-1.6514],[-0.1738,-0.6312],[-2.1474,0.6027],[1.3162,1.41],[0.0003,-0.635],[1.0803,0.8774],[-0.1051,-0.5368],[-0.4542,2.9433],[-0.5984,2.7038],[-0.5293,-0.3695],[-1.265,-1.6818],[-1.6018,1.2726],[-0.0668,0.5579],[-0.741,-0.6176],[0.9641,1.7692],[-0.1353,2.9909],[-0.0816,-0.0108],[-0.6033,-1.0474],[-1.9353,-2.3436],[-0.544,-0.9764],[-0.499,-0.7219],[0.6757,0.5116],[-2.169,-0.4307],[-1.6183,1.3119],[-0.957,-1.9875],[0.0241,-0.5593],[-1.8944,0.5322],[0.0781,0.0503],[1.234,1.02],[-1.0304,2.212],[0.0279,0.4623],[-0.3033,-0.0086],[-0.6413,1.4614],[-1.6879,1.2439],[-0.7695,2.5701],[0.026,0.2301],[-0.4519,-1.4045],[-1.5718,-2.2173],[-0.2266,-1.261],[-2.217,0.1283],[-0.7787,-0.9779],[-0.5035,-1.0994],[-0.6094,2.4055],[-0.7201,-1.1225],[-1.2544,1.2117],[1.1962,1.1776],[-0.5273,-1.0992],[-0.8385,-2.0174],[-2.4497,0.6202],[0.0672,0.4268],[-1.9462,0.6892],[-1.7471,0.8418],[0.6934,0.9687],[-1.2374,1.1307],[-1.0847,1.8531],[0.6181,0.3445],[0.4234,0.6546],[1.2764,2.13],[-0.7243,1.9397],[0.0644,0.2269],[-1.2528,1.4129],[-1.2586,2.5021],[-2.0632,2.267],[1.5012,1.4466],[-2.0287,1.4448],[-0.5999,-0.9094],[-1.944,1.9186],[0.4245,0.1732],[-1.7969,1.2382],[-1.2832,1.4056],[-1.5927,1.4856],[-0.2796,3.3672],[-1.6957,1.1181],[-0.7625,2.1224],[0.3737,0.3211],[0.4339,-0.622],[-1.9618,-0.0241],[-2.8526,0.3066],[-1.2317,0.759],[-1.5875,0.5392],[-0.5509,-0.2138],[-0.5282,-0.1087],[0.0566,0.3988],[-0.2795,-0.0704],[0.2273,1.597],[-2.3005,0.6762],[0.7842,1.0364],[-1.7827,1.3026],[0.1197,0.3453],[-0.8305,1.6026],[-0.7473,-0.6929],[-0.8915,-0.0462],[-1.5467,0.8436],[-1.65,0.6603],[-0.849,2.4742],[-1.1966,1.4964],[-1.0852,1.8029],[-1.5808,0.6337],[-1.3083,1.814],[-0.111,-0.992],[-0.3001,2.3096],[-1.2104,1.2865],[-1.7846,0.2639],[-2.0529,-0.4044],[1.7441,2.0217],[0.6524,0.6848],[0.11,-0.1965],[-0.0356,0.4157],[-0.0749,2.9341],[-1.5994,2.0179],[-1.0351,-1.8731],[-0.171,2.6711],[-0.4753,3.3274],[-2.3558,1.0955],[-2.0755,0.7241],[-1.8707,0.9155],[-2.8612,-0.5567],[-2.742,0.4937],[-0.3161,0.2769],[-1.7449,2.1745],[0.8469,-0.159],[-0.6866,2.0795],[-1.0555,1.2891],[-0.2582,0.0552],[-1.1792,-2.3321],[-0.2038,-0.2389],[-0.9721,1.4959],[-1.769,1.3658],[-0.4673,-0.3329],[-1.5077,-2.8246],[-2.2638,0.8664],[1.2534,1.3654],[1.0126,1.5736],[-0.5988,3.3043],[-1.898,-0.0087],[-1.4826,0.682],[0.3936,2.9382],[-1.7633,1.2016],[0.4435,0.7339],[-0.9929,1.6227],[-1.3397,-1.6588],[-0.7939,-0.9446],[-0.6199,-0.8483],[-1.2111,1.5388],[-2.0879,0.5661],[-0.6015,-0.4415],[-0.9774,-0.3869],[-2.8054,-0.2992],[0.2876,-0.0551],[-1.8222,0.4918],[-2.6337,-0.4616],[-1.6066,-1.8044],[0.9042,1.2706],[-2.4887,-0.5669],[-1.4762,2.1884],[-3.3941,-2.2838],[-0.0459,0.611],[0.3615,-0.6934],[-0.6614,-0.5482],[-0.9675,-1.1216],[-0.8114,1.6109],[-0.4249,-0.8102],[-0.3229,-0.2177],[-1.8453,1.2592],[0.6648,1.5428],[-2.018,1.4594],[-2.7884,0.4345],[0.257,0.8384],[-1.0581,1.6156],[-0.594,2.3061],[-0.8174,-0.8417],[2.0556,3.0074],[-1.0839,-1.2423],[0.5036,1.2528],[-2.7214,-0.3583],[-1.9335,1.9023],[0.093,0.2204],[-2.1258,0.0677],[1.971,3.5944],[0.2497,0.3212],[-0.5735,-1.5174],[-0.3901,2.4058],[0.0023,-0.8388],[-0.1752,0.0744],[0.4605,0.9383],[-0.7978,-0.7208],[-2.3619,0.0751],[-0.1661,3.8238],[0.2842,-0.7207],[0.7606,1.8876],[1.5808,4.2012],[0.2639,3.3232],[-1.2204,0.9527],[-1.7879,-1.6378],[-1.1302,1.8454],[1.2432,1.4315],[-2.1969,0.8246],[0.7229,0.4503],[0.3372,-0.0962],[-0.6671,2.5567],[-0.2091,-0.5909],[-0.5144,0.1183],[0.2037,0.7191],[0.4719,0.4481],[-0.723,2.1655],[0.0977,0.8399],[-0.4848,-0.1881],[0.3503,-0.8179],[-0.5901,-0.4508],[0.7263,1.0749],[-0.5648,-0.294],[-0.2182,-0.1902],[-0.9943,1.5624],[1.2449,1.9522],[0.623,1.2615],[-2.3547,-0.325],[-2.2113,0.7627],[-1.8254,1.4898],[1.2256,0.4739],[-0.4011,0.7219],[-0.1606,0.123],[-1.5009,1.0681],[0.0147,0.6807],[-0.0381,0.1796],[-1.0194,2.0814],[0.4602,2.7832],[-2.594,0.6801],[-1.0335,-1.261],[0.3909,3.2754],[0.0448,0.86],[-1.8919,1.0616],[0.0364,0.0331],[-0.9895,2.5191],[-0.8706,2.4464],[-1.3902,1.6082],[-1.3474,1.7972],[0.2534,0.1369],[-1.5536,0.9959],[-0.8877,2.0832],[-0.7537,-0.7697],[-0.9112,-0.787],[0.1413,-0.5064],[-1.9722,0.4258],[-2.2693,-0.1881],[-1.4793,0.6792],[-0.3873,2.6294],[0.1175,4.0415],[-2.4398,0.4219],[-0.1291,-0.6354],[1.6276,1.5164],[-1.9332,2.4903],[0.4905,1.1607],[-0.8633,2.2045],[-0.9049,1.9118],[0.6161,1.4715],[-2.5119,-0.4271],[0.8632,0.6314],[-1.8683,2.1036],[-1.8102,-0.5893],[-0.854,-1.0794],[-1.0491,2.6654],[-1.8057,1.3328],[-1.9607,0.4713],[0.1505,0.8971],[0.0872,0.6326],[1.8988,1.959],[-1.324,-1.569],[-1.2021,1.1349],[-2.8576,-0.3023],[-0.699,2.2929],[-0.0063,-0.3087],[-1.9344,0.5451],[3.455,2.5789],[2.9464,1.1252],[1.572,0.9878],[3.3295,1.4183],[2.6823,0.5125],[3.4773,-0.0434],[3.5826,-0.9733],[2.5003,0.448],[2.3398,1.39],[5.377,1.1889],[2.8108,1.2465],[2.4285,1.4879],[3.5583,-0.4105],[2.5828,-0.398],[4.0777,3.2047],[3.2711,1.3714],[2.0754,0.5936],[5.3566,1.4881],[3.2984,1.3158],[3.2566,0.7801],[2.8891,2.8666],[2.7887,0.6866],[2.8902,0.7226],[1.9401,1.0828],[2.7075,0.6922],[2.8139,1.626],[3.7183,2.3214],[4.9565,0.7885],[3.8149,0.6292],[3.6997,-0.9454],[5.07,1.7817],[4.1385,2.5373],[3.0302,2.0714],[4.3033,2.3705],[4.8936,-0.3876],[3.551,3.4775],[4.4062,1.343],[2.3845,-0.1819],[3.5879,1.9829],[3.8344,-0.7488],[0.8711,1.8362],[4.9687,0.496],[3.7413,1.3607],[3.7328,-0.899],[3.8221,1.9898],[4.2818,0.1049],[3.4561,-0.1544],[2.961,1.5276],[4.0127,1.3113],[3.4959,0.7797],[3.8522,2.5523],[3.0213,1.5568],[2.1626,1.2111],[3.7507,1.7961],[3.0274,0.5486],[1.8833,1.1989],[3.0987,-0.4492],[2.6516,0.7051],[2.2539,-0.3062],[4.3043,-0.2166],[2.5161,1.8014],[3.8524,0.3692],[3.5905,0.5108],[4.5218,1.9456],[3.7839,0.2999],[3.7484,0.9398],[3.1481,0.212],[3.5778,0.7153],[3.0116,0.8286],[2.0123,0.2944],[4.6336,1.7662],[2.2995,1.6708],[3.4188,2.0178],[4.3133,-0.2066],[2.5714,2.5642],[1.7659,0.452],[2.478,1.3285],[4.2293,1.1857],[3.7869,1.2753],[3.3752,0.7132]]","5.7","-3.7","4.5","-3.2",[11,14044,14045,14046,14049],{},"Notice the weights shown below the chart drift toward ",[87,14047,14048],{},"0.40, 0.21, 0.39"," as you click \"EM step\" repeatedly, and the ellipses go from generic circles (the initialization) to stretched shapes that trace the groups' real direction.",[79,14051,14053],{"id":14052},"anomaly-detection-for-free","Anomaly detection for free",[11,14055,14056],{},"One advantage of having a density model (not just a grouping): you can ask \"how likely is this point, given the model?\" and flag the least likely ones as anomalies.",[96,14058,14060],{"className":98,"code":14059,"language":100,"meta":57,"style":57},"densities = gm.score_samples(X)\ndensity_threshold = np.percentile(densities, 2)\nanomalies = X[densities \u003C density_threshold]\n",[87,14061,14062,14067,14072],{"__ignoreMap":57},[104,14063,14064],{"class":106,"line":107},[104,14065,14066],{},"densities = gm.score_samples(X)\n",[104,14068,14069],{"class":106,"line":58},[104,14070,14071],{},"density_threshold = np.percentile(densities, 2)\n",[104,14073,14074],{"class":106,"line":118},[104,14075,14076],{},"anomalies = X[densities \u003C density_threshold]\n",[11,14078,14079],{},"Points with density below the 2nd percentile (the least likely 2%) become anomaly candidates, and they capture exactly the small isolated group the professor placed on purpose far from the other two.",[79,14081,14083],{"id":14082},"how-many-groups-to-use-bic-aic-and-a-smarter-way","How many groups to use? BIC, AIC, and a smarter way",[96,14085,14087],{"className":98,"code":14086,"language":100,"meta":57,"style":57},"gm.bic(X), gm.aic(X)\n",[87,14088,14089],{"__ignoreMap":57},[104,14090,14091],{"class":106,"line":107},[104,14092,14086],{},[433,14094,14095],{},[11,14096,14097,14099],{},[15,14098,1629],{}," BIC = 8189.73, AIC = 8102.51.",[11,14101,14102,14103,14106,14107,14157,14158,14187,14188,14216],{},"Both are ",[15,14104,14105],{},"information criteria",": they measure how well the model explains the data, with a penalty for complexity (more groups = more parameters = bigger penalty), to avoid picking \"more groups is always better\" just because more groups always fits better. Running for ",[104,14108,14110,14127],{"className":14109},[148],[104,14111,14113],{"className":14112},[152],[154,14114,14115],{"xmlns":156},[158,14116,14117,14125],{},[161,14118,14119,14121,14123],{},[164,14120,5302],{},[168,14122,170],{},[483,14124,485],{},[186,14126,5487],{"encoding":188},[104,14128,14130,14148],{"className":14129,"ariaHidden":194},[193],[104,14131,14133,14136,14139,14142,14145],{"className":14132},[198],[104,14134],{"className":14135,"style":5314},[202],[104,14137,5302],{"className":14138,"style":5318},[207,208],[104,14140],{"className":14141,"style":214},[213],[104,14143,170],{"className":14144},[218],[104,14146],{"className":14147,"style":214},[213],[104,14149,14151,14154],{"className":14150},[198],[104,14152],{"className":14153,"style":2971},[202],[104,14155,485],{"className":14156},[207]," through ",[104,14159,14161,14175],{"className":14160},[148],[104,14162,14164],{"className":14163},[152],[154,14165,14166],{"xmlns":156},[158,14167,14168,14173],{},[161,14169,14170],{},[483,14171,14172],{},"9",[186,14174,14172],{"encoding":188},[104,14176,14178],{"className":14177,"ariaHidden":194},[193],[104,14179,14181,14184],{"className":14180},[198],[104,14182],{"className":14183,"style":2971},[202],[104,14185,14172],{"className":14186},[207]," and plotting both against ",[104,14189,14191,14204],{"className":14190},[148],[104,14192,14194],{"className":14193},[152],[154,14195,14196],{"xmlns":156},[158,14197,14198,14202],{},[161,14199,14200],{},[164,14201,5302],{},[186,14203,5302],{"encoding":188},[104,14205,14207],{"className":14206,"ariaHidden":194},[193],[104,14208,14210,14213],{"className":14209},[198],[104,14211],{"className":14212,"style":5314},[202],[104,14214,5302],{"className":14215,"style":5318},[207,208],", the shape of the curve points at the number of groups that balances fit against simplicity.",[11,14218,14219,14220,14223,14224,14227],{},"There's an even more direct way: ",[87,14221,14222],{},"BayesianGaussianMixture"," is handed a ",[15,14225,14226],{},"generous"," number of components (10, in this case) and prunes the unnecessary ones itself, zeroing out their weight:",[96,14229,14231],{"className":98,"code":14230,"language":100,"meta":57,"style":57},"from sklearn.mixture import BayesianGaussianMixture\nbgm = BayesianGaussianMixture(n_components=10, n_init=10, random_state=42)\nbgm.fit(X)\nprint(np.round(bgm.weights_, 2))\n",[87,14232,14233,14238,14243,14248],{"__ignoreMap":57},[104,14234,14235],{"class":106,"line":107},[104,14236,14237],{},"from sklearn.mixture import BayesianGaussianMixture\n",[104,14239,14240],{"class":106,"line":58},[104,14241,14242],{},"bgm = BayesianGaussianMixture(n_components=10, n_init=10, random_state=42)\n",[104,14244,14245],{"class":106,"line":118},[104,14246,14247],{},"bgm.fit(X)\n",[104,14249,14250],{"class":106,"line":124},[104,14251,14252],{},"print(np.round(bgm.weights_, 2))\n",[433,14254,14255],{},[11,14256,14257,1807,14259,14262,14263,14291],{},[15,14258,1629],{},[87,14260,14261],{},"[0.4, 0.21, 0.39, 0, 0, 0, 0, 0, 0, 0]",". Seven of the ten components zeroed out on their own, leaving the three real ones, no need for me to scan ",[104,14264,14266,14279],{"className":14265},[148],[104,14267,14269],{"className":14268},[152],[154,14270,14271],{"xmlns":156},[158,14272,14273,14277],{},[161,14274,14275],{},[164,14276,5302],{},[186,14278,5302],{"encoding":188},[104,14280,14282],{"className":14281,"ariaHidden":194},[193],[104,14283,14285,14288],{"className":14284},[198],[104,14286],{"className":14287,"style":5314},[202],[104,14289,5302],{"className":14290,"style":5318},[207,208]," by hand.",[11,14293,14294],{},"Bishop warns of a technical problem worth knowing: if a Gaussian \"collapses\" right on top of a single data point, its variance can go to zero and the model's likelihood goes to infinity, a singularity, not a good fit. Real implementations (like scikit-learn's) guard against this in practice with numerical safeguards, but it's a reminder that \"finding the maximum likelihood\" isn't always as well-behaved a problem as it sounds.",[79,14296,14298],{"id":14297},"switching-topics-other-ways-to-spot-an-anomaly","Switching topics: other ways to spot an anomaly",[11,14300,14301,14304,14305,14308],{},[87,14302,14303],{},"aula11b"," sets up a more direct anomaly-detection scenario: 980 \"normal\" points (3 well-behaved groups) plus 20 points scattered randomly across the space (the real anomalies), and compares three different detectors, each with the default ",[87,14306,14307],{},"contamination"," (0.1, meaning \"assume 10% of the data is anomalous\") and then tuned via Optuna:",[1872,14310,14311,14324],{},[1875,14312,14313],{},[1878,14314,14315,14318,14321],{},[1881,14316,14317],{"align":1883},"Detector",[1881,14319,14320],{"align":3186},"F1 (anomaly), default",[1881,14322,14323],{"align":3186},"F1 (anomaly), tuned",[1889,14325,14326,14339,14353],{},[1878,14327,14328,14333,14336],{},[1894,14329,14330],{"align":1883},[87,14331,14332],{},"IsolationForest",[1894,14334,14335],{"align":3186},"≈ 0.27",[1894,14337,14338],{"align":3186},"0.68",[1878,14340,14341,14346,14348],{},[1894,14342,14343],{"align":1883},[87,14344,14345],{},"LocalOutlierFactor",[1894,14347,14335],{"align":3186},[1894,14349,14350],{"align":3186},[15,14351,14352],{},"0.79",[1878,14354,14355,14360,14363],{},[1894,14356,14357],{"align":1883},[87,14358,14359],{},"OneClassSVM",[1894,14361,14362],{"align":3186},"≈ 0.24",[1894,14364,14365],{"align":3186},"0.64",[11,14367,14368,14369,14372,14373,14376,14377,14379,14380,14382,14383,14386],{},"The default ",[87,14370,14371],{},"contamination=0.1"," tells every detector to flag ",[15,14374,14375],{},"10%"," of the data as anomalous (98 points), but only 20 of the 1000 points (2%) are actually anomalies. Forcing the model to find 5 times more anomalies than actually exist guarantees a pile of false positives, hence the low F1 across all three before tuning. Once Optuna searches for the right ",[87,14378,14307],{}," (close to 0.02, the true value) and each model's other hyperparameters, all three improve substantially, and ",[87,14381,14345],{}," (which decides \"anomalous\" by comparing a point's local density against its neighbors', ",[20,14384,14385],{"href":12038},"the same density-based reasoning DBSCAN uses",") comes out ahead.",[11,14388,14389,14390,14392,14393,14396],{},"The practical lesson, matching the rest of the lecture: ",[87,14391,14307],{}," isn't a cosmetic detail, it's the most important piece of the tuning, because it tells the model ",[15,14394,14395],{},"how many"," anomalies to look for. Without knowing (or estimating well) that fraction ahead of time, any of these three detectors misses the mark.",[79,14398,1870],{"id":1869},[1872,14400,14401,14409],{},[1875,14402,14403],{},[1878,14404,14405,14407],{},[1881,14406,1884],{"align":1883},[1881,14408,1887],{"align":1883},[1889,14410,14411,14422,14430],{},[1878,14412,14413,14416],{},[1894,14414,14415],{"align":1883},"K-means groups by distance to a center",[1894,14417,14418,14419,14421],{"align":1883},"Gaussian mixtures group by density ",[15,14420,10059],{}," shape, with fractional responsibility instead of a rigid assignment",[1878,14423,14424,14427],{},[1894,14425,14426],{"align":1883},"DBSCAN finds anomalies as \"whoever isn't core to anything\"",[1894,14428,14429],{"align":1883},"Model density (GMM) and local density (LOF) are two other valid ways to define \"anomaly,\" each with its own bias",[1878,14431,14432,14435],{},[1894,14433,14434],{"align":1883},"Hyperparameters matter",[1894,14436,14437,14438,14440],{"align":1883},"For anomaly detection specifically, ",[87,14439,14307],{}," is the hyperparameter that matters most, because most algorithms need to know upfront how much anomaly to look for",[79,14442,1939],{"id":1938},[11,14444,14445],{},"I use the same density idea from the GMM anomaly-detection section, but compare four different percentile cutoffs on the same 3-group dataset, to see how much the cutoff choice changes how many points turn into \"anomalies.\"",[96,14447,14449],{"className":98,"code":14448,"language":100,"meta":57,"style":57},"for percentil in [1, 2, 5, 10]:\n    threshold = np.percentile(densities, percentil)\n    n_anomalias = (densities \u003C threshold).sum()\n    print(percentil, n_anomalias)\n",[87,14450,14451,14456,14461,14466],{"__ignoreMap":57},[104,14452,14453],{"class":106,"line":107},[104,14454,14455],{},"for percentil in [1, 2, 5, 10]:\n",[104,14457,14458],{"class":106,"line":58},[104,14459,14460],{},"    threshold = np.percentile(densities, percentil)\n",[104,14462,14463],{"class":106,"line":118},[104,14464,14465],{},"    n_anomalias = (densities \u003C threshold).sum()\n",[104,14467,14468],{"class":106,"line":124},[104,14469,14470],{},"    print(percentil, n_anomalias)\n",[433,14472,14473],{},[11,14474,14475,14478],{},[15,14476,14477],{},"Output (1250 points total, the same 750+250 generated in the lecture):"," with a 1% cutoff, 13 points flagged. With 2% (the value used in the notebook), 25 points. With 5%, 63 points. With 10%, 125 points.",[11,14480,14481],{},"The number of \"anomalies\" found scales almost linearly with the chosen percentile, because that's literally how a percentile cutoff works: it always finds exactly that fraction of the data, whether a real anomaly is sitting there or not. It's the same point the lecture's detector comparison already made a different way: deciding \"how much\" to look for is a choice that changes the result as much as the algorithm itself.",[1982,14483,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":14485},[14486,14487,14488,14489,14490,14491,14492,14493],{"id":13395,"depth":58,"text":13396},{"id":13402,"depth":58,"text":13403},{"id":14030,"depth":58,"text":14031},{"id":14052,"depth":58,"text":14053},{"id":14082,"depth":58,"text":14083},{"id":14297,"depth":58,"text":14298},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 11: the professor swaps K-means for Gaussian mixtures when groups aren't round, uses density to spot anomalies, then compares three different anomaly detectors, all needing careful tuning to work well.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection",{"title":13383,"description":14494},"en\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection",[14500,34,14501],"gaussian-mixtures","anomaly-detection","8w73rFcG_VUX93NK8-FsyDQoDaoMVN27u8wUk5jcUNQ",{"id":14504,"title":14505,"body":14506,"cover":3,"date":9148,"description":15155,"extension":61,"meta":15156,"navigation":63,"order":1713,"path":15157,"playlist":2000,"seo":15158,"status":66,"stem":15159,"tags":15160,"__hash__":15164},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud.md","Fraud Detection: When 99.8% Accuracy Means Nothing",{"type":8,"value":14507,"toc":15145},[14508,14517,14521,14536,14543,14564,14568,14619,14633,14640,14644,14658,14669,14672,14683,14687,14690,14803,14810,14817,14821,14827,14861,14868,14884,14888,14894,14941,14948,14974,14993,14995,15041,15043,15050,15065,15131,15136,15143],[11,14509,14510,14511,14516],{},"Lecture 12, and the dataset (Kaggle's ",[20,14512,14515],{"href":14513,"rel":14514},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fmlg-ulb\u002Fcreditcardfraud",[45],"Credit Card Fraud Detection",", real, anonymized European credit card transactions) is too large and gated for me to download and reproduce myself in this environment, with no Kaggle account. This post leans on the numbers the notebook itself already ran (genuinely executed, real outputs), and the practical application section rebuilds the lecture's most important finding on a synthetic dataset I can generate and verify on the spot.",[79,14518,14520],{"id":14519},"the-dataset-492-frauds-in-nearly-285-thousand-transactions","The dataset: 492 frauds in nearly 285 thousand transactions",[96,14522,14524],{"className":98,"code":14523,"language":100,"meta":57,"style":57},"df = pd.read_csv('creditcard.csv')\nprint(df['Class'].value_counts())\n",[87,14525,14526,14531],{"__ignoreMap":57},[104,14527,14528],{"class":106,"line":107},[104,14529,14530],{},"df = pd.read_csv('creditcard.csv')\n",[104,14532,14533],{"class":106,"line":58},[104,14534,14535],{},"print(df['Class'].value_counts())\n",[433,14537,14538],{},[11,14539,14540,14542],{},[15,14541,1629],{}," class 0 (normal transaction): 284315. Class 1 (fraud): 492.",[11,14544,14545,14546,14549,14550,14157,14553,14556,14557,1499,14560,14563],{},"That's ",[15,14547,14548],{},"0.17%"," fraud. The input variables already arrive PCA-transformed (",[87,14551,14552],{},"V1",[87,14554,14555],{},"V28",", no original name, for the bank's privacy), plus ",[87,14558,14559],{},"Time",[87,14561,14562],{},"Amount"," untransformed. This extreme imbalance is the whole post's subject.",[79,14565,14567],{"id":14566},"the-baseline-that-exposes-accuracys-lie","The baseline that exposes accuracy's lie",[96,14569,14571],{"className":98,"code":14570,"language":100,"meta":57,"style":57},"class ZeroR(BaseEstimator, TransformerMixin):\n    def fit(self, X, y):\n        self.most_frequent_class_ = y.value_counts().idxmax()\n        return self\n    def transform(self, X):\n        return [self.most_frequent_class_] * len(X)\n\nmodel = ZeroR()\nmodel.fit(X_train, y_train)\nprint(accuracy_score(y_test, model.transform(X_test)))\n",[87,14572,14573,14578,14582,14587,14591,14596,14601,14605,14610,14614],{"__ignoreMap":57},[104,14574,14575],{"class":106,"line":107},[104,14576,14577],{},"class ZeroR(BaseEstimator, TransformerMixin):\n",[104,14579,14580],{"class":106,"line":58},[104,14581,1469],{},[104,14583,14584],{"class":106,"line":118},[104,14585,14586],{},"        self.most_frequent_class_ = y.value_counts().idxmax()\n",[104,14588,14589],{"class":106,"line":124},[104,14590,1479],{},[104,14592,14593],{"class":106,"line":308},[104,14594,14595],{},"    def transform(self, X):\n",[104,14597,14598],{"class":106,"line":417},[104,14599,14600],{},"        return [self.most_frequent_class_] * len(X)\n",[104,14602,14603],{"class":106,"line":422},[104,14604,300],{"emptyLinePlaceholder":63},[104,14606,14607],{"class":106,"line":428},[104,14608,14609],{},"model = ZeroR()\n",[104,14611,14612],{"class":106,"line":1689},[104,14613,5810],{},[104,14615,14616],{"class":106,"line":1695},[104,14617,14618],{},"print(accuracy_score(y_test, model.transform(X_test)))\n",[433,14620,14621],{},[11,14622,14623,1807,14625,14628,14629,14632],{},[15,14624,1629],{},[87,14626,14627],{},"0.9983",". Always guessing \"not fraud,\" never looking at a single variable, the dumbest possible model gets ",[15,14630,14631],{},"99.83%"," right.",[11,14634,14635,14636,14639],{},"That's the most extreme baseline that's shown up in this playlist so far (",[20,14637,14638],{"href":8243},"Car Evaluation, in the decision trees post, had 70%",", here it's nearly 100%). Any headline claiming \"fraud model with 99% accuracy\" needs this ruler standing next to it, because without it the number says nothing.",[79,14641,14643],{"id":14642},"a-real-model-and-two-curves-better-than-accuracy","A real model, and two curves better than accuracy",[96,14645,14647],{"className":98,"code":14646,"language":100,"meta":57,"style":57},"model = LogisticRegression(tol=0.005)\nmodel.fit(X_train, y_train)\n",[87,14648,14649,14654],{"__ignoreMap":57},[104,14650,14651],{"class":106,"line":107},[104,14652,14653],{},"model = LogisticRegression(tol=0.005)\n",[104,14655,14656],{"class":106,"line":58},[104,14657,5810],{},[433,14659,14660],{},[11,14661,14662,14664,14665,14668],{},[15,14663,1629],{}," confusion matrix ",[87,14666,14667],{},"[[56832, 32], [26, 72]]",". Precision 0.69, recall 0.73, F1 0.71 (for the fraud class). ROC AUC 0.867. Precision-recall curve AUC: 0.613.",[11,14670,14671],{},"Notice I don't even cite accuracy here, it would hide everything (98 frauds among nearly 57 thousand transactions, so any reasonable model already clears 99.8%+ accuracy). Precision and recall, though, tell the right story: of the transactions the model flagged as fraud, 69% really were. Of the real frauds, the model caught 73%.",[11,14673,14674,14675,14678,14679,14682],{},"And between the two curves, the precision-recall curve (AUC 0.613) is more honest than the ROC curve (AUC 0.867) for this kind of problem. The ROC curve uses false positive rate on its axis, which is ",[87,14676,14677],{},"false positives \u002F total negatives",", and with nearly 57 thousand negatives, even a handful of false positives becomes a tiny fraction, the curve looks great almost for free. The precision-recall curve, by contrast, uses precision on its axis, which is ",[87,14680,14681],{},"true positives \u002F (true positives + false positives)",", directly sensitive to how many false positives exist compared to the few real frauds, with no dilution in the sea of negatives.",[79,14684,14686],{"id":14685},"resampling-the-remedy-that-makes-things-worse","Resampling: the remedy that makes things worse",[11,14688,14689],{},"The most common idea for handling imbalance is resampling the training set, one way or another, to even out the classes:",[1872,14691,14692,14711],{},[1875,14693,14694],{},[1878,14695,14696,14699,14702,14705,14708],{},[1881,14697,14698],{"align":1883},"Technique",[1881,14700,14701],{"align":1883},"How it works",[1881,14703,14704],{"align":3186},"Precision",[1881,14706,14707],{"align":3186},"Recall",[1881,14709,14710],{"align":3186},"F1",[1889,14712,14713,14730,14748,14766,14785],{},[1878,14714,14715,14718,14721,14724,14727],{},[1894,14716,14717],{"align":1883},"None (baseline)",[1894,14719,14720],{"align":1883},"-",[1894,14722,14723],{"align":3186},"0.69",[1894,14725,14726],{"align":3186},"0.73",[1894,14728,14729],{"align":3186},"0.71",[1878,14731,14732,14737,14740,14743,14745],{},[1894,14733,14734],{"align":1883},[87,14735,14736],{},"RandomOverSampler",[1894,14738,14739],{"align":1883},"duplicates minority-class examples",[1894,14741,14742],{"align":3186},"0.04",[1894,14744,5583],{"align":3186},[1894,14746,14747],{"align":3186},"0.08",[1878,14749,14750,14755,14758,14761,14763],{},[1894,14751,14752],{"align":1883},[87,14753,14754],{},"RandomUnderSampler",[1894,14756,14757],{"align":1883},"discards majority-class examples",[1894,14759,14760],{"align":3186},"0.03",[1894,14762,5583],{"align":3186},[1894,14764,14765],{"align":3186},"0.06",[1878,14767,14768,14773,14776,14779,14782],{},[1894,14769,14770],{"align":1883},[87,14771,14772],{},"SMOTE",[1894,14774,14775],{"align":1883},"creates synthetic examples interpolating minority neighbors",[1894,14777,14778],{"align":3186},"0.07",[1894,14780,14781],{"align":3186},"0.91",[1894,14783,14784],{"align":3186},"0.12",[1878,14786,14787,14792,14795,14797,14800],{},[1894,14788,14789],{"align":1883},[87,14790,14791],{},"NearMiss",[1894,14793,14794],{"align":1883},"discards majority examples near the boundary",[1894,14796,14760],{"align":3186},[1894,14798,14799],{"align":3186},"0.90",[1894,14801,14802],{"align":3186},"0.05",[11,14804,14805,14806,14809],{},"Across all four techniques, recall climbs (from 0.73 to ~0.90), the model catches more real frauds. But precision ",[15,14807,14808],{},"craters"," (from 0.69 to 0.03-0.07), the model starts screaming \"fraud!\" at a lot of normal transactions too. F1 (which balances the two) gets notably worse in all four cases. Resampling isn't a silver bullet, it's a trade-off, and on this specific dataset the trade-off loses.",[11,14811,14812,14813,14816],{},"The reason precision crashes this badly is a mix of two simple things. First, resampling only touches the ",[15,14814,14815],{},"training"," set: the model learns in an artificially balanced world (close to 50\u002F50), but keeps getting tested against the real world, where fraud is 0.17% of transactions. Second, with so many normal transactions in the test set (nearly 57 thousand), even a small error rate on them turns into a huge number of cases in absolute terms: if the model, calibrated for a world where fraud is common, starts getting \"suspicious\" of anything that looks even a little like fraud, even a 1-2% false-positive rate against 57 thousand normal transactions already generates hundreds of false alarms, far more than the handful of real frauds there are to catch. It's that mismatch between \"how many normal transactions exist\" and \"how many false positives the model now makes\" that sends precision off a cliff.",[79,14818,14820],{"id":14819},"the-wrong-way-to-resample-and-why-it-fools-you","The wrong way to resample (and why it fools you)",[11,14822,14823,14824,1863],{},"The notebook has a section with the most direct title in the whole course: ",[15,14825,14826],{},"\"WRONG Approach, don't do it this way!\"",[96,14828,14830],{"className":98,"code":14829,"language":100,"meta":57,"style":57},"smote = SMOTE(random_state=42)\nX_resampled, y_resampled = smote.fit_resample(X, y)  # resamples BEFORE splitting train\u002Ftest\n\nX_train_resampled, X_test_resampled, y_train_resampled, y_test_resampled = train_test_split(\n    X_resampled, y_resampled, test_size=0.2, random_state=42)\nmodel.fit(X_train_resampled, y_train_resampled)\n",[87,14831,14832,14837,14842,14846,14851,14856],{"__ignoreMap":57},[104,14833,14834],{"class":106,"line":107},[104,14835,14836],{},"smote = SMOTE(random_state=42)\n",[104,14838,14839],{"class":106,"line":58},[104,14840,14841],{},"X_resampled, y_resampled = smote.fit_resample(X, y)  # resamples BEFORE splitting train\u002Ftest\n",[104,14843,14844],{"class":106,"line":118},[104,14845,300],{"emptyLinePlaceholder":63},[104,14847,14848],{"class":106,"line":124},[104,14849,14850],{},"X_train_resampled, X_test_resampled, y_train_resampled, y_test_resampled = train_test_split(\n",[104,14852,14853],{"class":106,"line":308},[104,14854,14855],{},"    X_resampled, y_resampled, test_size=0.2, random_state=42)\n",[104,14857,14858],{"class":106,"line":417},[104,14859,14860],{},"model.fit(X_train_resampled, y_train_resampled)\n",[433,14862,14863],{},[11,14864,14865,14867],{},[15,14866,1629],{}," precision 0.98, recall 0.97, F1 0.97. An impressive result.",[11,14869,14870,14871,7536,14874,14876,14877,14880,14881,1863],{},"Impressive and ",[15,14872,14873],{},"invalid",[87,14875,14772],{}," was called on the entire dataset, before splitting train and test. Since SMOTE creates a synthetic example by interpolating between real neighbors of the minority class, some of the synthetic examples that end up in \"training\" after the split are nearly identical to real examples that ended up in \"test.\" The model isn't generalizing to unseen data, it's recognizing near-identical copies of what it already trained on, ",[20,14878,14879],{"href":5610},"the same kind of leakage I already saw before",", just hiding inside a resampling technique this time instead of a misplaced ",[87,14882,14883],{},"fit_transform",[79,14885,14887],{"id":14886},"the-right-way-resampling-inside-the-pipeline","The right way: resampling inside the pipeline",[11,14889,14890,14891,14893],{},"The fix is resampling ",[15,14892,4847],{}," each cross-validation split, never before, exactly like any normalization or feature selection should be done:",[96,14895,14897],{"className":98,"code":14896,"language":100,"meta":57,"style":57},"from imblearn.pipeline import Pipeline\nfrom sklearn.model_selection import cross_validate, StratifiedKFold\n\npipe = Pipeline([\n    ('sampling', SMOTE(random_state=42)),\n    ('model', LogisticRegression(tol=0.005))\n])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nscores = cross_validate(pipe, X, y, cv=cv, scoring=['precision', 'recall', 'f1'])\n",[87,14898,14899,14904,14909,14913,14918,14923,14928,14932,14936],{"__ignoreMap":57},[104,14900,14901],{"class":106,"line":107},[104,14902,14903],{},"from imblearn.pipeline import Pipeline\n",[104,14905,14906],{"class":106,"line":58},[104,14907,14908],{},"from sklearn.model_selection import cross_validate, StratifiedKFold\n",[104,14910,14911],{"class":106,"line":118},[104,14912,300],{"emptyLinePlaceholder":63},[104,14914,14915],{"class":106,"line":124},[104,14916,14917],{},"pipe = Pipeline([\n",[104,14919,14920],{"class":106,"line":308},[104,14921,14922],{},"    ('sampling', SMOTE(random_state=42)),\n",[104,14924,14925],{"class":106,"line":417},[104,14926,14927],{},"    ('model', LogisticRegression(tol=0.005))\n",[104,14929,14930],{"class":106,"line":422},[104,14931,5868],{},[104,14933,14934],{"class":106,"line":428},[104,14935,9057],{},[104,14937,14938],{"class":106,"line":1689},[104,14939,14940],{},"scores = cross_validate(pipe, X, y, cv=cv, scoring=['precision', 'recall', 'f1'])\n",[433,14942,14943],{},[11,14944,14945,14947],{},[15,14946,1629],{}," mean precision ≈ 0.07, mean recall ≈ 0.89, mean F1 ≈ 0.14, across the 5 folds.",[11,14949,14950,14951,14954,14955,14957,14958,14960,14961,14963,14964,14966,14967,1807,14969,1807,14971,14973],{},"Much closer to plain SMOTE's honest result (F1 0.12) than to the inflated 0.97 from the wrong approach. Notice the technical detail: this is ",[87,14952,14953],{},"imblearn","'s ",[87,14956,1498],{},", not scikit-learn's own. The regular ",[87,14959,1498],{}," only accepts steps that transform ",[87,14962,133],{},", but ",[87,14965,14772],{}," needs to touch ",[87,14968,133],{},[15,14970,10059],{},[87,14972,166],{}," together (it creates new rows in both), so it needs a pipeline variant that knows how to propagate that size change downstream.",[11,14975,14976,14977,14979,14980,14982,14983,14986,14987,1807,14989,14992],{},"One bonus noted in passing: adding a ",[87,14978,5828],{}," before ",[87,14981,14772],{}," in the pipeline doesn't change the result much, but makes fitting ",[15,14984,14985],{},"much faster"," (from around 15-19 seconds per fold to 2-3 seconds). That tracks: ",[87,14988,14772],{},[20,14990,14991],{"href":5766},"needs to find nearest neighbors to interpolate"," to create every synthetic example, and searching for neighbors on unnormalized data, with variables at very different scales, is more expensive to compute.",[79,14994,1870],{"id":1869},[1872,14996,14997,15005],{},[1875,14998,14999],{},[1878,15000,15001,15003],{},[1881,15002,1884],{"align":1883},[1881,15004,1887],{"align":1883},[1889,15006,15007,15015,15023],{},[1878,15008,15009,15012],{},[1894,15010,15011],{"align":1883},"A dumb baseline helps interpret accuracy",[1894,15013,15014],{"align":1883},"With extreme imbalance (0.17% fraud), accuracy practically loses meaning, and precision\u002Frecall\u002FPR curve have to take its place",[1878,15016,15017,15020],{},[1894,15018,15019],{"align":1883},"Leakage happens when the same data influences training and evaluation",[1894,15021,15022],{"align":1883},"Resampling before splitting train\u002Ftest is a leak just as serious as normalizing wrong, just easier to miss",[1878,15024,15025,15030],{},[1894,15026,15027,15029],{"align":1883},[87,15028,1498],{}," prevents leakage between steps",[1894,15031,15032,15035,15036,15038,15039],{"align":1883},[87,15033,15034],{},"imblearn.Pipeline"," extends the same idea to techniques that also change ",[87,15037,166],{},", not just ",[87,15040,133],{},[79,15042,1939],{"id":1938},[11,15044,15045,15046,15049],{},"I can't download the real fraud dataset here (needs a Kaggle login), so I rebuilt the lecture's most important finding, SMOTE-before-split leakage, on a synthetic dataset I control and can verify: 20 thousand examples, 2% positive class (scikit-learn's ",[87,15047,15048],{},"make_classification",", with a fraction of flipped labels on purpose so it isn't too clean).",[96,15051,15053],{"className":98,"code":15052,"language":100,"meta":57,"style":57},"from sklearn.datasets import make_classification\nX, y = make_classification(n_samples=20000, weights=[0.98, 0.02], flip_y=0.01, random_state=42)\n",[87,15054,15055,15060],{"__ignoreMap":57},[104,15056,15057],{"class":106,"line":107},[104,15058,15059],{},"from sklearn.datasets import make_classification\n",[104,15061,15062],{"class":106,"line":58},[104,15063,15064],{},"X, y = make_classification(n_samples=20000, weights=[0.98, 0.02], flip_y=0.01, random_state=42)\n",[1872,15066,15067,15079],{},[1875,15068,15069],{},[1878,15070,15071,15073,15075,15077],{},[1881,15072,11201],{"align":1883},[1881,15074,14704],{"align":3186},[1881,15076,14707],{"align":3186},[1881,15078,14710],{"align":3186},[1889,15080,15081,15095,15117],{},[1878,15082,15083,15086,15089,15092],{},[1894,15084,15085],{"align":1883},"Logistic regression, no resampling",[1894,15087,15088],{"align":3186},"1.000",[1894,15090,15091],{"align":3186},"0.190",[1894,15093,15094],{"align":3186},"0.319",[1878,15096,15097,15102,15107,15112],{},[1894,15098,15099],{"align":1883},[15,15100,15101],{},"SMOTE before split (wrong)",[1894,15103,15104],{"align":3186},[15,15105,15106],{},"0.801",[1894,15108,15109],{"align":3186},[15,15110,15111],{},"0.791",[1894,15113,15114],{"align":3186},[15,15115,15116],{},"0.796",[1878,15118,15119,15122,15125,15128],{},[1894,15120,15121],{"align":1883},"SMOTE inside the pipeline + cross-validation (right)",[1894,15123,15124],{"align":3186},"0.086",[1894,15126,15127],{"align":3186},"0.731",[1894,15129,15130],{"align":3186},"0.154",[11,15132,470,15133,15135],{},[87,15134,7041],{},", always guessing \"not fraud,\" hits 0.975 accuracy while detecting zero frauds, the same dumb baseline as always, now at a smaller scale.)",[11,15137,15138,15139,15142],{},"The gap between \"wrong\" and \"right\" here is even more dramatic than on the real fraud dataset: an F1 of 0.796 (looks excellent) versus 0.154 (the honest number), just by changing ",[15,15140,15141],{},"when"," SMOTE runs. Same conclusion as the lecture, confirmed on a dataset I built from scratch: the leak from resampling before splitting train and test isn't a theoretical footnote, it inflates the result enough to turn a mediocre model into one that looks production-ready without being one.",[1982,15144,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":15146},[15147,15148,15149,15150,15151,15152,15153,15154],{"id":14519,"depth":58,"text":14520},{"id":14566,"depth":58,"text":14567},{"id":14642,"depth":58,"text":14643},{"id":14685,"depth":58,"text":14686},{"id":14819,"depth":58,"text":14820},{"id":14886,"depth":58,"text":14887},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 12: the professor uses Kaggle's real credit card fraud dataset (492 frauds in nearly 285 thousand transactions) to show why accuracy lies on imbalanced data, and a data leak so subtle the wrong result looks great.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud",{"title":14505,"description":15155},"en\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud",[15161,15162,15163],"imbalanced-data","fraud-detection","data-leakage","2ES8pEqXjfSmaMxYcphSQFy_exC4QiRqsgpwreAF2EY",{"id":15166,"title":15167,"body":15168,"cover":3,"date":9148,"description":16005,"extension":61,"meta":16006,"navigation":63,"order":1719,"path":16007,"playlist":2000,"seo":16008,"status":66,"stem":16009,"tags":16010,"__hash__":16014},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection.md","Feature Selection: Letting the Label Choose, Not Just Variance",{"type":8,"value":15169,"toc":15995},[15170,15177,15181,15187,15223,15231,15237,15241,15256,15267,15280,15284,15553,15598,15628,15632,15635,15642,15645,15649,15696,15703,15710,15717,15721,15728,15763,15774,15845,15847,15886,15888,15898,15928,15990,15993],[11,15171,15172,15173,15176],{},"Lecture 13, and the topic closes a loop opened ",[20,15174,15175],{"href":13373},"back in the PCA post",": reducing dimensionality without using the label can throw away important information. Feature selection fixes that in the most direct way possible, using the label to decide what to drop.",[79,15178,15180],{"id":15179},"the-dataset-digits-pixel-by-pixel","The dataset: digits, pixel by pixel",[11,15182,15183,15184,15186],{},"MNIST again, 60 thousand training images, 784 pixels each. The professor prints a digit as ASCII art (",[87,15185,133],{}," where there's ink, blank where there isn't), a pretty literal reminder that each of these 784 \"variables\" is just one pixel, and most of them (the image borders, for instance) probably never have any ink at all.",[96,15188,15190],{"className":98,"code":15189,"language":100,"meta":57,"style":57},"class Normalizer255(BaseEstimator, TransformerMixin):\n    def fit(self, X, y=None):\n        return self\n    def transform(self, X, y=None):\n        return X \u002F 255.0\n\npipeline = make_pipeline(Normalizer255(), RidgeClassifier())\n",[87,15191,15192,15197,15201,15205,15209,15214,15218],{"__ignoreMap":57},[104,15193,15194],{"class":106,"line":107},[104,15195,15196],{},"class Normalizer255(BaseEstimator, TransformerMixin):\n",[104,15198,15199],{"class":106,"line":58},[104,15200,9458],{},[104,15202,15203],{"class":106,"line":118},[104,15204,1479],{},[104,15206,15207],{"class":106,"line":124},[104,15208,9467],{},[104,15210,15211],{"class":106,"line":308},[104,15212,15213],{},"        return X \u002F 255.0\n",[104,15215,15216],{"class":106,"line":417},[104,15217,300],{"emptyLinePlaceholder":63},[104,15219,15220],{"class":106,"line":422},[104,15221,15222],{},"pipeline = make_pipeline(Normalizer255(), RidgeClassifier())\n",[433,15224,15225],{},[11,15226,15227,15230],{},[15,15228,15229],{},"Output (all 784 pixels):"," 0.8604 test accuracy.",[11,15232,15233,15234,15236],{},"Notice the normalization: instead of ",[87,15235,5828],{}," (which needs to compute the training set's mean and spread), the professor uses a fixed constant, 255, because he already knows ahead of time that a grayscale image pixel ranges from 0 to 255. Domain knowledge replacing a computation that would otherwise be unnecessary.",[79,15238,15240],{"id":15239},"dropping-by-variance-doesnt-always-help","Dropping by variance: doesn't always help",[96,15242,15244],{"className":98,"code":15243,"language":100,"meta":57,"style":57},"from sklearn.feature_selection import VarianceThreshold\nX_transformed = VarianceThreshold(0.1).fit_transform(X_train)\n",[87,15245,15246,15251],{"__ignoreMap":57},[104,15247,15248],{"class":106,"line":107},[104,15249,15250],{},"from sklearn.feature_selection import VarianceThreshold\n",[104,15252,15253],{"class":106,"line":58},[104,15254,15255],{},"X_transformed = VarianceThreshold(0.1).fit_transform(X_train)\n",[433,15257,15258],{},[11,15259,15260,15262,15263,15266],{},[15,15261,1629],{}," from 784 pixels down to 695 (89 dropped, probably the borders that almost never have ink). Accuracy after the cut: 0.8449, ",[15,15264,15265],{},"worse"," than the original 784.",[11,15268,15269,15270,15273,15274,15276,15277,15279],{},"Worth stating that honestly: even a cut that looks obvious (drop a pixel that almost never changes) made the result worse here, instead of simplifying for free. ",[87,15271,15272],{},"VarianceThreshold"," only looks at ",[87,15275,133],{},", never at the label ",[87,15278,166],{},", so it has no way of knowing whether that small bit of variance it dropped was carrying signal relevant to telling the digits apart.",[79,15281,15283],{"id":15282},"the-homemade-ruler-within-class-versus-between-class","The homemade ruler: within-class versus between-class",[11,15285,15286,15287,15552],{},"The professor builds a vectorized Euclidean distance function from scratch (the same idea you can derive from ",[104,15288,15290,15350],{"className":15289},[148],[104,15291,15293],{"className":15292},[152],[154,15294,15295],{"xmlns":156},[158,15296,15297,15347],{},[161,15298,15299,15301,15303,15305,15307,15313,15315,15317,15319,15325,15327,15329,15331,15333,15335,15337,15339,15341],{},[164,15300,10153],{"mathvariant":959},[164,15302,20],{},[168,15304,530],{},[164,15306,184],{},[539,15308,15309,15311],{},[164,15310,10153],{"mathvariant":959},[483,15312,488],{},[168,15314,170],{},[164,15316,10153],{"mathvariant":959},[164,15318,20],{},[539,15320,15321,15323],{},[164,15322,10153],{"mathvariant":959},[483,15324,488],{},[168,15326,530],{},[483,15328,488],{},[164,15330,20],{},[168,15332,175],{},[164,15334,184],{},[168,15336,181],{},[164,15338,10153],{"mathvariant":959},[164,15340,184],{},[539,15342,15343,15345],{},[164,15344,10153],{"mathvariant":959},[483,15346,488],{},[186,15348,15349],{"encoding":188},"\\|a-b\\|^2 = \\|a\\|^2 - 2a\\cdot b + \\|b\\|^2",[104,15351,15353,15374,15421,15471,15492,15511],{"className":15352,"ariaHidden":194},[193],[104,15354,15356,15359,15362,15365,15368,15371],{"className":15355},[198],[104,15357],{"className":15358,"style":558},[202],[104,15360,10153],{"className":15361},[207],[104,15363,20],{"className":15364},[207,208],[104,15366],{"className":15367,"style":235},[213],[104,15369,530],{"className":15370},[239],[104,15372],{"className":15373,"style":235},[213],[104,15375,15377,15380,15383,15412,15415,15418],{"className":15376},[198],[104,15378],{"className":15379,"style":838},[202],[104,15381,184],{"className":15382},[207,208],[104,15384,15386,15389],{"className":15385},[207],[104,15387,10153],{"className":15388},[207],[104,15390,15392],{"className":15391},[697],[104,15393,15395],{"className":15394},[605],[104,15396,15398],{"className":15397},[610],[104,15399,15401],{"className":15400,"style":897},[614],[104,15402,15403,15406],{"style":900},[104,15404],{"className":15405,"style":714},[622],[104,15407,15409],{"className":15408},[627,628,629,630],[104,15410,488],{"className":15411},[207,630],[104,15413],{"className":15414,"style":214},[213],[104,15416,170],{"className":15417},[218],[104,15419],{"className":15420,"style":214},[213],[104,15422,15424,15427,15430,15433,15462,15465,15468],{"className":15423},[198],[104,15425],{"className":15426,"style":838},[202],[104,15428,10153],{"className":15429},[207],[104,15431,20],{"className":15432},[207,208],[104,15434,15436,15439],{"className":15435},[207],[104,15437,10153],{"className":15438},[207],[104,15440,15442],{"className":15441},[697],[104,15443,15445],{"className":15444},[605],[104,15446,15448],{"className":15447},[610],[104,15449,15451],{"className":15450,"style":897},[614],[104,15452,15453,15456],{"style":900},[104,15454],{"className":15455,"style":714},[622],[104,15457,15459],{"className":15458},[627,628,629,630],[104,15460,488],{"className":15461},[207,630],[104,15463],{"className":15464,"style":235},[213],[104,15466,530],{"className":15467},[239],[104,15469],{"className":15470,"style":235},[213],[104,15472,15474,15477,15480,15483,15486,15489],{"className":15473},[198],[104,15475],{"className":15476,"style":2971},[202],[104,15478,488],{"className":15479},[207],[104,15481,20],{"className":15482},[207,208],[104,15484],{"className":15485,"style":235},[213],[104,15487,175],{"className":15488},[239],[104,15490],{"className":15491,"style":235},[213],[104,15493,15495,15499,15502,15505,15508],{"className":15494},[198],[104,15496],{"className":15497,"style":15498},[202],"height:0.7778em;vertical-align:-0.0833em;",[104,15500,184],{"className":15501},[207,208],[104,15503],{"className":15504,"style":235},[213],[104,15506,181],{"className":15507},[239],[104,15509],{"className":15510,"style":235},[213],[104,15512,15514,15517,15520,15523],{"className":15513},[198],[104,15515],{"className":15516,"style":838},[202],[104,15518,10153],{"className":15519},[207],[104,15521,184],{"className":15522},[207,208],[104,15524,15526,15529],{"className":15525},[207],[104,15527,10153],{"className":15528},[207],[104,15530,15532],{"className":15531},[697],[104,15533,15535],{"className":15534},[605],[104,15536,15538],{"className":15537},[610],[104,15539,15541],{"className":15540,"style":897},[614],[104,15542,15543,15546],{"style":900},[104,15544],{"className":15545,"style":714},[622],[104,15547,15549],{"className":15548},[627,628,629,630],[104,15550,488],{"className":15551},[207,630],", avoiding a loop over every pair of points) and uses it to define a quality ruler per variable:",[96,15554,15556],{"className":98,"code":15555,"language":100,"meta":57,"style":57},"def distance_score(X, y):\n    in_dist, out_dist = 0, 0\n    for label in np.unique(y):\n        in_class = X[y == label]\n        out_class = X[y != label]\n        in_dist += pairwise_distances(in_class, in_class).mean()\n        out_dist += pairwise_distances(in_class, out_class).mean()\n    return in_dist \u002F (out_dist + 1e-8)\n",[87,15557,15558,15563,15568,15573,15578,15583,15588,15593],{"__ignoreMap":57},[104,15559,15560],{"class":106,"line":107},[104,15561,15562],{},"def distance_score(X, y):\n",[104,15564,15565],{"class":106,"line":58},[104,15566,15567],{},"    in_dist, out_dist = 0, 0\n",[104,15569,15570],{"class":106,"line":118},[104,15571,15572],{},"    for label in np.unique(y):\n",[104,15574,15575],{"class":106,"line":124},[104,15576,15577],{},"        in_class = X[y == label]\n",[104,15579,15580],{"class":106,"line":308},[104,15581,15582],{},"        out_class = X[y != label]\n",[104,15584,15585],{"class":106,"line":417},[104,15586,15587],{},"        in_dist += pairwise_distances(in_class, in_class).mean()\n",[104,15589,15590],{"class":106,"line":422},[104,15591,15592],{},"        out_dist += pairwise_distances(in_class, out_class).mean()\n",[104,15594,15595],{"class":106,"line":428},[104,15596,15597],{},"    return in_dist \u002F (out_dist + 1e-8)\n",[11,15599,15600,15601,15604,15605,15608,15609,15611,15612,15615,15616,15619,15620,15623,15624,15627],{},"The idea: for every variable, measure the average distance ",[15,15602,15603],{},"within"," each class (do all the \"3\"s look alike on this variable?) and the average distance ",[15,15606,15607],{},"between"," different classes (are a \"3\" and a \"7\" pretty different on this variable?). A good variable has ",[15,15610,7872],{}," within-class distance and ",[15,15613,15614],{},"high"," between-class distance, so the ",[87,15617,15618],{},"within\u002Fbetween"," ratio comes out small. That's essentially the same idea as the ",[15,15621,15622],{},"Fisher discriminant"," Bishop describes (chapter 4.1.4, ",[20,15625,15626],{"href":5113},"which I already cited back in the classification post"," in a different form): separation between classes divided by variation within each class, just applied variable by variable here, instead of along a learned projection direction.",[79,15629,15631],{"id":15630},"the-cost-of-computing-this-over-everything","The cost of computing this over everything",[11,15633,15634],{},"Computing pairwise distance across 60 thousand points is expensive (the cost grows with the square of the point count), so the professor cuts down to a sample of 1200 training examples before ranking:",[433,15636,15637],{},[11,15638,15639,15641],{},[15,15640,1629],{}," ranking the 784 variables on that smaller sample took around 20 seconds. Still much faster than trying this on all 60 thousand points.",[11,15643,15644],{},"A direct reminder that \"how much this costs to compute\" is also part of a technique's design, not just \"what it measures.\"",[79,15646,15648],{"id":15647},"actually-selecting","Actually selecting",[96,15650,15652],{"className":98,"code":15651,"language":100,"meta":57,"style":57},"class UnivariatedRanking(BaseEstimator, TransformerMixin):\n    def __init__(self, n_features):\n        self.n_features = n_features\n    def fit(self, X, y=None):\n        self.scores_ = univariate_ranking(X, y)\n        return self\n    def transform(self, X, y=None):\n        X_sorted = X[:, self.scores_.argsort()]\n        return X_sorted[:, :self.n_features]\n",[87,15653,15654,15659,15664,15669,15673,15678,15682,15686,15691],{"__ignoreMap":57},[104,15655,15656],{"class":106,"line":107},[104,15657,15658],{},"class UnivariatedRanking(BaseEstimator, TransformerMixin):\n",[104,15660,15661],{"class":106,"line":58},[104,15662,15663],{},"    def __init__(self, n_features):\n",[104,15665,15666],{"class":106,"line":118},[104,15667,15668],{},"        self.n_features = n_features\n",[104,15670,15671],{"class":106,"line":124},[104,15672,9458],{},[104,15674,15675],{"class":106,"line":308},[104,15676,15677],{},"        self.scores_ = univariate_ranking(X, y)\n",[104,15679,15680],{"class":106,"line":417},[104,15681,1479],{},[104,15683,15684],{"class":106,"line":422},[104,15685,9467],{},[104,15687,15688],{"class":106,"line":428},[104,15689,15690],{},"        X_sorted = X[:, self.scores_.argsort()]\n",[104,15692,15693],{"class":106,"line":1689},[104,15694,15695],{},"        return X_sorted[:, :self.n_features]\n",[11,15697,15698,15699,15702],{},"With only 1200 training examples (much less than the 60 thousand from before, to fit the time budget), the baseline with all 784 variables already drops quite a bit, to 0.7678, the price of training on less data. Selecting only the ",[15,15700,15701],{},"180 best"," variables (23% of the total):",[433,15704,15705],{},[11,15706,15707,15709],{},[15,15708,1629],{}," 0.7681. Practically tied with using everything, with less than a quarter of the variables.",[11,15711,15712,15713,15716],{},"That's the core difference from PCA: here the label participates in the choice from the start, so the technique doesn't risk throwing away exactly the information that separates the classes, ",[20,15714,15715],{"href":13373},"the same risk that dropped accuracy from 100% to 47% on that artificial dataset in the PCA post",". Supervised variable selection and unsupervised dimensionality reduction solve similar-looking problems, but with very different guarantees.",[79,15718,15720],{"id":15719},"automating-how-many-variables-to-keep","Automating \"how many variables to keep\"",[11,15722,15723,15724,15727],{},"There was still the question of picking 180 somehow, and the professor solves that too: ",[87,15725,15726],{},"HybridRanking"," uses the univariate ranking to order the variables, then tests incrementally (1 variable, 2 variables, 3...) on a separate validation set, keeping whichever size gave the best accuracy:",[96,15729,15731],{"className":98,"code":15730,"language":100,"meta":57,"style":57},"for i in range(1, len(score_idxs)):\n    X_selected = X_tr[:, score_idxs[0:i]]\n    self.estimator.fit(X_selected, y_tr)\n    acc = self.estimator.score(X_val[:, score_idxs[0:i]], y_val)\n    if acc > best_score:\n        best_score, best_set = acc, i\n",[87,15732,15733,15738,15743,15748,15753,15758],{"__ignoreMap":57},[104,15734,15735],{"class":106,"line":107},[104,15736,15737],{},"for i in range(1, len(score_idxs)):\n",[104,15739,15740],{"class":106,"line":58},[104,15741,15742],{},"    X_selected = X_tr[:, score_idxs[0:i]]\n",[104,15744,15745],{"class":106,"line":118},[104,15746,15747],{},"    self.estimator.fit(X_selected, y_tr)\n",[104,15749,15750],{"class":106,"line":124},[104,15751,15752],{},"    acc = self.estimator.score(X_val[:, score_idxs[0:i]], y_val)\n",[104,15754,15755],{"class":106,"line":308},[104,15756,15757],{},"    if acc > best_score:\n",[104,15759,15760],{"class":106,"line":417},[104,15761,15762],{},"        best_score, best_set = acc, i\n",[433,15764,15765],{},[11,15766,15767,15769,15770,15773],{},[15,15768,1629],{}," the algorithm picked ",[15,15771,15772],{},"199"," variables on its own, with accuracy 0.7702551020408164, exactly matching, down to the last decimal, the result of manually choosing the ranking's top 199 variables.",[11,15775,15776,15777,15780,15781,15844],{},"The combination (a cheap ranking first, then testing sizes incrementally) is a classic feature-selection pattern: a cheap filter (",[87,15778,15779],{},"distance_score",", computed once, sorted) shrinks the search space, and a more expensive method (actually training and validating) decides the cutoff, without needing to train a model for every one of the ",[104,15782,15784,15803],{"className":15783},[148],[104,15785,15787],{"className":15786},[152],[154,15788,15789],{"xmlns":156},[158,15790,15791,15800],{},[161,15792,15793],{},[539,15794,15795,15797],{},[483,15796,488],{},[483,15798,15799],{},"784",[186,15801,15802],{"encoding":188},"2^{784}",[104,15804,15806],{"className":15805,"ariaHidden":194},[193],[104,15807,15809,15812],{"className":15808},[198],[104,15810],{"className":15811,"style":897},[202],[104,15813,15815,15818],{"className":15814},[207],[104,15816,488],{"className":15817},[207],[104,15819,15821],{"className":15820},[697],[104,15822,15824],{"className":15823},[605],[104,15825,15827],{"className":15826},[610],[104,15828,15830],{"className":15829,"style":897},[614],[104,15831,15832,15835],{"style":900},[104,15833],{"className":15834,"style":714},[622],[104,15836,15838],{"className":15837},[627,628,629,630],[104,15839,15841],{"className":15840},[207,630],[104,15842,15799],{"className":15843},[207,630]," possible variable combinations.",[79,15846,1870],{"id":1869},[1872,15848,15849,15857],{},[1875,15850,15851],{},[1878,15852,15853,15855],{},[1881,15854,1884],{"align":1883},[1881,15856,1887],{"align":1883},[1889,15858,15859,15867,15878],{},[1878,15860,15861,15864],{},[1894,15862,15863],{"align":1883},"PCA reduces dimensionality by maximizing variance",[1894,15865,15866],{"align":1883},"Feature selection can reduce dimensionality using the label, avoiding the risk of dropping what actually matters for classifying",[1878,15868,15869,15872],{},[1894,15870,15871],{"align":1883},"Dropping an \"obvious\" variable always helps",[1894,15873,15874,15875,15877],{"align":1883},"Not always: ",[87,15876,15272],{}," made the result worse here, because low variance isn't a synonym for useless",[1878,15879,15880,15883],{},[1894,15881,15882],{"align":1883},"Fisher measures separation between classes over variation within them",[1894,15884,15885],{"align":1883},"The same idea gives a simple ruler for ranking individual variables, not just for finding one projection direction",[79,15887,1939],{"id":1938},[11,15889,15890,15891,15893,15894,15897],{},"I reproduced the same ruler (",[87,15892,15779],{},", univariate ranking) on a smaller digits dataset, ",[20,15895,15896],{"href":11413},"the same Digits (1797 images, 8×8 pixels) that already showed up in the K-means post",", fast enough to run on all 64 pixels without needing to shrink the sample.",[96,15899,15901],{"className":98,"code":15900,"language":100,"meta":57,"style":57},"scores = univariate_ranking(X_train, y_train)\nsorted_idxs = np.argsort(scores)\nfor n in [10, 20, 32]:\n    model = RidgeClassifier().fit(X_train[:, sorted_idxs[:n]], y_train)\n    acc = accuracy_score(y_test, model.predict(X_test[:, sorted_idxs[:n]]))\n",[87,15902,15903,15908,15913,15918,15923],{"__ignoreMap":57},[104,15904,15905],{"class":106,"line":107},[104,15906,15907],{},"scores = univariate_ranking(X_train, y_train)\n",[104,15909,15910],{"class":106,"line":58},[104,15911,15912],{},"sorted_idxs = np.argsort(scores)\n",[104,15914,15915],{"class":106,"line":118},[104,15916,15917],{},"for n in [10, 20, 32]:\n",[104,15919,15920],{"class":106,"line":124},[104,15921,15922],{},"    model = RidgeClassifier().fit(X_train[:, sorted_idxs[:n]], y_train)\n",[104,15924,15925],{"class":106,"line":308},[104,15926,15927],{},"    acc = accuracy_score(y_test, model.predict(X_test[:, sorted_idxs[:n]]))\n",[1872,15929,15930,15942],{},[1875,15931,15932],{},[1878,15933,15934,15937,15940],{},[1881,15935,15936],{"align":5561},"Variables used",[1881,15938,15939],{"align":5561},"Fraction of total",[1881,15941,11204],{"align":3186},[1889,15943,15944,15955,15968,15979],{},[1878,15945,15946,15949,15952],{},[1894,15947,15948],{"align":5561},"64 (all)",[1894,15950,15951],{"align":5561},"100%",[1894,15953,15954],{"align":3186},"0.9389",[1878,15956,15957,15960,15963],{},[1894,15958,15959],{"align":5561},"32 (best)",[1894,15961,15962],{"align":5561},"50%",[1894,15964,15965],{"align":3186},[15,15966,15967],{},"0.9278",[1878,15969,15970,15973,15976],{},[1894,15971,15972],{"align":5561},"20 (best)",[1894,15974,15975],{"align":5561},"31%",[1894,15977,15978],{"align":3186},"0.8806",[1878,15980,15981,15984,15987],{},[1894,15982,15983],{"align":5561},"10 (best)",[1894,15985,15986],{"align":5561},"16%",[1894,15988,15989],{"align":3186},"0.7722",[11,15991,15992],{},"With half the variables (32 of 64), accuracy drops by just one percentage point (0.9389 to 0.9278). The curve isn't linear: cutting from 64 to 32 barely hurts, but cutting from 32 to 20 and then to 10 starts hurting fast, a sign that most of the useful signal really is concentrated in a fraction of the variables, exactly the assumption that makes feature selection a technique worth using.",[1982,15994,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":15996},[15997,15998,15999,16000,16001,16002,16003,16004],{"id":15179,"depth":58,"text":15180},{"id":15239,"depth":58,"text":15240},{"id":15282,"depth":58,"text":15283},{"id":15630,"depth":58,"text":15631},{"id":15647,"depth":58,"text":15648},{"id":15719,"depth":58,"text":15720},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 13: the professor builds a ruler from scratch to rank which MNIST pixels actually help tell one digit from another, and shows that dropping 77% of the variables barely costs any accuracy.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection",{"title":15167,"description":16005},"en\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection",[16011,16012,16013],"feature-selection","mnist","fisher","ADirBj0jRBBT8n1zxImiOkzFfyAaTnRAACeYcIG3M6s",{"id":16016,"title":16017,"body":16018,"cover":3,"date":9148,"description":16744,"extension":61,"meta":16745,"navigation":63,"order":1724,"path":16746,"playlist":2000,"seo":16747,"status":66,"stem":16748,"tags":16749,"__hash__":16752},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnlp-intro.md","Text Becomes a Vector: My First Steps in NLP",{"type":8,"value":16019,"toc":16734},[16020,16036,16040,16043,16086,16095,16102,16106,16198,16205,16219,16229,16233,16242,16257,16269,16273,16276,16359,16366,16381,16388,16392,16400,16472,16493,16502,16509,16515,16521,16525,16528,16569,16582,16584,16625,16632,16634,16651,16676,16692,16729,16732],[11,16021,16022,16023,16026,16027,16030,16031,16035],{},"Lecture 14. The last one in the course, and the topic closes the entire playlist with a nice twist: every model I've used so far, ",[20,16024,16025],{"href":1999},"from linear regression back in the first post"," to ",[20,16028,16029],{"href":16007},"feature selection in the previous post",", expects a vector of numbers as input. A pixel is already a number. A wine's chemical measurement is already a number. But text, a movie review written by a real person, isn't a number at all. This lecture is about the first step of any ",[445,16032,16034],{"definition":16033},"Natural Language Processing, the ML field dealing with text: from Google search to your phone's keyboard autocorrect","NLP"," (Natural Language Processing) task: how to turn text into a vector without losing what matters.",[79,16037,16039],{"id":16038},"standardize-and-tokenize-the-bare-minimum","Standardize and tokenize: the bare minimum",[11,16041,16042],{},"Before any counting, the professor cleans the text: everything lowercase, punctuation gone.",[96,16044,16046],{"className":98,"code":16045,"language":100,"meta":57,"style":57},"def standardize(text):\n    text = text.lower()\n    return \"\".join(c for c in text if c not in string.punctuation)\n\ndef tokenize(text):\n    return standardize(text).split()\n\ntokenize(\"I write, erase, rewrite, erase again, and then a poppy blooms!\")\n",[87,16047,16048,16053,16058,16063,16067,16072,16077,16081],{"__ignoreMap":57},[104,16049,16050],{"class":106,"line":107},[104,16051,16052],{},"def standardize(text):\n",[104,16054,16055],{"class":106,"line":58},[104,16056,16057],{},"    text = text.lower()\n",[104,16059,16060],{"class":106,"line":118},[104,16061,16062],{},"    return \"\".join(c for c in text if c not in string.punctuation)\n",[104,16064,16065],{"class":106,"line":124},[104,16066,300],{"emptyLinePlaceholder":63},[104,16068,16069],{"class":106,"line":308},[104,16070,16071],{},"def tokenize(text):\n",[104,16073,16074],{"class":106,"line":417},[104,16075,16076],{},"    return standardize(text).split()\n",[104,16078,16079],{"class":106,"line":422},[104,16080,300],{"emptyLinePlaceholder":63},[104,16082,16083],{"class":106,"line":428},[104,16084,16085],{},"tokenize(\"I write, erase, rewrite, erase again, and then a poppy blooms!\")\n",[433,16087,16088],{},[11,16089,16090,1807,16092,1863],{},[15,16091,1629],{},[87,16093,16094],{},"['i', 'write', 'erase', 'rewrite', 'erase', 'again', 'and', 'then', 'a', 'poppy', 'blooms']",[11,16096,16097,16098,16101],{},"Notice \"Erase\" and \"erase\" are now the same word, and the comma right after the first \"erase\" is gone instead of sticking to it. Without this step, the model would treat \"erase\" and \"erase,\" (comma glued on) as two completely different words, and \"Write\" and \"write\" too, inflating the vocabulary with duplicates that shouldn't exist. It's the same spirit as the normalization I already saw ",[20,16099,16100],{"href":5766},"back in the KNN post",", just rescaling text into a canonical form instead of rescaling numbers.",[79,16103,16105],{"id":16104},"the-homemade-vectorizer-word-becomes-index","The homemade vectorizer: word becomes index",[96,16107,16109],{"className":98,"code":16108,"language":100,"meta":57,"style":57},"class Vectorizer:\n    def standardize(self, text):\n        ...\n    def tokenize(self, text):\n        ...\n    def make_vocabolary(self, dataset):\n        self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n        for text in dataset:\n            text = self.standardize(text)\n            tokens = self.tokenize(text)\n            for token in tokens:\n                if token not in self.vocabulary:\n                    self.vocabulary[token] = len(self.vocabulary)\n\n    def encode(self, text):\n        text = self.standardize(text)\n        tokens = self.tokenize(text)\n        return [self.vocabulary.get(token, 1) for token in tokens]\n",[87,16110,16111,16116,16121,16125,16130,16134,16139,16144,16149,16154,16159,16164,16169,16174,16178,16183,16188,16193],{"__ignoreMap":57},[104,16112,16113],{"class":106,"line":107},[104,16114,16115],{},"class Vectorizer:\n",[104,16117,16118],{"class":106,"line":58},[104,16119,16120],{},"    def standardize(self, text):\n",[104,16122,16123],{"class":106,"line":118},[104,16124,7931],{},[104,16126,16127],{"class":106,"line":124},[104,16128,16129],{},"    def tokenize(self, text):\n",[104,16131,16132],{"class":106,"line":308},[104,16133,7931],{},[104,16135,16136],{"class":106,"line":417},[104,16137,16138],{},"    def make_vocabolary(self, dataset):\n",[104,16140,16141],{"class":106,"line":422},[104,16142,16143],{},"        self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n",[104,16145,16146],{"class":106,"line":428},[104,16147,16148],{},"        for text in dataset:\n",[104,16150,16151],{"class":106,"line":1689},[104,16152,16153],{},"            text = self.standardize(text)\n",[104,16155,16156],{"class":106,"line":1695},[104,16157,16158],{},"            tokens = self.tokenize(text)\n",[104,16160,16161],{"class":106,"line":1701},[104,16162,16163],{},"            for token in tokens:\n",[104,16165,16166],{"class":106,"line":1707},[104,16167,16168],{},"                if token not in self.vocabulary:\n",[104,16170,16171],{"class":106,"line":1713},[104,16172,16173],{},"                    self.vocabulary[token] = len(self.vocabulary)\n",[104,16175,16176],{"class":106,"line":1719},[104,16177,300],{"emptyLinePlaceholder":63},[104,16179,16180],{"class":106,"line":1724},[104,16181,16182],{},"    def encode(self, text):\n",[104,16184,16185],{"class":106,"line":1729},[104,16186,16187],{},"        text = self.standardize(text)\n",[104,16189,16190],{"class":106,"line":1734},[104,16191,16192],{},"        tokens = self.tokenize(text)\n",[104,16194,16195],{"class":106,"line":10027},[104,16196,16197],{},"        return [self.vocabulary.get(token, 1) for token in tokens]\n",[11,16199,16200,16201,16204],{},"The idea is literal: every word seen during training gets an integer, an index in a dictionary. ",[87,16202,16203],{},"\"[UNK]\""," (unknown) is reserved at index 1 from the start, for any new word that shows up later, in a test sentence, that wasn't in the training vocabulary. The professor tests this with a sentence using the word \"still\", absent from the 3-sentence training dataset:",[433,16206,16207],{},[11,16208,16209,16211,16212,16214,16215,16218],{},[15,16210,1629],{}," the word \"still\" becomes index ",[87,16213,485],{},", the same as ",[87,16216,16217],{},"[UNK]",", while the other words in the sentence (already seen during training) become their real indices.",[11,16220,16221,16222,4296,16225,16228],{},"That's the same problem an unseen category solved ",[20,16223,16224],{"href":9908},"back in the Titanic post",[87,16226,16227],{},"OneHotEncoder",": what to do when production data brings something training never saw. There it was an embarkation category, here it's a word.",[79,16230,16232],{"id":16231},"the-dataset-50-thousand-real-movie-reviews","The dataset: 50 thousand real movie reviews",[11,16234,16235,16236,16241],{},"The professor downloads the ",[20,16237,16240],{"href":16238,"rel":16239},"https:\u002F\u002Fai.stanford.edu\u002F~amaas\u002Fdata\u002Fsentiment\u002F",[45],"IMDB Large Movie Review Dataset",", from Stanford, one of the most cited datasets in sentiment analysis: 50 thousand real IMDB reviews, half labeled positive, half negative, already split 25 thousand train \u002F 25 thousand test.",[96,16243,16245],{"className":98,"code":16244,"language":100,"meta":57,"style":57},"train_df = text_dataset_from_directory('aclImdb\u002Ftrain')\ntest_df = text_dataset_from_directory('aclImdb\u002Ftest')\n",[87,16246,16247,16252],{"__ignoreMap":57},[104,16248,16249],{"class":106,"line":107},[104,16250,16251],{},"train_df = text_dataset_from_directory('aclImdb\u002Ftrain')\n",[104,16253,16254],{"class":106,"line":58},[104,16255,16256],{},"test_df = text_dataset_from_directory('aclImdb\u002Ftest')\n",[433,16258,16259],{},[11,16260,16261,16263,16264,1499,16266,1863],{},[15,16262,1629],{}," 25000 training rows, 25000 test rows, columns ",[87,16265,1009],{},[87,16267,16268],{},"label",[79,16270,16272],{"id":16271},"bag-of-words-every-word-is-a-vote-no-order","Bag-of-words: every word is a vote, no order",[11,16274,16275],{},"The first strategy for vectorizing an entire review: a vector the size of the vocabulary, with 1 in every position whose word appears in the review, 0 everywhere else. Completely ignores word order, just marks presence.",[96,16277,16279],{"className":98,"code":16278,"language":100,"meta":57,"style":57},"class UnigramTransformer(BaseEstimator, TransformerMixin):\n    def fit(self, X, y=None):\n        self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n        for text in X:\n            for token in set(self.tokenize(text)):\n                if token not in self.vocabulary and len(self.vocabulary) \u003C self.max_features:\n                    self.vocabulary[token] = len(self.vocabulary)\n        return self\n\n    def transform(self, X, y=None):\n        rows, cols, data = [], [], []\n        for row, text in enumerate(X):\n            for token in set(self.tokenize(text)):\n                rows.append(row)\n                cols.append(self.vocabulary.get(token, 1))\n                data.append(1)\n        return csr_matrix((data, (rows, cols)), shape=(len(X), len(self.vocabulary)))\n",[87,16280,16281,16286,16290,16294,16299,16304,16309,16313,16317,16321,16325,16330,16335,16339,16344,16349,16354],{"__ignoreMap":57},[104,16282,16283],{"class":106,"line":107},[104,16284,16285],{},"class UnigramTransformer(BaseEstimator, TransformerMixin):\n",[104,16287,16288],{"class":106,"line":58},[104,16289,9458],{},[104,16291,16292],{"class":106,"line":118},[104,16293,16143],{},[104,16295,16296],{"class":106,"line":124},[104,16297,16298],{},"        for text in X:\n",[104,16300,16301],{"class":106,"line":308},[104,16302,16303],{},"            for token in set(self.tokenize(text)):\n",[104,16305,16306],{"class":106,"line":417},[104,16307,16308],{},"                if token not in self.vocabulary and len(self.vocabulary) \u003C self.max_features:\n",[104,16310,16311],{"class":106,"line":422},[104,16312,16173],{},[104,16314,16315],{"class":106,"line":428},[104,16316,1479],{},[104,16318,16319],{"class":106,"line":1689},[104,16320,300],{"emptyLinePlaceholder":63},[104,16322,16323],{"class":106,"line":1695},[104,16324,9467],{},[104,16326,16327],{"class":106,"line":1701},[104,16328,16329],{},"        rows, cols, data = [], [], []\n",[104,16331,16332],{"class":106,"line":1707},[104,16333,16334],{},"        for row, text in enumerate(X):\n",[104,16336,16337],{"class":106,"line":1713},[104,16338,16303],{},[104,16340,16341],{"class":106,"line":1719},[104,16342,16343],{},"                rows.append(row)\n",[104,16345,16346],{"class":106,"line":1724},[104,16347,16348],{},"                cols.append(self.vocabulary.get(token, 1))\n",[104,16350,16351],{"class":106,"line":1729},[104,16352,16353],{},"                data.append(1)\n",[104,16355,16356],{"class":106,"line":1734},[104,16357,16358],{},"        return csr_matrix((data, (rows, cols)), shape=(len(X), len(self.vocabulary)))\n",[11,16360,16361,16362,16365],{},"Notice the ",[87,16363,16364],{},"csr_matrix",": with a 10-thousand-word vocabulary and each review only using a few hundred of those words, the matrix is almost entirely zero. Storing a dense vector of 10 thousand positions, almost all zero, for each of the 20 thousand reviews would waste a huge amount of memory. The sparse format only stores the positions with a nonzero value.",[96,16367,16369],{"className":98,"code":16368,"language":100,"meta":57,"style":57},"pipeline = Pipeline([(\"vectorizer\", UnigramTransformer(10000)), (\"classifier\", RandomForestClassifier(random_state=42))])\npipeline.fit(train_texts, train_labels)\n",[87,16370,16371,16376],{"__ignoreMap":57},[104,16372,16373],{"class":106,"line":107},[104,16374,16375],{},"pipeline = Pipeline([(\"vectorizer\", UnigramTransformer(10000)), (\"classifier\", RandomForestClassifier(random_state=42))])\n",[104,16377,16378],{"class":106,"line":58},[104,16379,16380],{},"pipeline.fit(train_texts, train_labels)\n",[433,16382,16383],{},[11,16384,16385,16387],{},[15,16386,1629],{}," 0.832 validation accuracy, just from which words show up, with no order at all.",[79,16389,16391],{"id":16390},"tf-idf-not-every-word-carries-the-same-weight","TF-IDF: not every word carries the same weight",[11,16393,16394,16395,16399],{},"Bag-of-words treats \"the\" (shows up in almost every review) and \"wonderful\" (shows up mostly in the good ones) the same way: both count as 1 if present. But \"the\" carries no information at all about whether a review is good or bad, while \"wonderful\" carries plenty. ",[445,16396,16398],{"definition":16397},"Term Frequency times Inverse Document Frequency: weighs each word by how often it appears in the text, multiplied by how rare it is across the rest of the corpus","TF-IDF"," fixes this with two multiplied quantities:",[96,16401,16403],{"className":98,"code":16402,"language":100,"meta":57,"style":57},"class TfidfTransformer(BaseEstimator, TransformerMixin):\n    def fit(self, X, y=None):\n        doc_freq = {}\n        for text in X:\n            for token in set(self.tokenize(text)):\n                doc_freq[token] = doc_freq.get(token, 0) + 1\n        ...\n        self.idf = {token: math.log(len(X) \u002F freq) for token, freq in doc_freq.items()}\n        return self\n\n    def transform(self, X, y=None):\n        ...\n        for token, count in token_counts.items():\n            tf = count \u002F total_tokens_in_document\n            value = tf * self.idf[token]\n",[87,16404,16405,16410,16414,16419,16423,16427,16432,16436,16441,16445,16449,16453,16457,16462,16467],{"__ignoreMap":57},[104,16406,16407],{"class":106,"line":107},[104,16408,16409],{},"class TfidfTransformer(BaseEstimator, TransformerMixin):\n",[104,16411,16412],{"class":106,"line":58},[104,16413,9458],{},[104,16415,16416],{"class":106,"line":118},[104,16417,16418],{},"        doc_freq = {}\n",[104,16420,16421],{"class":106,"line":124},[104,16422,16298],{},[104,16424,16425],{"class":106,"line":308},[104,16426,16303],{},[104,16428,16429],{"class":106,"line":417},[104,16430,16431],{},"                doc_freq[token] = doc_freq.get(token, 0) + 1\n",[104,16433,16434],{"class":106,"line":422},[104,16435,7931],{},[104,16437,16438],{"class":106,"line":428},[104,16439,16440],{},"        self.idf = {token: math.log(len(X) \u002F freq) for token, freq in doc_freq.items()}\n",[104,16442,16443],{"class":106,"line":1689},[104,16444,1479],{},[104,16446,16447],{"class":106,"line":1695},[104,16448,300],{"emptyLinePlaceholder":63},[104,16450,16451],{"class":106,"line":1701},[104,16452,9467],{},[104,16454,16455],{"class":106,"line":1707},[104,16456,7931],{},[104,16458,16459],{"class":106,"line":1713},[104,16460,16461],{},"        for token, count in token_counts.items():\n",[104,16463,16464],{"class":106,"line":1719},[104,16465,16466],{},"            tf = count \u002F total_tokens_in_document\n",[104,16468,16469],{"class":106,"line":1724},[104,16470,16471],{},"            value = tf * self.idf[token]\n",[11,16473,16474,16477,16478,16481,16482,16485,16486,16489,16490,16492],{},[15,16475,16476],{},"TF"," (term frequency) is simple: how many times the word appears in the review, divided by the review's total word count, the \"local\" weight. ",[15,16479,16480],{},"IDF"," (inverse document frequency) is the \"global\" weight: ",[87,16483,16484],{},"log(total documents \u002F documents containing the word)",". A word appearing in almost every document (like \"the\") has an IDF near zero, barely counting at all. A rare word, present in few documents, has a high IDF. Multiplying the two, a word only gets a high weight if it's frequent in ",[15,16487,16488],{},"that specific"," review ",[15,16491,10059],{}," rare across the rest of the corpus, exactly the kind of word that helps tell one review apart from another.",[96,16494,16496],{"className":98,"code":16495,"language":100,"meta":57,"style":57},"pipeline = Pipeline([(\"vectorizer\", TfidfTransformer(10000)), (\"classifier\", RandomForestClassifier(random_state=42))])\n",[87,16497,16498],{"__ignoreMap":57},[104,16499,16500],{"class":106,"line":107},[104,16501,16495],{},[433,16503,16504],{},[11,16505,16506,16508],{},[15,16507,1629],{}," 0.8316 validation accuracy, essentially tied with bag-of-words (0.832).",[11,16510,16511,16512,16514],{},"Interactive: type a sentence below and watch the TF-IDF weight of each word, computed against a small 16-review example corpus. A common word like \"the\" or \"was\" gets a short bar, a rare, loaded word like \"wonderful\", \"terrible\" or \"boring\" gets a much taller one, and a word that this tiny example corpus never saw shows up marked as ",[87,16513,16217],{},", with no weight, the same problem the homemade vectorizer had above.",[16516,16517],"text-vectorizer-explorer",{"label":16518,"readout-suffix":16519,"unk-label":16520},"Type a sentence (in English, same vocabulary as the example corpus)","token(s) recognized in the example vocabulary","UNK",[79,16522,16524],{"id":16523},"on-the-full-dataset-against-the-official-tfidfvectorizer","On the full dataset, against the official TfidfVectorizer",[11,16526,16527],{},"With both techniques tested on validation, the professor retrains on the entire training set (25 thousand reviews, not just the 20-thousand slice) and measures on the real test set:",[1872,16529,16530,16538],{},[1875,16531,16532],{},[1878,16533,16534,16536],{},[1881,16535,14698],{"align":1883},[1881,16537,11204],{"align":3186},[1889,16539,16540,16548,16558],{},[1878,16541,16542,16545],{},[1894,16543,16544],{"align":1883},"Bag-of-words (unigram)",[1894,16546,16547],{"align":3186},"0.83804",[1878,16549,16550,16553],{},[1894,16551,16552],{"align":1883},"Homemade TF-IDF",[1894,16554,16555],{"align":3186},[15,16556,16557],{},"0.84292",[1878,16559,16560,16566],{},[1894,16561,16562,16563],{"align":1883},"scikit-learn's ",[87,16564,16565],{},"TfidfVectorizer",[1894,16567,16568],{"align":3186},"0.83872",[11,16570,16571,16572,1499,16575,16578,16579,16581],{},"TF-IDF wins by a small margin over bag-of-words, and more importantly: my homemade TF-IDF implementation (0.84292) lands less than half a percentage point from scikit-learn's own official implementation (0.83872), the same \"matches the professional tool, down to the decimal\" test I already saw ",[20,16573,16574],{"href":1925},"back in the normal equation post",[20,16576,16577],{"href":13373},"in the PCA post",". The real ",[87,16580,16565],{}," has extra optimizations and details (L2 normalization of the final vector, for instance), but the core idea, the same TF times IDF math, is identical.",[79,16583,1870],{"id":1869},[1872,16585,16586,16594],{},[1875,16587,16588],{},[1878,16589,16590,16592],{},[1881,16591,1884],{"align":1883},[1881,16593,1887],{"align":1883},[1889,16595,16596,16604,16614],{},[1878,16597,16598,16601],{},[1894,16599,16600],{"align":1883},"Every model expects a vector of numbers",[1894,16602,16603],{"align":1883},"Text has to become a vector too, and how you do it is a design choice, not an automatic detail",[1878,16605,16606,16609],{},[1894,16607,16608],{"align":1883},"An unseen category in production needs a plan",[1894,16610,16611,16613],{"align":1883},[87,16612,16217],{}," solves for words the same problem an unknown category solved for the Titanic",[1878,16615,16616,16619],{},[1894,16617,16618],{"align":1883},"Matching scikit-learn validates a homemade implementation",[1894,16620,16621,16622,16624],{"align":1883},"Hand-built TF-IDF (0.84292) lands close enough to the official ",[87,16623,16565],{}," (0.83872) to confirm the logic is right",[11,16626,16627,16628,16631],{},"And with that, the 14 lectures in this course close out. I started ",[20,16629,16630],{"href":1999},"fitting a line with the normal equation"," and end here, turning a movie review into a vector of weighted words. Along the way there were trees, ensembles, clusters, dimensionality reduction, imbalanced data, with Bishop holding down the theoretical side of almost all of it, except the last three lectures (DBSCAN, semi-supervised learning, and NLP), which came after his 2006 book and still fit right into the same recurring logic: data becomes a vector, a vector becomes a decision. Thanks, professor Boldt.",[79,16633,1939],{"id":1938},[11,16635,16636,16637,16639,16640,16643,16644,1499,16647,16650],{},"To confirm the lecture's finding (bag-of-words, homemade TF-IDF, and the official ",[87,16638,16565],{}," landing on similar results) on a dataset different from IMDB, I used scikit-learn's own ",[87,16641,16642],{},"20newsgroups",": real forum posts, on two clearly distinct topics, ",[87,16645,16646],{},"sci.space",[87,16648,16649],{},"rec.sport.baseball",", 1190 training posts and 791 test posts.",[96,16652,16654],{"className":98,"code":16653,"language":100,"meta":57,"style":57},"from sklearn.datasets import fetch_20newsgroups\ncats = ['sci.space', 'rec.sport.baseball']\ntrain = fetch_20newsgroups(subset='train', categories=cats, remove=('headers','footers','quotes'), random_state=42)\ntest = fetch_20newsgroups(subset='test', categories=cats, remove=('headers','footers','quotes'), random_state=42)\n",[87,16655,16656,16661,16666,16671],{"__ignoreMap":57},[104,16657,16658],{"class":106,"line":107},[104,16659,16660],{},"from sklearn.datasets import fetch_20newsgroups\n",[104,16662,16663],{"class":106,"line":58},[104,16664,16665],{},"cats = ['sci.space', 'rec.sport.baseball']\n",[104,16667,16668],{"class":106,"line":118},[104,16669,16670],{},"train = fetch_20newsgroups(subset='train', categories=cats, remove=('headers','footers','quotes'), random_state=42)\n",[104,16672,16673],{"class":106,"line":124},[104,16674,16675],{},"test = fetch_20newsgroups(subset='test', categories=cats, remove=('headers','footers','quotes'), random_state=42)\n",[11,16677,16678,16679,1499,16682,16685,16686,6470,16688,16691],{},"I reproduced all three approaches (",[87,16680,16681],{},"UnigramTransformer",[87,16683,16684],{},"TfidfTransformer"," exactly as the professor built them, plus the official ",[87,16687,16565],{},[87,16689,16690],{},"RandomForestClassifier(random_state=42)"," on top of a 5000-word vocabulary:",[1872,16693,16694,16702],{},[1875,16695,16696],{},[1878,16697,16698,16700],{},[1881,16699,14698],{"align":1883},[1881,16701,11204],{"align":3186},[1889,16703,16704,16711,16718],{},[1878,16705,16706,16708],{},[1894,16707,16544],{"align":1883},[1894,16709,16710],{"align":3186},"0.8786",[1878,16712,16713,16715],{},[1894,16714,16552],{"align":1883},[1894,16716,16717],{"align":3186},"0.8774",[1878,16719,16720,16724],{},[1894,16721,16562,16722],{"align":1883},[87,16723,16565],{},[1894,16725,16726],{"align":3186},[15,16727,16728],{},"0.8963",[11,16730,16731],{},"Three close numbers again, on a completely different topic and dataset from the IMDB movies, which gives confidence the lecture's pattern wasn't a coincidence of that specific dataset: any reasonable text vectorization (word presence or TF-IDF) already delivers most of the useful signal for separating two clearly distinct classes, here rocket science against baseball bats.",[1982,16733,1984],{},{"title":57,"searchDepth":58,"depth":58,"links":16735},[16736,16737,16738,16739,16740,16741,16742,16743],{"id":16038,"depth":58,"text":16039},{"id":16104,"depth":58,"text":16105},{"id":16231,"depth":58,"text":16232},{"id":16271,"depth":58,"text":16272},{"id":16390,"depth":58,"text":16391},{"id":16523,"depth":58,"text":16524},{"id":1869,"depth":58,"text":1870},{"id":1938,"depth":58,"text":1939},"Lecture 14, the last one in the course: the professor turns a movie review into numbers by hand, and shows a homemade TF-IDF matching scikit-learn's official TfidfVectorizer.",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fnlp-intro",{"title":16017,"description":16744},"en\u002Fplaylists\u002Fpattern-recognition\u002Fnlp-intro",[16750,16751,1009],"nlp","tfidf","RXe164tiQy1aq47XAa3jbNwnSt8Thn5i-WQZyCLxqj8",1787338982721]