[{"data":1,"prerenderedAt":959},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection":3,"post-en-pattern-recognition-feature-selection":4},"\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection",{"id":5,"title":6,"body":7,"cover":943,"date":944,"description":945,"extension":946,"meta":947,"navigation":79,"order":948,"path":949,"playlist":950,"seo":951,"status":952,"stem":953,"tags":954,"__hash__":958},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection.md","Feature Selection: Letting the Label Choose, Not Just Variance",{"type":8,"value":9,"toc":933},"minimark",[10,20,25,33,87,97,104,108,123,135,149,153,468,514,546,550,553,560,563,567,615,622,629,636,640,647,682,693,764,768,816,820,831,861,926,929],[11,12,13,14,19],"p",{},"Lecture 13, and the topic closes a loop opened ",[15,16,18],"a",{"href":17},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpca","back in the PCA post",": reducing dimensionality without using the label can throw away important information. Feature selection fixes that in the most direct way possible, using the label to decide what to drop.",[21,22,24],"h2",{"id":23},"the-dataset-digits-pixel-by-pixel","The dataset: digits, pixel by pixel",[11,26,27,28,32],{},"MNIST again, 60 thousand training images, 784 pixels each. The professor prints a digit as ASCII art (",[29,30,31],"code",{},"X"," where there's ink, blank where there isn't), a pretty literal reminder that each of these 784 \"variables\" is just one pixel, and most of them (the image borders, for instance) probably never have any ink at all.",[34,35,40],"pre",{"className":36,"code":37,"language":38,"meta":39,"style":39},"language-python shiki shiki-themes github-light github-dark","class Normalizer255(BaseEstimator, TransformerMixin):\n    def fit(self, X, y=None):\n        return self\n    def transform(self, X, y=None):\n        return X \u002F 255.0\n\npipeline = make_pipeline(Normalizer255(), RidgeClassifier())\n","python","",[29,41,42,50,56,62,68,74,81],{"__ignoreMap":39},[43,44,47],"span",{"class":45,"line":46},"line",1,[43,48,49],{},"class Normalizer255(BaseEstimator, TransformerMixin):\n",[43,51,53],{"class":45,"line":52},2,[43,54,55],{},"    def fit(self, X, y=None):\n",[43,57,59],{"class":45,"line":58},3,[43,60,61],{},"        return self\n",[43,63,65],{"class":45,"line":64},4,[43,66,67],{},"    def transform(self, X, y=None):\n",[43,69,71],{"class":45,"line":70},5,[43,72,73],{},"        return X \u002F 255.0\n",[43,75,77],{"class":45,"line":76},6,[43,78,80],{"emptyLinePlaceholder":79},true,"\n",[43,82,84],{"class":45,"line":83},7,[43,85,86],{},"pipeline = make_pipeline(Normalizer255(), RidgeClassifier())\n",[88,89,90],"blockquote",{},[11,91,92,96],{},[93,94,95],"strong",{},"Output (all 784 pixels):"," 0.8604 test accuracy.",[11,98,99,100,103],{},"Notice the normalization: instead of ",[29,101,102],{},"StandardScaler"," (which needs to compute the training set's mean and spread), the professor uses a fixed constant, 255, because he already knows ahead of time that a grayscale image pixel ranges from 0 to 255. Domain knowledge replacing a computation that would otherwise be unnecessary.",[21,105,107],{"id":106},"dropping-by-variance-doesnt-always-help","Dropping by variance: doesn't always help",[34,109,111],{"className":36,"code":110,"language":38,"meta":39,"style":39},"from sklearn.feature_selection import VarianceThreshold\nX_transformed = VarianceThreshold(0.1).fit_transform(X_train)\n",[29,112,113,118],{"__ignoreMap":39},[43,114,115],{"class":45,"line":46},[43,116,117],{},"from sklearn.feature_selection import VarianceThreshold\n",[43,119,120],{"class":45,"line":52},[43,121,122],{},"X_transformed = VarianceThreshold(0.1).fit_transform(X_train)\n",[88,124,125],{},[11,126,127,130,131,134],{},[93,128,129],{},"Output:"," from 784 pixels down to 695 (89 dropped, probably the borders that almost never have ink). Accuracy after the cut: 0.8449, ",[93,132,133],{},"worse"," than the original 784.",[11,136,137,138,141,142,144,145,148],{},"Worth stating that honestly: even a cut that looks obvious (drop a pixel that almost never changes) made the result worse here, instead of simplifying for free. ",[29,139,140],{},"VarianceThreshold"," only looks at ",[29,143,31],{},", never at the label ",[29,146,147],{},"y",", so it has no way of knowing whether that small bit of variance it dropped was carrying signal relevant to telling the digits apart.",[21,150,152],{"id":151},"the-homemade-ruler-within-class-versus-between-class","The homemade ruler: within-class versus between-class",[11,154,155,156,467],{},"The professor builds a vectorized Euclidean distance function from scratch (the same idea you can derive from ",[43,157,160,239],{"className":158},[159],"katex",[43,161,164],{"className":162},[163],"katex-mathml",[165,166,168],"math",{"xmlns":167},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[169,170,171,234],"semantics",{},[172,173,174,179,181,185,188,197,200,202,204,210,212,214,216,219,221,224,226,228],"mrow",{},[175,176,178],"mi",{"mathvariant":177},"normal","∥",[175,180,15],{},[182,183,184],"mo",{},"−",[175,186,187],{},"b",[189,190,191,193],"msup",{},[175,192,178],{"mathvariant":177},[194,195,196],"mn",{},"2",[182,198,199],{},"=",[175,201,178],{"mathvariant":177},[175,203,15],{},[189,205,206,208],{},[175,207,178],{"mathvariant":177},[194,209,196],{},[182,211,184],{},[194,213,196],{},[175,215,15],{},[182,217,218],{},"⋅",[175,220,187],{},[182,222,223],{},"+",[175,225,178],{"mathvariant":177},[175,227,187],{},[189,229,230,232],{},[175,231,178],{"mathvariant":177},[194,233,196],{},[235,236,238],"annotation",{"encoding":237},"application\u002Fx-tex","\\|a-b\\|^2 = \\|a\\|^2 - 2a\\cdot b + \\|b\\|^2",[43,240,244,273,335,385,407,426],{"className":241,"ariaHidden":243},[242],"katex-html","true",[43,245,248,253,257,261,266,270],{"className":246},[247],"base",[43,249],{"className":250,"style":252},[251],"strut","height:1em;vertical-align:-0.25em;",[43,254,178],{"className":255},[256],"mord",[43,258,15],{"className":259},[256,260],"mathnormal",[43,262],{"className":263,"style":265},[264],"mspace","margin-right:0.2222em;",[43,267,184],{"className":268},[269],"mbin",[43,271],{"className":272,"style":265},[264],[43,274,276,280,283,324,328,332],{"className":275},[247],[43,277],{"className":278,"style":279},[251],"height:1.0641em;vertical-align:-0.25em;",[43,281,187],{"className":282},[256,260],[43,284,286,289],{"className":285},[256],[43,287,178],{"className":288},[256],[43,290,293],{"className":291},[292],"msupsub",[43,294,297],{"className":295},[296],"vlist-t",[43,298,301],{"className":299},[300],"vlist-r",[43,302,306],{"className":303,"style":305},[304],"vlist","height:0.8141em;",[43,307,309,314],{"style":308},"top:-3.063em;margin-right:0.05em;",[43,310],{"className":311,"style":313},[312],"pstrut","height:2.7em;",[43,315,321],{"className":316},[317,318,319,320],"sizing","reset-size6","size3","mtight",[43,322,196],{"className":323},[256,320],[43,325],{"className":326,"style":327},[264],"margin-right:0.2778em;",[43,329,199],{"className":330},[331],"mrel",[43,333],{"className":334,"style":327},[264],[43,336,338,341,344,347,376,379,382],{"className":337},[247],[43,339],{"className":340,"style":279},[251],[43,342,178],{"className":343},[256],[43,345,15],{"className":346},[256,260],[43,348,350,353],{"className":349},[256],[43,351,178],{"className":352},[256],[43,354,356],{"className":355},[292],[43,357,359],{"className":358},[296],[43,360,362],{"className":361},[300],[43,363,365],{"className":364,"style":305},[304],[43,366,367,370],{"style":308},[43,368],{"className":369,"style":313},[312],[43,371,373],{"className":372},[317,318,319,320],[43,374,196],{"className":375},[256,320],[43,377],{"className":378,"style":265},[264],[43,380,184],{"className":381},[269],[43,383],{"className":384,"style":265},[264],[43,386,388,392,395,398,401,404],{"className":387},[247],[43,389],{"className":390,"style":391},[251],"height:0.6444em;",[43,393,196],{"className":394},[256],[43,396,15],{"className":397},[256,260],[43,399],{"className":400,"style":265},[264],[43,402,218],{"className":403},[269],[43,405],{"className":406,"style":265},[264],[43,408,410,414,417,420,423],{"className":409},[247],[43,411],{"className":412,"style":413},[251],"height:0.7778em;vertical-align:-0.0833em;",[43,415,187],{"className":416},[256,260],[43,418],{"className":419,"style":265},[264],[43,421,223],{"className":422},[269],[43,424],{"className":425,"style":265},[264],[43,427,429,432,435,438],{"className":428},[247],[43,430],{"className":431,"style":279},[251],[43,433,178],{"className":434},[256],[43,436,187],{"className":437},[256,260],[43,439,441,444],{"className":440},[256],[43,442,178],{"className":443},[256],[43,445,447],{"className":446},[292],[43,448,450],{"className":449},[296],[43,451,453],{"className":452},[300],[43,454,456],{"className":455,"style":305},[304],[43,457,458,461],{"style":308},[43,459],{"className":460,"style":313},[312],[43,462,464],{"className":463},[317,318,319,320],[43,465,196],{"className":466},[256,320],", avoiding a loop over every pair of points) and uses it to define a quality ruler per variable:",[34,469,471],{"className":36,"code":470,"language":38,"meta":39,"style":39},"def distance_score(X, y):\n    in_dist, out_dist = 0, 0\n    for label in np.unique(y):\n        in_class = X[y == label]\n        out_class = X[y != label]\n        in_dist += pairwise_distances(in_class, in_class).mean()\n        out_dist += pairwise_distances(in_class, out_class).mean()\n    return in_dist \u002F (out_dist + 1e-8)\n",[29,472,473,478,483,488,493,498,503,508],{"__ignoreMap":39},[43,474,475],{"class":45,"line":46},[43,476,477],{},"def distance_score(X, y):\n",[43,479,480],{"class":45,"line":52},[43,481,482],{},"    in_dist, out_dist = 0, 0\n",[43,484,485],{"class":45,"line":58},[43,486,487],{},"    for label in np.unique(y):\n",[43,489,490],{"class":45,"line":64},[43,491,492],{},"        in_class = X[y == label]\n",[43,494,495],{"class":45,"line":70},[43,496,497],{},"        out_class = X[y != label]\n",[43,499,500],{"class":45,"line":76},[43,501,502],{},"        in_dist += pairwise_distances(in_class, in_class).mean()\n",[43,504,505],{"class":45,"line":83},[43,506,507],{},"        out_dist += pairwise_distances(in_class, out_class).mean()\n",[43,509,511],{"class":45,"line":510},8,[43,512,513],{},"    return in_dist \u002F (out_dist + 1e-8)\n",[11,515,516,517,520,521,524,525,528,529,532,533,536,537,540,541,545],{},"The idea: for every variable, measure the average distance ",[93,518,519],{},"within"," each class (do all the \"3\"s look alike on this variable?) and the average distance ",[93,522,523],{},"between"," different classes (are a \"3\" and a \"7\" pretty different on this variable?). A good variable has ",[93,526,527],{},"low"," within-class distance and ",[93,530,531],{},"high"," between-class distance, so the ",[29,534,535],{},"within\u002Fbetween"," ratio comes out small. That's essentially the same idea as the ",[93,538,539],{},"Fisher discriminant"," Bishop describes (chapter 4.1.4, ",[15,542,544],{"href":543},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fclassification-threshold","which I already cited back in the classification post"," in a different form): separation between classes divided by variation within each class, just applied variable by variable here, instead of along a learned projection direction.",[21,547,549],{"id":548},"the-cost-of-computing-this-over-everything","The cost of computing this over everything",[11,551,552],{},"Computing pairwise distance across 60 thousand points is expensive (the cost grows with the square of the point count), so the professor cuts down to a sample of 1200 training examples before ranking:",[88,554,555],{},[11,556,557,559],{},[93,558,129],{}," ranking the 784 variables on that smaller sample took around 20 seconds. Still much faster than trying this on all 60 thousand points.",[11,561,562],{},"A direct reminder that \"how much this costs to compute\" is also part of a technique's design, not just \"what it measures.\"",[21,564,566],{"id":565},"actually-selecting","Actually selecting",[34,568,570],{"className":36,"code":569,"language":38,"meta":39,"style":39},"class UnivariatedRanking(BaseEstimator, TransformerMixin):\n    def __init__(self, n_features):\n        self.n_features = n_features\n    def fit(self, X, y=None):\n        self.scores_ = univariate_ranking(X, y)\n        return self\n    def transform(self, X, y=None):\n        X_sorted = X[:, self.scores_.argsort()]\n        return X_sorted[:, :self.n_features]\n",[29,571,572,577,582,587,591,596,600,604,609],{"__ignoreMap":39},[43,573,574],{"class":45,"line":46},[43,575,576],{},"class UnivariatedRanking(BaseEstimator, TransformerMixin):\n",[43,578,579],{"class":45,"line":52},[43,580,581],{},"    def __init__(self, n_features):\n",[43,583,584],{"class":45,"line":58},[43,585,586],{},"        self.n_features = n_features\n",[43,588,589],{"class":45,"line":64},[43,590,55],{},[43,592,593],{"class":45,"line":70},[43,594,595],{},"        self.scores_ = univariate_ranking(X, y)\n",[43,597,598],{"class":45,"line":76},[43,599,61],{},[43,601,602],{"class":45,"line":83},[43,603,67],{},[43,605,606],{"class":45,"line":510},[43,607,608],{},"        X_sorted = X[:, self.scores_.argsort()]\n",[43,610,612],{"class":45,"line":611},9,[43,613,614],{},"        return X_sorted[:, :self.n_features]\n",[11,616,617,618,621],{},"With only 1200 training examples (much less than the 60 thousand from before, to fit the time budget), the baseline with all 784 variables already drops quite a bit, to 0.7678, the price of training on less data. Selecting only the ",[93,619,620],{},"180 best"," variables (23% of the total):",[88,623,624],{},[11,625,626,628],{},[93,627,129],{}," 0.7681. Practically tied with using everything, with less than a quarter of the variables.",[11,630,631,632,635],{},"That's the core difference from PCA: here the label participates in the choice from the start, so the technique doesn't risk throwing away exactly the information that separates the classes, ",[15,633,634],{"href":17},"the same risk that dropped accuracy from 100% to 47% on that artificial dataset in the PCA post",". Supervised variable selection and unsupervised dimensionality reduction solve similar-looking problems, but with very different guarantees.",[21,637,639],{"id":638},"automating-how-many-variables-to-keep","Automating \"how many variables to keep\"",[11,641,642,643,646],{},"There was still the question of picking 180 somehow, and the professor solves that too: ",[29,644,645],{},"HybridRanking"," uses the univariate ranking to order the variables, then tests incrementally (1 variable, 2 variables, 3...) on a separate validation set, keeping whichever size gave the best accuracy:",[34,648,650],{"className":36,"code":649,"language":38,"meta":39,"style":39},"for i in range(1, len(score_idxs)):\n    X_selected = X_tr[:, score_idxs[0:i]]\n    self.estimator.fit(X_selected, y_tr)\n    acc = self.estimator.score(X_val[:, score_idxs[0:i]], y_val)\n    if acc > best_score:\n        best_score, best_set = acc, i\n",[29,651,652,657,662,667,672,677],{"__ignoreMap":39},[43,653,654],{"class":45,"line":46},[43,655,656],{},"for i in range(1, len(score_idxs)):\n",[43,658,659],{"class":45,"line":52},[43,660,661],{},"    X_selected = X_tr[:, score_idxs[0:i]]\n",[43,663,664],{"class":45,"line":58},[43,665,666],{},"    self.estimator.fit(X_selected, y_tr)\n",[43,668,669],{"class":45,"line":64},[43,670,671],{},"    acc = self.estimator.score(X_val[:, score_idxs[0:i]], y_val)\n",[43,673,674],{"class":45,"line":70},[43,675,676],{},"    if acc > best_score:\n",[43,678,679],{"class":45,"line":76},[43,680,681],{},"        best_score, best_set = acc, i\n",[88,683,684],{},[11,685,686,688,689,692],{},[93,687,129],{}," the algorithm picked ",[93,690,691],{},"199"," variables on its own, with accuracy 0.7702551020408164, exactly matching, down to the last decimal, the result of manually choosing the ranking's top 199 variables.",[11,694,695,696,699,700,763],{},"The combination (a cheap ranking first, then testing sizes incrementally) is a classic feature-selection pattern: a cheap filter (",[29,697,698],{},"distance_score",", computed once, sorted) shrinks the search space, and a more expensive method (actually training and validating) decides the cutoff, without needing to train a model for every one of the ",[43,701,703,722],{"className":702},[159],[43,704,706],{"className":705},[163],[165,707,708],{"xmlns":167},[169,709,710,719],{},[172,711,712],{},[189,713,714,716],{},[194,715,196],{},[194,717,718],{},"784",[235,720,721],{"encoding":237},"2^{784}",[43,723,725],{"className":724,"ariaHidden":243},[242],[43,726,728,731],{"className":727},[247],[43,729],{"className":730,"style":305},[251],[43,732,734,737],{"className":733},[256],[43,735,196],{"className":736},[256],[43,738,740],{"className":739},[292],[43,741,743],{"className":742},[296],[43,744,746],{"className":745},[300],[43,747,749],{"className":748,"style":305},[304],[43,750,751,754],{"style":308},[43,752],{"className":753,"style":313},[312],[43,755,757],{"className":756},[317,318,319,320],[43,758,760],{"className":759},[256,320],[43,761,718],{"className":762},[256,320]," possible variable combinations.",[21,765,767],{"id":766},"wrapping-up","Wrapping up",[769,770,771,785],"table",{},[772,773,774],"thead",{},[775,776,777,782],"tr",{},[778,779,781],"th",{"align":780},"left","What I already knew",[778,783,784],{"align":780},"What this lecture settled",[786,787,788,797,808],"tbody",{},[775,789,790,794],{},[791,792,793],"td",{"align":780},"PCA reduces dimensionality by maximizing variance",[791,795,796],{"align":780},"Feature selection can reduce dimensionality using the label, avoiding the risk of dropping what actually matters for classifying",[775,798,799,802],{},[791,800,801],{"align":780},"Dropping an \"obvious\" variable always helps",[791,803,804,805,807],{"align":780},"Not always: ",[29,806,140],{}," made the result worse here, because low variance isn't a synonym for useless",[775,809,810,813],{},[791,811,812],{"align":780},"Fisher measures separation between classes over variation within them",[791,814,815],{"align":780},"The same idea gives a simple ruler for ranking individual variables, not just for finding one projection direction",[21,817,819],{"id":818},"practical-application","Practical application",[11,821,822,823,825,826,830],{},"I reproduced the same ruler (",[29,824,698],{},", univariate ranking) on a smaller digits dataset, ",[15,827,829],{"href":828},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fkmeans","the same Digits (1797 images, 8×8 pixels) that already showed up in the K-means post",", fast enough to run on all 64 pixels without needing to shrink the sample.",[34,832,834],{"className":36,"code":833,"language":38,"meta":39,"style":39},"scores = univariate_ranking(X_train, y_train)\nsorted_idxs = np.argsort(scores)\nfor n in [10, 20, 32]:\n    model = RidgeClassifier().fit(X_train[:, sorted_idxs[:n]], y_train)\n    acc = accuracy_score(y_test, model.predict(X_test[:, sorted_idxs[:n]]))\n",[29,835,836,841,846,851,856],{"__ignoreMap":39},[43,837,838],{"class":45,"line":46},[43,839,840],{},"scores = univariate_ranking(X_train, y_train)\n",[43,842,843],{"class":45,"line":52},[43,844,845],{},"sorted_idxs = np.argsort(scores)\n",[43,847,848],{"class":45,"line":58},[43,849,850],{},"for n in [10, 20, 32]:\n",[43,852,853],{"class":45,"line":64},[43,854,855],{},"    model = RidgeClassifier().fit(X_train[:, sorted_idxs[:n]], y_train)\n",[43,857,858],{"class":45,"line":70},[43,859,860],{},"    acc = accuracy_score(y_test, model.predict(X_test[:, sorted_idxs[:n]]))\n",[769,862,863,878],{},[772,864,865],{},[775,866,867,871,874],{},[778,868,870],{"align":869},"center","Variables used",[778,872,873],{"align":869},"Fraction of total",[778,875,877],{"align":876},"right","Test accuracy",[786,879,880,891,904,915],{},[775,881,882,885,888],{},[791,883,884],{"align":869},"64 (all)",[791,886,887],{"align":869},"100%",[791,889,890],{"align":876},"0.9389",[775,892,893,896,899],{},[791,894,895],{"align":869},"32 (best)",[791,897,898],{"align":869},"50%",[791,900,901],{"align":876},[93,902,903],{},"0.9278",[775,905,906,909,912],{},[791,907,908],{"align":869},"20 (best)",[791,910,911],{"align":869},"31%",[791,913,914],{"align":876},"0.8806",[775,916,917,920,923],{},[791,918,919],{"align":869},"10 (best)",[791,921,922],{"align":869},"16%",[791,924,925],{"align":876},"0.7722",[11,927,928],{},"With half the variables (32 of 64), accuracy drops by just one percentage point (0.9389 to 0.9278). The curve isn't linear: cutting from 64 to 32 barely hurts, but cutting from 32 to 20 and then to 10 starts hurting fast, a sign that most of the useful signal really is concentrated in a fraction of the variables, exactly the assumption that makes feature selection a technique worth using.",[930,931,932],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":39,"searchDepth":52,"depth":52,"links":934},[935,936,937,938,939,940,941,942],{"id":23,"depth":52,"text":24},{"id":106,"depth":52,"text":107},{"id":151,"depth":52,"text":152},{"id":548,"depth":52,"text":549},{"id":565,"depth":52,"text":566},{"id":638,"depth":52,"text":639},{"id":766,"depth":52,"text":767},{"id":818,"depth":52,"text":819},null,"2026-08-20","Lecture 13: the professor builds a ruler from scratch to rank which MNIST pixels actually help tell one digit from another, and shows that dropping 77% of the variables barely costs any accuracy.","md",{},14,"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection","pattern-recognition",{"title":6,"description":945},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Ffeature-selection",[955,956,957],"feature-selection","mnist","fisher","ADirBj0jRBBT8n1zxImiOkzFfyAaTnRAACeYcIG3M6s",1787338984538]