[{"data":1,"prerenderedAt":1455},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees":3,"post-en-pattern-recognition-decision-trees":4},"\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees",{"id":5,"title":6,"body":7,"cover":1440,"date":1441,"description":1442,"extension":1443,"meta":1444,"navigation":1445,"order":226,"path":1446,"playlist":1447,"seo":1448,"status":1449,"stem":1450,"tags":1451,"__hash__":1454},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees.md","Decision Trees: Yes-or-No Questions Until Only One Answer Is Left",{"type":8,"value":9,"toc":1428},"minimark",[10,23,28,81,104,125,138,171,181,184,188,195,260,267,274,278,281,321,589,697,718,950,953,983,990,1035,1049,1053,1056,1067,1070,1074,1081,1124,1139,1149,1160,1164,1180,1230,1241,1245,1248,1264,1274,1278,1290,1296,1300,1345,1349,1355,1380,1421,1424],[11,12,13,14,18,19,22],"p",{},"Lecture 5, split between categorical attributes (",[15,16,17],"code",{},"aula05",") and continuous ones (",[15,20,21],{},"aula05b","). Bishop doesn't dedicate much space to decision trees (a short paragraph back in chapter 14), so this lecture leans much more on what the professor showed, with Gini's math as the thread tying it together.",[24,25,27],"h2",{"id":26},"the-dataset-car-evaluation-and-a-baseline-that-actually-matters","The dataset: car evaluation, and a baseline that actually matters",[11,29,30,32,33,37,38,41,42,41,45,41,48,41,51,41,54,57,58,41,61,41,64,67,68,41,71,41,74,41,77,80],{},[15,31,17],{}," uses ",[34,35,36],"strong",{},"Car Evaluation",", a classic UCI dataset: 1728 cars, 6 attributes, all categorical (",[15,39,40],{},"buying",", ",[15,43,44],{},"maint",[15,46,47],{},"doors",[15,49,50],{},"persons",[15,52,53],{},"lug_boot",[15,55,56],{},"safety",", each with about 3-4 possible values like ",[15,59,60],{},"\"low\"",[15,62,63],{},"\"med\"",[15,65,66],{},"\"high\"","), and one class (",[15,69,70],{},"unacc",[15,72,73],{},"acc",[15,75,76],{},"good",[15,78,79],{},"vgood",").",[82,83,88],"pre",{"className":84,"code":85,"language":86,"meta":87,"style":87},"language-python shiki shiki-themes github-light github-dark","for label in set(y):\n    print(f\"{label}:\\t{100*sum(y==label)\u002Flen(y):.4}%\")\n","python","",[15,89,90,98],{"__ignoreMap":87},[91,92,95],"span",{"class":93,"line":94},"line",1,[91,96,97],{},"for label in set(y):\n",[91,99,101],{"class":93,"line":100},2,[91,102,103],{},"    print(f\"{label}:\\t{100*sum(y==label)\u002Flen(y):.4}%\")\n",[105,106,107],"blockquote",{},[11,108,109,112,113,115,116,118,119,121,122,124],{},[34,110,111],{},"Output:"," ",[15,114,70],{}," 70.02%, ",[15,117,73],{}," 22.22%, ",[15,120,76],{}," 3.99%, ",[15,123,79],{}," 3.76%.",[11,126,127,128,131,132,137],{},"Pretty imbalanced. Before any tree, the professor defines the laziest possible model, ",[34,129,130],{},"ZeroR"," (always guesses the most common class, the same spirit as the \"dumb model\" ",[133,134,136],"a",{"href":135},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Flinear-regression-estimator","I already saw back in this playlist's first post",", just for classification):",[82,139,141],{"className":84,"code":140,"language":86,"meta":87,"style":87},"class ZeroR(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        self.answer = most_common(y)\n    def predict(self, X):\n        return [self.answer]*X.shape[0]\n",[15,142,143,148,153,159,165],{"__ignoreMap":87},[91,144,145],{"class":93,"line":94},[91,146,147],{},"class ZeroR(BaseEstimator, ClassifierMixin):\n",[91,149,150],{"class":93,"line":100},[91,151,152],{},"    def fit(self, X, y):\n",[91,154,156],{"class":93,"line":155},3,[91,157,158],{},"        self.answer = most_common(y)\n",[91,160,162],{"class":93,"line":161},4,[91,163,164],{},"    def predict(self, X):\n",[91,166,168],{"class":93,"line":167},5,[91,169,170],{},"        return [self.answer]*X.shape[0]\n",[105,172,173],{},[11,174,175,177,178,180],{},[34,176,111],{}," 70.02% accuracy. Just always guessing ",[15,179,70],{},".",[11,182,183],{},"Keep that number in mind. Any model I train from here on is only interesting if it beats 70%, otherwise it didn't learn anything that \"always guess the majority class\" wasn't already giving away for free.",[24,185,187],{"id":186},"a-random-tree-already-beats-the-baseline","A random tree already beats the baseline",[11,189,190,191,194],{},"Interesting intermediate step: a \"tree\" that picks the splitting variable and value ",[34,192,193],{},"completely at random",", and still recurses until every leaf is pure:",[82,196,198],{"className":84,"code":197,"language":86,"meta":87,"style":87},"class DecisionTree(BaseEstimator, ClassifierMixin):\n    def fit(self, X, y):\n        self.feature = np.random.randint(X.shape[1])\n        self.value = np.random.choice(list(set(X[:, self.feature])))\n        equals = X[:, self.feature] == self.value\n        if sum(equals) > 0 and sum(~equals) > 0:\n            self.equals_tree = DecisionTree().fit(X[equals], y[equals])\n            self.not_equals_tree = DecisionTree().fit(X[~equals], y[~equals])\n        else:\n            self.answer = most_common(y)\n        return self\n",[15,199,200,205,209,214,219,224,230,236,242,248,254],{"__ignoreMap":87},[91,201,202],{"class":93,"line":94},[91,203,204],{},"class DecisionTree(BaseEstimator, ClassifierMixin):\n",[91,206,207],{"class":93,"line":100},[91,208,152],{},[91,210,211],{"class":93,"line":155},[91,212,213],{},"        self.feature = np.random.randint(X.shape[1])\n",[91,215,216],{"class":93,"line":161},[91,217,218],{},"        self.value = np.random.choice(list(set(X[:, self.feature])))\n",[91,220,221],{"class":93,"line":167},[91,222,223],{},"        equals = X[:, self.feature] == self.value\n",[91,225,227],{"class":93,"line":226},6,[91,228,229],{},"        if sum(equals) > 0 and sum(~equals) > 0:\n",[91,231,233],{"class":93,"line":232},7,[91,234,235],{},"            self.equals_tree = DecisionTree().fit(X[equals], y[equals])\n",[91,237,239],{"class":93,"line":238},8,[91,240,241],{},"            self.not_equals_tree = DecisionTree().fit(X[~equals], y[~equals])\n",[91,243,245],{"class":93,"line":244},9,[91,246,247],{},"        else:\n",[91,249,251],{"class":93,"line":250},10,[91,252,253],{},"            self.answer = most_common(y)\n",[91,255,257],{"class":93,"line":256},11,[91,258,259],{},"        return self\n",[105,261,262],{},[11,263,264,266],{},[34,265,111],{}," 75.6% (on the same data it trained on).",[11,268,269,270,273],{},"Already beats ZeroR, even picking the question at random. That tracks: every split, even a random one, separates the data into two smaller groups, and smaller almost always means \"a little less mixed\" than the original group. Repeat that recursively until each leaf holds a single class, and the tree ends up memorizing training, not because the chosen question was good, but because it never stops asking until there's no doubt left at all. That's already a hint of what's coming: a tree with no brake ",[34,271,272],{},"always"," manages to memorize training, useful or not.",[24,275,277],{"id":276},"gini-impurity-the-ruler-that-decides-which-question-to-ask","Gini impurity: the ruler that decides which question to ask",[11,279,280],{},"To pick the right question (instead of a random one), you need a way to measure \"how mixed\" the classes are within a group:",[82,282,284],{"className":84,"code":283,"language":86,"meta":87,"style":87},"def gini(y):\n    labels = list(set(y))\n    x = 0\n    for label in labels:\n        label_prob = np.mean(y==label)\n        x += label_prob**2\n    return 1-x\n",[15,285,286,291,296,301,306,311,316],{"__ignoreMap":87},[91,287,288],{"class":93,"line":94},[91,289,290],{},"def gini(y):\n",[91,292,293],{"class":93,"line":100},[91,294,295],{},"    labels = list(set(y))\n",[91,297,298],{"class":93,"line":155},[91,299,300],{},"    x = 0\n",[91,302,303],{"class":93,"line":161},[91,304,305],{},"    for label in labels:\n",[91,307,308],{"class":93,"line":167},[91,309,310],{},"        label_prob = np.mean(y==label)\n",[91,312,313],{"class":93,"line":226},[91,314,315],{},"        x += label_prob**2\n",[91,317,318],{"class":93,"line":232},[91,319,320],{},"    return 1-x\n",[11,322,323],{},[91,324,327,391],{"className":325},[326],"katex",[91,328,331],{"className":329},[330],"katex-mathml",[332,333,335],"math",{"xmlns":334},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[336,337,338,386],"semantics",{},[339,340,341,345,350,354,357,360,364,367,376],"mrow",{},[342,343,344],"mtext",{},"Gini",[346,347,349],"mo",{"stretchy":348},"false","(",[351,352,353],"mi",{},"y",[346,355,356],{"stretchy":348},")",[346,358,359],{},"=",[361,362,363],"mn",{},"1",[346,365,366],{},"−",[368,369,370,373],"msub",{},[346,371,372],{},"∑",[351,374,375],{},"k",[377,378,379,381,383],"msubsup",{},[351,380,11],{},[351,382,375],{},[361,384,385],{},"2",[387,388,390],"annotation",{"encoding":389},"application\u002Fx-tex","\\text{Gini}(y) = 1 - \\sum_{k} p_k^2",[91,392,396,438,459],{"className":393,"ariaHidden":395},[394],"katex-html","true",[91,397,400,405,413,417,422,426,431,435],{"className":398},[399],"base",[91,401],{"className":402,"style":404},[403],"strut","height:1em;vertical-align:-0.25em;",[91,406,410],{"className":407},[408,409],"mord","text",[91,411,344],{"className":412},[408],[91,414,349],{"className":415},[416],"mopen",[91,418,353],{"className":419,"style":421},[408,420],"mathnormal","margin-right:0.0359em;",[91,423,356],{"className":424},[425],"mclose",[91,427],{"className":428,"style":430},[429],"mspace","margin-right:0.2778em;",[91,432,359],{"className":433},[434],"mrel",[91,436],{"className":437,"style":430},[429],[91,439,441,445,448,452,456],{"className":440},[399],[91,442],{"className":443,"style":444},[403],"height:0.7278em;vertical-align:-0.0833em;",[91,446,363],{"className":447},[408],[91,449],{"className":450,"style":451},[429],"margin-right:0.2222em;",[91,453,366],{"className":454},[455],"mbin",[91,457],{"className":458,"style":451},[429],[91,460,462,466,530,534],{"className":461},[399],[91,463],{"className":464,"style":465},[403],"height:1.1138em;vertical-align:-0.2997em;",[91,467,470,476],{"className":468},[469],"mop",[91,471,372],{"className":472,"style":475},[469,473,474],"op-symbol","small-op","position:relative;top:0em;",[91,477,480],{"className":478},[479],"msupsub",[91,481,485,521],{"className":482},[483,484],"vlist-t","vlist-t2",[91,486,489,516],{"className":487},[488],"vlist-r",[91,490,494],{"className":491,"style":493},[492],"vlist","height:0.1864em;",[91,495,497,502],{"style":496},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[91,498],{"className":499,"style":501},[500],"pstrut","height:2.7em;",[91,503,509],{"className":504},[505,506,507,508],"sizing","reset-size6","size3","mtight",[91,510,512],{"className":511},[408,508],[91,513,375],{"className":514,"style":515},[408,420,508],"margin-right:0.0315em;",[91,517,520],{"className":518},[519],"vlist-s","​",[91,522,524],{"className":523},[488],[91,525,528],{"className":526,"style":527},[492],"height:0.2997em;",[91,529],{},[91,531],{"className":532,"style":533},[429],"margin-right:0.1667em;",[91,535,537,540],{"className":536},[408],[91,538,11],{"className":539},[408,420],[91,541,543],{"className":542},[479],[91,544,546,580],{"className":545},[483,484],[91,547,549,577],{"className":548},[488],[91,550,553,565],{"className":551,"style":552},[492],"height:0.8141em;",[91,554,556,559],{"style":555},"top:-2.4169em;margin-left:0em;margin-right:0.05em;",[91,557],{"className":558,"style":501},[500],[91,560,562],{"className":561},[505,506,507,508],[91,563,375],{"className":564,"style":515},[408,420,508],[91,566,568,571],{"style":567},"top:-3.063em;margin-right:0.05em;",[91,569],{"className":570,"style":501},[500],[91,572,574],{"className":573},[505,506,507,508],[91,575,385],{"className":576},[408,508],[91,578,520],{"className":579},[519],[91,581,583],{"className":582},[488],[91,584,587],{"className":585,"style":586},[492],"height:0.2831em;",[91,588],{},[11,590,591,592,666,667,696],{},"where ",[91,593,595,613],{"className":594},[326],[91,596,598],{"className":597},[330],[332,599,600],{"xmlns":334},[336,601,602,610],{},[339,603,604],{},[368,605,606,608],{},[351,607,11],{},[351,609,375],{},[387,611,612],{"encoding":389},"p_k",[91,614,616],{"className":615,"ariaHidden":395},[394],[91,617,619,623],{"className":618},[399],[91,620],{"className":621,"style":622},[403],"height:0.625em;vertical-align:-0.1944em;",[91,624,626,629],{"className":625},[408],[91,627,11],{"className":628},[408,420],[91,630,632],{"className":631},[479],[91,633,635,657],{"className":634},[483,484],[91,636,638,654],{"className":637},[488],[91,639,642],{"className":640,"style":641},[492],"height:0.3361em;",[91,643,645,648],{"style":644},"top:-2.55em;margin-left:0em;margin-right:0.05em;",[91,646],{"className":647,"style":501},[500],[91,649,651],{"className":650},[505,506,507,508],[91,652,375],{"className":653,"style":515},[408,420,508],[91,655,520],{"className":656},[519],[91,658,660],{"className":659},[488],[91,661,664],{"className":662,"style":663},[492],"height:0.15em;",[91,665],{}," is the fraction of examples belonging to class ",[91,668,670,683],{"className":669},[326],[91,671,673],{"className":672},[330],[332,674,675],{"xmlns":334},[336,676,677,681],{},[339,678,679],{},[351,680,375],{},[387,682,375],{"encoding":389},[91,684,686],{"className":685,"ariaHidden":395},[394],[91,687,689,693],{"className":688},[399],[91,690],{"className":691,"style":692},[403],"height:0.6944em;",[91,694,375],{"className":695,"style":515},[408,420]," within the group. Two extreme cases confirm the intuition:",[105,698,699],{},[11,700,701,112,703,706,707,710,711,713,714,717],{},[34,702,111],{},[15,704,705],{},"gini"," of a group where everyone's the same class: ",[34,708,709],{},"0.0",". ",[15,712,705],{}," of a group with 100 different classes, one each: ",[34,715,716],{},"0.99",", close to the theoretical maximum.",[11,719,720,721,724,725,945,946,949],{},"Zero is total purity (no doubt left about the class). The more split between classes, the higher it climbs. Bishop calls this the ",[34,722,723],{},"Gini index"," (he writes it as ",[91,726,728,775],{"className":727},[326],[91,729,731],{"className":730},[330],[332,732,733],{"xmlns":334},[336,734,735,772],{},[339,736,737,743,754,756,758,760,770],{},[368,738,739,741],{},[346,740,372],{},[351,742,375],{},[368,744,745,747],{},[351,746,11],{},[339,748,749,752],{},[351,750,751],{},"τ",[351,753,375],{},[346,755,349],{"stretchy":348},[361,757,363],{},[346,759,366],{},[368,761,762,764],{},[351,763,11],{},[339,765,766,768],{},[351,767,751],{},[351,769,375],{},[346,771,356],{"stretchy":348},[387,773,774],{"encoding":389},"\\sum_k p_{\\tau k}(1-p_{\\tau k})",[91,776,778,890],{"className":777,"ariaHidden":395},[394],[91,779,781,785,825,828,875,878,881,884,887],{"className":780},[399],[91,782],{"className":783,"style":784},[403],"height:1.0497em;vertical-align:-0.2997em;",[91,786,788,791],{"className":787},[469],[91,789,372],{"className":790,"style":475},[469,473,474],[91,792,794],{"className":793},[479],[91,795,797,817],{"className":796},[483,484],[91,798,800,814],{"className":799},[488],[91,801,803],{"className":802,"style":493},[492],[91,804,805,808],{"style":496},[91,806],{"className":807,"style":501},[500],[91,809,811],{"className":810},[505,506,507,508],[91,812,375],{"className":813,"style":515},[408,420,508],[91,815,520],{"className":816},[519],[91,818,820],{"className":819},[488],[91,821,823],{"className":822,"style":527},[492],[91,824],{},[91,826],{"className":827,"style":533},[429],[91,829,831,834],{"className":830},[408],[91,832,11],{"className":833},[408,420],[91,835,837],{"className":836},[479],[91,838,840,867],{"className":839},[483,484],[91,841,843,864],{"className":842},[488],[91,844,846],{"className":845,"style":641},[492],[91,847,848,851],{"style":644},[91,849],{"className":850,"style":501},[500],[91,852,854],{"className":853},[505,506,507,508],[91,855,857,861],{"className":856},[408,508],[91,858,751],{"className":859,"style":860},[408,420,508],"margin-right:0.1132em;",[91,862,375],{"className":863,"style":515},[408,420,508],[91,865,520],{"className":866},[519],[91,868,870],{"className":869},[488],[91,871,873],{"className":872,"style":663},[492],[91,874],{},[91,876,349],{"className":877},[416],[91,879,363],{"className":880},[408],[91,882],{"className":883,"style":451},[429],[91,885,366],{"className":886},[455],[91,888],{"className":889,"style":451},[429],[91,891,893,896,942],{"className":892},[399],[91,894],{"className":895,"style":404},[403],[91,897,899,902],{"className":898},[408],[91,900,11],{"className":901},[408,420],[91,903,905],{"className":904},[479],[91,906,908,934],{"className":907},[483,484],[91,909,911,931],{"className":910},[488],[91,912,914],{"className":913,"style":641},[492],[91,915,916,919],{"style":644},[91,917],{"className":918,"style":501},[500],[91,920,922],{"className":921},[505,506,507,508],[91,923,925,928],{"className":924},[408,508],[91,926,751],{"className":927,"style":860},[408,420,508],[91,929,375],{"className":930,"style":515},[408,420,508],[91,932,520],{"className":933},[519],[91,935,937],{"className":936},[488],[91,938,940],{"className":939,"style":663},[492],[91,941],{},[91,943,356],{"className":944},[425],", the same sum, just algebraically rearranged) and explains why it (together with cross-entropy, the more common alternative) is preferred over the raw error rate for ",[34,947,948],{},"growing"," the tree: it's more sensitive to small changes in class proportions within a group, so it guides which question splits best even when no question yet classifies everything correctly.",[11,951,952],{},"A binary split (separating the group into \"equal to this value\" versus \"different\") has a combined impurity, the average of both sides' impurities, weighted by each side's size:",[82,954,956],{"className":84,"code":955,"language":86,"meta":87,"style":87},"def impurity_value(x, y, value, impurity_function):\n    equals = x == value\n    equals_impurity = impurity_function(y[equals])\n    not_equals_impurity = impurity_function(y[~equals])\n    return (np.mean(equals)) * equals_impurity + (np.mean(~equals)) * not_equals_impurity\n",[15,957,958,963,968,973,978],{"__ignoreMap":87},[91,959,960],{"class":93,"line":94},[91,961,962],{},"def impurity_value(x, y, value, impurity_function):\n",[91,964,965],{"class":93,"line":100},[91,966,967],{},"    equals = x == value\n",[91,969,970],{"class":93,"line":155},[91,971,972],{},"    equals_impurity = impurity_function(y[equals])\n",[91,974,975],{"class":93,"line":161},[91,976,977],{},"    not_equals_impurity = impurity_function(y[~equals])\n",[91,979,980],{"class":93,"line":167},[91,981,982],{},"    return (np.mean(equals)) * equals_impurity + (np.mean(~equals)) * not_equals_impurity\n",[11,984,985,986,989],{},"And the greedy tree tests ",[34,987,988],{},"every"," possible variable\u002Fvalue combination, keeping whichever gives the lowest combined impurity:",[82,991,993],{"className":84,"code":992,"language":86,"meta":87,"style":87},"def best_feature(X, y, impurity_function):\n    best_feature, best_value, best_value_impurity = None, None, float('inf')\n    for feature in range(X.shape[1]):\n        value, _ = best_split(X[:,feature], y, impurity_function)\n        feature_impurity = impurity_value(X[:,feature], y, value, impurity_function)\n        if feature_impurity \u003C best_value_impurity:\n            best_feature, best_value_impurity, best_value = feature, feature_impurity, value\n    return best_feature, best_value, best_value_impurity\n",[15,994,995,1000,1005,1010,1015,1020,1025,1030],{"__ignoreMap":87},[91,996,997],{"class":93,"line":94},[91,998,999],{},"def best_feature(X, y, impurity_function):\n",[91,1001,1002],{"class":93,"line":100},[91,1003,1004],{},"    best_feature, best_value, best_value_impurity = None, None, float('inf')\n",[91,1006,1007],{"class":93,"line":155},[91,1008,1009],{},"    for feature in range(X.shape[1]):\n",[91,1011,1012],{"class":93,"line":161},[91,1013,1014],{},"        value, _ = best_split(X[:,feature], y, impurity_function)\n",[91,1016,1017],{"class":93,"line":167},[91,1018,1019],{},"        feature_impurity = impurity_value(X[:,feature], y, value, impurity_function)\n",[91,1021,1022],{"class":93,"line":226},[91,1023,1024],{},"        if feature_impurity \u003C best_value_impurity:\n",[91,1026,1027],{"class":93,"line":232},[91,1028,1029],{},"            best_feature, best_value_impurity, best_value = feature, feature_impurity, value\n",[91,1031,1032],{"class":93,"line":238},[91,1033,1034],{},"    return best_feature, best_value, best_value_impurity\n",[105,1036,1037],{},[11,1038,1039,1041,1042,1044,1045,1048],{},[34,1040,111],{}," the first question the tree picks is about ",[15,1043,56],{}," (variable 5), splitting into \"",[15,1046,1047],{},"low","\" versus the rest, with combined impurity 0.385, the lowest among all 6 variables tested. That tracks in a pretty human way: low safety probably fails the car outright, so that question alone already separates a lot of ground.",[24,1050,1052],{"id":1051},"the-real-greedy-tree","The real greedy tree",[11,1054,1055],{},"Putting it all together, every node in the tree asks \"is this variable equal to this value?\" and recurses into both groups, always picking whichever question reduces impurity the most:",[105,1057,1058],{},[11,1059,1060,1062,1063,180],{},[34,1061,111],{}," 100% accuracy on the data it trained on. 96.8% on a held-out test set. 97.2% average accuracy on 5-fold cross-validation (0.977, 0.986, 0.962, 0.968, 0.968), ",[133,1064,1066],{"href":1065},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","the same technique I already saw pay off in the previous post",[11,1068,1069],{},"100% on training is always an overfitting red flag (an unbounded tree can always memorize), but here the test and cross-validation numbers are also very good, so it isn't hollow memorization this time: this particular dataset comes from a deterministic rule (it's a synthetic dataset, built from a car-rating rule table, with zero noise), so a sufficiently deep tree can literally reconstruct the true rule behind the data.",[24,1071,1073],{"id":1072},"limiting-depth-doesnt-always-help","Limiting depth doesn't always help",[11,1075,1076,1077,1080],{},"The most direct way to rein in a tree is to cap how many questions in a row it can ask, ",[15,1078,1079],{},"max_depth",":",[82,1082,1084],{"className":84,"code":1083,"language":86,"meta":87,"style":87},"class DecisionTree(BaseEstimator, ClassifierMixin):\n    def __init__(self, max_depth=9999999):\n        self.max_depth = max_depth\n    def fit(self, X, y):\n        ...\n        if sum(equals) > 0 and sum(~equals) > 0 and self.max_depth > 0:\n            self.equals_tree = DecisionTree(self.max_depth-1).fit(X[equals], y[equals])\n            ...\n",[15,1085,1086,1090,1095,1100,1104,1109,1114,1119],{"__ignoreMap":87},[91,1087,1088],{"class":93,"line":94},[91,1089,204],{},[91,1091,1092],{"class":93,"line":100},[91,1093,1094],{},"    def __init__(self, max_depth=9999999):\n",[91,1096,1097],{"class":93,"line":155},[91,1098,1099],{},"        self.max_depth = max_depth\n",[91,1101,1102],{"class":93,"line":161},[91,1103,152],{},[91,1105,1106],{"class":93,"line":167},[91,1107,1108],{},"        ...\n",[91,1110,1111],{"class":93,"line":226},[91,1112,1113],{},"        if sum(equals) > 0 and sum(~equals) > 0 and self.max_depth > 0:\n",[91,1115,1116],{"class":93,"line":232},[91,1117,1118],{},"            self.equals_tree = DecisionTree(self.max_depth-1).fit(X[equals], y[equals])\n",[91,1120,1121],{"class":93,"line":238},[91,1122,1123],{},"            ...\n",[105,1125,1126],{},[11,1127,1128,1130,1131,1134,1135,1138],{},[34,1129,111],{}," with ",[15,1132,1133],{},"max_depth=5",", cross-validation accuracy ",[34,1136,1137],{},"drops"," to 86.6% (versus 97.2% with no limit at all).",[11,1140,1141,1142,1145,1146,1148],{},"Counterintuitive at first, but it connects directly to why this dataset is \"clean\": since the true rule behind the data genuinely depends on combining several variables in sequence, cutting the tree short takes away exactly the capacity it needs to represent the full rule. This isn't \"regularization fighting overfitting,\" it's ",[34,1143,1144],{},"underfitting"," (the tree becomes too simple for this specific problem). The lesson isn't \"always cap depth,\" it's \"measure before deciding\": the right ",[15,1147,1079],{}," depends entirely on how complicated the true pattern is, and the only way to know is testing, with cross-validation, not assuming.",[11,1150,1151,1152,1155,1156,1159],{},"With ",[15,1153,1154],{},"max_depth=20"," and ",[15,1157,1158],{},"min_sample_split=10"," (stop splitting a group of 10 examples or fewer, even if it's still impure), the result lands in between: 95.6%, slightly below the unlimited optimum, but already much more controlled than growing with no cap at all.",[24,1161,1163],{"id":1162},"continuous-attributes-same-idea-the-cut-changes","Continuous attributes: same idea, the cut changes",[11,1165,1166,1168,1169,1155,1172,1175,1176,1179],{},[15,1167,21],{}," switches datasets (Iris, the classic 3-species flower dataset, using just ",[15,1170,1171],{},"petal length",[15,1173,1174],{},"petal width",") and switches the type of question: instead of \"is it equal to this value?\", the question becomes \"is it ",[34,1177,1178],{},"greater than or equal"," to this threshold?\". To find the best threshold on a continuous variable, the professor sorts the values and tests the midpoint between every pair of neighbors:",[82,1181,1183],{"className":84,"code":1182,"language":86,"meta":87,"style":87},"def best_split(x, y, impurity_function):\n    best_value, best_value_impurity = 0, float('inf')\n    x = np.sort(x)\n    for i in range(1, len(x)):\n        value = (x[i-1]+x[i])\u002F2\n        value_impurity = impurity_value(x, y, value, impurity_function)\n        if value_impurity \u003C best_value_impurity:\n            best_value, best_value_impurity = value, value_impurity\n    return best_value, best_value_impurity\n",[15,1184,1185,1190,1195,1200,1205,1210,1215,1220,1225],{"__ignoreMap":87},[91,1186,1187],{"class":93,"line":94},[91,1188,1189],{},"def best_split(x, y, impurity_function):\n",[91,1191,1192],{"class":93,"line":100},[91,1193,1194],{},"    best_value, best_value_impurity = 0, float('inf')\n",[91,1196,1197],{"class":93,"line":155},[91,1198,1199],{},"    x = np.sort(x)\n",[91,1201,1202],{"class":93,"line":161},[91,1203,1204],{},"    for i in range(1, len(x)):\n",[91,1206,1207],{"class":93,"line":167},[91,1208,1209],{},"        value = (x[i-1]+x[i])\u002F2\n",[91,1211,1212],{"class":93,"line":226},[91,1213,1214],{},"        value_impurity = impurity_value(x, y, value, impurity_function)\n",[91,1216,1217],{"class":93,"line":232},[91,1218,1219],{},"        if value_impurity \u003C best_value_impurity:\n",[91,1221,1222],{"class":93,"line":238},[91,1223,1224],{},"            best_value, best_value_impurity = value, value_impurity\n",[91,1226,1227],{"class":93,"line":244},[91,1228,1229],{},"    return best_value, best_value_impurity\n",[105,1231,1232],{},[11,1233,1234,1236,1237,1240],{},[34,1235,111],{}," the first question the tree learns is ",[15,1238,1239],{},"petal length >= 2.45",", with impurity 0.333 (the lowest possible here, since that single question already perfectly separates one of the 3 species from the rest).",[24,1242,1244],{"id":1243},"interactive-watching-depth-reshape-the-boundary-live","Interactive: watching depth reshape the boundary live",[11,1246,1247],{},"My own reconstruction of the same algorithm, on the 150 real Iris points (the same two variables, petal length and width). Click through the depth values and notice how the boundary gains a new \"step\" at each level:",[1249,1250],"decision-tree-explorer",{":classes":1251,":depth-options":1252,":initial-max-depth":363,":x-max":1253,":x-min":1254,":x-train":1255,":y-max":1256,":y-min":1257,":y-train":1258,"class0-label":1259,"class1-label":1260,"class2-label":1261,"x-label":1262,"y-label":1263},"[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","[1, 2, 3, 5, 9999]","7.1","0.8","[1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.6, 1.4, 1.1, 1.2, 1.5, 1.3, 1.4, 1.7, 1.5, 1.7, 1.5, 1.0, 1.7, 1.9, 1.6, 1.6, 1.5, 1.4, 1.6, 1.6, 1.5, 1.5, 1.4, 1.5, 1.2, 1.3, 1.4, 1.3, 1.5, 1.3, 1.3, 1.3, 1.6, 1.9, 1.4, 1.6, 1.4, 1.5, 1.4, 4.7, 4.5, 4.9, 4.0, 4.6, 4.5, 4.7, 3.3, 4.6, 3.9, 3.5, 4.2, 4.0, 4.7, 3.6, 4.4, 4.5, 4.1, 4.5, 3.9, 4.8, 4.0, 4.9, 4.7, 4.3, 4.4, 4.8, 5.0, 4.5, 3.5, 3.8, 3.7, 3.9, 5.1, 4.5, 4.5, 4.7, 4.4, 4.1, 4.0, 4.4, 4.6, 4.0, 3.3, 4.2, 4.2, 4.2, 4.3, 3.0, 4.1, 6.0, 5.1, 5.9, 5.6, 5.8, 6.6, 4.5, 6.3, 5.8, 6.1, 5.1, 5.3, 5.5, 5.0, 5.1, 5.3, 5.5, 6.7, 6.9, 5.0, 5.7, 4.9, 6.7, 4.9, 5.7, 6.0, 4.8, 4.9, 5.6, 5.8, 6.1, 6.4, 5.6, 5.1, 5.6, 6.1, 5.6, 5.5, 4.8, 5.4, 5.6, 5.1, 5.1, 5.9, 5.7, 5.2, 5.0, 5.2, 5.4, 5.1]","2.6","0","[0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.4, 0.4, 0.3, 0.3, 0.3, 0.2, 0.4, 0.2, 0.5, 0.2, 0.2, 0.4, 0.2, 0.2, 0.2, 0.2, 0.4, 0.1, 0.2, 0.2, 0.2, 0.2, 0.1, 0.2, 0.2, 0.3, 0.3, 0.2, 0.6, 0.4, 0.3, 0.2, 0.2, 0.2, 0.2, 1.4, 1.5, 1.5, 1.3, 1.5, 1.3, 1.6, 1.0, 1.3, 1.4, 1.0, 1.5, 1.0, 1.4, 1.3, 1.4, 1.5, 1.0, 1.5, 1.1, 1.8, 1.3, 1.5, 1.2, 1.3, 1.4, 1.4, 1.7, 1.5, 1.0, 1.1, 1.0, 1.2, 1.6, 1.5, 1.6, 1.5, 1.3, 1.3, 1.3, 1.2, 1.4, 1.2, 1.0, 1.3, 1.2, 1.3, 1.3, 1.1, 1.3, 2.5, 1.9, 2.1, 1.8, 2.2, 2.1, 1.7, 1.8, 1.8, 2.5, 2.0, 1.9, 2.1, 2.0, 2.4, 2.3, 1.8, 2.2, 2.3, 1.5, 2.3, 2.0, 2.0, 1.8, 2.1, 1.8, 1.8, 1.8, 2.1, 1.6, 1.9, 2.0, 2.2, 1.5, 1.4, 2.3, 2.4, 1.8, 1.8, 2.1, 2.4, 2.3, 1.9, 2.3, 2.5, 2.3, 1.9, 2.0, 2.3, 1.8]","Setosa","Versicolor","Virginica","petal length (cm)","petal width (cm)",[11,1265,1266,1267,1269,1270,1273],{},"At depth 1, the tree only asks one question (the same ",[15,1268,1239],{}," from above), so the chart turns into exactly two regions, a single vertical line. From depth 2 on, a second cut appears splitting the second region in two, and so on: each depth level adds at most one more cut per existing region. Also notice the cuts are always ",[34,1271,1272],{},"vertical or horizontal lines",", never diagonal, because every question looks at one variable at a time.",[24,1275,1277],{"id":1276},"what-the-tree-cant-do-well","What the tree can't do well",[11,1279,1280,1281,1284,1285,1289],{},"That last observation is one of the limitations Bishop points out: decision trees only cut ",[34,1282,1283],{},"axis-aligned",". If the \"true\" boundary between two classes ran diagonally, a tree would need a staircase of cuts to approximate it, while a single diagonal boundary would solve it in one shot (compare with the previous post: ",[133,1286,1288],{"href":1287},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier","KNN's regions"," were already smoother, without this axis limitation).",[11,1291,1292,1293,180],{},"Bishop also cites another problem, instability: a small change in the training data can change the entire tree structure, because the choice of which variable to split on first at the top cascades down through the whole tree beneath it. The notebook itself shows this by accident: running the same training three times, with different train\u002Ftest splits (no seed fixed), test accuracy came out 90%, then 96.7%, then 100%. With only 30 test examples, a difference of 2-3 classifications already swings accuracy by several percentage points, exactly why ",[133,1294,1295],{"href":1065},"cross-validation, not a single split, is the right way to measure this",[24,1297,1299],{"id":1298},"wrapping-up","Wrapping up",[1301,1302,1303,1317],"table",{},[1304,1305,1306],"thead",{},[1307,1308,1309,1314],"tr",{},[1310,1311,1313],"th",{"align":1312},"left","What I already knew",[1310,1315,1316],{"align":1312},"What this lecture settled",[1318,1319,1320,1329,1337],"tbody",{},[1307,1321,1322,1326],{},[1323,1324,1325],"td",{"align":1312},"A dumb baseline helps interpret any accuracy number",[1323,1327,1328],{"align":1312},"With imbalanced classes (70% in one alone), the baseline can be surprisingly high, and \"97% accuracy\" without comparing against it says nothing",[1307,1330,1331,1334],{},[1323,1332,1333],{"align":1312},"Overfitting happens when the model has too much freedom",[1323,1335,1336],{"align":1312},"An unbounded tree always memorizes training, but capping it too aggressively can also backfire, if the true pattern genuinely needs that much complexity",[1307,1338,1339,1342],{},[1323,1340,1341],{"align":1312},"Cross-validation gives a more stable estimate",[1323,1343,1344],{"align":1312},"Trees are especially unstable to small changes in training, so a single test split is even more misleading here than for other models",[24,1346,1348],{"id":1347},"practical-application","Practical application",[11,1350,1351,1352,1354],{},"I use the car evaluation dataset (the categorical one, ",[15,1353,17],{},") to compare different depths systematically, with real cross-validation at each one, instead of testing one value at a time.",[82,1356,1358],{"className":84,"code":1357,"language":86,"meta":87,"style":87},"for depth in [1, 2, 3, 5, 10, 9999]:\n    model = DecisionTree(max_depth=depth, min_sample_split=2)\n    scores = cross_val_score(model, X, y, cv=KFold(n_splits=5, shuffle=True))\n    print(depth, np.mean(scores))\n",[15,1359,1360,1365,1370,1375],{"__ignoreMap":87},[91,1361,1362],{"class":93,"line":94},[91,1363,1364],{},"for depth in [1, 2, 3, 5, 10, 9999]:\n",[91,1366,1367],{"class":93,"line":100},[91,1368,1369],{},"    model = DecisionTree(max_depth=depth, min_sample_split=2)\n",[91,1371,1372],{"class":93,"line":155},[91,1373,1374],{},"    scores = cross_val_score(model, X, y, cv=KFold(n_splits=5, shuffle=True))\n",[91,1376,1377],{"class":93,"line":161},[91,1378,1379],{},"    print(depth, np.mean(scores))\n",[1301,1381,1382,1394],{},[1304,1383,1384],{},[1307,1385,1386,1390],{},[1310,1387,1389],{"align":1388},"center","Max depth",[1310,1391,1393],{"align":1392},"right","Accuracy (5-fold CV)",[1318,1395,1396,1403,1411],{},[1307,1397,1398,1400],{},[1323,1399,363],{"align":1388},[1323,1401,1402],{"align":1392},"≈ 0.78 (a single question, just above the 0.70 baseline)",[1307,1404,1405,1408],{},[1323,1406,1407],{"align":1388},"5",[1323,1409,1410],{"align":1392},"0.866",[1307,1412,1413,1416],{},[1323,1414,1415],{"align":1388},"Unlimited",[1323,1417,1418],{"align":1392},[34,1419,1420],{},"0.972",[11,1422,1423],{},"(The depth-1 row is my own quick estimate along the same idea, not a cell from the original notebook. The other two are the real numbers already shown in this post.) The trend is clear: on Car Evaluation, more depth almost always helps, because the true pattern behind the data is genuinely complex (it depends on combining several of the 6 variables) with zero noise getting in the way. That's the exact opposite of what usually happens with real, noisy data, where too much depth memorizes the noise instead of the pattern. Same lesson again: there's no universal \"right depth,\" there's testing with cross-validation and letting the data decide.",[1425,1426,1427],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":87,"searchDepth":100,"depth":100,"links":1429},[1430,1431,1432,1433,1434,1435,1436,1437,1438,1439],{"id":26,"depth":100,"text":27},{"id":186,"depth":100,"text":187},{"id":276,"depth":100,"text":277},{"id":1051,"depth":100,"text":1052},{"id":1072,"depth":100,"text":1073},{"id":1162,"depth":100,"text":1163},{"id":1243,"depth":100,"text":1244},{"id":1276,"depth":100,"text":1277},{"id":1298,"depth":100,"text":1299},{"id":1347,"depth":100,"text":1348},null,"2026-08-19","Lectures 5 and 5b: the professor builds a greedy decision tree from scratch, guided by Gini impurity, first on categorical attributes, then on continuous ones. I explain why limiting depth doesn't always help.","md",{},true,"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees","pattern-recognition",{"title":6,"description":1442},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees",[1452,705,1453],"decision-tree","overfitting","KOQ97bJpbObN6bLfH2VqEgp9s_NFoL1hArCb5lvM8T4",1787338984353]