[{"data":1,"prerenderedAt":827},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic":3,"post-en-pattern-recognition-titanic":4},"\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic",{"id":5,"title":6,"body":7,"cover":812,"date":813,"description":814,"extension":815,"meta":816,"navigation":306,"order":339,"path":817,"playlist":818,"seo":819,"status":820,"stem":821,"tags":822,"__hash__":826},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic.md","The Titanic: My First Genuinely Messy Dataset",{"type":8,"value":9,"toc":802},"minimark",[10,14,19,49,91,100,115,119,134,150,164,184,197,201,237,257,269,273,292,349,364,419,426,459,466,473,531,535,550,557,570,585,592,595,599,606,613,630,637,644,648,693,697,700,725,791,798],[11,12,13],"p",{},"Lecture 7, and the professor switches gears: every dataset up to now already arrived ready, no gaps, no useless column, no loose text. The Titanic (Kaggle's most famous dataset, probably everyone's first data science project) doesn't get that luxury.",[15,16,18],"h2",{"id":17},"messy-data-891-passengers-not-every-column-is-useful","Messy data: 891 passengers, not every column is useful",[20,21,26],"pre",{"className":22,"code":23,"language":24,"meta":25,"style":25},"language-python shiki shiki-themes github-light github-dark","df = pd.read_csv('train.csv')\ny = df['Survived']\nX = df.drop('Survived', axis=1)\n","python","",[27,28,29,37,43],"code",{"__ignoreMap":25},[30,31,34],"span",{"class":32,"line":33},"line",1,[30,35,36],{},"df = pd.read_csv('train.csv')\n",[30,38,40],{"class":32,"line":39},2,[30,41,42],{},"y = df['Survived']\n",[30,44,46],{"class":32,"line":45},3,[30,47,48],{},"X = df.drop('Survived', axis=1)\n",[11,50,51,52,55,56,59,60,63,64,63,67,63,70,73,74,77,78,63,81,63,84,63,87,90],{},"891 passengers, ",[27,53,54],{},"Survived"," (0 or 1) as the target, and a handful of pretty mixed input columns: ",[27,57,58],{},"Pclass"," (ticket class), ",[27,61,62],{},"Name",", ",[27,65,66],{},"Sex",[27,68,69],{},"Age",[27,71,72],{},"SibSp"," (siblings\u002Fspouse aboard), ",[27,75,76],{},"Parch"," (parents\u002Fchildren aboard), ",[27,79,80],{},"Ticket",[27,82,83],{},"Fare",[27,85,86],{},"Cabin",[27,88,89],{},"Embarked"," (port of embarkation). The professor drops four of them right away:",[20,92,94],{"className":22,"code":93,"language":24,"meta":25,"style":25},"caracteristicas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n",[27,95,96],{"__ignoreMap":25},[30,97,98],{"class":32,"line":33},[30,99,93],{},[11,101,102,105,106,108,109,111,112,114],{},[27,103,104],{},"PassengerId"," is just a sequential number, carries no information about the person at all. ",[27,107,62],{}," and ",[27,110,80],{}," are free text, almost every value is unique (891 different names for 891 passengers), so a model has no way to generalize from that without processing the text first (extracting a title, like \"Mr.\"\u002F\"Mrs.\", would be a valid move, but that's extra work the lecture didn't do). ",[27,113,86],{}," is missing for most rows and is also far too specific.",[15,116,118],{"id":117},"missing-data-is-the-rule-not-the-exception","Missing data is the rule, not the exception",[20,120,122],{"className":22,"code":121,"language":24,"meta":25,"style":25},"for col in Xnum.columns:\n    print(f\"{col:>12} {Xnum[col].isnull().sum():2}\")\n",[27,123,124,129],{"__ignoreMap":25},[30,125,126],{"class":32,"line":33},[30,127,128],{},"for col in Xnum.columns:\n",[30,130,131],{"class":32,"line":39},[30,132,133],{},"    print(f\"{col:>12} {Xnum[col].isnull().sum():2}\")\n",[135,136,137],"blockquote",{},[11,138,139,143,144,146,147,149],{},[140,141,142],"strong",{},"Output:"," ",[27,145,69],{}," is missing for 177 of 891 passengers (almost 20%). ",[27,148,89],{}," is missing for 2.",[11,151,152,153,156,157,159,160,163],{},"That never happened in this playlist before now: every previous dataset already arrived complete. Here, nearly 1 in every 5 ages is ",[27,154,155],{},"NaN",". Dropping those rows would throw away 20% of the data, and dropping the ",[27,158,69],{}," column entirely would throw away a probably important variable (children had rescue priority). The professor's solution is ",[140,161,162],{},"imputation",": fill the gap with an estimated value.",[20,165,167],{"className":22,"code":166,"language":24,"meta":25,"style":25},"from sklearn.impute import SimpleImputer\nimputer = SimpleImputer(strategy='median')\nXnumTratado = imputer.fit_transform(Xnum)\n",[27,168,169,174,179],{"__ignoreMap":25},[30,170,171],{"class":32,"line":33},[30,172,173],{},"from sklearn.impute import SimpleImputer\n",[30,175,176],{"class":32,"line":39},[30,177,178],{},"imputer = SimpleImputer(strategy='median')\n",[30,180,181],{"class":32,"line":45},[30,182,183],{},"XnumTratado = imputer.fit_transform(Xnum)\n",[11,185,186,187,189,190,192,193,196],{},"For numeric variables, the median (not the mean): the age and fare (",[27,188,83],{},") distributions have people well outside the norm (infants a few months old, very expensive first-class fares), and the median doesn't get pulled by those extremes the way the mean does. For categorical variables (",[27,191,89],{},"), there's no such thing as a \"median\" port, so the strategy becomes ",[27,194,195],{},"'most_frequent'",", whichever value shows up most.",[15,198,200],{"id":199},"a-category-isnt-a-number-even-when-it-looks-like-one","A category isn't a number, even when it looks like one",[11,202,203,108,205,207,208,211,212,63,215,211,218,211,221,224,225,228,229,232,233,236],{},[27,204,66],{},[27,206,89],{}," are text (",[27,209,210],{},"\"male\"","\u002F",[27,213,214],{},"\"female\"",[27,216,217],{},"\"S\"",[27,219,220],{},"\"C\"",[27,222,223],{},"\"Q\"","). A model only understands numbers, but simply numbering them (",[27,226,227],{},"male=0, female=1",", or ",[27,230,231],{},"S=0, C=1, Q=2",") would invent an order that doesn't exist: why would \"Q\" be \"bigger\" than \"S\"? The right move is ",[140,234,235],{},"one-hot encoding",": every category becomes its own binary column.",[20,238,240],{"className":22,"code":239,"language":24,"meta":25,"style":25},"from sklearn.preprocessing import OneHotEncoder\nencoder = OneHotEncoder()\nXcatTratadoHot = encoder.fit_transform(XcatTratado)\n",[27,241,242,247,252],{"__ignoreMap":25},[30,243,244],{"class":32,"line":33},[30,245,246],{},"from sklearn.preprocessing import OneHotEncoder\n",[30,248,249],{"class":32,"line":39},[30,250,251],{},"encoder = OneHotEncoder()\n",[30,253,254],{"class":32,"line":45},[30,255,256],{},"XcatTratadoHot = encoder.fit_transform(XcatTratado)\n",[135,258,259],{},[11,260,261,263,264,63,266,268],{},[140,262,142],{}," the 2 categorical columns (",[27,265,66],{},[27,267,89],{},") turn into 5 binary columns (2 for sex, since one is already redundant with the other, and 3 for port of embarkation). None of them carries an implicit numeric order, they're just \"is\" or \"isn't\" that category.",[15,270,272],{"id":271},"my-own-transformer-not-everything-is-a-classifier","My own Transformer: not everything is a classifier",[11,274,275,276,279,280,283,284,287,288,291],{},"Up to now, every custom class in this playlist inherited from ",[27,277,278],{},"ClassifierMixin"," or ",[27,281,282],{},"RegressorMixin",", because it always ended in ",[27,285,286],{},".predict()",". Here the goal is different: just transform the data, without predicting anything. That's what ",[27,289,290],{},"TransformerMixin"," is for:",[20,293,295],{"className":22,"code":294,"language":24,"meta":25,"style":25},"from sklearn.base import BaseEstimator, TransformerMixin\n\nclass AtributosDesejados(BaseEstimator, TransformerMixin):\n    def __init__(self):\n        self.colunas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n    def fit(self, X, y=None):\n        return self\n    def transform(self, X, y=None):\n        return X.drop(self.colunas_indesejadas, axis=1)\n",[27,296,297,302,308,313,319,325,331,337,343],{"__ignoreMap":25},[30,298,299],{"class":32,"line":33},[30,300,301],{},"from sklearn.base import BaseEstimator, TransformerMixin\n",[30,303,304],{"class":32,"line":39},[30,305,307],{"emptyLinePlaceholder":306},true,"\n",[30,309,310],{"class":32,"line":45},[30,311,312],{},"class AtributosDesejados(BaseEstimator, TransformerMixin):\n",[30,314,316],{"class":32,"line":315},4,[30,317,318],{},"    def __init__(self):\n",[30,320,322],{"class":32,"line":321},5,[30,323,324],{},"        self.colunas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']\n",[30,326,328],{"class":32,"line":327},6,[30,329,330],{},"    def fit(self, X, y=None):\n",[30,332,334],{"class":32,"line":333},7,[30,335,336],{},"        return self\n",[30,338,340],{"class":32,"line":339},8,[30,341,342],{},"    def transform(self, X, y=None):\n",[30,344,346],{"class":32,"line":345},9,[30,347,348],{},"        return X.drop(self.colunas_indesejadas, axis=1)\n",[11,350,351,352,355,356,359,360,363],{},"Along with ",[27,353,354],{},"AtributosNumericos"," (keeps only the numeric columns) and ",[27,357,358],{},"AtributosCategoricos"," (keeps only the categorical ones), the professor builds two parallel ",[27,361,362],{},"Pipeline","s, one for each kind of data:",[20,365,367],{"className":22,"code":366,"language":24,"meta":25,"style":25},"pipenum = Pipeline([\n    ('atributos_numericos', AtributosNumericos()),\n    ('imputer', SimpleImputer(strategy='median')),\n    ('scaler', StandardScaler())\n])\npipecat = Pipeline([\n    ('atributos_categoricos', AtributosCategoricos()),\n    ('imputer', SimpleImputer(strategy='most_frequent')),\n    ('encoder', OneHotEncoder())\n])\n",[27,368,369,374,379,384,389,394,399,404,409,414],{"__ignoreMap":25},[30,370,371],{"class":32,"line":33},[30,372,373],{},"pipenum = Pipeline([\n",[30,375,376],{"class":32,"line":39},[30,377,378],{},"    ('atributos_numericos', AtributosNumericos()),\n",[30,380,381],{"class":32,"line":45},[30,382,383],{},"    ('imputer', SimpleImputer(strategy='median')),\n",[30,385,386],{"class":32,"line":315},[30,387,388],{},"    ('scaler', StandardScaler())\n",[30,390,391],{"class":32,"line":321},[30,392,393],{},"])\n",[30,395,396],{"class":32,"line":327},[30,397,398],{},"pipecat = Pipeline([\n",[30,400,401],{"class":32,"line":333},[30,402,403],{},"    ('atributos_categoricos', AtributosCategoricos()),\n",[30,405,406],{"class":32,"line":339},[30,407,408],{},"    ('imputer', SimpleImputer(strategy='most_frequent')),\n",[30,410,411],{"class":32,"line":345},[30,412,413],{},"    ('encoder', OneHotEncoder())\n",[30,415,417],{"class":32,"line":416},10,[30,418,393],{},[11,420,421,422,425],{},"And ",[27,423,424],{},"FeatureUnion"," joins both outputs into one set of columns, side by side:",[20,427,429],{"className":22,"code":428,"language":24,"meta":25,"style":25},"from sklearn.pipeline import FeatureUnion\n\nunecaracteristicas = FeatureUnion([\n    ('pipenum', pipenum),\n    ('pipecat', pipecat)\n])\n",[27,430,431,436,440,445,450,455],{"__ignoreMap":25},[30,432,433],{"class":32,"line":33},[30,434,435],{},"from sklearn.pipeline import FeatureUnion\n",[30,437,438],{"class":32,"line":39},[30,439,307],{"emptyLinePlaceholder":306},[30,441,442],{"class":32,"line":45},[30,443,444],{},"unecaracteristicas = FeatureUnion([\n",[30,446,447],{"class":32,"line":315},[30,448,449],{},"    ('pipenum', pipenum),\n",[30,451,452],{"class":32,"line":321},[30,453,454],{},"    ('pipecat', pipecat)\n",[30,456,457],{"class":32,"line":327},[30,458,393],{},[135,460,461],{},[11,462,463,465],{},[140,464,142],{}," the 7 original columns (after dropping the 4 useless ones) become 10 processed columns: 5 numeric (already imputed and normalized) plus 5 categorical (already imputed and one-hot encoded).",[11,467,468,469,472],{},"The final pipeline chains everything, from raw data to classifier, in a single ",[27,470,471],{},".fit()"," call:",[20,474,476],{"className":22,"code":475,"language":24,"meta":25,"style":25},"preproc = Pipeline([\n    ('atributos_desejados', AtributosDesejados()),\n    ('unecaracteristicas', unecaracteristicas),\n    ('to_dense', DenseTransformer())\n])\n\nclf = Pipeline([\n    ('preproc', preproc),\n    ('classificador', RandomForestClassifier())\n])\nclf.fit(X, y)\n",[27,477,478,483,488,493,498,502,506,511,516,521,525],{"__ignoreMap":25},[30,479,480],{"class":32,"line":33},[30,481,482],{},"preproc = Pipeline([\n",[30,484,485],{"class":32,"line":39},[30,486,487],{},"    ('atributos_desejados', AtributosDesejados()),\n",[30,489,490],{"class":32,"line":45},[30,491,492],{},"    ('unecaracteristicas', unecaracteristicas),\n",[30,494,495],{"class":32,"line":315},[30,496,497],{},"    ('to_dense', DenseTransformer())\n",[30,499,500],{"class":32,"line":321},[30,501,393],{},[30,503,504],{"class":32,"line":327},[30,505,307],{"emptyLinePlaceholder":306},[30,507,508],{"class":32,"line":333},[30,509,510],{},"clf = Pipeline([\n",[30,512,513],{"class":32,"line":339},[30,514,515],{},"    ('preproc', preproc),\n",[30,517,518],{"class":32,"line":345},[30,519,520],{},"    ('classificador', RandomForestClassifier())\n",[30,522,523],{"class":32,"line":416},[30,524,393],{},[30,526,528],{"class":32,"line":527},11,[30,529,530],{},"clf.fit(X, y)\n",[15,532,534],{"id":533},"the-same-old-mistake-with-a-real-consequence-this-time","The same old mistake, with a real consequence this time",[20,536,538],{"className":22,"code":537,"language":24,"meta":25,"style":25},"y_pred = clf.predict(X)\naccuracy_score(y, y_pred)\n",[27,539,540,545],{"__ignoreMap":25},[30,541,542],{"class":32,"line":33},[30,543,544],{},"y_pred = clf.predict(X)\n",[30,546,547],{"class":32,"line":39},[30,548,549],{},"accuracy_score(y, y_pred)\n",[135,551,552],{},[11,553,554,556],{},[140,555,142],{}," 0.9798. Almost 98% correct.",[11,558,559,560,565,566,569],{},"If you ",[561,562,564],"a",{"href":563},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","already read this playlist's post on cross-validation",", that accuracy number should set off an alarm: it's measured on the ",[140,567,568],{},"same"," data it trained on. The proof it's memorization, not learning:",[20,571,573],{"className":22,"code":572,"language":24,"meta":25,"style":25},"from sklearn.model_selection import cross_val_score\nscores = cross_val_score(clf, X, y)\n",[27,574,575,580],{"__ignoreMap":25},[30,576,577],{"class":32,"line":33},[30,578,579],{},"from sklearn.model_selection import cross_val_score\n",[30,581,582],{"class":32,"line":39},[30,583,584],{},"scores = cross_val_score(clf, X, y)\n",[135,586,587],{},[11,588,589,591],{},[140,590,142],{}," 0.807 average (0.765, 0.815, 0.854, 0.775, 0.826 across the 5 folds).",[11,593,594],{},"A nearly 17-percentage-point drop between \"accuracy on training\" and \"honest accuracy.\" The Random Forest, with no tree limit at all, memorizes a good chunk of the 891 passengers individually, and that 98% figure never measured the ability to generalize, it measured the ability to memorize.",[15,596,598],{"id":597},"trying-to-beat-80-stacking-and-a-neural-network","Trying to beat 80%: stacking and a neural network",[11,600,601,602,605],{},"The professor tries two things to improve on the honest 0.807. First, ",[27,603,604],{},"StackingClassifier"," with 8 quite different models at once (Random Forest, Extra Trees, a neural network, SGD, Ridge, KNN, Naive Bayes, logistic regression):",[135,607,608],{},[11,609,610,612],{},[140,611,142],{}," 0.824 cross-validation. A real, if small, improvement.",[11,614,615,616,619,620,624,625,629],{},"Then, a neural network alone (",[27,617,618],{},"MLPClassifier",", short for ",[621,622,623],"em",{},"Multi-Layer Perceptron",", a Perceptron ",[561,626,628],{"href":627},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fensembles","I already saw earlier in this playlist"," stacked into several layers, the subject of its own course that isn't the focus here):",[135,631,632],{},[11,633,634,636],{},[140,635,142],{}," 0.822 cross-validation, practically tied with the 8-model stack.",[11,638,639,640,643],{},"Neither is a huge leap over the lone forest's 0.807, but both beat the simpler version consistently. Unlike ",[561,641,642],{"href":627},"the previous post"," (where a simple model beat every ensemble tried), here the extra complexity genuinely helped, a little. The earlier lesson still holds: there's no knowing which one wins without measuring both by the same ruler.",[15,645,647],{"id":646},"wrapping-up","Wrapping up",[649,650,651,665],"table",{},[652,653,654],"thead",{},[655,656,657,662],"tr",{},[658,659,661],"th",{"align":660},"left","What I already knew",[658,663,664],{"align":660},"What this lecture settled",[666,667,668,677,685],"tbody",{},[655,669,670,674],{},[671,672,673],"td",{"align":660},"Every dataset up to now already arrived complete",[671,675,676],{"align":660},"Missing data is normal in real data, and the fix (imputation) needs a deliberate strategy, not just \"drop the row\"",[655,678,679,682],{},[671,680,681],{"align":660},"A category becomes a number somehow",[671,683,684],{"align":660},"One-hot encoding avoids inventing an order between categories that doesn't exist",[655,686,687,690],{},[671,688,689],{"align":660},"Cross-validation is more honest than measuring on training",[671,691,692],{"align":660},"The same old warning, but this time on a real Random Forest: 98% on training against 81% for real",[15,694,696],{"id":695},"practical-application","Practical application",[11,698,699],{},"I reproduced the entire pipeline with a fixed seed (the original notebook fixes none), to get a stable number and compare all three approaches under the exact same cross-validation criterion.",[20,701,703],{"className":22,"code":702,"language":24,"meta":25,"style":25},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nscores_rf = cross_val_score(clf_rf, X, y, cv=cv)\nscores_stacking = cross_val_score(clf_stacking, X, y, cv=cv)\nscores_mlp = cross_val_score(clf_mlp, X, y, cv=cv)\n",[27,704,705,710,715,720],{"__ignoreMap":25},[30,706,707],{"class":32,"line":33},[30,708,709],{},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",[30,711,712],{"class":32,"line":39},[30,713,714],{},"scores_rf = cross_val_score(clf_rf, X, y, cv=cv)\n",[30,716,717],{"class":32,"line":45},[30,718,719],{},"scores_stacking = cross_val_score(clf_stacking, X, y, cv=cv)\n",[30,721,722],{"class":32,"line":315},[30,723,724],{},"scores_mlp = cross_val_score(clf_mlp, X, y, cv=cv)\n",[649,726,727,744],{},[652,728,729],{},[655,730,731,734,738,741],{},[658,732,733],{"align":660},"Model",[658,735,737],{"align":736},"right","Accuracy (training, memorized)",[658,739,740],{"align":736},"Accuracy (cross-validation)",[658,742,743],{"align":736},"Standard deviation",[666,745,746,760,778],{},[655,747,748,751,754,757],{},[671,749,750],{"align":660},"Random Forest",[671,752,753],{"align":736},"0.9798",[671,755,756],{"align":736},"0.8092",[671,758,759],{"align":736},"0.0268",[655,761,762,765,768,773],{},[671,763,764],{"align":660},"Stacking (8 models)",[671,766,767],{"align":736},"(didn't measure)",[671,769,770],{"align":736},[140,771,772],{},"0.8283",[671,774,775],{"align":736},[140,776,777],{},"0.0077",[655,779,780,783,785,788],{},[671,781,782],{"align":660},"Neural network (MLP)",[671,784,767],{"align":736},[671,786,787],{"align":736},"0.8182",[671,789,790],{"align":736},"0.0197",[11,792,793,794,797],{},"The numbers land close to the original notebook's (0.807, 0.824, 0.822), confirming the difference between the three wasn't luck from one specific run. And notice the standard deviation: the stack didn't just have the best average, it's also the most ",[140,795,796],{},"stable"," fold to fold (0.0077 versus the lone forest's 0.0268), a sign that combining several different models cushioned some of the variation each one carries on its own.",[799,800,801],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":25,"searchDepth":39,"depth":39,"links":803},[804,805,806,807,808,809,810,811],{"id":17,"depth":39,"text":18},{"id":117,"depth":39,"text":118},{"id":199,"depth":39,"text":200},{"id":271,"depth":39,"text":272},{"id":533,"depth":39,"text":534},{"id":597,"depth":39,"text":598},{"id":646,"depth":39,"text":647},{"id":695,"depth":39,"text":696},null,"2026-08-20","Lecture 7: the professor builds a full preprocessing pipeline on the classic Titanic dataset, missing data and all, and shows the same train-set-accuracy mistake I'd already seen before, except this time with a real consequence.","md",{},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic","pattern-recognition",{"title":6,"description":814},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Ftitanic",[823,824,825],"missing-data","one-hot-encoding","pipeline","JT8M33VOJ6EeHVrdY6FV3C4JD81tPXez0hpeK5txcbw",1787338984424]