The Titanic: My First Genuinely Messy Dataset

Lecture 7, and the professor switches gears: every dataset up to now already arrived ready, no gaps, no useless column, no loose text. The Titanic (Kaggle's most famous dataset, probably everyone's first data science project) doesn't get that luxury.

Messy data: 891 passengers, not every column is useful

df = pd.read_csv('train.csv')
y = df['Survived']
X = df.drop('Survived', axis=1)

891 passengers, Survived (0 or 1) as the target, and a handful of pretty mixed input columns: Pclass (ticket class), Name, Sex, Age, SibSp (siblings/spouse aboard), Parch (parents/children aboard), Ticket, Fare, Cabin, Embarked (port of embarkation). The professor drops four of them right away:

caracteristicas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']

PassengerId is just a sequential number, carries no information about the person at all. Name and Ticket are free text, almost every value is unique (891 different names for 891 passengers), so a model has no way to generalize from that without processing the text first (extracting a title, like "Mr."/"Mrs.", would be a valid move, but that's extra work the lecture didn't do). Cabin is missing for most rows and is also far too specific.

Missing data is the rule, not the exception

for col in Xnum.columns:
    print(f"{col:>12} {Xnum[col].isnull().sum():2}")

Output: Age is missing for 177 of 891 passengers (almost 20%). Embarked is missing for 2.

That never happened in this playlist before now: every previous dataset already arrived complete. Here, nearly 1 in every 5 ages is NaN. Dropping those rows would throw away 20% of the data, and dropping the Age column entirely would throw away a probably important variable (children had rescue priority). The professor's solution is imputation: fill the gap with an estimated value.

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='median')
XnumTratado = imputer.fit_transform(Xnum)

For numeric variables, the median (not the mean): the age and fare (Fare) distributions have people well outside the norm (infants a few months old, very expensive first-class fares), and the median doesn't get pulled by those extremes the way the mean does. For categorical variables (Embarked), there's no such thing as a "median" port, so the strategy becomes 'most_frequent', whichever value shows up most.

A category isn't a number, even when it looks like one

Sex and Embarked are text ("male"/"female", "S"/"C"/"Q"). A model only understands numbers, but simply numbering them (male=0, female=1, or S=0, C=1, Q=2) would invent an order that doesn't exist: why would "Q" be "bigger" than "S"? The right move is one-hot encoding: every category becomes its own binary column.

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
XcatTratadoHot = encoder.fit_transform(XcatTratado)

Output: the 2 categorical columns (Sex, Embarked) turn into 5 binary columns (2 for sex, since one is already redundant with the other, and 3 for port of embarkation). None of them carries an implicit numeric order, they're just "is" or "isn't" that category.

My own Transformer: not everything is a classifier

Up to now, every custom class in this playlist inherited from ClassifierMixin or RegressorMixin, because it always ended in .predict(). Here the goal is different: just transform the data, without predicting anything. That's what TransformerMixin is for:

from sklearn.base import BaseEstimator, TransformerMixin

class AtributosDesejados(BaseEstimator, TransformerMixin):
    def __init__(self):
        self.colunas_indesejadas = ['PassengerId', 'Name', 'Ticket', 'Cabin']
    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        return X.drop(self.colunas_indesejadas, axis=1)

Along with AtributosNumericos (keeps only the numeric columns) and AtributosCategoricos (keeps only the categorical ones), the professor builds two parallel Pipelines, one for each kind of data:

pipenum = Pipeline([
    ('atributos_numericos', AtributosNumericos()),
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])
pipecat = Pipeline([
    ('atributos_categoricos', AtributosCategoricos()),
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder())
])

And FeatureUnion joins both outputs into one set of columns, side by side:

from sklearn.pipeline import FeatureUnion

unecaracteristicas = FeatureUnion([
    ('pipenum', pipenum),
    ('pipecat', pipecat)
])

Output: the 7 original columns (after dropping the 4 useless ones) become 10 processed columns: 5 numeric (already imputed and normalized) plus 5 categorical (already imputed and one-hot encoded).

The final pipeline chains everything, from raw data to classifier, in a single .fit() call:

preproc = Pipeline([
    ('atributos_desejados', AtributosDesejados()),
    ('unecaracteristicas', unecaracteristicas),
    ('to_dense', DenseTransformer())
])

clf = Pipeline([
    ('preproc', preproc),
    ('classificador', RandomForestClassifier())
])
clf.fit(X, y)

The same old mistake, with a real consequence this time

y_pred = clf.predict(X)
accuracy_score(y, y_pred)

Output: 0.9798. Almost 98% correct.

If you already read this playlist's post on cross-validation, that accuracy number should set off an alarm: it's measured on the same data it trained on. The proof it's memorization, not learning:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, y)

Output: 0.807 average (0.765, 0.815, 0.854, 0.775, 0.826 across the 5 folds).

A nearly 17-percentage-point drop between "accuracy on training" and "honest accuracy." The Random Forest, with no tree limit at all, memorizes a good chunk of the 891 passengers individually, and that 98% figure never measured the ability to generalize, it measured the ability to memorize.

Trying to beat 80%: stacking and a neural network

The professor tries two things to improve on the honest 0.807. First, StackingClassifier with 8 quite different models at once (Random Forest, Extra Trees, a neural network, SGD, Ridge, KNN, Naive Bayes, logistic regression):

Output: 0.824 cross-validation. A real, if small, improvement.

Then, a neural network alone (MLPClassifier, short for Multi-Layer Perceptron, a Perceptron I already saw earlier in this playlist stacked into several layers, the subject of its own course that isn't the focus here):

Output: 0.822 cross-validation, practically tied with the 8-model stack.

Neither is a huge leap over the lone forest's 0.807, but both beat the simpler version consistently. Unlike the previous post (where a simple model beat every ensemble tried), here the extra complexity genuinely helped, a little. The earlier lesson still holds: there's no knowing which one wins without measuring both by the same ruler.

Wrapping up

Practical application

I reproduced the entire pipeline with a fixed seed (the original notebook fixes none), to get a stable number and compare all three approaches under the exact same cross-validation criterion.

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores_rf = cross_val_score(clf_rf, X, y, cv=cv)
scores_stacking = cross_val_score(clf_stacking, X, y, cv=cv)
scores_mlp = cross_val_score(clf_mlp, X, y, cv=cv)

The numbers land close to the original notebook's (0.807, 0.824, 0.822), confirming the difference between the three wasn't luck from one specific run. And notice the standard deviation: the stack didn't just have the best average, it's also the most stable fold to fold (0.0077 versus the lone forest's 0.0268), a sign that combining several different models cushioned some of the variation each one carries on its own.