Ensembles: the Wisdom of Crowds (and Its Limits)

Lecture 6, and the professor leaves "one model" behind for the first time in the course: instead of picking the best classifier, he trains several and lets them vote. The dataset changes shape too: real face recognition.

The dataset: faces, not numbers

fetch_olivetti_faces brings 400 photos (64×64 pixels, so each photo turns into a vector of 4096 numbers, one per pixel) of 40 different people, 10 photos each. The task is to say, from the photo, which of the 40 people it is. That's already a much harder problem than anything I've seen so far in this playlist: 40 classes (against 3 or 4 in earlier lectures) and 4096 input variables (against 2, 6, 13, or 30).

Three different guesses, one vote

The professor trains three very different classifiers on the same normalized data:

None crosses 88%. But instead of picking the best of the three, he combines all three predictions and votes for the most common one:

hits = np.stack((knn_hits, gnb_hits, ppn_hits))
y_pred = np.stack((knn_pred, gnb_pred, ppn_pred))

from scipy.stats import mode
y_pred_mode, _ = mode(y_pred, axis=0)
vote_hits = y_pred_mode == y_test

Output: 0.9625. Better than any of the three alone, by a wide margin.

Bishop calls this a committee (chapter 14.2), and the math behind it is neat: if each model's errors are uncorrelated (one makes mistakes in a different place than the other), the committee's average error drops by a factor of $1/M$ ($M$ = number of models) compared to each model's own average error. $1/3$ of the error is too optimistic to happen in practice (Bishop himself warns: "in practice the errors are typically highly correlated, and the reduction in overall error is generally small"), but the core idea holds: where KNN misses, the Perceptron sometimes gets it right, and vice versa, so the majority tends to be right even when any one of the three individually is wrong.

Scikit-learn's VotingClassifier runs exactly this computation, and matches the same 0.9625, confirming the hand-rolled version is correct.

The catch: voting doesn't help if everyone thinks alike

Here comes the most important experiment in the post. First, the professor swaps the three different models for three KNNs with different K (1, 3, 5):

Output: 0.90. Worse than the three different models (0.9625).

Then something even more revealing. A single decision tree (no limit, nothing special): 0.50, pretty weak on this 4096-variable dataset. What if I train 10 copies of that same tree and have them vote?

base_estimators = [(f"dtc({i})", DecisionTreeClassifier()) for i in range(10)]
voting_clf = VotingClassifier(base_estimators, voting='hard')

Output: 0.50. Exactly the same as the single tree.

That makes complete sense, and it's exactly what Bishop warned about: scikit-learn's default DecisionTreeClassifier() is deterministic, it always picks the same optimal question to split each group. Ten copies of the same algorithm, on the same data, always land on the same tree, always miss the exact same examples. There's no "majority" when everyone votes identically: 10 identical votes are worth the same as 1. A committee only works when its members genuinely disagree, and disagreement requires some real source of variation between the models.

The professor introduces that variation by swapping how the split gets picked from 'best' (always the optimal question) to 'random' (the tree picks a good question, but at random, among the candidates):

Output: a single random tree: 0.6125, already better than the deterministic one alone. Ten random trees voting: 0.7875, a serious jump.

Now every tree is genuinely different from the others (real randomness in picking the question), so the errors stop being identical, and voting starts genuinely paying off.

Bagging, Random Forest, and Extra Trees: organized randomness

BaggingClassifier generalizes that idea: it trains several copies of the same model, each one seeing a random sample with replacement of the training data (bootstrap), so each tree sees a slightly different set, on top of picking different splits:

Output: BaggingClassifier with 100 random trees: 0.9125.

RandomForestClassifier is essentially that same recipe pre-packaged (bootstrap + trees, but also sampling a random subset of the variables at every split, not just shuffling which examples each tree sees):

Output: Random Forest, 100 trees: 0.9125, tied with manual bagging.

And ExtraTreesClassifier goes one step further, also randomizing the threshold of each split instead of searching for the optimal one (even more randomness):

Output: 0.9625, the best among all the forests tested, tying the original vote of three different models.

These numbers aren't just illustration, they confirm the previous section's thesis: what makes a committee worth having is how much the trees disagree with each other. Bootstrap alone (BaggingClassifier) already samples which examples each tree sees, and also sampling which variable each tree can split on (RandomForestClassifier) ties that exactly, 0.9125 both, which suggests that on this dataset the bootstrap variation was already generating nearly all the useful disagreement, and forcing trees to look at different columns on top of that didn't add much more. Also randomizing the split threshold itself (ExtraTreesClassifier), not just which examples and which variables, but literally the value that separates "yes" from "no" at each split, is a stronger source of disagreement: two trees can see the same data, the same variables, and still cut in quite different places. That's why the jump from 0.9125 to 0.9625 comes specifically from ExtraTrees: more randomness in how the trees get built, more genuinely different trees, more chance their mistakes don't line up.

Boosting: learning from the last mistake

AdaBoostClassifier changes the strategy: instead of training everyone in parallel and voting, it trains in sequence, and each new model gives more weight to the examples the previous model got wrong. Bishop describes the algorithm formally: every example carries a weight $w_n$, starting equal for everyone. After each classifier is trained, the weight of the examples it missed increases (multiplied by $e^{\alpha_m}$), so the next classifier in the sequence is forced to pay more attention to them. In the end, the final prediction is a weighted vote, where more accurate classifiers (larger $\alpha_m$) count for more than the weak ones.

Output: AdaBoost with deep random trees: 0.925 on test, 1.0 on training.

100% on training is a sign that boosting, given a weak enough base and enough rounds, can also memorize training (the same overfitting I already saw happen with an unbounded-depth tree), but the test result still comes out much better than any individual tree alone.

Stacking: fancier doesn't always win

StackingClassifier tries to be smarter than voting: instead of counting votes, it trains an extra model (the "meta-model") that learns to combine the base models' predictions, using cross-validation internally so the training data doesn't leak straight into the meta-model.

Output: stacking the three original models (KNN, Naive Bayes, Perceptron): 0.80. Stacking of voting + Random Forest + Extra Trees: 0.7625.

Worse than the simple vote of the same models (0.9625) both times the professor tried it. Worth stating plainly: the fancier technique did not win here. With few training examples per class (10 photos per person, and stacking's cv=3 still carves off a slice of that just to train the meta-model), there simply isn't enough data left for that meta-model to learn a combination better than "everyone gets one vote."

The humbling twist: a simple model beats everyone

After all that ensembling, the professor tries plain LogisticRegression, no committee at all:

Output: 0.975 on test. The best result in the entire post, beating every ensemble tried, including Extra Trees (0.9625) and AdaBoost (0.925).

And to make sure it wasn't luck from a single test split, 5-fold cross-validation confirms it: 0.965 average (1.0, 0.95, 0.95, 0.975, 0.95). The professor still tries tuning hyperparameters with Optuna (50 trials, searching AdaBoost's tree depth, learning rate, and number of estimators), and the best result found after all that is 0.9125 on test, still behind raw logistic regression.

Why does logistic regression win here? A good guess: with 4096 variables (pixels) and only 400 examples, the problem is already nearly linear in practice, images of the same person form a "cloud" that a hyperplane separates reasonably well, and there isn't much genuinely non-linear pattern left for trees and forests to exploit. Tree ensembles shine when the true pattern is genuinely complex and non-linear (like in the previous post, with Car Evaluation). Here, the problem already had a simple, effective way to be solved, and the plain tool beat the fancy one.

Wrapping up

The lesson that sticks: an ensemble is a powerful tool, not a silver bullet. Before stacking models, it's worth checking whether the problem already has a simple, effective solution, because sometimes it does.

Practical application

The whole post compared ensembles against LogisticRegression using only the single test split the notebook had already run. To close with more confidence, I ran real 5-fold cross-validation on the post's three best models (Random Forest, Extra Trees, and logistic regression), all under the same criterion, something the original notebook only did for logistic regression.

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
    print(name, scores.mean(), scores.std())

Logistic regression's win wasn't luck from the notebook's specific split: it wins across all 5 folds tested, and on top of that with the lowest standard deviation of the three, meaning the most consistent result fold to fold. That confirms the previous section's lesson numerically, with the extra care of measuring everyone by the same ruler.