Fraud Detection: When 99.8% Accuracy Means Nothing

Lecture 12, and the dataset (Kaggle's Credit Card Fraud Detection, real, anonymized European credit card transactions) is too large and gated for me to download and reproduce myself in this environment, with no Kaggle account. This post leans on the numbers the notebook itself already ran (genuinely executed, real outputs), and the practical application section rebuilds the lecture's most important finding on a synthetic dataset I can generate and verify on the spot.

The dataset: 492 frauds in nearly 285 thousand transactions

df = pd.read_csv('creditcard.csv')
print(df['Class'].value_counts())

Output: class 0 (normal transaction): 284315. Class 1 (fraud): 492.

That's 0.17% fraud. The input variables already arrive PCA-transformed (V1 through V28, no original name, for the bank's privacy), plus Time and Amount untransformed. This extreme imbalance is the whole post's subject.

The baseline that exposes accuracy's lie

class ZeroR(BaseEstimator, TransformerMixin):
    def fit(self, X, y):
        self.most_frequent_class_ = y.value_counts().idxmax()
        return self
    def transform(self, X):
        return [self.most_frequent_class_] * len(X)

model = ZeroR()
model.fit(X_train, y_train)
print(accuracy_score(y_test, model.transform(X_test)))

Output: 0.9983. Always guessing "not fraud," never looking at a single variable, the dumbest possible model gets 99.83% right.

That's the most extreme baseline that's shown up in this playlist so far (Car Evaluation, in the decision trees post, had 70%, here it's nearly 100%). Any headline claiming "fraud model with 99% accuracy" needs this ruler standing next to it, because without it the number says nothing.

A real model, and two curves better than accuracy

model = LogisticRegression(tol=0.005)
model.fit(X_train, y_train)

Output: confusion matrix [[56832, 32], [26, 72]]. Precision 0.69, recall 0.73, F1 0.71 (for the fraud class). ROC AUC 0.867. Precision-recall curve AUC: 0.613.

Notice I don't even cite accuracy here, it would hide everything (98 frauds among nearly 57 thousand transactions, so any reasonable model already clears 99.8%+ accuracy). Precision and recall, though, tell the right story: of the transactions the model flagged as fraud, 69% really were. Of the real frauds, the model caught 73%.

And between the two curves, the precision-recall curve (AUC 0.613) is more honest than the ROC curve (AUC 0.867) for this kind of problem. The ROC curve uses false positive rate on its axis, which is false positives / total negatives, and with nearly 57 thousand negatives, even a handful of false positives becomes a tiny fraction, the curve looks great almost for free. The precision-recall curve, by contrast, uses precision on its axis, which is true positives / (true positives + false positives), directly sensitive to how many false positives exist compared to the few real frauds, with no dilution in the sea of negatives.

Resampling: the remedy that makes things worse

The most common idea for handling imbalance is resampling the training set, one way or another, to even out the classes:

Across all four techniques, recall climbs (from 0.73 to ~0.90), the model catches more real frauds. But precision craters (from 0.69 to 0.03-0.07), the model starts screaming "fraud!" at a lot of normal transactions too. F1 (which balances the two) gets notably worse in all four cases. Resampling isn't a silver bullet, it's a trade-off, and on this specific dataset the trade-off loses.

The reason precision crashes this badly is a mix of two simple things. First, resampling only touches the training set: the model learns in an artificially balanced world (close to 50/50), but keeps getting tested against the real world, where fraud is 0.17% of transactions. Second, with so many normal transactions in the test set (nearly 57 thousand), even a small error rate on them turns into a huge number of cases in absolute terms: if the model, calibrated for a world where fraud is common, starts getting "suspicious" of anything that looks even a little like fraud, even a 1-2% false-positive rate against 57 thousand normal transactions already generates hundreds of false alarms, far more than the handful of real frauds there are to catch. It's that mismatch between "how many normal transactions exist" and "how many false positives the model now makes" that sends precision off a cliff.

The wrong way to resample (and why it fools you)

The notebook has a section with the most direct title in the whole course: "WRONG Approach, don't do it this way!".

smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X, y)  # resamples BEFORE splitting train/test

X_train_resampled, X_test_resampled, y_train_resampled, y_test_resampled = train_test_split(
    X_resampled, y_resampled, test_size=0.2, random_state=42)
model.fit(X_train_resampled, y_train_resampled)

Output: precision 0.98, recall 0.97, F1 0.97. An impressive result.

Impressive and invalid. SMOTE was called on the entire dataset, before splitting train and test. Since SMOTE creates a synthetic example by interpolating between real neighbors of the minority class, some of the synthetic examples that end up in "training" after the split are nearly identical to real examples that ended up in "test." The model isn't generalizing to unseen data, it's recognizing near-identical copies of what it already trained on, the same kind of leakage I already saw before, just hiding inside a resampling technique this time instead of a misplaced fit_transform.

The right way: resampling inside the pipeline

The fix is resampling after each cross-validation split, never before, exactly like any normalization or feature selection should be done:

from imblearn.pipeline import Pipeline
from sklearn.model_selection import cross_validate, StratifiedKFold

pipe = Pipeline([
    ('sampling', SMOTE(random_state=42)),
    ('model', LogisticRegression(tol=0.005))
])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(pipe, X, y, cv=cv, scoring=['precision', 'recall', 'f1'])

Output: mean precision ≈ 0.07, mean recall ≈ 0.89, mean F1 ≈ 0.14, across the 5 folds.

Much closer to plain SMOTE's honest result (F1 0.12) than to the inflated 0.97 from the wrong approach. Notice the technical detail: this is imblearn's Pipeline, not scikit-learn's own. The regular Pipeline only accepts steps that transform X, but SMOTE needs to touch X and y together (it creates new rows in both), so it needs a pipeline variant that knows how to propagate that size change downstream.

One bonus noted in passing: adding a StandardScaler before SMOTE in the pipeline doesn't change the result much, but makes fitting much faster (from around 15-19 seconds per fold to 2-3 seconds). That tracks: SMOTE needs to find nearest neighbors to interpolate to create every synthetic example, and searching for neighbors on unnormalized data, with variables at very different scales, is more expensive to compute.

Wrapping up

Practical application

I can't download the real fraud dataset here (needs a Kaggle login), so I rebuilt the lecture's most important finding, SMOTE-before-split leakage, on a synthetic dataset I control and can verify: 20 thousand examples, 2% positive class (scikit-learn's make_classification, with a fraction of flipped labels on purpose so it isn't too clean).

from sklearn.datasets import make_classification
X, y = make_classification(n_samples=20000, weights=[0.98, 0.02], flip_y=0.01, random_state=42)

(ZeroR, always guessing "not fraud," hits 0.975 accuracy while detecting zero frauds, the same dumb baseline as always, now at a smaller scale.)

The gap between "wrong" and "right" here is even more dramatic than on the real fraud dataset: an F1 of 0.796 (looks excellent) versus 0.154 (the honest number), just by changing when SMOTE runs. Same conclusion as the lecture, confirmed on a dataset I built from scratch: the leak from resampling before splitting train and test isn't a theoretical footnote, it inflates the result enough to turn a mediocre model into one that looks production-ready without being one.