[{"data":1,"prerenderedAt":738},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud":3,"post-en-pattern-recognition-credit-card-fraud":4},"\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud",{"id":5,"title":6,"body":7,"cover":722,"date":723,"description":724,"extension":725,"meta":726,"navigation":132,"order":727,"path":728,"playlist":729,"seo":730,"status":731,"stem":732,"tags":733,"__hash__":737},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud.md","Fraud Detection: When 99.8% Accuracy Means Nothing",{"type":8,"value":9,"toc":712},"minimark",[10,22,27,51,61,84,88,152,167,175,179,193,204,207,218,222,225,347,354,361,365,372,406,413,431,435,442,491,498,528,549,553,601,605,612,627,694,701,708],[11,12,13,14,21],"p",{},"Lecture 12, and the dataset (Kaggle's ",[15,16,20],"a",{"href":17,"rel":18},"https:\u002F\u002Fwww.kaggle.com\u002Fdatasets\u002Fmlg-ulb\u002Fcreditcardfraud",[19],"nofollow","Credit Card Fraud Detection",", real, anonymized European credit card transactions) is too large and gated for me to download and reproduce myself in this environment, with no Kaggle account. This post leans on the numbers the notebook itself already ran (genuinely executed, real outputs), and the practical application section rebuilds the lecture's most important finding on a synthetic dataset I can generate and verify on the spot.",[23,24,26],"h2",{"id":25},"the-dataset-492-frauds-in-nearly-285-thousand-transactions","The dataset: 492 frauds in nearly 285 thousand transactions",[28,29,34],"pre",{"className":30,"code":31,"language":32,"meta":33,"style":33},"language-python shiki shiki-themes github-light github-dark","df = pd.read_csv('creditcard.csv')\nprint(df['Class'].value_counts())\n","python","",[35,36,37,45],"code",{"__ignoreMap":33},[38,39,42],"span",{"class":40,"line":41},"line",1,[38,43,44],{},"df = pd.read_csv('creditcard.csv')\n",[38,46,48],{"class":40,"line":47},2,[38,49,50],{},"print(df['Class'].value_counts())\n",[52,53,54],"blockquote",{},[11,55,56,60],{},[57,58,59],"strong",{},"Output:"," class 0 (normal transaction): 284315. Class 1 (fraud): 492.",[11,62,63,64,67,68,71,72,75,76,79,80,83],{},"That's ",[57,65,66],{},"0.17%"," fraud. The input variables already arrive PCA-transformed (",[35,69,70],{},"V1"," through ",[35,73,74],{},"V28",", no original name, for the bank's privacy), plus ",[35,77,78],{},"Time"," and ",[35,81,82],{},"Amount"," untransformed. This extreme imbalance is the whole post's subject.",[23,85,87],{"id":86},"the-baseline-that-exposes-accuracys-lie","The baseline that exposes accuracy's lie",[28,89,91],{"className":30,"code":90,"language":32,"meta":33,"style":33},"class ZeroR(BaseEstimator, TransformerMixin):\n    def fit(self, X, y):\n        self.most_frequent_class_ = y.value_counts().idxmax()\n        return self\n    def transform(self, X):\n        return [self.most_frequent_class_] * len(X)\n\nmodel = ZeroR()\nmodel.fit(X_train, y_train)\nprint(accuracy_score(y_test, model.transform(X_test)))\n",[35,92,93,98,103,109,115,121,127,134,140,146],{"__ignoreMap":33},[38,94,95],{"class":40,"line":41},[38,96,97],{},"class ZeroR(BaseEstimator, TransformerMixin):\n",[38,99,100],{"class":40,"line":47},[38,101,102],{},"    def fit(self, X, y):\n",[38,104,106],{"class":40,"line":105},3,[38,107,108],{},"        self.most_frequent_class_ = y.value_counts().idxmax()\n",[38,110,112],{"class":40,"line":111},4,[38,113,114],{},"        return self\n",[38,116,118],{"class":40,"line":117},5,[38,119,120],{},"    def transform(self, X):\n",[38,122,124],{"class":40,"line":123},6,[38,125,126],{},"        return [self.most_frequent_class_] * len(X)\n",[38,128,130],{"class":40,"line":129},7,[38,131,133],{"emptyLinePlaceholder":132},true,"\n",[38,135,137],{"class":40,"line":136},8,[38,138,139],{},"model = ZeroR()\n",[38,141,143],{"class":40,"line":142},9,[38,144,145],{},"model.fit(X_train, y_train)\n",[38,147,149],{"class":40,"line":148},10,[38,150,151],{},"print(accuracy_score(y_test, model.transform(X_test)))\n",[52,153,154],{},[11,155,156,158,159,162,163,166],{},[57,157,59],{}," ",[35,160,161],{},"0.9983",". Always guessing \"not fraud,\" never looking at a single variable, the dumbest possible model gets ",[57,164,165],{},"99.83%"," right.",[11,168,169,170,174],{},"That's the most extreme baseline that's shown up in this playlist so far (",[15,171,173],{"href":172},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdecision-trees","Car Evaluation, in the decision trees post, had 70%",", here it's nearly 100%). Any headline claiming \"fraud model with 99% accuracy\" needs this ruler standing next to it, because without it the number says nothing.",[23,176,178],{"id":177},"a-real-model-and-two-curves-better-than-accuracy","A real model, and two curves better than accuracy",[28,180,182],{"className":30,"code":181,"language":32,"meta":33,"style":33},"model = LogisticRegression(tol=0.005)\nmodel.fit(X_train, y_train)\n",[35,183,184,189],{"__ignoreMap":33},[38,185,186],{"class":40,"line":41},[38,187,188],{},"model = LogisticRegression(tol=0.005)\n",[38,190,191],{"class":40,"line":47},[38,192,145],{},[52,194,195],{},[11,196,197,199,200,203],{},[57,198,59],{}," confusion matrix ",[35,201,202],{},"[[56832, 32], [26, 72]]",". Precision 0.69, recall 0.73, F1 0.71 (for the fraud class). ROC AUC 0.867. Precision-recall curve AUC: 0.613.",[11,205,206],{},"Notice I don't even cite accuracy here, it would hide everything (98 frauds among nearly 57 thousand transactions, so any reasonable model already clears 99.8%+ accuracy). Precision and recall, though, tell the right story: of the transactions the model flagged as fraud, 69% really were. Of the real frauds, the model caught 73%.",[11,208,209,210,213,214,217],{},"And between the two curves, the precision-recall curve (AUC 0.613) is more honest than the ROC curve (AUC 0.867) for this kind of problem. The ROC curve uses false positive rate on its axis, which is ",[35,211,212],{},"false positives \u002F total negatives",", and with nearly 57 thousand negatives, even a handful of false positives becomes a tiny fraction, the curve looks great almost for free. The precision-recall curve, by contrast, uses precision on its axis, which is ",[35,215,216],{},"true positives \u002F (true positives + false positives)",", directly sensitive to how many false positives exist compared to the few real frauds, with no dilution in the sea of negatives.",[23,219,221],{"id":220},"resampling-the-remedy-that-makes-things-worse","Resampling: the remedy that makes things worse",[11,223,224],{},"The most common idea for handling imbalance is resampling the training set, one way or another, to even out the classes:",[226,227,228,252],"table",{},[229,230,231],"thead",{},[232,233,234,239,242,246,249],"tr",{},[235,236,238],"th",{"align":237},"left","Technique",[235,240,241],{"align":237},"How it works",[235,243,245],{"align":244},"right","Precision",[235,247,248],{"align":244},"Recall",[235,250,251],{"align":244},"F1",[253,254,255,273,292,310,329],"tbody",{},[232,256,257,261,264,267,270],{},[258,259,260],"td",{"align":237},"None (baseline)",[258,262,263],{"align":237},"-",[258,265,266],{"align":244},"0.69",[258,268,269],{"align":244},"0.73",[258,271,272],{"align":244},"0.71",[232,274,275,280,283,286,289],{},[258,276,277],{"align":237},[35,278,279],{},"RandomOverSampler",[258,281,282],{"align":237},"duplicates minority-class examples",[258,284,285],{"align":244},"0.04",[258,287,288],{"align":244},"0.92",[258,290,291],{"align":244},"0.08",[232,293,294,299,302,305,307],{},[258,295,296],{"align":237},[35,297,298],{},"RandomUnderSampler",[258,300,301],{"align":237},"discards majority-class examples",[258,303,304],{"align":244},"0.03",[258,306,288],{"align":244},[258,308,309],{"align":244},"0.06",[232,311,312,317,320,323,326],{},[258,313,314],{"align":237},[35,315,316],{},"SMOTE",[258,318,319],{"align":237},"creates synthetic examples interpolating minority neighbors",[258,321,322],{"align":244},"0.07",[258,324,325],{"align":244},"0.91",[258,327,328],{"align":244},"0.12",[232,330,331,336,339,341,344],{},[258,332,333],{"align":237},[35,334,335],{},"NearMiss",[258,337,338],{"align":237},"discards majority examples near the boundary",[258,340,304],{"align":244},[258,342,343],{"align":244},"0.90",[258,345,346],{"align":244},"0.05",[11,348,349,350,353],{},"Across all four techniques, recall climbs (from 0.73 to ~0.90), the model catches more real frauds. But precision ",[57,351,352],{},"craters"," (from 0.69 to 0.03-0.07), the model starts screaming \"fraud!\" at a lot of normal transactions too. F1 (which balances the two) gets notably worse in all four cases. Resampling isn't a silver bullet, it's a trade-off, and on this specific dataset the trade-off loses.",[11,355,356,357,360],{},"The reason precision crashes this badly is a mix of two simple things. First, resampling only touches the ",[57,358,359],{},"training"," set: the model learns in an artificially balanced world (close to 50\u002F50), but keeps getting tested against the real world, where fraud is 0.17% of transactions. Second, with so many normal transactions in the test set (nearly 57 thousand), even a small error rate on them turns into a huge number of cases in absolute terms: if the model, calibrated for a world where fraud is common, starts getting \"suspicious\" of anything that looks even a little like fraud, even a 1-2% false-positive rate against 57 thousand normal transactions already generates hundreds of false alarms, far more than the handful of real frauds there are to catch. It's that mismatch between \"how many normal transactions exist\" and \"how many false positives the model now makes\" that sends precision off a cliff.",[23,362,364],{"id":363},"the-wrong-way-to-resample-and-why-it-fools-you","The wrong way to resample (and why it fools you)",[11,366,367,368,371],{},"The notebook has a section with the most direct title in the whole course: ",[57,369,370],{},"\"WRONG Approach, don't do it this way!\"",".",[28,373,375],{"className":30,"code":374,"language":32,"meta":33,"style":33},"smote = SMOTE(random_state=42)\nX_resampled, y_resampled = smote.fit_resample(X, y)  # resamples BEFORE splitting train\u002Ftest\n\nX_train_resampled, X_test_resampled, y_train_resampled, y_test_resampled = train_test_split(\n    X_resampled, y_resampled, test_size=0.2, random_state=42)\nmodel.fit(X_train_resampled, y_train_resampled)\n",[35,376,377,382,387,391,396,401],{"__ignoreMap":33},[38,378,379],{"class":40,"line":41},[38,380,381],{},"smote = SMOTE(random_state=42)\n",[38,383,384],{"class":40,"line":47},[38,385,386],{},"X_resampled, y_resampled = smote.fit_resample(X, y)  # resamples BEFORE splitting train\u002Ftest\n",[38,388,389],{"class":40,"line":105},[38,390,133],{"emptyLinePlaceholder":132},[38,392,393],{"class":40,"line":111},[38,394,395],{},"X_train_resampled, X_test_resampled, y_train_resampled, y_test_resampled = train_test_split(\n",[38,397,398],{"class":40,"line":117},[38,399,400],{},"    X_resampled, y_resampled, test_size=0.2, random_state=42)\n",[38,402,403],{"class":40,"line":123},[38,404,405],{},"model.fit(X_train_resampled, y_train_resampled)\n",[52,407,408],{},[11,409,410,412],{},[57,411,59],{}," precision 0.98, recall 0.97, F1 0.97. An impressive result.",[11,414,415,416,419,420,422,423,427,428,371],{},"Impressive and ",[57,417,418],{},"invalid",". ",[35,421,316],{}," was called on the entire dataset, before splitting train and test. Since SMOTE creates a synthetic example by interpolating between real neighbors of the minority class, some of the synthetic examples that end up in \"training\" after the split are nearly identical to real examples that ended up in \"test.\" The model isn't generalizing to unseen data, it's recognizing near-identical copies of what it already trained on, ",[15,424,426],{"href":425},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fpipeline-cross-validation","the same kind of leakage I already saw before",", just hiding inside a resampling technique this time instead of a misplaced ",[35,429,430],{},"fit_transform",[23,432,434],{"id":433},"the-right-way-resampling-inside-the-pipeline","The right way: resampling inside the pipeline",[11,436,437,438,441],{},"The fix is resampling ",[57,439,440],{},"after"," each cross-validation split, never before, exactly like any normalization or feature selection should be done:",[28,443,445],{"className":30,"code":444,"language":32,"meta":33,"style":33},"from imblearn.pipeline import Pipeline\nfrom sklearn.model_selection import cross_validate, StratifiedKFold\n\npipe = Pipeline([\n    ('sampling', SMOTE(random_state=42)),\n    ('model', LogisticRegression(tol=0.005))\n])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nscores = cross_validate(pipe, X, y, cv=cv, scoring=['precision', 'recall', 'f1'])\n",[35,446,447,452,457,461,466,471,476,481,486],{"__ignoreMap":33},[38,448,449],{"class":40,"line":41},[38,450,451],{},"from imblearn.pipeline import Pipeline\n",[38,453,454],{"class":40,"line":47},[38,455,456],{},"from sklearn.model_selection import cross_validate, StratifiedKFold\n",[38,458,459],{"class":40,"line":105},[38,460,133],{"emptyLinePlaceholder":132},[38,462,463],{"class":40,"line":111},[38,464,465],{},"pipe = Pipeline([\n",[38,467,468],{"class":40,"line":117},[38,469,470],{},"    ('sampling', SMOTE(random_state=42)),\n",[38,472,473],{"class":40,"line":123},[38,474,475],{},"    ('model', LogisticRegression(tol=0.005))\n",[38,477,478],{"class":40,"line":129},[38,479,480],{},"])\n",[38,482,483],{"class":40,"line":136},[38,484,485],{},"cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",[38,487,488],{"class":40,"line":142},[38,489,490],{},"scores = cross_validate(pipe, X, y, cv=cv, scoring=['precision', 'recall', 'f1'])\n",[52,492,493],{},[11,494,495,497],{},[57,496,59],{}," mean precision ≈ 0.07, mean recall ≈ 0.89, mean F1 ≈ 0.14, across the 5 folds.",[11,499,500,501,504,505,508,509,511,512,515,516,518,519,158,521,158,524,527],{},"Much closer to plain SMOTE's honest result (F1 0.12) than to the inflated 0.97 from the wrong approach. Notice the technical detail: this is ",[35,502,503],{},"imblearn","'s ",[35,506,507],{},"Pipeline",", not scikit-learn's own. The regular ",[35,510,507],{}," only accepts steps that transform ",[35,513,514],{},"X",", but ",[35,517,316],{}," needs to touch ",[35,520,514],{},[57,522,523],{},"and",[35,525,526],{},"y"," together (it creates new rows in both), so it needs a pipeline variant that knows how to propagate that size change downstream.",[11,529,530,531,534,535,537,538,541,542,158,544,548],{},"One bonus noted in passing: adding a ",[35,532,533],{},"StandardScaler"," before ",[35,536,316],{}," in the pipeline doesn't change the result much, but makes fitting ",[57,539,540],{},"much faster"," (from around 15-19 seconds per fold to 2-3 seconds). That tracks: ",[35,543,316],{},[15,545,547],{"href":546},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier","needs to find nearest neighbors to interpolate"," to create every synthetic example, and searching for neighbors on unnormalized data, with variables at very different scales, is more expensive to compute.",[23,550,552],{"id":551},"wrapping-up","Wrapping up",[226,554,555,565],{},[229,556,557],{},[232,558,559,562],{},[235,560,561],{"align":237},"What I already knew",[235,563,564],{"align":237},"What this lecture settled",[253,566,567,575,583],{},[232,568,569,572],{},[258,570,571],{"align":237},"A dumb baseline helps interpret accuracy",[258,573,574],{"align":237},"With extreme imbalance (0.17% fraud), accuracy practically loses meaning, and precision\u002Frecall\u002FPR curve have to take its place",[232,576,577,580],{},[258,578,579],{"align":237},"Leakage happens when the same data influences training and evaluation",[258,581,582],{"align":237},"Resampling before splitting train\u002Ftest is a leak just as serious as normalizing wrong, just easier to miss",[232,584,585,590],{},[258,586,587,589],{"align":237},[35,588,507],{}," prevents leakage between steps",[258,591,592,595,596,598,599],{"align":237},[35,593,594],{},"imblearn.Pipeline"," extends the same idea to techniques that also change ",[35,597,526],{},", not just ",[35,600,514],{},[23,602,604],{"id":603},"practical-application","Practical application",[11,606,607,608,611],{},"I can't download the real fraud dataset here (needs a Kaggle login), so I rebuilt the lecture's most important finding, SMOTE-before-split leakage, on a synthetic dataset I control and can verify: 20 thousand examples, 2% positive class (scikit-learn's ",[35,609,610],{},"make_classification",", with a fraction of flipped labels on purpose so it isn't too clean).",[28,613,615],{"className":30,"code":614,"language":32,"meta":33,"style":33},"from sklearn.datasets import make_classification\nX, y = make_classification(n_samples=20000, weights=[0.98, 0.02], flip_y=0.01, random_state=42)\n",[35,616,617,622],{"__ignoreMap":33},[38,618,619],{"class":40,"line":41},[38,620,621],{},"from sklearn.datasets import make_classification\n",[38,623,624],{"class":40,"line":47},[38,625,626],{},"X, y = make_classification(n_samples=20000, weights=[0.98, 0.02], flip_y=0.01, random_state=42)\n",[226,628,629,642],{},[229,630,631],{},[232,632,633,636,638,640],{},[235,634,635],{"align":237},"Approach",[235,637,245],{"align":244},[235,639,248],{"align":244},[235,641,251],{"align":244},[253,643,644,658,680],{},[232,645,646,649,652,655],{},[258,647,648],{"align":237},"Logistic regression, no resampling",[258,650,651],{"align":244},"1.000",[258,653,654],{"align":244},"0.190",[258,656,657],{"align":244},"0.319",[232,659,660,665,670,675],{},[258,661,662],{"align":237},[57,663,664],{},"SMOTE before split (wrong)",[258,666,667],{"align":244},[57,668,669],{},"0.801",[258,671,672],{"align":244},[57,673,674],{},"0.791",[258,676,677],{"align":244},[57,678,679],{},"0.796",[232,681,682,685,688,691],{},[258,683,684],{"align":237},"SMOTE inside the pipeline + cross-validation (right)",[258,686,687],{"align":244},"0.086",[258,689,690],{"align":244},"0.731",[258,692,693],{"align":244},"0.154",[11,695,696,697,700],{},"(",[35,698,699],{},"ZeroR",", always guessing \"not fraud,\" hits 0.975 accuracy while detecting zero frauds, the same dumb baseline as always, now at a smaller scale.)",[11,702,703,704,707],{},"The gap between \"wrong\" and \"right\" here is even more dramatic than on the real fraud dataset: an F1 of 0.796 (looks excellent) versus 0.154 (the honest number), just by changing ",[57,705,706],{},"when"," SMOTE runs. Same conclusion as the lecture, confirmed on a dataset I built from scratch: the leak from resampling before splitting train and test isn't a theoretical footnote, it inflates the result enough to turn a mediocre model into one that looks production-ready without being one.",[709,710,711],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":33,"searchDepth":47,"depth":47,"links":713},[714,715,716,717,718,719,720,721],{"id":25,"depth":47,"text":26},{"id":86,"depth":47,"text":87},{"id":177,"depth":47,"text":178},{"id":220,"depth":47,"text":221},{"id":363,"depth":47,"text":364},{"id":433,"depth":47,"text":434},{"id":551,"depth":47,"text":552},{"id":603,"depth":47,"text":604},null,"2026-08-20","Lecture 12: the professor uses Kaggle's real credit card fraud dataset (492 frauds in nearly 285 thousand transactions) to show why accuracy lies on imbalanced data, and a data leak so subtle the wrong result looks great.","md",{},13,"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud","pattern-recognition",{"title":6,"description":724},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Fcredit-card-fraud",[734,735,736],"imbalanced-data","fraud-detection","data-leakage","2ES8pEqXjfSmaMxYcphSQFy_exC4QiRqsgpwreAF2EY",1787338984174]