[{"data":1,"prerenderedAt":1259},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection":3,"post-en-pattern-recognition-gaussian-mixtures-anomaly-detection":4},"\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection",{"id":5,"title":6,"body":7,"cover":1242,"date":1243,"description":1244,"extension":1245,"meta":1246,"navigation":1247,"order":1248,"path":1249,"playlist":1250,"seo":1251,"status":1252,"stem":1253,"tags":1254,"__hash__":1258},"posts\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection.md","Gaussian Mixtures and Anomaly Detection: Clustering With Shape",{"type":8,"value":9,"toc":1232},"minimark",[10,20,25,28,32,62,121,530,609,737,740,753,757,760,771,778,782,785,805,808,812,821,828,947,958,984,1023,1026,1030,1040,1105,1127,1137,1141,1185,1189,1192,1217,1225,1228],[11,12,13,14,19],"p",{},"Lectures 11a and 11b. A direct continuation of ",[15,16,18],"a",{"href":17},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fkmeans","the K-means post",": what if the groups aren't round?",[21,22,24],"h2",{"id":23},"where-k-means-limps-again","Where K-means limps again",[11,26,27],{},"The professor generates a dataset on purpose: two stretched, rotated groups (multiplying the points by a rotation matrix), plus a much smaller third group set apart from the other two. K-means, even with a good initialization (centers picked by hand, close to the right spot), struggles: since it only ever sees \"distance to a center,\" it tends to cut the elongated groups into rounder pieces than they really are, because its boundary between two groups is always a straight line perpendicular to the line joining the centers, never an ellipse.",[21,29,31],{"id":30},"gaussian-mixtures-every-group-becomes-an-ellipse-not-a-point","Gaussian mixtures: every group becomes an ellipse, not a point",[33,34,39],"pre",{"className":35,"code":36,"language":37,"meta":38,"style":38},"language-python shiki shiki-themes github-light github-dark","from sklearn.mixture import GaussianMixture\ngm = GaussianMixture(n_components=3, n_init=10, random_state=42)\ngm.fit(X)\n","python","",[40,41,42,50,56],"code",{"__ignoreMap":38},[43,44,47],"span",{"class":45,"line":46},"line",1,[43,48,49],{},"from sklearn.mixture import GaussianMixture\n",[43,51,53],{"class":45,"line":52},2,[43,54,55],{},"gm = GaussianMixture(n_components=3, n_init=10, random_state=42)\n",[43,57,59],{"class":45,"line":58},3,[43,60,61],{},"gm.fit(X)\n",[11,63,64,65,69,70,73,74,120],{},"The core difference from K-means: instead of storing just one center per group, a ",[66,67,68],"strong",{},"Gaussian mixture"," stores an entire normal distribution per group, with its own mean ",[66,71,72],{},"and"," covariance matrix (which captures the shape, how elongated the group is and in which direction). The full model is a weighted sum of ",[43,75,78,100],{"className":76},[77],"katex",[43,79,82],{"className":80},[81],"katex-mathml",[83,84,86],"math",{"xmlns":85},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[87,88,89,96],"semantics",{},[90,91,92],"mrow",{},[93,94,95],"mi",{},"K",[97,98,95],"annotation",{"encoding":99},"application\u002Fx-tex",[43,101,105],{"className":102,"ariaHidden":104},[103],"katex-html","true",[43,106,109,114],{"className":107},[108],"base",[43,110],{"className":111,"style":113},[112],"strut","height:0.6833em;",[43,115,95],{"className":116,"style":119},[117,118],"mord","mathnormal","margin-right:0.0715em;"," Gaussians:",[11,122,123],{},[43,124,126,217],{"className":125},[77],[43,127,129],{"className":128},[81],[83,130,131],{"xmlns":85},[87,132,133,214],{},[90,134,135,137,142,146,149,152,171,179,183,187,189,191,194,202,205,212],{},[93,136,11],{},[138,139,141],"mo",{"stretchy":140},"false","(",[93,143,145],{"mathvariant":144},"bold","x",[138,147,148],{"stretchy":140},")",[138,150,151],{},"=",[153,154,155,158,169],"msubsup",{},[138,156,157],{},"∑",[90,159,160,163,165],{},[93,161,162],{},"k",[138,164,151],{},[166,167,168],"mn",{},"1",[93,170,95],{},[172,173,174,177],"msub",{},[93,175,176],{},"π",[93,178,162],{},[180,181,182],"mtext",{}," ",[93,184,186],{"mathvariant":185},"script","N",[138,188,141],{"stretchy":140},[93,190,145],{"mathvariant":144},[138,192,193],{},"∣",[172,195,196,200],{},[93,197,199],{"mathvariant":198},"bold-italic","μ",[93,201,162],{},[138,203,204],{"separator":104},",",[172,206,207,210],{},[93,208,209],{"mathvariant":144},"Σ",[93,211,162],{},[138,213,148],{"stretchy":140},[97,215,216],{"encoding":99},"p(\\mathbf{x}) = \\sum_{k=1}^{K} \\pi_k \\, \\mathcal{N}(\\mathbf{x} \\mid \\boldsymbol{\\mu}_k, \\boldsymbol{\\Sigma}_k)",[43,218,220,254,417],{"className":219,"ariaHidden":104},[103],[43,221,223,227,230,234,238,242,247,251],{"className":222},[108],[43,224],{"className":225,"style":226},[112],"height:1em;vertical-align:-0.25em;",[43,228,11],{"className":229},[117,118],[43,231,141],{"className":232},[233],"mopen",[43,235,145],{"className":236},[117,237],"mathbf",[43,239,148],{"className":240},[241],"mclose",[43,243],{"className":244,"style":246},[245],"mspace","margin-right:0.2778em;",[43,248,151],{"className":249},[250],"mrel",[43,252],{"className":253,"style":246},[245],[43,255,257,261,346,350,394,397,402,405,408,411,414],{"className":256},[108],[43,258],{"className":259,"style":260},[112],"height:1.2809em;vertical-align:-0.2997em;",[43,262,265,271],{"className":263},[264],"mop",[43,266,157],{"className":267,"style":270},[264,268,269],"op-symbol","small-op","position:relative;top:0em;",[43,272,275],{"className":273},[274],"msupsub",[43,276,280,337],{"className":277},[278,279],"vlist-t","vlist-t2",[43,281,284,332],{"className":282},[283],"vlist-r",[43,285,289,317],{"className":286,"style":288},[287],"vlist","height:0.9812em;",[43,290,292,297],{"style":291},"top:-2.4003em;margin-left:0em;margin-right:0.05em;",[43,293],{"className":294,"style":296},[295],"pstrut","height:2.7em;",[43,298,304],{"className":299},[300,301,302,303],"sizing","reset-size6","size3","mtight",[43,305,307,311,314],{"className":306},[117,303],[43,308,162],{"className":309,"style":310},[117,118,303],"margin-right:0.0315em;",[43,312,151],{"className":313},[250,303],[43,315,168],{"className":316},[117,303],[43,318,320,323],{"style":319},"top:-3.2029em;margin-right:0.05em;",[43,321],{"className":322,"style":296},[295],[43,324,326],{"className":325},[300,301,302,303],[43,327,329],{"className":328},[117,303],[43,330,95],{"className":331,"style":119},[117,118,303],[43,333,336],{"className":334},[335],"vlist-s","​",[43,338,340],{"className":339},[283],[43,341,344],{"className":342,"style":343},[287],"height:0.2997em;",[43,345],{},[43,347],{"className":348,"style":349},[245],"margin-right:0.1667em;",[43,351,353,357],{"className":352},[117],[43,354,176],{"className":355,"style":356},[117,118],"margin-right:0.0359em;",[43,358,360],{"className":359},[274],[43,361,363,385],{"className":362},[278,279],[43,364,366,382],{"className":365},[283],[43,367,370],{"className":368,"style":369},[287],"height:0.3361em;",[43,371,373,376],{"style":372},"top:-2.55em;margin-left:-0.0359em;margin-right:0.05em;",[43,374],{"className":375,"style":296},[295],[43,377,379],{"className":378},[300,301,302,303],[43,380,162],{"className":381,"style":310},[117,118,303],[43,383,336],{"className":384},[335],[43,386,388],{"className":387},[283],[43,389,392],{"className":390,"style":391},[287],"height:0.15em;",[43,393],{},[43,395],{"className":396,"style":349},[245],[43,398,186],{"className":399,"style":401},[117,400],"mathcal","margin-right:0.1474em;",[43,403,141],{"className":404},[233],[43,406,145],{"className":407},[117,237],[43,409],{"className":410,"style":246},[245],[43,412,193],{"className":413},[250],[43,415],{"className":416,"style":246},[245],[43,418,420,423,473,477,480,527],{"className":419},[108],[43,421],{"className":422,"style":226},[112],[43,424,426,436],{"className":425},[117],[43,427,429],{"className":428},[117],[43,430,432],{"className":431},[117],[43,433,199],{"className":434},[117,435],"boldsymbol",[43,437,439],{"className":438},[274],[43,440,442,464],{"className":441},[278,279],[43,443,445,461],{"className":444},[283],[43,446,449],{"className":447,"style":448},[287],"height:0.242em;",[43,450,452,455],{"style":451},"top:-2.4559em;margin-right:0.05em;",[43,453],{"className":454,"style":296},[295],[43,456,458],{"className":457},[300,301,302,303],[43,459,162],{"className":460,"style":310},[117,118,303],[43,462,336],{"className":463},[335],[43,465,467],{"className":466},[283],[43,468,471],{"className":469,"style":470},[287],"height:0.2441em;",[43,472],{},[43,474,204],{"className":475},[476],"mpunct",[43,478],{"className":479,"style":349},[245],[43,481,483,492],{"className":482},[117],[43,484,486],{"className":485},[117],[43,487,489],{"className":488},[117],[43,490,209],{"className":491},[117,237],[43,493,495],{"className":494},[274],[43,496,498,519],{"className":497},[278,279],[43,499,501,516],{"className":500},[283],[43,502,504],{"className":503,"style":369},[287],[43,505,507,510],{"style":506},"top:-2.55em;margin-right:0.05em;",[43,508],{"className":509,"style":296},[295],[43,511,513],{"className":512},[300,301,302,303],[43,514,162],{"className":515,"style":310},[117,118,303],[43,517,336],{"className":518},[335],[43,520,522],{"className":521},[283],[43,523,525],{"className":524,"style":391},[287],[43,526],{},[43,528,148],{"className":529},[241],[11,531,532,533,604,605,608],{},"where ",[43,534,536,554],{"className":535},[77],[43,537,539],{"className":538},[81],[83,540,541],{"xmlns":85},[87,542,543,551],{},[90,544,545],{},[172,546,547,549],{},[93,548,176],{},[93,550,162],{},[97,552,553],{"encoding":99},"\\pi_k",[43,555,557],{"className":556,"ariaHidden":104},[103],[43,558,560,564],{"className":559},[108],[43,561],{"className":562,"style":563},[112],"height:0.5806em;vertical-align:-0.15em;",[43,565,567,570],{"className":566},[117],[43,568,176],{"className":569,"style":356},[117,118],[43,571,573],{"className":572},[274],[43,574,576,596],{"className":575},[278,279],[43,577,579,593],{"className":578},[283],[43,580,582],{"className":581,"style":369},[287],[43,583,584,587],{"style":372},[43,585],{"className":586,"style":296},[295],[43,588,590],{"className":589},[300,301,302,303],[43,591,162],{"className":592,"style":310},[117,118,303],[43,594,336],{"className":595},[335],[43,597,599],{"className":598},[283],[43,600,602],{"className":601,"style":391},[287],[43,603],{}," is the weight (the fraction of data belonging to that group). Bishop derives fitting this model via an algorithm called ",[66,606,607],{},"EM"," (expectation-maximization), which alternates two steps until it converges:",[610,611,612,731],"ul",{},[613,614,615,618,619,622,623,722,723,726,727,730],"li",{},[66,616,617],{},"E step (expectation)",": for every point, compute each component's ",[66,620,621],{},"responsibility"," ",[43,624,626,657],{"className":625},[77],[43,627,629],{"className":628},[81],[83,630,631],{"xmlns":85},[87,632,633,654],{},[90,634,635,638,640,652],{},[93,636,637],{},"γ",[138,639,141],{"stretchy":140},[172,641,642,645],{},[93,643,644],{},"z",[90,646,647,650],{},[93,648,649],{},"n",[93,651,162],{},[138,653,148],{"stretchy":140},[97,655,656],{"encoding":99},"\\gamma(z_{nk})",[43,658,660],{"className":659,"ariaHidden":104},[103],[43,661,663,666,670,673,719],{"className":662},[108],[43,664],{"className":665,"style":226},[112],[43,667,637],{"className":668,"style":669},[117,118],"margin-right:0.0556em;",[43,671,141],{"className":672},[233],[43,674,676,680],{"className":675},[117],[43,677,644],{"className":678,"style":679},[117,118],"margin-right:0.044em;",[43,681,683],{"className":682},[274],[43,684,686,711],{"className":685},[278,279],[43,687,689,708],{"className":688},[283],[43,690,692],{"className":691,"style":369},[287],[43,693,695,698],{"style":694},"top:-2.55em;margin-left:-0.044em;margin-right:0.05em;",[43,696],{"className":697,"style":296},[295],[43,699,701],{"className":700},[300,301,302,303],[43,702,704],{"className":703},[117,303],[43,705,707],{"className":706,"style":310},[117,118,303],"nk",[43,709,336],{"className":710},[335],[43,712,714],{"className":713},[283],[43,715,717],{"className":716,"style":391},[287],[43,718],{},[43,720,148],{"className":721},[241],", the probability (via Bayes) that the point came from that specific group, given where the Gaussians currently sit. Unlike K-means, which assigns each point to ",[66,724,725],{},"one"," group only, EM assigns a ",[66,728,729],{},"fractional"," responsibility across all groups (a point on the border between two groups might get 60% responsibility from one and 40% from the other).",[613,732,733,736],{},[66,734,735],{},"M step (maximization)",": recompute each Gaussian's mean, covariance, and weight, using those responsibilities as weights. A point with 0.9 responsibility toward group 1 counts almost fully toward group 1's mean and covariance, while one split 0.5\u002F0.5 counts half toward each.",[11,738,739],{},"This is literally the \"soft\" version of K-means: swap \"each point belongs to exactly one group\" for \"each point belongs a little to every group,\" and swap \"a group is just a mean\" for \"a group is a mean plus a shape.\"",[741,742,743],"blockquote",{},[11,744,745,748,749,752],{},[66,746,747],{},"Output:"," weights found, ",[40,750,751],{},"[0.40, 0.21, 0.39]"," (matches the true proportion of the three generated groups). Converged in just 4 iterations.",[21,754,756],{"id":755},"interactive-the-ellipses-settling-into-place","Interactive: the ellipses settling into place",[11,758,759],{},"My own reconstruction of the EM algorithm (on the same 380 points, 3 groups, two of them stretched). Click \"EM step\" and watch the ellipses (each the 1-standard-deviation outline of that Gaussian) rotate and stretch until they fit the data's real shape:",[761,762],"gmm-explorer",{":n-components":763,":points":764,":x-max":765,":x-min":766,":y-max":767,":y-min":768,"x-label":769,"y-label":770},"3","[[-1.397,0.6689],[-1.5544,0.9572],[0.4216,0.4219],[-1.4608,0.7519],[-2.2469,0.7798],[-2.6499,0.1603],[-0.5571,0.0035],[-0.3012,1.944],[-2.1017,0.3399],[-0.7528,-1.1562],[-0.3647,2.9444],[-1.8702,0.663],[-1.3084,-0.5343],[-0.1774,0.0732],[-0.4236,-0.1319],[-1.8018,1.6157],[-0.3874,3.4638],[-0.3632,2.2896],[-1.5163,0.9009],[-0.2556,3.0671],[0.5872,0.5437],[-1.5002,-2.5391],[-1.691,1.0455],[0.1699,-0.2883],[0.2436,1.5866],[-1.8124,0.5865],[0.4329,1.4138],[-1.0305,1.3562],[-0.6777,-1.6514],[-0.1738,-0.6312],[-2.1474,0.6027],[1.3162,1.41],[0.0003,-0.635],[1.0803,0.8774],[-0.1051,-0.5368],[-0.4542,2.9433],[-0.5984,2.7038],[-0.5293,-0.3695],[-1.265,-1.6818],[-1.6018,1.2726],[-0.0668,0.5579],[-0.741,-0.6176],[0.9641,1.7692],[-0.1353,2.9909],[-0.0816,-0.0108],[-0.6033,-1.0474],[-1.9353,-2.3436],[-0.544,-0.9764],[-0.499,-0.7219],[0.6757,0.5116],[-2.169,-0.4307],[-1.6183,1.3119],[-0.957,-1.9875],[0.0241,-0.5593],[-1.8944,0.5322],[0.0781,0.0503],[1.234,1.02],[-1.0304,2.212],[0.0279,0.4623],[-0.3033,-0.0086],[-0.6413,1.4614],[-1.6879,1.2439],[-0.7695,2.5701],[0.026,0.2301],[-0.4519,-1.4045],[-1.5718,-2.2173],[-0.2266,-1.261],[-2.217,0.1283],[-0.7787,-0.9779],[-0.5035,-1.0994],[-0.6094,2.4055],[-0.7201,-1.1225],[-1.2544,1.2117],[1.1962,1.1776],[-0.5273,-1.0992],[-0.8385,-2.0174],[-2.4497,0.6202],[0.0672,0.4268],[-1.9462,0.6892],[-1.7471,0.8418],[0.6934,0.9687],[-1.2374,1.1307],[-1.0847,1.8531],[0.6181,0.3445],[0.4234,0.6546],[1.2764,2.13],[-0.7243,1.9397],[0.0644,0.2269],[-1.2528,1.4129],[-1.2586,2.5021],[-2.0632,2.267],[1.5012,1.4466],[-2.0287,1.4448],[-0.5999,-0.9094],[-1.944,1.9186],[0.4245,0.1732],[-1.7969,1.2382],[-1.2832,1.4056],[-1.5927,1.4856],[-0.2796,3.3672],[-1.6957,1.1181],[-0.7625,2.1224],[0.3737,0.3211],[0.4339,-0.622],[-1.9618,-0.0241],[-2.8526,0.3066],[-1.2317,0.759],[-1.5875,0.5392],[-0.5509,-0.2138],[-0.5282,-0.1087],[0.0566,0.3988],[-0.2795,-0.0704],[0.2273,1.597],[-2.3005,0.6762],[0.7842,1.0364],[-1.7827,1.3026],[0.1197,0.3453],[-0.8305,1.6026],[-0.7473,-0.6929],[-0.8915,-0.0462],[-1.5467,0.8436],[-1.65,0.6603],[-0.849,2.4742],[-1.1966,1.4964],[-1.0852,1.8029],[-1.5808,0.6337],[-1.3083,1.814],[-0.111,-0.992],[-0.3001,2.3096],[-1.2104,1.2865],[-1.7846,0.2639],[-2.0529,-0.4044],[1.7441,2.0217],[0.6524,0.6848],[0.11,-0.1965],[-0.0356,0.4157],[-0.0749,2.9341],[-1.5994,2.0179],[-1.0351,-1.8731],[-0.171,2.6711],[-0.4753,3.3274],[-2.3558,1.0955],[-2.0755,0.7241],[-1.8707,0.9155],[-2.8612,-0.5567],[-2.742,0.4937],[-0.3161,0.2769],[-1.7449,2.1745],[0.8469,-0.159],[-0.6866,2.0795],[-1.0555,1.2891],[-0.2582,0.0552],[-1.1792,-2.3321],[-0.2038,-0.2389],[-0.9721,1.4959],[-1.769,1.3658],[-0.4673,-0.3329],[-1.5077,-2.8246],[-2.2638,0.8664],[1.2534,1.3654],[1.0126,1.5736],[-0.5988,3.3043],[-1.898,-0.0087],[-1.4826,0.682],[0.3936,2.9382],[-1.7633,1.2016],[0.4435,0.7339],[-0.9929,1.6227],[-1.3397,-1.6588],[-0.7939,-0.9446],[-0.6199,-0.8483],[-1.2111,1.5388],[-2.0879,0.5661],[-0.6015,-0.4415],[-0.9774,-0.3869],[-2.8054,-0.2992],[0.2876,-0.0551],[-1.8222,0.4918],[-2.6337,-0.4616],[-1.6066,-1.8044],[0.9042,1.2706],[-2.4887,-0.5669],[-1.4762,2.1884],[-3.3941,-2.2838],[-0.0459,0.611],[0.3615,-0.6934],[-0.6614,-0.5482],[-0.9675,-1.1216],[-0.8114,1.6109],[-0.4249,-0.8102],[-0.3229,-0.2177],[-1.8453,1.2592],[0.6648,1.5428],[-2.018,1.4594],[-2.7884,0.4345],[0.257,0.8384],[-1.0581,1.6156],[-0.594,2.3061],[-0.8174,-0.8417],[2.0556,3.0074],[-1.0839,-1.2423],[0.5036,1.2528],[-2.7214,-0.3583],[-1.9335,1.9023],[0.093,0.2204],[-2.1258,0.0677],[1.971,3.5944],[0.2497,0.3212],[-0.5735,-1.5174],[-0.3901,2.4058],[0.0023,-0.8388],[-0.1752,0.0744],[0.4605,0.9383],[-0.7978,-0.7208],[-2.3619,0.0751],[-0.1661,3.8238],[0.2842,-0.7207],[0.7606,1.8876],[1.5808,4.2012],[0.2639,3.3232],[-1.2204,0.9527],[-1.7879,-1.6378],[-1.1302,1.8454],[1.2432,1.4315],[-2.1969,0.8246],[0.7229,0.4503],[0.3372,-0.0962],[-0.6671,2.5567],[-0.2091,-0.5909],[-0.5144,0.1183],[0.2037,0.7191],[0.4719,0.4481],[-0.723,2.1655],[0.0977,0.8399],[-0.4848,-0.1881],[0.3503,-0.8179],[-0.5901,-0.4508],[0.7263,1.0749],[-0.5648,-0.294],[-0.2182,-0.1902],[-0.9943,1.5624],[1.2449,1.9522],[0.623,1.2615],[-2.3547,-0.325],[-2.2113,0.7627],[-1.8254,1.4898],[1.2256,0.4739],[-0.4011,0.7219],[-0.1606,0.123],[-1.5009,1.0681],[0.0147,0.6807],[-0.0381,0.1796],[-1.0194,2.0814],[0.4602,2.7832],[-2.594,0.6801],[-1.0335,-1.261],[0.3909,3.2754],[0.0448,0.86],[-1.8919,1.0616],[0.0364,0.0331],[-0.9895,2.5191],[-0.8706,2.4464],[-1.3902,1.6082],[-1.3474,1.7972],[0.2534,0.1369],[-1.5536,0.9959],[-0.8877,2.0832],[-0.7537,-0.7697],[-0.9112,-0.787],[0.1413,-0.5064],[-1.9722,0.4258],[-2.2693,-0.1881],[-1.4793,0.6792],[-0.3873,2.6294],[0.1175,4.0415],[-2.4398,0.4219],[-0.1291,-0.6354],[1.6276,1.5164],[-1.9332,2.4903],[0.4905,1.1607],[-0.8633,2.2045],[-0.9049,1.9118],[0.6161,1.4715],[-2.5119,-0.4271],[0.8632,0.6314],[-1.8683,2.1036],[-1.8102,-0.5893],[-0.854,-1.0794],[-1.0491,2.6654],[-1.8057,1.3328],[-1.9607,0.4713],[0.1505,0.8971],[0.0872,0.6326],[1.8988,1.959],[-1.324,-1.569],[-1.2021,1.1349],[-2.8576,-0.3023],[-0.699,2.2929],[-0.0063,-0.3087],[-1.9344,0.5451],[3.455,2.5789],[2.9464,1.1252],[1.572,0.9878],[3.3295,1.4183],[2.6823,0.5125],[3.4773,-0.0434],[3.5826,-0.9733],[2.5003,0.448],[2.3398,1.39],[5.377,1.1889],[2.8108,1.2465],[2.4285,1.4879],[3.5583,-0.4105],[2.5828,-0.398],[4.0777,3.2047],[3.2711,1.3714],[2.0754,0.5936],[5.3566,1.4881],[3.2984,1.3158],[3.2566,0.7801],[2.8891,2.8666],[2.7887,0.6866],[2.8902,0.7226],[1.9401,1.0828],[2.7075,0.6922],[2.8139,1.626],[3.7183,2.3214],[4.9565,0.7885],[3.8149,0.6292],[3.6997,-0.9454],[5.07,1.7817],[4.1385,2.5373],[3.0302,2.0714],[4.3033,2.3705],[4.8936,-0.3876],[3.551,3.4775],[4.4062,1.343],[2.3845,-0.1819],[3.5879,1.9829],[3.8344,-0.7488],[0.8711,1.8362],[4.9687,0.496],[3.7413,1.3607],[3.7328,-0.899],[3.8221,1.9898],[4.2818,0.1049],[3.4561,-0.1544],[2.961,1.5276],[4.0127,1.3113],[3.4959,0.7797],[3.8522,2.5523],[3.0213,1.5568],[2.1626,1.2111],[3.7507,1.7961],[3.0274,0.5486],[1.8833,1.1989],[3.0987,-0.4492],[2.6516,0.7051],[2.2539,-0.3062],[4.3043,-0.2166],[2.5161,1.8014],[3.8524,0.3692],[3.5905,0.5108],[4.5218,1.9456],[3.7839,0.2999],[3.7484,0.9398],[3.1481,0.212],[3.5778,0.7153],[3.0116,0.8286],[2.0123,0.2944],[4.6336,1.7662],[2.2995,1.6708],[3.4188,2.0178],[4.3133,-0.2066],[2.5714,2.5642],[1.7659,0.452],[2.478,1.3285],[4.2293,1.1857],[3.7869,1.2753],[3.3752,0.7132]]","5.7","-3.7","4.5","-3.2","x1","x2",[11,772,773,774,777],{},"Notice the weights shown below the chart drift toward ",[40,775,776],{},"0.40, 0.21, 0.39"," as you click \"EM step\" repeatedly, and the ellipses go from generic circles (the initialization) to stretched shapes that trace the groups' real direction.",[21,779,781],{"id":780},"anomaly-detection-for-free","Anomaly detection for free",[11,783,784],{},"One advantage of having a density model (not just a grouping): you can ask \"how likely is this point, given the model?\" and flag the least likely ones as anomalies.",[33,786,788],{"className":35,"code":787,"language":37,"meta":38,"style":38},"densities = gm.score_samples(X)\ndensity_threshold = np.percentile(densities, 2)\nanomalies = X[densities \u003C density_threshold]\n",[40,789,790,795,800],{"__ignoreMap":38},[43,791,792],{"class":45,"line":46},[43,793,794],{},"densities = gm.score_samples(X)\n",[43,796,797],{"class":45,"line":52},[43,798,799],{},"density_threshold = np.percentile(densities, 2)\n",[43,801,802],{"class":45,"line":58},[43,803,804],{},"anomalies = X[densities \u003C density_threshold]\n",[11,806,807],{},"Points with density below the 2nd percentile (the least likely 2%) become anomaly candidates, and they capture exactly the small isolated group the professor placed on purpose far from the other two.",[21,809,811],{"id":810},"how-many-groups-to-use-bic-aic-and-a-smarter-way","How many groups to use? BIC, AIC, and a smarter way",[33,813,815],{"className":35,"code":814,"language":37,"meta":38,"style":38},"gm.bic(X), gm.aic(X)\n",[40,816,817],{"__ignoreMap":38},[43,818,819],{"class":45,"line":46},[43,820,814],{},[741,822,823],{},[11,824,825,827],{},[66,826,747],{}," BIC = 8189.73, AIC = 8102.51.",[11,829,830,831,834,835,887,888,917,918,946],{},"Both are ",[66,832,833],{},"information criteria",": they measure how well the model explains the data, with a penalty for complexity (more groups = more parameters = bigger penalty), to avoid picking \"more groups is always better\" just because more groups always fits better. Running for ",[43,836,838,856],{"className":837},[77],[43,839,841],{"className":840},[81],[83,842,843],{"xmlns":85},[87,844,845,853],{},[90,846,847,849,851],{},[93,848,95],{},[138,850,151],{},[166,852,168],{},[97,854,855],{"encoding":99},"K=1",[43,857,859,877],{"className":858,"ariaHidden":104},[103],[43,860,862,865,868,871,874],{"className":861},[108],[43,863],{"className":864,"style":113},[112],[43,866,95],{"className":867,"style":119},[117,118],[43,869],{"className":870,"style":246},[245],[43,872,151],{"className":873},[250],[43,875],{"className":876,"style":246},[245],[43,878,880,884],{"className":879},[108],[43,881],{"className":882,"style":883},[112],"height:0.6444em;",[43,885,168],{"className":886},[117]," through ",[43,889,891,905],{"className":890},[77],[43,892,894],{"className":893},[81],[83,895,896],{"xmlns":85},[87,897,898,903],{},[90,899,900],{},[166,901,902],{},"9",[97,904,902],{"encoding":99},[43,906,908],{"className":907,"ariaHidden":104},[103],[43,909,911,914],{"className":910},[108],[43,912],{"className":913,"style":883},[112],[43,915,902],{"className":916},[117]," and plotting both against ",[43,919,921,934],{"className":920},[77],[43,922,924],{"className":923},[81],[83,925,926],{"xmlns":85},[87,927,928,932],{},[90,929,930],{},[93,931,95],{},[97,933,95],{"encoding":99},[43,935,937],{"className":936,"ariaHidden":104},[103],[43,938,940,943],{"className":939},[108],[43,941],{"className":942,"style":113},[112],[43,944,95],{"className":945,"style":119},[117,118],", the shape of the curve points at the number of groups that balances fit against simplicity.",[11,948,949,950,953,954,957],{},"There's an even more direct way: ",[40,951,952],{},"BayesianGaussianMixture"," is handed a ",[66,955,956],{},"generous"," number of components (10, in this case) and prunes the unnecessary ones itself, zeroing out their weight:",[33,959,961],{"className":35,"code":960,"language":37,"meta":38,"style":38},"from sklearn.mixture import BayesianGaussianMixture\nbgm = BayesianGaussianMixture(n_components=10, n_init=10, random_state=42)\nbgm.fit(X)\nprint(np.round(bgm.weights_, 2))\n",[40,962,963,968,973,978],{"__ignoreMap":38},[43,964,965],{"class":45,"line":46},[43,966,967],{},"from sklearn.mixture import BayesianGaussianMixture\n",[43,969,970],{"class":45,"line":52},[43,971,972],{},"bgm = BayesianGaussianMixture(n_components=10, n_init=10, random_state=42)\n",[43,974,975],{"class":45,"line":58},[43,976,977],{},"bgm.fit(X)\n",[43,979,981],{"class":45,"line":980},4,[43,982,983],{},"print(np.round(bgm.weights_, 2))\n",[741,985,986],{},[11,987,988,622,990,993,994,1022],{},[66,989,747],{},[40,991,992],{},"[0.4, 0.21, 0.39, 0, 0, 0, 0, 0, 0, 0]",". Seven of the ten components zeroed out on their own, leaving the three real ones, no need for me to scan ",[43,995,997,1010],{"className":996},[77],[43,998,1000],{"className":999},[81],[83,1001,1002],{"xmlns":85},[87,1003,1004,1008],{},[90,1005,1006],{},[93,1007,95],{},[97,1009,95],{"encoding":99},[43,1011,1013],{"className":1012,"ariaHidden":104},[103],[43,1014,1016,1019],{"className":1015},[108],[43,1017],{"className":1018,"style":113},[112],[43,1020,95],{"className":1021,"style":119},[117,118]," by hand.",[11,1024,1025],{},"Bishop warns of a technical problem worth knowing: if a Gaussian \"collapses\" right on top of a single data point, its variance can go to zero and the model's likelihood goes to infinity, a singularity, not a good fit. Real implementations (like scikit-learn's) guard against this in practice with numerical safeguards, but it's a reminder that \"finding the maximum likelihood\" isn't always as well-behaved a problem as it sounds.",[21,1027,1029],{"id":1028},"switching-topics-other-ways-to-spot-an-anomaly","Switching topics: other ways to spot an anomaly",[11,1031,1032,1035,1036,1039],{},[40,1033,1034],{},"aula11b"," sets up a more direct anomaly-detection scenario: 980 \"normal\" points (3 well-behaved groups) plus 20 points scattered randomly across the space (the real anomalies), and compares three different detectors, each with the default ",[40,1037,1038],{},"contamination"," (0.1, meaning \"assume 10% of the data is anomalous\") and then tuned via Optuna:",[1041,1042,1043,1061],"table",{},[1044,1045,1046],"thead",{},[1047,1048,1049,1054,1058],"tr",{},[1050,1051,1053],"th",{"align":1052},"left","Detector",[1050,1055,1057],{"align":1056},"right","F1 (anomaly), default",[1050,1059,1060],{"align":1056},"F1 (anomaly), tuned",[1062,1063,1064,1078,1092],"tbody",{},[1047,1065,1066,1072,1075],{},[1067,1068,1069],"td",{"align":1052},[40,1070,1071],{},"IsolationForest",[1067,1073,1074],{"align":1056},"≈ 0.27",[1067,1076,1077],{"align":1056},"0.68",[1047,1079,1080,1085,1087],{},[1067,1081,1082],{"align":1052},[40,1083,1084],{},"LocalOutlierFactor",[1067,1086,1074],{"align":1056},[1067,1088,1089],{"align":1056},[66,1090,1091],{},"0.79",[1047,1093,1094,1099,1102],{},[1067,1095,1096],{"align":1052},[40,1097,1098],{},"OneClassSVM",[1067,1100,1101],{"align":1056},"≈ 0.24",[1067,1103,1104],{"align":1056},"0.64",[11,1106,1107,1108,1111,1112,1115,1116,1118,1119,1121,1122,1126],{},"The default ",[40,1109,1110],{},"contamination=0.1"," tells every detector to flag ",[66,1113,1114],{},"10%"," of the data as anomalous (98 points), but only 20 of the 1000 points (2%) are actually anomalies. Forcing the model to find 5 times more anomalies than actually exist guarantees a pile of false positives, hence the low F1 across all three before tuning. Once Optuna searches for the right ",[40,1117,1038],{}," (close to 0.02, the true value) and each model's other hyperparameters, all three improve substantially, and ",[40,1120,1084],{}," (which decides \"anomalous\" by comparing a point's local density against its neighbors', ",[15,1123,1125],{"href":1124},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised","the same density-based reasoning DBSCAN uses",") comes out ahead.",[11,1128,1129,1130,1132,1133,1136],{},"The practical lesson, matching the rest of the lecture: ",[40,1131,1038],{}," isn't a cosmetic detail, it's the most important piece of the tuning, because it tells the model ",[66,1134,1135],{},"how many"," anomalies to look for. Without knowing (or estimating well) that fraction ahead of time, any of these three detectors misses the mark.",[21,1138,1140],{"id":1139},"wrapping-up","Wrapping up",[1041,1142,1143,1153],{},[1044,1144,1145],{},[1047,1146,1147,1150],{},[1050,1148,1149],{"align":1052},"What I already knew",[1050,1151,1152],{"align":1052},"What this lecture settled",[1062,1154,1155,1166,1174],{},[1047,1156,1157,1160],{},[1067,1158,1159],{"align":1052},"K-means groups by distance to a center",[1067,1161,1162,1163,1165],{"align":1052},"Gaussian mixtures group by density ",[66,1164,72],{}," shape, with fractional responsibility instead of a rigid assignment",[1047,1167,1168,1171],{},[1067,1169,1170],{"align":1052},"DBSCAN finds anomalies as \"whoever isn't core to anything\"",[1067,1172,1173],{"align":1052},"Model density (GMM) and local density (LOF) are two other valid ways to define \"anomaly,\" each with its own bias",[1047,1175,1176,1179],{},[1067,1177,1178],{"align":1052},"Hyperparameters matter",[1067,1180,1181,1182,1184],{"align":1052},"For anomaly detection specifically, ",[40,1183,1038],{}," is the hyperparameter that matters most, because most algorithms need to know upfront how much anomaly to look for",[21,1186,1188],{"id":1187},"practical-application","Practical application",[11,1190,1191],{},"I use the same density idea from the GMM anomaly-detection section, but compare four different percentile cutoffs on the same 3-group dataset, to see how much the cutoff choice changes how many points turn into \"anomalies.\"",[33,1193,1195],{"className":35,"code":1194,"language":37,"meta":38,"style":38},"for percentil in [1, 2, 5, 10]:\n    threshold = np.percentile(densities, percentil)\n    n_anomalias = (densities \u003C threshold).sum()\n    print(percentil, n_anomalias)\n",[40,1196,1197,1202,1207,1212],{"__ignoreMap":38},[43,1198,1199],{"class":45,"line":46},[43,1200,1201],{},"for percentil in [1, 2, 5, 10]:\n",[43,1203,1204],{"class":45,"line":52},[43,1205,1206],{},"    threshold = np.percentile(densities, percentil)\n",[43,1208,1209],{"class":45,"line":58},[43,1210,1211],{},"    n_anomalias = (densities \u003C threshold).sum()\n",[43,1213,1214],{"class":45,"line":980},[43,1215,1216],{},"    print(percentil, n_anomalias)\n",[741,1218,1219],{},[11,1220,1221,1224],{},[66,1222,1223],{},"Output (1250 points total, the same 750+250 generated in the lecture):"," with a 1% cutoff, 13 points flagged. With 2% (the value used in the notebook), 25 points. With 5%, 63 points. With 10%, 125 points.",[11,1226,1227],{},"The number of \"anomalies\" found scales almost linearly with the chosen percentile, because that's literally how a percentile cutoff works: it always finds exactly that fraction of the data, whether a real anomaly is sitting there or not. It's the same point the lecture's detector comparison already made a different way: deciding \"how much\" to look for is a choice that changes the result as much as the algorithm itself.",[1229,1230,1231],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":38,"searchDepth":52,"depth":52,"links":1233},[1234,1235,1236,1237,1238,1239,1240,1241],{"id":23,"depth":52,"text":24},{"id":30,"depth":52,"text":31},{"id":755,"depth":52,"text":756},{"id":780,"depth":52,"text":781},{"id":810,"depth":52,"text":811},{"id":1028,"depth":52,"text":1029},{"id":1139,"depth":52,"text":1140},{"id":1187,"depth":52,"text":1188},null,"2026-08-20","Lecture 11: the professor swaps K-means for Gaussian mixtures when groups aren't round, uses density to spot anomalies, then compares three different anomaly detectors, all needing careful tuning to work well.","md",{},true,12,"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection","pattern-recognition",{"title":6,"description":1244},"published","en\u002Fplaylists\u002Fpattern-recognition\u002Fgaussian-mixtures-anomaly-detection",[1255,1256,1257],"gaussian-mixtures","em","anomaly-detection","8w73rFcG_VUX93NK8-FsyDQoDaoMVN27u8wUk5jcUNQ",1787338984529]