[{"data":1,"prerenderedAt":446},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron":3,"post-en-neural-networks-xor-o-limite-do-perceptron":4},"\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron",{"id":5,"title":6,"body":7,"cover":430,"date":431,"description":432,"extension":433,"meta":434,"navigation":435,"order":69,"path":436,"playlist":437,"seo":438,"status":439,"stem":440,"tags":441,"__hash__":445},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron.md","XOR: Where a Single Neuron Just Hits a Wall",{"type":8,"value":9,"toc":421},"minimark",[10,14,19,79,106,110,130,148,166,178,189,193,211,224,237,254,258,265,279,282,286,293,301,305,350,354,362,377,414,417],[11,12,13],"p",{},"Lecture 3d, and it's the last one in the repository as of today. Before closing out this batch of posts, the professor leaves a cliffhanger that's almost poetic: the same perceptron that solved everything up to this point, effortlessly, gets stuck hard on a 4-point problem.",[15,16,18],"h2",{"id":17},"the-dataset-logic-gates-geometrically","The dataset: logic gates, geometrically",[20,21,26],"pre",{"className":22,"code":23,"language":24,"meta":25,"style":25},"language-python shiki shiki-themes github-light github-dark","def createLogicalDataset(n=40, func=lambda a, b: bool(a) or bool(b)):\n  X, y = make_blobs(n_samples=n,\n                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n                    n_features=2,\n                    cluster_std=0.05,\n                    center_box=(0,1))\n  y = np.array([func(a>0.5, b>0.5) for a, b in X], dtype=int)\n  return X, y\n","python","",[27,28,29,37,43,49,55,61,67,73],"code",{"__ignoreMap":25},[30,31,34],"span",{"class":32,"line":33},"line",1,[30,35,36],{},"def createLogicalDataset(n=40, func=lambda a, b: bool(a) or bool(b)):\n",[30,38,40],{"class":32,"line":39},2,[30,41,42],{},"  X, y = make_blobs(n_samples=n,\n",[30,44,46],{"class":32,"line":45},3,[30,47,48],{},"                    centers=[[0.2,0.2], [0.8, 0.2], [0.2, 0.8], [0.8, 0.8]],\n",[30,50,52],{"class":32,"line":51},4,[30,53,54],{},"                    n_features=2,\n",[30,56,58],{"class":32,"line":57},5,[30,59,60],{},"                    cluster_std=0.05,\n",[30,62,64],{"class":32,"line":63},6,[30,65,66],{},"                    center_box=(0,1))\n",[30,68,70],{"class":32,"line":69},7,[30,71,72],{},"  y = np.array([func(a>0.5, b>0.5) for a, b in X], dtype=int)\n",[30,74,76],{"class":32,"line":75},8,[30,77,78],{},"  return X, y\n",[11,80,81,86,87,90,91,90,94,97,98,101,102,105],{},[82,83,85],"a",{"href":84},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmulticlasse","The same 4-blob dataset from the previous post",", except now the label comes from applying a logical function (",[27,88,89],{},"or",", ",[27,92,93],{},"and",[27,95,96],{},"!=",") to \"is this point to the right of 0.5?\" and \"is this point above 0.5?\". Every blob effectively becomes a logic gate value: the bottom-left corner is ",[27,99,100],{},"(false, false)",", bottom-right is ",[27,103,104],{},"(true, false)",", and so on.",[15,107,109],{"id":108},"or-and-and-no-drama","OR and AND: no drama",[20,111,113],{"className":22,"code":112,"language":24,"meta":25,"style":25},"X, y = createLogicalDataset(func=lambda a, b: bool(a) or bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[27,114,115,120,125],{"__ignoreMap":25},[30,116,117],{"class":32,"line":33},[30,118,119],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) or bool(b))\n",[30,121,122],{"class":32,"line":39},[30,123,124],{},"model = NeuralNetwork()\n",[30,126,127],{"class":32,"line":45},[30,128,129],{},"model.fit(X, y)\n",[131,132,133],"blockquote",{},[11,134,135,139,140,143,144,147],{},[136,137,138],"strong",{},"Output (OR):"," accuracy ",[136,141,142],{},"1.0",", weights ",[27,145,146],{},"[-1.14, 1.72, 1.57]",".",[20,149,151],{"className":22,"code":150,"language":24,"meta":25,"style":25},"X, y = createLogicalDataset(func=lambda a, b: bool(a) and bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[27,152,153,158,162],{"__ignoreMap":25},[30,154,155],{"class":32,"line":33},[30,156,157],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) and bool(b))\n",[30,159,160],{"class":32,"line":39},[30,161,124],{},[30,163,164],{"class":32,"line":45},[30,165,129],{},[131,167,168],{},[11,169,170,139,173,143,175,147],{},[136,171,172],{},"Output (AND):",[136,174,142],{},[27,176,177],{},"[-2.16, 1.55, 1.79]",[11,179,180,181,184,185,188],{},"Both gates are linearly separable: for OR, a single line separates the bottom-left blob (the only ",[27,182,183],{},"false",") from the other three. For AND, it's the same thing, just isolating the top-right blob (the only ",[27,186,187],{},"true","). Nothing this perceptron hasn't already done throughout this whole post series.",[15,190,192],{"id":191},"xor-the-same-recipe-the-same-code-and-it-gets-stuck","XOR: the same recipe, the same code, and it gets stuck",[20,194,196],{"className":22,"code":195,"language":24,"meta":25,"style":25},"X, y = createLogicalDataset(func=lambda a, b: bool(a) != bool(b))\nmodel = NeuralNetwork()\nmodel.fit(X, y)\n",[27,197,198,203,207],{"__ignoreMap":25},[30,199,200],{"class":32,"line":33},[30,201,202],{},"X, y = createLogicalDataset(func=lambda a, b: bool(a) != bool(b))\n",[30,204,205],{"class":32,"line":39},[30,206,124],{},[30,208,209],{"class":32,"line":45},[30,210,129],{},[131,212,213],{},[11,214,215,139,218,143,221,147],{},[136,216,217],{},"Output (XOR):",[136,219,220],{},"0.5",[27,222,223],{},"[0.006, -0.011, -0.002]",[11,225,226,227,229,230,232,233,236],{},"Accuracy of 0.5 with 2 classes is exactly coin-flip level. And notice the weights: essentially zero across all three, the model basically gave up, shrinking the weight vector down to almost nothing instead of converging to anything useful. I reproduced this myself, with a seeded dataset, and hit the exact same wall: OR and AND at ",[27,228,142],{},", XOR pinned at ",[27,231,220],{},", equally shriveled weights (",[27,234,235],{},"[-0.025, -0.066, 0.118]",").",[11,238,239,240,243,244,246,247,249,250,253],{},"The reason is geometric, and you can see it without any formula: XOR marks the two ",[136,241,242],{},"opposite"," diagonal corners (bottom-right and top-left) as ",[27,245,187],{}," and the other two corners, also opposite each other (bottom-left and top-right), as ",[27,248,183],{},". There's no line that separates \"the two corners on one diagonal\" from \"the two corners on the other diagonal\": any line I draw cuts through one of the diagonals, mixing both classes on either side. XOR is ",[136,251,252],{},"not linearly separable",", full stop, and no amount of training is going to change that, because the problem isn't the learning algorithm, it's what a single neuron, with a single straight boundary, is capable of representing at all.",[15,255,257],{"id":256},"interactive-watch-the-perceptron-never-converge","Interactive: watch the perceptron never converge",[11,259,260,261,264],{},"Same seeded dataset from above, now on XOR. Click \"Process next point\" a bunch of times: notice that, unlike every other component in this playlist, this one ",[136,262,263],{},"never"," shows the convergence message. There will always be at least one point on the wrong side, no matter how much I click.",[266,267],"perceptron-explorer",{":classes":268,":points":269,":update-bias":187,":x-max":270,":x-min":271,":y-max":270,":y-min":271,"converged-label":272,"negative-label":273,"positive-label":274,"reset-label":275,"step-label":276,"x-label":277,"y-label":278},"[1, 1, -1, 1, -1, 1, 1, 1, 1, -1, -1, -1, -1, 1, 1, 1, -1, -1, -1, -1, 1, 1, -1, -1, 1, -1, -1, 1, -1, -1, 1, -1, 1, 1, 1, -1, 1, 1, -1, -1]","[[0.6968, 0.1669], [0.7174, 0.2268], [0.785, 0.741], [0.2956, 0.8119], [0.2277, 0.2062], [0.7398, 0.2731], [0.1953, 0.8597], [0.8825, 0.2077], [0.242, 0.791], [0.2, 0.1123], [0.1687, 0.1914], [0.2845, 0.1767], [0.1606, 0.2001], [0.8525, 0.1792], [0.2051, 0.8126], [0.7629, 0.2536], [0.795, 0.885], [0.8166, 0.8368], [0.1879, 0.1273], [0.8749, 0.7859], [0.7797, 0.0856], [0.1146, 0.7098], [0.7403, 0.7475], [0.7904, 0.7111], [0.8883, 0.1835], [0.7816, 0.7047], [0.7808, 0.7555], [0.1283, 0.8251], [0.8054, 0.8719], [0.2016, 0.2204], [0.2135, 0.7738], [0.2253, 0.1869], [0.7806, 0.3015], [0.7977, 0.1275], [0.1934, 0.7845], [0.2509, 0.23], [0.2284, 0.7624], [0.2192, 0.9124], [0.2137, 0.1237], [0.8752, 0.7894]]","1","0","converged, zero mistakes in a full pass","false (XOR)","true (XOR)","Reset (new random draw)","Process next point","x0","x1",[11,280,281],{},"Feel free to click a lot. The boundary will keep rotating and sliding forever, trying to find a spot that doesn't exist.",[15,283,285],{"id":284},"what-this-means-and-what-comes-next","What this means (and what comes next)",[11,287,288,289,292],{},"This result isn't an isolated curiosity, it's historically ",[136,290,291],{},"the"," limit that defined the early years of neural networks: a single neuron, with a linear decision boundary, has a ceiling on what it can express, and XOR sits above that ceiling. The way out, and this is where this playlist hasn't reached yet, is stacking neurons: an MLP (multi-layer network) can solve XOR by combining two linear boundaries in a hidden layer before deciding the final output. I saved two papers exactly for this moment, in this playlist's reference folder: Paul Werbos's doctoral thesis (1974), the earliest known derivation of the backpropagation algorithm, and Rumelhart, Hinton, and Williams's paper (Nature, 1986), the one that actually popularized the algorithm and unlocked practical MLP training.",[11,294,295,296,300],{},"The professor's repository doesn't have that notebook yet. The 3 lectures I covered here (perceptron, Adaline, cost functions, multiclass) are everything published as of right now, and the XOR cliffhanger is exactly where his course also currently stands. ",[82,297,299],{"href":298},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","As I said back in this playlist's first post",", this is a living playlist: as soon as the professor publishes the MLP and backpropagation lecture, I'll come back here to continue where I left off, with Werbos and Rumelhart-Hinton-Williams already waiting to step in.",[15,302,304],{"id":303},"wrapping-up","Wrapping up",[306,307,308,322],"table",{},[309,310,311],"thead",{},[312,313,314,319],"tr",{},[315,316,318],"th",{"align":317},"left","What I already knew",[315,320,321],{"align":317},"What this lecture settled",[323,324,325,334,342],"tbody",{},[312,326,327,331],{},[328,329,330],"td",{"align":317},"A perceptron separates anything given enough time",[328,332,333],{"align":317},"It only separates what's linearly separable: XOR never converges, no matter how much I train it",[312,335,336,339],{},[328,337,338],{"align":317},"Weights grow when the model is wrong",[328,340,341],{"align":317},"On XOR, the weights shrink toward zero instead, the model gives up rather than keep chasing something that doesn't exist",[312,343,344,347],{},[328,345,346],{"align":317},"A neural network is \"just\" one neuron with weights",[328,348,349],{"align":317},"A single neuron has an expressiveness ceiling: stacking layers (MLP) is what exists to break through it",[15,351,353],{"id":352},"practical-application","Practical application",[11,355,356,357,361],{},"There's no classic real dataset for \"XOR\" to go fetch (it's a logical function, not a real-world phenomenon), so the practical application here is showing the same linear-separability wall shows up in real data too, not just toy logic gates. I use the two-moons dataset ",[82,358,360],{"href":359},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fdbscan-semi-supervised","that already showed up in Pattern Recognition",", known to not be linearly separable.",[20,363,365],{"className":22,"code":364,"language":24,"meta":25,"style":25},"from sklearn.datasets import make_moons\nX, y = make_moons(n_samples=300, noise=0.05, random_state=42)\n",[27,366,367,372],{"__ignoreMap":25},[30,368,369],{"class":32,"line":33},[30,370,371],{},"from sklearn.datasets import make_moons\n",[30,373,374],{"class":32,"line":39},[30,375,376],{},"X, y = make_moons(n_samples=300, noise=0.05, random_state=42)\n",[306,378,379,390],{},[309,380,381],{},[312,382,383,386],{},[315,384,385],{"align":317},"Model",[315,387,389],{"align":388},"right","Accuracy",[323,391,392,400],{},[312,393,394,397],{},[328,395,396],{"align":317},"Perceptron (linear boundary)",[328,398,399],{"align":388},"0.867",[312,401,402,410],{},[328,403,404,405,409],{"align":317},"KNN (K=5, ",[82,406,408],{"href":407},"\u002Fen\u002Fplaylists\u002Fpattern-recognition\u002Fknn-classifier","already seen in Pattern Recognition"," as a contrast)",[328,411,412],{"align":388},[136,413,142],{},[11,415,416],{},"The perceptron doesn't get stuck as badly as on XOR (0.867 isn't 0.5, because the two moons have an approximately linear separation over most of the space, it's only the curvature at the tips that fools a straight line), but it lags well behind KNN, which has no such ceiling because it doesn't depend on a single straight boundary. It's the same structural limit as XOR, just in a softer version: any problem where the true boundary isn't a line will expose this limitation, sooner or later.",[418,419,420],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":25,"searchDepth":39,"depth":39,"links":422},[423,424,425,426,427,428,429],{"id":17,"depth":39,"text":18},{"id":108,"depth":39,"text":109},{"id":191,"depth":39,"text":192},{"id":256,"depth":39,"text":257},{"id":284,"depth":39,"text":285},{"id":303,"depth":39,"text":304},{"id":352,"depth":39,"text":353},null,"2026-08-20","Lecture 3d, the last one the professor has published so far: the same perceptron that learns OR and AND effortlessly gets stuck on XOR, pinned at 50% accuracy, weights collapsed to nearly zero. I close this first batch of the playlist exactly where the real course also stands: right at the threshold before the MLP.","md",{},true,"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron","neural-networks",{"title":6,"description":432},"published","en\u002Fplaylists\u002Fneural-networks\u002Fxor-o-limite-do-perceptron",[442,443,444],"xor","linear-separability","mlp","oQBoUTmug8QCiLbYCtWQrqgS5dp764djz4TYa0q3Daw",1787338984664]