[{"data":1,"prerenderedAt":1474},["ShallowReactive",2],{"lang-switch-post-\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo":3,"post-en-neural-networks-funcoes-de-custo":4},"\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",{"id":5,"title":6,"body":7,"cover":1459,"date":1460,"description":1461,"extension":1462,"meta":1463,"navigation":65,"order":62,"path":1464,"playlist":1465,"seo":1466,"status":1467,"stem":1468,"tags":1469,"__hash__":1473},"posts\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo.md","Swapping the Cost Function Like Changing Clothes",{"type":8,"value":9,"toc":1451},"minimark",[10,19,24,158,192,221,230,234,307,340,344,380,622,658,672,693,753,784,800,1081,1119,1130,1145,1149,1249,1258,1275,1279,1340,1344,1362,1439,1447],[11,12,13,14,18],"p",{},"Lectures 3a and 3b. So far I've seen 4 different algorithms (perceptron, vectorized perceptron, Adaline via pseudo-inverse, Adaline via gradient) as somewhat separate things. This lecture shows they're actually the ",[15,16,17],"strong",{},"same recipe",", just swapping one ingredient: the cost function.",[20,21,23],"h2",{"id":22},"first-the-training-algorithm-becomes-pluggable","First, the training algorithm becomes pluggable",[25,26,31],"pre",{"className":27,"code":28,"language":29,"meta":30,"style":30},"language-python shiki shiki-themes github-light github-dark","class TrainingAlgorithm(ABC):\n  @abstractmethod\n  def get_w(self, X, y):\n    pass\n\nclass PseudoInverse(TrainingAlgorithm):\n  def get_w(self, X, y):\n    return np.linalg.pinv(X) @ y\n\nclass NeuralNetwork(BaseEstimator, ClassifierMixin):\n  def __init__(self, training_algorithm=PseudoInverse()):\n    self.training_algorithm = training_algorithm\n\n  def fit(self, X, y):\n    X = include_bias(X)\n    self.w_ = self.training_algorithm.get_w(X, y)\n    return self\n\n  def predict(self, X):\n    X = include_bias(X)\n    return np.sign(X @ self.w_)\n","python","",[32,33,34,42,48,54,60,67,73,78,84,89,95,101,107,112,118,124,130,136,141,147,152],"code",{"__ignoreMap":30},[35,36,39],"span",{"class":37,"line":38},"line",1,[35,40,41],{},"class TrainingAlgorithm(ABC):\n",[35,43,45],{"class":37,"line":44},2,[35,46,47],{},"  @abstractmethod\n",[35,49,51],{"class":37,"line":50},3,[35,52,53],{},"  def get_w(self, X, y):\n",[35,55,57],{"class":37,"line":56},4,[35,58,59],{},"    pass\n",[35,61,63],{"class":37,"line":62},5,[35,64,66],{"emptyLinePlaceholder":65},true,"\n",[35,68,70],{"class":37,"line":69},6,[35,71,72],{},"class PseudoInverse(TrainingAlgorithm):\n",[35,74,76],{"class":37,"line":75},7,[35,77,53],{},[35,79,81],{"class":37,"line":80},8,[35,82,83],{},"    return np.linalg.pinv(X) @ y\n",[35,85,87],{"class":37,"line":86},9,[35,88,66],{"emptyLinePlaceholder":65},[35,90,92],{"class":37,"line":91},10,[35,93,94],{},"class NeuralNetwork(BaseEstimator, ClassifierMixin):\n",[35,96,98],{"class":37,"line":97},11,[35,99,100],{},"  def __init__(self, training_algorithm=PseudoInverse()):\n",[35,102,104],{"class":37,"line":103},12,[35,105,106],{},"    self.training_algorithm = training_algorithm\n",[35,108,110],{"class":37,"line":109},13,[35,111,66],{"emptyLinePlaceholder":65},[35,113,115],{"class":37,"line":114},14,[35,116,117],{},"  def fit(self, X, y):\n",[35,119,121],{"class":37,"line":120},15,[35,122,123],{},"    X = include_bias(X)\n",[35,125,127],{"class":37,"line":126},16,[35,128,129],{},"    self.w_ = self.training_algorithm.get_w(X, y)\n",[35,131,133],{"class":37,"line":132},17,[35,134,135],{},"    return self\n",[35,137,139],{"class":37,"line":138},18,[35,140,66],{"emptyLinePlaceholder":65},[35,142,144],{"class":37,"line":143},19,[35,145,146],{},"  def predict(self, X):\n",[35,148,150],{"class":37,"line":149},20,[35,151,123],{},[35,153,155],{"class":37,"line":154},21,[35,156,157],{},"    return np.sign(X @ self.w_)\n",[11,159,160,161,164,165,168,169,172,173,176,177,180,181,184,185,188,189,191],{},"This is the ",[15,162,163],{},"Strategy"," design pattern: ",[32,166,167],{},"NeuralNetwork"," no longer knows ",[15,170,171],{},"how"," the weights get computed, just that there's a ",[32,174,175],{},"training_algorithm"," object with a ",[32,178,179],{},"get_w"," method. Swapping ",[32,182,183],{},"PseudoInverse()"," for ",[32,186,187],{},"SGD()"," in the constructor swaps out the entire training algorithm, without touching ",[32,190,167],{},".",[193,194,195],"blockquote",{},[11,196,197,204,205,208,209,212,213,204,218,220],{},[15,198,199,200,203],{},"Output (",[32,201,202],{},"PseudoInverse","):"," accuracy ",[32,206,207],{},"0.95",", weights ",[32,210,211],{},"[-1.905, 2.656, 1.049]",".\n",[15,214,199,215,203],{},[32,216,217],{},"SGD",[32,219,207],{},", essentially identical weights.",[11,222,223,224,229],{},"Confirms again, now with more cleanly organized code, what ",[225,226,228],"a",{"href":227},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fadaline-regra-delta","I already saw in the last two posts",": pseudo-inverse and gradient descent solve the exact same problem.",[20,231,233],{"id":232},"now-the-cost-function-becomes-pluggable-too","Now the cost function becomes pluggable too",[25,235,237],{"className":27,"code":236,"language":29,"meta":30,"style":30},"class CostFunction(ABC):\n  @abstractstaticmethod\n  def get_cost(y, y_pred):\n    pass\n  @abstractstaticmethod\n  def get_gradient(X, y, y_pred):\n    pass\n\nclass WidrowHoff(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.mean((y-y_pred)**2)\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y-y_pred)\n",[32,238,239,244,249,254,258,262,267,271,275,280,285,289,294,298,302],{"__ignoreMap":30},[35,240,241],{"class":37,"line":38},[35,242,243],{},"class CostFunction(ABC):\n",[35,245,246],{"class":37,"line":44},[35,247,248],{},"  @abstractstaticmethod\n",[35,250,251],{"class":37,"line":50},[35,252,253],{},"  def get_cost(y, y_pred):\n",[35,255,256],{"class":37,"line":56},[35,257,59],{},[35,259,260],{"class":37,"line":62},[35,261,248],{},[35,263,264],{"class":37,"line":69},[35,265,266],{},"  def get_gradient(X, y, y_pred):\n",[35,268,269],{"class":37,"line":75},[35,270,59],{},[35,272,273],{"class":37,"line":80},[35,274,66],{"emptyLinePlaceholder":65},[35,276,277],{"class":37,"line":86},[35,278,279],{},"class WidrowHoff(CostFunction):\n",[35,281,282],{"class":37,"line":91},[35,283,284],{},"  @staticmethod\n",[35,286,287],{"class":37,"line":97},[35,288,253],{},[35,290,291],{"class":37,"line":103},[35,292,293],{},"    return np.mean((y-y_pred)**2)\n",[35,295,296],{"class":37,"line":109},[35,297,284],{},[35,299,300],{"class":37,"line":114},[35,301,266],{},[35,303,304],{"class":37,"line":120},[35,305,306],{},"    return X.T @ (y-y_pred)\n",[11,308,309,311,312,315,316,319,320,323,324,327,328,331,332,335,336,339],{},[32,310,217],{}," now also takes a ",[32,313,314],{},"cost_function",", and uses ",[32,317,318],{},"self.cost_function.get_gradient(...)"," instead of computing the gradient by hand. ",[32,321,322],{},"WidrowHoff"," is ",[225,325,326],{"href":227},"exactly the delta rule from the last post",": continuous error (",[32,329,330],{},"y - y_pred",", no ",[32,333,334],{},"sign()",") times the input. Swapping the cost function here means swapping ",[15,337,338],{},"what \"error\" means",", without touching the training loop.",[20,341,343],{"id":342},"every-cost-function-recovers-a-different-algorithm","Every cost function recovers a different algorithm",[25,345,347],{"className":27,"code":346,"language":29,"meta":30,"style":30},"class SmoothedSurrogate(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - np.sign(y_pred))\n",[32,348,349,354,358,362,367,371,375],{"__ignoreMap":30},[35,350,351],{"class":37,"line":38},[35,352,353],{},"class SmoothedSurrogate(CostFunction):\n",[35,355,356],{"class":37,"line":44},[35,357,284],{},[35,359,360],{"class":37,"line":50},[35,361,253],{},[35,363,364],{"class":37,"line":56},[35,365,366],{},"    return np.sum(np.maximum(np.zeros(y.shape), -y * y_pred))\n",[35,368,369],{"class":37,"line":62},[35,370,284],{},[35,372,373],{"class":37,"line":69},[35,374,266],{},[35,376,377],{"class":37,"line":75},[35,378,379],{},"    return X.T @ (y - np.sign(y_pred))\n",[11,381,382,383,386,387,390,391,395,396,399,400,403,404,613,614,617,618,621],{},"Notice the ",[32,384,385],{},"np.sign(y_pred)"," inside the gradient: this goes back to measuring error ",[15,388,389],{},"after"," the threshold, exactly like ",[225,392,394],{"href":393},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Fmcculloch-pitts-perceptron","Rosenblatt's perceptron",". The name ",[32,397,398],{},"SmoothedSurrogate"," matches what Aggarwal calls the ",[15,401,402],{},"perceptron criterion",": ",[35,405,408,473],{"className":406},[407],"katex",[35,409,412],{"className":410},[411],"katex-mathml",[413,414,416],"math",{"xmlns":415},"http:\u002F\u002Fwww.w3.org\u002F1998\u002FMath\u002FMathML",[417,418,419,468],"semantics",{},[420,421,422,426,430,433,436,440,444,448,451,454,457,465],"mrow",{},[423,424,425],"mi",{},"L",[427,428,429],"mo",{},"=",[423,431,432],{},"max",[427,434,435],{},"⁡",[427,437,439],{"stretchy":438},"false","(",[441,442,443],"mn",{},"0",[427,445,447],{"separator":446},"true",",",[427,449,450],{},"−",[423,452,453],{},"y",[427,455,456],{},"⋅",[458,459,460,462],"mover",{"accent":446},[423,461,453],{},[427,463,464],{},"^",[427,466,467],{"stretchy":438},")",[469,470,472],"annotation",{"encoding":471},"application\u002Fx-tex","L = \\max(0, -y \\cdot \\hat{y})",[35,474,477,503,547],{"className":475,"ariaHidden":446},[476],"katex-html",[35,478,481,486,491,496,500],{"className":479},[480],"base",[35,482],{"className":483,"style":485},[484],"strut","height:0.6833em;",[35,487,425],{"className":488},[489,490],"mord","mathnormal",[35,492],{"className":493,"style":495},[494],"mspace","margin-right:0.2778em;",[35,497,429],{"className":498},[499],"mrel",[35,501],{"className":502,"style":495},[494],[35,504,506,510,514,518,521,525,529,532,536,540,544],{"className":505},[480],[35,507],{"className":508,"style":509},[484],"height:1em;vertical-align:-0.25em;",[35,511,432],{"className":512},[513],"mop",[35,515,439],{"className":516},[517],"mopen",[35,519,443],{"className":520},[489],[35,522,447],{"className":523},[524],"mpunct",[35,526],{"className":527,"style":528},[494],"margin-right:0.1667em;",[35,530,450],{"className":531},[489],[35,533,453],{"className":534,"style":535},[489,490],"margin-right:0.0359em;",[35,537],{"className":538,"style":539},[494],"margin-right:0.2222em;",[35,541,456],{"className":542},[543],"mbin",[35,545],{"className":546,"style":539},[494],[35,548,550,553,609],{"className":549},[480],[35,551],{"className":552,"style":509},[484],[35,554,557],{"className":555},[489,556],"accent",[35,558,562,600],{"className":559},[560,561],"vlist-t","vlist-t2",[35,563,566,595],{"className":564},[565],"vlist-r",[35,567,571,582],{"className":568,"style":570},[569],"vlist","height:0.6944em;",[35,572,574,579],{"style":573},"top:-3em;",[35,575],{"className":576,"style":578},[577],"pstrut","height:3em;",[35,580,453],{"className":581,"style":535},[489,490],[35,583,584,587],{"style":573},[35,585],{"className":586,"style":578},[577],[35,588,592],{"className":589,"style":591},[590],"accent-body","left:-0.1944em;",[35,593,464],{"className":594},[489],[35,596,599],{"className":597},[598],"vlist-s","​",[35,601,603],{"className":602},[565],[35,604,607],{"className":605,"style":606},[569],"height:0.1944em;",[35,608],{},[35,610,467],{"className":611},[612],"mclose",", zero once the point is on the right side, growing linearly when it's wrong. No coincidence the accuracy hits ",[32,615,616],{},"1.0",": this cost function, plugged into this generic framework, ",[15,619,620],{},"is"," the original perceptron again, just expressed in the language of \"cost function\" instead of \"update rule\".",[25,623,625],{"className":27,"code":624,"language":29,"meta":30,"style":30},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_cost(y, y_pred):\n    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - expit(y_pred))\n",[32,626,627,632,636,640,645,649,653],{"__ignoreMap":30},[35,628,629],{"class":37,"line":38},[35,630,631],{},"class LogLikehood(CostFunction):\n",[35,633,634],{"class":37,"line":44},[35,635,284],{},[35,637,638],{"class":37,"line":50},[35,639,253],{},[35,641,642],{"class":37,"line":56},[35,643,644],{},"    return np.sum(np.maximum(np.zeros(y.shape), 1 - y * y_pred))\n",[35,646,647],{"class":37,"line":62},[35,648,284],{},[35,650,651],{"class":37,"line":69},[35,652,266],{},[35,654,655],{"class":37,"line":75},[35,656,657],{},"    return X.T @ (y - expit(y_pred))\n",[193,659,660],{},[11,661,662,204,665,208,668,671],{},[15,663,664],{},"Output:",[15,666,667],{},"0.65",[32,669,670],{},"[-60.78, 27.44, -24.79]",". Much worse than anything I've seen so far, and the weights got huge.",[11,673,674,675,678,679,682,683,686,687,689,690,692],{},"Two things wrong here, worth separating. First, a detail that doesn't affect the outcome: this class's ",[32,676,677],{},"get_cost"," uses the hinge loss formula (",[32,680,681],{},"max(0, 1 - y·ŷ)","), not an actual log-likelihood formula. That doesn't break anything in practice because ",[32,684,685],{},"get_gradient"," is the only thing ",[32,688,217],{}," calls, ",[32,691,677],{}," never gets used during training, it's leftover residue from copying and pasting from another cell.",[11,694,695,696,699,700,702,703,706,707,323,709,712,713,716,717,720,721,724,725,728,729,731,732,735,736,738,739,741,742,745,746,749,750,752],{},"The second problem is real, and explains the bad accuracy: ",[32,697,698],{},"expit"," (the sigmoid function) only returns values between ",[32,701,443],{}," and ",[32,704,705],{},"1",", but the label ",[32,708,453],{},[32,710,711],{},"-1"," or ",[32,714,715],{},"+1",". For class ",[32,718,719],{},"y=-1",", the error ",[32,722,723],{},"y - expit(y_pred)"," can ",[15,726,727],{},"never"," get close to zero, because ",[32,730,698],{}," never goes negative: even with an infinitely confident correctly-classified prediction, ",[32,733,734],{},"-1 - expit(y_pred)"," stays pinned near ",[32,737,711],{},", never ",[32,740,443],{},". I checked this by hand: ",[32,743,744],{},"expit(-1000) = 0.0",", so ",[32,747,748],{},"-1 - expit(-1000) = -1.0"," exactly, not ",[32,751,443],{},". The gradient for half the points never vanishes, so training never settles, and the weights keep growing trying to compensate for an error that's structurally impossible to zero out.",[25,754,756],{"className":27,"code":755,"language":29,"meta":30,"style":30},"class LogLikehood(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    return X.T @ (y - tanh(y_pred))\n\nmodel = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[32,757,758,762,766,770,775,779],{"__ignoreMap":30},[35,759,760],{"class":37,"line":38},[35,761,631],{},[35,763,764],{"class":37,"line":44},[35,765,284],{},[35,767,768],{"class":37,"line":50},[35,769,266],{},[35,771,772],{"class":37,"line":56},[35,773,774],{},"    return X.T @ (y - tanh(y_pred))\n",[35,776,777],{"class":37,"line":62},[35,778,66],{"emptyLinePlaceholder":65},[35,780,781],{"class":37,"line":69},[35,782,783],{},"model = NeuralNetwork(training_algorithm=SGD(max_iter=10000, cost_function=LogLikehood()))\n",[193,785,786],{},[11,787,788,204,790,208,792,795,796,799],{},[15,789,664],{},[15,791,616],{},[32,793,794],{},"[-9.84, 12.40, 8.37]"," (with ",[32,797,798],{},"max_iter=10000",", ten times more iterations).",[11,801,802,803,184,805,808,809,811,812,814,815,817,818,745,821,824,825,966,967,814,1019,1076,1077,1080],{},"Swapping ",[32,804,698],{},[32,806,807],{},"tanh"," (same class redefined, Python just lets that run live in a notebook session), the problem disappears. Makes sense: ",[32,810,807],{}," ranges from ",[32,813,711],{}," to ",[32,816,715],{},", exactly the labels' range. Checked it by hand again: ",[32,819,820],{},"tanh(-1000) = -1.0",[32,822,823],{},"-1 - tanh(-1000) = 0.0",", the error genuinely reaches zero this time. Aggarwal states this relationship directly in chapter 1: ",[35,826,828,874],{"className":827},[407],[35,829,831],{"className":830},[411],[413,832,833],{"xmlns":415},[417,834,835,871],{},[420,836,837,839,841,843,846,848,850,853,855,859,861,863,865,867,869],{},[423,838,807],{},[427,840,435],{},[427,842,439],{"stretchy":438},[423,844,845],{},"v",[427,847,467],{"stretchy":438},[427,849,429],{},[441,851,852],{},"2",[427,854,456],{},[856,857,858],"mtext",{},"sigmoid",[427,860,439],{"stretchy":438},[441,862,852],{},[423,864,845],{},[427,866,467],{"stretchy":438},[427,868,450],{},[441,870,705],{},[469,872,873],{"encoding":471},"\\tanh(v) = 2 \\cdot \\text{sigmoid}(2v) - 1",[35,875,877,904,923,957],{"className":876,"ariaHidden":446},[476],[35,878,880,883,886,889,892,895,898,901],{"className":879},[480],[35,881],{"className":882,"style":509},[484],[35,884,807],{"className":885},[513],[35,887,439],{"className":888},[517],[35,890,845],{"className":891,"style":535},[489,490],[35,893,467],{"className":894},[612],[35,896],{"className":897,"style":495},[494],[35,899,429],{"className":900},[499],[35,902],{"className":903,"style":495},[494],[35,905,907,911,914,917,920],{"className":906},[480],[35,908],{"className":909,"style":910},[484],"height:0.6444em;",[35,912,852],{"className":913},[489],[35,915],{"className":916,"style":539},[494],[35,918,456],{"className":919},[543],[35,921],{"className":922,"style":539},[494],[35,924,926,929,936,939,942,945,948,951,954],{"className":925},[480],[35,927],{"className":928,"style":509},[484],[35,930,933],{"className":931},[489,932],"text",[35,934,858],{"className":935},[489],[35,937,439],{"className":938},[517],[35,940,852],{"className":941},[489],[35,943,845],{"className":944,"style":535},[489,490],[35,946,467],{"className":947},[612],[35,949],{"className":950,"style":539},[494],[35,952,450],{"className":953},[543],[35,955],{"className":956,"style":539},[494],[35,958,960,963],{"className":959},[480],[35,961],{"className":962,"style":910},[484],[35,964,705],{"className":965},[489],", and it's exactly that range shift, from ",[35,968,970,992],{"className":969},[407],[35,971,973],{"className":972},[411],[413,974,975],{"xmlns":415},[417,976,977,989],{},[420,978,979,981,983,985,987],{},[427,980,439],{"stretchy":438},[441,982,443],{},[427,984,447],{"separator":446},[441,986,705],{},[427,988,467],{"stretchy":438},[469,990,991],{"encoding":471},"(0,1)",[35,993,995],{"className":994,"ariaHidden":446},[476],[35,996,998,1001,1004,1007,1010,1013,1016],{"className":997},[480],[35,999],{"className":1000,"style":509},[484],[35,1002,439],{"className":1003},[517],[35,1005,443],{"className":1006},[489],[35,1008,447],{"className":1009},[524],[35,1011],{"className":1012,"style":528},[494],[35,1014,705],{"className":1015},[489],[35,1017,467],{"className":1018},[612],[35,1020,1022,1046],{"className":1021},[407],[35,1023,1025],{"className":1024},[411],[413,1026,1027],{"xmlns":415},[417,1028,1029,1043],{},[420,1030,1031,1033,1035,1037,1039,1041],{},[427,1032,439],{"stretchy":438},[427,1034,450],{},[441,1036,705],{},[427,1038,447],{"separator":446},[441,1040,705],{},[427,1042,467],{"stretchy":438},[469,1044,1045],{"encoding":471},"(-1,1)",[35,1047,1049],{"className":1048,"ariaHidden":446},[476],[35,1050,1052,1055,1058,1061,1064,1067,1070,1073],{"className":1051},[480],[35,1053],{"className":1054,"style":509},[484],[35,1056,439],{"className":1057},[517],[35,1059,450],{"className":1060},[489],[35,1062,705],{"className":1063},[489],[35,1065,447],{"className":1066},[524],[35,1068],{"className":1069,"style":528},[494],[35,1071,705],{"className":1072},[489],[35,1074,467],{"className":1075},[612],", that fixes the mismatch with ",[32,1078,1079],{},"±1"," labels.",[25,1082,1084],{"className":27,"code":1083,"language":29,"meta":30,"style":30},"class HingeLoss(CostFunction):\n  @staticmethod\n  def get_gradient(X, y, y_pred):\n    marginal_errors = (y * y_pred) \u003C 1\n    marginal_ys = np.copy(y)\n    marginal_ys[~marginal_errors] = 0\n    return X.T @ marginal_ys\n",[32,1085,1086,1091,1095,1099,1104,1109,1114],{"__ignoreMap":30},[35,1087,1088],{"class":37,"line":38},[35,1089,1090],{},"class HingeLoss(CostFunction):\n",[35,1092,1093],{"class":37,"line":44},[35,1094,284],{},[35,1096,1097],{"class":37,"line":50},[35,1098,266],{},[35,1100,1101],{"class":37,"line":56},[35,1102,1103],{},"    marginal_errors = (y * y_pred) \u003C 1\n",[35,1105,1106],{"class":37,"line":62},[35,1107,1108],{},"    marginal_ys = np.copy(y)\n",[35,1110,1111],{"class":37,"line":69},[35,1112,1113],{},"    marginal_ys[~marginal_errors] = 0\n",[35,1115,1116],{"class":37,"line":75},[35,1117,1118],{},"    return X.T @ marginal_ys\n",[193,1120,1121],{},[11,1122,1123,204,1125,208,1127,191],{},[15,1124,664],{},[15,1126,616],{},[32,1128,1129],{},"[-8.70, 11.59, 7.71]",[11,1131,1132,1133,1136,1137,1140,1141,1144],{},"Notice ",[32,1134,1135],{},"marginal_ys[~marginal_errors] = 0",": points that are already well classified, with room to spare (",[32,1138,1139],{},"y · ŷ ≥ 1","), get zeroed out and ",[15,1142,1143],{},"contribute nothing"," to the gradient. Only points inside the margin (or misclassified) participate in the update. That's literally SVM's central idea: only the points near the boundary (the \"support vectors\") matter for deciding where it sits. The rest of the dataset gets ignored once it's already well separated.",[20,1146,1148],{"id":1147},"all-four-curves-side-by-side","All four curves, side by side",[11,1150,1151,1152,1248],{},"The professor has a reference image saved in the notebook comparing the four penalty curves as a function of the \"margin\" (",[35,1153,1155,1177],{"className":1154},[407],[35,1156,1158],{"className":1157},[411],[413,1159,1160],{"xmlns":415},[417,1161,1162,1174],{},[420,1163,1164,1166,1168],{},[423,1165,453],{},[427,1167,456],{},[458,1169,1170,1172],{"accent":446},[423,1171,453],{},[427,1173,464],{},[469,1175,1176],{"encoding":471},"y \\cdot \\hat{y}",[35,1178,1180,1199],{"className":1179,"ariaHidden":446},[476],[35,1181,1183,1187,1190,1193,1196],{"className":1182},[480],[35,1184],{"className":1185,"style":1186},[484],"height:0.6389em;vertical-align:-0.1944em;",[35,1188,453],{"className":1189,"style":535},[489,490],[35,1191],{"className":1192,"style":539},[494],[35,1194,456],{"className":1195},[543],[35,1197],{"className":1198,"style":539},[494],[35,1200,1202,1206],{"className":1201},[480],[35,1203],{"className":1204,"style":1205},[484],"height:0.8889em;vertical-align:-0.1944em;",[35,1207,1209],{"className":1208},[489,556],[35,1210,1212,1240],{"className":1211},[560,561],[35,1213,1215,1237],{"className":1214},[565],[35,1216,1218,1226],{"className":1217,"style":570},[569],[35,1219,1220,1223],{"style":573},[35,1221],{"className":1222,"style":578},[577],[35,1224,453],{"className":1225,"style":535},[489,490],[35,1227,1228,1231],{"style":573},[35,1229],{"className":1230,"style":578},[577],[35,1232,1234],{"className":1233,"style":591},[590],[35,1235,464],{"className":1236},[489],[35,1238,599],{"className":1239},[598],[35,1241,1243],{"className":1242},[565],[35,1244,1246],{"className":1245,"style":606},[569],[35,1247],{},": positive and large means a confident correct call, negative means a mistake). I recreated the same idea here, interactively:",[1250,1251],"margin-loss-chart",{"hinge-label":1252,"logistic-label":1253,"perceptron-label":1254,"widrow-hoff-label":1255,"x-label":1256,"y-label":1257},"Hinge (SVM)","Logistic","Perceptron","Widrow-Hoff","margin (y · ŷ)","penalty",[11,1259,1260,1261,1264,1265,1268,1269,1271,1272,1274],{},"Hover over any point on the x-axis and compare all four. Notice the shapes: Widrow-Hoff is a parabola, it keeps penalizing even a point that's already correctly classified with room to spare (margin ",[32,1262,1263],{},"> 1","), because it doesn't know \"correct is correct\", it only knows how to measure distance to a continuous target. Perceptron and Hinge are the only two that ",[15,1266,1267],{},"fully zero out"," once a point is well classified (perceptron zeros as soon as it crosses ",[32,1270,443],{},", hinge requires crossing ",[32,1273,705],{},", with room to spare). Logistic never truly zeroes out, it only approaches zero, which is the price it pays for returning a smooth probability instead of a binary decision.",[20,1276,1278],{"id":1277},"wrapping-up","Wrapping up",[1280,1281,1282,1296],"table",{},[1283,1284,1285],"thead",{},[1286,1287,1288,1293],"tr",{},[1289,1290,1292],"th",{"align":1291},"left","What I already knew",[1289,1294,1295],{"align":1291},"What this lecture settled",[1297,1298,1299,1318,1332],"tbody",{},[1286,1300,1301,1305],{},[1302,1303,1304],"td",{"align":1291},"Perceptron, Adaline, batch gradient descent looked like separate algorithms",[1302,1306,1307,1308,1310,1311,1310,1314,1317],{"align":1291},"They're the same framework (",[32,1309,167],{}," + ",[32,1312,1313],{},"TrainingAlgorithm",[32,1315,1316],{},"CostFunction","), just swapping which cost function gets plugged in",[1286,1319,1320,1323],{},[1302,1321,1322],{"align":1291},"Sigmoid returns a probability between 0 and 1",[1302,1324,1325,1326,1328,1329,1331],{"align":1291},"Using sigmoid directly against a ",[32,1327,1079],{}," label locks up the gradient, because the output range doesn't match the target's range. ",[32,1330,807],{}," fixes it",[1286,1333,1334,1337],{},[1302,1335,1336],{"align":1291},"SVM uses \"support vectors\"",[1302,1338,1339],{"align":1291},"That's not empty jargon: the hinge loss gradient literally zeros out the contribution of every point that isn't a support vector",[20,1341,1343],{"id":1342},"practical-application","Practical application",[11,1345,1346,1347,1349,1350,1354,1355,1358,1359,1361],{},"I ran all four cost functions (Widrow-Hoff, perceptron criterion, hinge, and the log-likelihood version with ",[32,1348,807],{},") on Iris (",[1351,1352,1353],"em",{},"setosa"," vs. ",[1351,1356,1357],{},"versicolor","), plus the ",[32,1360,698],{}," version on purpose, to confirm the sigmoid problem isn't exclusive to the notebook's toy synthetic dataset.",[1280,1363,1364,1378],{},[1283,1365,1366],{},[1286,1367,1368,1371,1375],{},[1289,1369,1370],{"align":1291},"Cost function",[1289,1372,1374],{"align":1373},"right","Train accuracy",[1289,1376,1377],{"align":1373},"Test accuracy",[1297,1379,1380,1390,1400,1411,1424],{},[1286,1381,1382,1384,1386],{},[1302,1383,1255],{"align":1291},[1302,1385,616],{"align":1373},[1302,1387,1388],{"align":1373},[15,1389,616],{},[1286,1391,1392,1394,1396],{},[1302,1393,1254],{"align":1291},[1302,1395,616],{"align":1373},[1302,1397,1398],{"align":1373},[15,1399,616],{},[1286,1401,1402,1405,1407],{},[1302,1403,1404],{"align":1291},"Hinge",[1302,1406,616],{"align":1373},[1302,1408,1409],{"align":1373},[15,1410,616],{},[1286,1412,1413,1418,1420],{},[1302,1414,1415,1416,467],{"align":1291},"Log-likelihood (",[32,1417,807],{},[1302,1419,616],{"align":1373},[1302,1421,1422],{"align":1373},[15,1423,616],{},[1286,1425,1426,1431,1434],{},[1302,1427,1415,1428,1430],{"align":1291},[32,1429,698],{},", sigmoid)",[1302,1432,1433],{"align":1373},"0.843",[1302,1435,1436],{"align":1373},[15,1437,1438],{},"0.933",[11,1440,1441,1442,702,1444,1446],{},"Four out of five hit 100% (Iris has a generous enough margin for any of them to find a perfect boundary), and sigmoid falls behind again, this time on real data, not just the notebook's synthetic dataset. Confirms it wasn't a one-run coincidence: the range mismatch between ",[32,1443,698],{},[32,1445,1079],{}," labels genuinely hurts convergence, on every dataset I tried it on.",[1448,1449,1450],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":30,"searchDepth":44,"depth":44,"links":1452},[1453,1454,1455,1456,1457,1458],{"id":22,"depth":44,"text":23},{"id":232,"depth":44,"text":233},{"id":342,"depth":44,"text":343},{"id":1147,"depth":44,"text":1148},{"id":1277,"depth":44,"text":1278},{"id":1342,"depth":44,"text":1343},null,"2026-08-20","Lectures 3a and 3b: the professor generalizes training to accept any pluggable cost function, and each choice (Widrow-Hoff, perceptron criterion, log-likelihood, hinge) recovers a different algorithm from this playlist. Along the way, I found a real mismatch between activation function and label encoding.","md",{},"\u002Fen\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo","neural-networks",{"title":6,"description":1461},"published","en\u002Fplaylists\u002Fneural-networks\u002Ffuncoes-de-custo",[1470,1471,1472],"cost-functions","hinge-loss","logistic-regression","alBoIF9HT5FQ7eR5ld8q-Pe1ojBlQBX2mpc5QwQQgnk",1787338982677]