Classification by Threshold: When Regression Becomes a Decision

Lecture 3, split into two parts: classification (3a) and normalization (3b). And the professor opens with a provocation, reusing the exact same regression class from the previous post to solve a problem that isn't quite regression.

The dataset: breast cancer, 569 patients

The professor switches datasets: now it's scikit-learn's load_breast_cancer, 569 patients, 30 variables measured from a biopsy image (radius, texture, perimeter, area, and so on), and a binary target, malignant or benign. Before picking any variable, he computes each of the 30's correlation with the target and sorts them.

correlations = df.corr()['target'].drop('target')

Output: the most correlated is worst concave points (-0.79), followed by worst perimeter (-0.78) and mean concave points (-0.78). That tracks clinically: concave points along the tumor's contour and its perimeter are exactly the kind of thing a pathologist looks at to suspect malignancy.

He keeps worst concave points as the only input variable for now, saving the other 29 for later.

The binary target, flipped on purpose

Scikit-learn ships this dataset with a convention already: target=0 is malignant, target=1 is benign. The professor flips that by hand:

y = np.array(y==0, dtype=int)  # now 1 = malignant

Why? Because the more common convention in a detection problem (fraud, disease, defect) is to let label 1 represent the case you want to flag, the rare, concerning event, not the "normal" case. After the flip, the fraction of malignant patients is 0.373, meaning 37% of the 569 cases.

Regression to classify?

This is where the lecture gets provocative. Instead of building a classifier from scratch, the professor reuses the exact same LinearRegressor class from the previous post (the pseudo-inverse one) and just trains it on the 0/1 target as if it were any continuous number:

regressor = LinearRegressor()
regressor.fit(X_train, y_train)
y_pred = regressor.predict(X_train)

Output: w_ = [-0.303, 5.844], MSE = 0.0863.

An MSE of 0.0863 looks great at first glance (way below 1), but watch what happens when the professor measures accuracy on that same result:

def accuracy(y, y_pred):
    return np.sum(y == y_pred) / len(y)

print(accuracy(y_train, y_pred))

Output: 0.0. Zero percent correct.

That's not a bug, it's a deliberate trap. y_pred is regression's continuous output, something like 0.312 or 0.847, never exactly 0 or 1. Comparing y == y_pred with == between an integer and a float almost never matches, so accuracy computed this way always lands on zero, no matter how good the model actually is. One piece is missing: turning that continuous number into a decision.

The threshold: from continuous number to decision

The fix is small: compare the regression's output against 0.5 before turning it into a label.

class LinearClassifier(BaseEstimator, ClassifierMixin):
    def fit(self, X, y):
        X = include_bias(X)
        self.w_ = np.linalg.pinv(X) @ y
        return self
    def predict(self, X):
        X = include_bias(X)
        y_pred = X @ self.w_
        return (y_pred.reshape(X.shape[0],) > 0.5).astype(int)

Output: training accuracy is now 0.9165. On the test set, 0.8947.

The exact same fit (the same w_), just now with a threshold deciding the final label. This is precisely what Bishop calls a linear discriminant function: $y(\mathbf{x}) = \mathbf{w}^{\mathsf{T}}\mathbf{x} + w_0$, and the point where $y(\mathbf{x})$ crosses the decision threshold is the decision boundary. In Bishop's classic case that boundary sits at $y(\mathbf{x})=0$, here it sits at $y(\mathbf{x})=0.5$ because the target is coded 0/1 instead of centered at zero, but the geometric idea is identical: on either side of a line, the decision flips.

An elegant identity

The professor adds accuracy to MSE (both computed over the already-thresholded predictions, 0 or 1):

accuracy(y_train, y_pred) + mean_squared_error(y_train, y_pred)

Output: exactly 1.0.

That's not a coincidence. When y and y_pred are only 0 or 1, the squared error (y - y_pred)² equals 0 when it's right and 1 when it's wrong, exactly the same as the absolute error. So the mean MSE is literally the error rate, and error rate plus accuracy always adds up to 1. A cute bit of algebra, but it shows something worth remembering: MSE over a thresholded binary label is just another name for "fraction of mistakes."

The best fit for the continuous error isn't the best fit for classifying

Here's the subtlest point of the lecture. The professor takes the w_ the pseudo-inverse found (the one minimizing squared error on the continuous 0/1 target) and sweeps values near coefficient w_[1], measuring MSE after thresholding at each one:

w1_values = np.linspace(original_w[1] - 3, original_w[1] + 3, 100)
for w1_candidate in w1_values:
    classifier.w_ = np.array([original_w[0], w1_candidate])
    mse_values.append(mean_squared_error(y_train, classifier.predict(X_train)))

Output: the lowest MSE found (0.0791) happens at w_[1] = 5.5709, not the w_[1] = 5.8436 the pseudo-inverse returned.

The fit that minimizes squared error before thresholding isn't the same fit that minimizes error after thresholding. Two similar-looking objectives, but not identical ones. Bishop explains why: least squares corresponds to assuming the noise in the data follows a Gaussian distribution (chapter 3), which makes complete sense for predicting a continuous number. But a binary label doesn't have Gaussian noise, and that's why least squares applied directly to classification suffers a specific problem: points that are already "too correct," sitting far on the right side of the boundary, still pull the fit, because they contribute squared error even though they don't need to. Bishop shows this with an example where adding extra points, all on the correct side of the boundary, moves the decision boundary for the worse, something that doesn't happen with a method actually designed for classification (logistic regression, which the course hasn't reached yet, but exists precisely to fix this flaw).

All 30 variables

Same LinearClassifier, now trained on all 30 columns instead of just 1:

modelo = LinearClassifier()
modelo.fit(X_train, y_train)
print(accuracy_score(y_test, modelo.predict(X_test)))

Output: test accuracy 0.9561, versus 0.8947 with just one variable.

The same jump I've already seen in the previous two posts: more information, same algorithm, same threshold, and the result improves a lot.

Normalizing, but for a different reason

The second half of the lecture (aula03b) switches topics: instead of thresholded regression, the professor uses scikit-learn's KNeighborsClassifier (KNN, K nearest neighbors, a topic that gets its own whole lecture later, here it just makes a cameo) straight on the cancer dataset, no normalization at all:

modelo = KNeighborsClassifier()
modelo.fit(X_train, y_train)
modelo.score(X_test, y_test)

Output: 0.9298.

Then he normalizes (min-max, into the 0-1 range, computed from training statistics only and applied the same way to test) and repeats:

Output: 0.9649.

And confirms that standardizing (z-score, the same normalization from the specialization playlist) gives the same result: 0.9649 too.

I'd already seen normalization matter before, but for a different reason: back in the specialization playlist, normalizing helped gradient descent converge faster, because the cost bowl was less elongated. Here the reason is different. KNN decides the class by looking at the distance between patients, usually Euclidean distance, the square root of the sum of squared differences across each variable. If one variable (say mean area, ranging from 143 to 2501) has a scale hundreds of times bigger than another (say mean smoothness, ranging from 0.05 to 0.16), the distance ends up being decided almost entirely by the large-scale variable, the other 29 barely register. Normalizing puts everyone on the same ruler before measuring distance, and suddenly all 30 variables genuinely contribute, not just one.

Wrapping up

And it leaves a clean hook for the next lecture, which this playlist hasn't reached yet: least squares applied to classification has a structural bias (Bishop showed why), and the right fix is swapping the error function for one actually designed for classification, logistic regression.

Practical application

I repeat the single-variable classifier (worst concave points) and visualize, patient by patient in the test set, where it gets it right and where it doesn't, together with the actual decision boundary.

classifier = LinearClassifier()
classifier.fit(X_train, y_train)  # 1 variable: worst concave points
y_pred_test = classifier.predict(X_test)

I redid this fit with random_state=42 (the original notebook doesn't fix the seed, so the numbers shift on every run, here I use a fixed train/test split so I can show the exact patients) and got w_ = [-0.296, 5.841], which puts the decision boundary at worst concave points ≈ 0.136: below that the model predicts benign, above it, malignant.

The red triangles (the mistakes) cluster right around the dashed boundary, on both sides, exactly where I'd expect: those are the ambiguous cases, where worst concave points alone doesn't separate malignant from benign well. Far from the boundary, on either side, the model is right almost every time. That tracks: one variable alone carries plenty of signal (it had a 0.79 correlation, after all), but it isn't enough to never miss, and that's exactly why all 30 variables together (0.9561) beat the single variable (0.8947).