From Classification to Regression, and the One-Line Solution

Lectures 2c and 2d. The code structure barely changes, but the meaning shifts a lot: sign() goes out, a continuous prediction comes in, and at the end a closed-form computation replaces hundreds of iterations.

Swapping classification for regression: just drop the sign()

def createRegressionDataset(n=20):
  X = np.random.rand(n,1)
  coef = 0.7
  intercept = 0.2
  noise = np.random.randn(n,1) * 0.1
  y = X * coef + intercept + noise
  return X, y.reshape(-1)

Unlike the earlier datasets, y here isn't -1 or +1 anymore, it's a continuous number, 0.7 * x + 0.2 plus a small Gaussian noise. There's no class to get right, there's a line to find.

class LinearRegression(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    self.w_ = np.random.rand(X.shape[1])
    self.b_ = np.random.rand()
    for _ in range(self.max_iter):
      y_pred = self.predict(X)
      error = y - y_pred
      self.w_ += np.dot(X.T, error) * self.learning_rate
      self.b_ += np.sum(error) * self.learning_rate
    return self

  def predict(self, X):
    return X @ self.w_ + self.b_

Notice how this is almost identical to the vectorized perceptron from last post: same X.T @ error to update the weight, same sum(error) to update the bias. The two differences are small in code but change the whole meaning: first, predict no longer goes through sign(), the prediction stays continuous (linear activation, the same idea Aggarwal describes as the simplest activation function there is, $\Phi(v) = v$). Second, a learning_rate shows up multiplying the update. In the perceptron this learning rate didn't even exist, it was implicitly 1. Aggarwal calls this an interesting quirk of the perceptron, you can fix the rate at 1 because it only rescales the weight, not the direction of the adjustment. Here, with a continuous error instead of an error in {-2,0,+2}, the adjustment's magnitude can end up too big or too small depending on the error's scale, and that's why an explicit learning rate becomes necessary to control the step size, the exact subject that already earned a whole post in the other playlist.

Output: RMSE 0.0948, weights [0.706], bias 0.204. Pretty close to the true generator (0.7 and 0.2), the difference is just the noise baked in on purpose.

One honest note: the notebook's next cell calls createDataset (not createRegressionDataset), accuracy_score, and plotHyperplan, names that don't exist in this notebook, only in the previous lecture's. That only ran because the professor's Colab still had the previous session in memory (a variable reused lecture to lecture). Running this notebook fresh would break that cell. I don't reproduce it here, since it doesn't actually test the regression model that was just trained.

The normal equation: the same question, solved without iterating

def include_bias(X):
  return np.hstack((np.ones((X.shape[0],1)), X))

class NormalEquation(BaseEstimator, ClassifierMixin):
  def fit(self, X, y):
    X = include_bias(X)
    self.w_ = np.linalg.pinv(X) @ y
    return self

  def predict(self, X):
    X = include_bias(X)
    return X @ self.w_

Two new things here. The first is include_bias: sticks a column of 1s in front of X. This is literally the trick I described in words in the last post, bias as the weight of a phantom variable always worth 1, now written as real code instead of a separate b_. With that extra column, the bias just becomes another regular weight inside w_.

The second is NormalEquation itself: no loop, no learning_rate, just np.linalg.pinv(X) @ y. That's the closed-form solution I already explored in detail back in Pattern Recognition, the same question ("which line minimizes squared error") solved directly, in a single computation, instead of stepping down gradually. The technical difference here is the method: there, I implemented it via Gauss-Jordan elimination with pivoting. The professor uses the pseudo-inverse (pinv), which solves via SVD decomposition under the hood. The advantage of the pseudo-inverse is that it never gets stuck: even if X has redundant (linearly dependent) columns and X^T X can't be inverted the traditional way, pinv still returns a valid answer (the smallest-norm one among the infinitely many possible solutions). Plain Gauss-Jordan elimination, in that same case, simply breaks.

Output: RMSE 0.12364226682065012, weights [0.25402951 0.61056989] (bias and coefficient, in that order, because of include_bias).

And the gradient descent from the previous cell, on the same dataset, landed at RMSE 0.12364226680246916, essentially identical weights. Two completely different computations (one iterative, one closed-form) converging to the exact same place, down to the seventh decimal. I reproduced this myself with a seeded dataset (np.random.seed(7)) to confirm it wasn't a one-run coincidence: GD gave weights [0.6478, bias 0.2041], normal equation gave [bias 0.2041, coef 0.6478], identical RMSE down to the ninth decimal in both.

Interactive: find the line yourself

Before watching the machine solve it, try solving it by hand. This is the same seeded dataset from above (20 real points generated by coef=0.7, intercept=0.2 plus noise), drag the w and b sliders and watch the total error change live.

Wrapping up

Practical application

I tested gradient descent against the normal equation on the real 50-house dataset that already showed up in the other playlist, using just square_feet to predict price, with nothing normalized first.

w, b = fit_gd(X, y, max_iter=1000, learning_rate=0.01)  # not normalized

Output: the weight becomes -inf by iteration 71. Gradient descent diverges completely.

Expected: it's the same feature-scaling lesson from the other playlist, just rediscovered here inside a neural-network context. square_feet lives in the hundreds and price in the hundreds of thousands, so the gradient is huge and a 0.01 step blows up.

Normalizing square_feet before running gradient descent, it converges and lands at exactly the same RMSE the normal equation finds directly on the raw data, no normalization needed. Makes sense: the normal equation solves the linear system in one shot, so the variables' scale only affects the computation's numerical stability, not whether it converges at all (unlike gradient descent, which can literally diverge if the step is too big for the data's scale).