Neural Networks
Second playlist with Dr. Francisco Boldt (the first was Pattern Recognition), now in his Neural Networks course. Same style as always: code by hand, live, in class, no pre-baked formula slides. If you've already read the previous playlist, you know exactly what to expect.
The reference book changes: here I use Neural Networks and Deep Learning: A Textbook, by Charu Aggarwal (2018), to play the role Bishop played in the previous playlist, filling in the foundation the notebook only shows in code. A lot of the material also connects straight back to what I've already covered: the delta rule and linear regression already showed up in Pattern Recognition, and loss functions already showed up in Andrew Ng's specialization, so whenever it fits I'll pull those threads instead of reteaching from scratch.
The notebooks come from the course repository, pablobelmiro/aulasann, a fork of Dr. Boldt's own repository, fboldt/aulasann, where he publishes each lecture's code. Same convention as the previous playlist: whoever "wrote" the code is always him, the professor; the foundational explanation is mine.
One important detail this time: this course is being taught right now, live, and the repository only has the beginning of the course as of this moment (perceptron, Adaline, cost functions, and a cliffhanger right at the edge of what a single neuron can solve). Unlike the Pattern Recognition playlist, which I only started once the whole course had already ended, this one is a living playlist: it grows every time the professor publishes a new lecture, and I come back to keep going. We start at the very beginning: the simplest neuron there is.
Playlists · 7 posts
- Perceptron: the First Neuron That Learns on Its Own
Lecture 1a: the professor implements the Perceptron Learning Algorithm from scratch, no bias yet. I tell the story of two papers that came before it and find a real bug hiding in the algorithm's stopping condition.
- Bias and Vectorization: Unlocking the Perceptron
Lectures 2a and 2b: the professor adds bias to the perceptron (solving exactly the limit that closed last lecture) and then vectorizes the whole weight update, swapping the point-by-point loop for a single computation over the entire dataset.
- From Classification to Regression, and the One-Line Solution
Lectures 2c and 2d: the professor swaps the binary sign for a continuous line (linear regression) and then replaces the whole gradient descent loop with a single closed-form computation, the normal equation via pseudo-inverse.
- Adaline: Train on the Line, Classify on the Sign
Lectures 2e and 2f: the professor trains Adaline by fitting a continuous line to the ±1 labels (instead of fitting directly on the sign), and I connect this to the delta rule that already showed up in the other playlist. In the end, a deliberately biased dataset reveals the problem was never the training algorithm.
- Swapping the Cost Function Like Changing Clothes
Lectures 3a and 3b: the professor generalizes training to accept any pluggable cost function, and each choice (Widrow-Hoff, perceptron criterion, log-likelihood, hinge) recovers a different algorithm from this playlist. Along the way, I found a real mismatch between activation function and label encoding.
- Multiclass: When a Weight Becomes a Matrix
Lecture 3c: reusing the binary classifier directly on a 4-class problem lands exactly at random-guess accuracy, 25%. The fix is generalizing the weight from a vector to a matrix, one set of weights per class, and swapping sign() for argmax.
- XOR: Where a Single Neuron Just Hits a Wall
Lecture 3d, the last one the professor has published so far: the same perceptron that learns OR and AND effortlessly gets stuck on XOR, pinned at 50% accuracy, weights collapsed to nearly zero. I close this first batch of the playlist exactly where the real course also stands: right at the threshold before the MLP.