Backpropagation Visual Notes

Backpropagation

The core algorithm for training neural networks — an intuitive walkthrough without formulas

🧠 Network Anatomy

Input layer: 784 pixels (28×28 handwritten digit)
Hidden layers: 2 layers, 16 neurons each
Output layer: 10 neurons (0–9)

Weights + biases = ~13,000 parameters. The cost function measures how far the network's guesses are from true labels.

🎯 Core Idea

Backpropagation computes the gradient of the cost function — a vector that tells us how to tweak each weight and bias to reduce error most efficiently.

The gradient's components represent sensitivity: a weight with a component of 3.2 is 32× more influential on the cost than one with 0.1.

It's not magic — it's just systematically propagating "desired adjustments" backward from the output.

💡 Key Insight

"The cost function is sensitive to some weights far more than others. Backpropagation reveals exactly which levers to pull."

🔍 How It Works — One Example at a Time

Take an image of a handwritten "2". The network's output is random (e.g., 0.5, 0.8…). We want:
→ Increase activation for neuron #2
→ Decrease all others, proportionally to how far they are from their target.

Three complementary ways to increase a neuron's activation:

Adjust the bias — simplest, direct effect.
Tweak the weights — connections from brighter (more active) neurons matter most; they carry bigger signals.
Modify activations of the previous layer — amplify positive-weight connections, dampen negative-weight ones.

We can't change activations directly — only weights and biases. But we record what changes would help, then propagate those wishes backward.

⚡ Hebbian Analogy

"Cells that fire together, wire together." — The most reinforced connections are between highly active neurons and the ones we want to activate more. This mirrors a biological learning principle (simplified).

Not a claim about brains, but a helpful intuition: the network strengthens the paths that lead to correct answers.

🔄 The Backward Flow

Each output neuron has a "wish list" for the previous layer. To compute the adjustment for layer L-1:

Combine the wishes from all 10 output neurons
Weight each wish by the connection strength (weights)
Scale by how much each neuron wants to change

Then repeat the process for the next layer backward — hence the name backpropagation.

Without listening to all training examples at once, we'd bias the network toward only one digit. So we average the adjustments across examples.

📉 Stochastic Gradient Descent (SGD)

Running backprop on every single training example per step is computationally expensive. Instead:

Shuffle the training data
Split into mini-batches (e.g., 100 examples each)
Compute one gradient step per mini-batch

The path down the cost landscape looks less like a precise hiker and more like a drunken stagger — but it's dramatically faster and converges well.

Each mini-batch gives an approximate gradient; the overall process still reaches a good local minimum.

📊 Key Data — MNIST

The handwritten digit dataset (MNIST) provides tens of thousands of labeled 28×28 grayscale images. This abundance of labeled data is what makes training possible — and remains a central challenge in machine learning.

70k

total images

digit classes

28×28

pixels per image

784

input neurons

Getting high-quality labeled data is often the hardest part of real-world ML.

📌 Quick Summary

🔹 Backpropagation computes how each training example wants to adjust every weight and bias — not just the direction, but the relative magnitude.
🔹 Theoretical approach: average the adjustments over all training examples per gradient step.
🔹 Practical approach (SGD): use mini-batches to approximate the gradient — much faster, still converges.
🔹 The network converges to a local minimum of the cost function, performing well on training data.

🧩 Final Thought

Understanding backpropagation intuitively means grasping: why some adjustments are larger than others, how they propagate backward, and how they accumulate across examples. The math (derivatives, chain rule) formalizes this — but the intuition comes first.

The next step: express these same ideas using calculus notation, so the symbols become familiar when encountered elsewhere.

⚠️ Real-world challenge: Backpropagation and modern ML require massive labeled datasets. MNIST is a clean example — collecting and labeling data is often the bottleneck.

Generated by CuteNote.app Visual Notes