How Neural Networks Learn – Summary

How Neural Networks Learn

From architecture to gradient descent — what really happens inside the hidden layers.

Deep Learning · MNIST · Visual Summary

Architecture Recap

A network that recognizes handwritten digits: 28×28 pixel grayscale images feed into 784 input neurons. Each neuron's activation is a weighted sum of the previous layer plus a bias, passed through a sigmoid or ReLU function.

784

Input neurons

~13,000

Parameters (weights + biases)

Hidden layers (16 neurons each)

Output neurons (digits 0–9)

The brightest neuron in the final layer is the network's answer. Our hypothesis: layer 2 detects small edges, layer 3 assembles them into loops and lines, and the final layer combines these into digits.

The Cost Function — Measuring Failure

We define a cost function that punishes the network for bad guesses. For a single training example: subtract the desired activation from the actual activation for each output neuron, square the differences, and sum them.

        Cost per example = Σ (desired − actual)²

        Total cost = average cost over thousands of training examples

Small cost = confident correct answer. Large cost = random guessing. The cost function depends on all 13,000 weights and biases — it's a 13,000-dimensional function that outputs a single number describing how badly the network performs.

Gradient Descent — Finding the Minimum

Instead of brute-forcing a 13,000-dimensional space, we use calculus: compute the gradient (the direction of steepest ascent), then move in the opposite direction — the steepest descent. Repeat.

Core Idea

At each point, the gradient tells you both which direction is steepest and how fast the function is changing. Step size is proportional to the slope — smaller steps near flat regions to avoid overshooting.

For a 13,000-parameter function, the gradient is a vector where each component tells you:

Sign — whether to increase or decrease that parameter
Relative magnitude — how much that parameter influences the cost

This process — repeatedly nudging all parameters in the direction of steepest descent — is called gradient descent. It converges to a local minimum of the cost function.

"Training any network is nothing more than minimizing its cost function."

Training Results — What Works

With random initial weights, the network performs poorly. After gradient descent on the MNIST training set:

96%

Accuracy (2 hidden layers, 16 units)

98%

Achievable with tuned architecture

Many of the images it gets wrong are ones humans might also mistake. The network generalizes well to unseen images — magical given the difficulty of the task.

What Hidden Layers Actually Learn

Surprise: The weights do not form neat edge detectors or shape templates. They look nearly random — with no interpretable patterns.

The network found a local minimum in the 13,000-dimensional weight space that works well, but it did not use the hierarchical mechanism we hypothesized (edges → shapes → digits).

Reality Check

Feed the network a random noise image — it confidently classifies it as a "5". It has no idea how to draw a digit, only how to recognize them within a narrow training distribution.

The network's world is entirely small, centered, handwritten digits. The cost function never rewards doubt or cautious uncertainty — only confident correct answers.

Research Insight — Memorization vs. Learning

Two recent papers shed light on what gradient descent actually does:

Paper 1: Train a network on randomly labeled images — accuracy on new data is near random, but training accuracy stays high. The network can memorize random labels using its millions of parameters.
Paper 2: The key difference is learning speed. With real (correctly labeled) data, the network finds a local minimum much faster. With random labels, progress is painfully slow.

"Different local minima tend to have similar quality when training data has a true pattern — making them easier to find."

— Discussion with Lisha Li, PhD in deep learning

Why This Matters

This simple network — studied since the 1990s — is not the end goal. It's a starting point. Understanding it is essential for grasping modern architectures. Key takeaways:

Gradient descent works even in 13,000-dimensional spaces
The network's "intelligence" is brittle and narrow
Hidden layers do not automatically learn interpretable features
Training can emphasize memorization over genuine understanding

Actionable Next Steps

To truly internalize how neural networks learn, go beyond watching:

Read Michael Nielsen's free online book on deep learning — contains the exact code and training data for this MNIST example. (Consider donating to support his work.)
Explore Chris Olah's writings and the Distill journal for intuitive, visual explanations of neural network internals.
Pause and think: If you wanted the network to detect edges and loops, how would you design the training process differently?

The network discussed uses two hidden layers of 16 neurons each, with ~13,000 parameters total. All results are based on the MNIST handwritten digit dataset.

Generated by CuteNote.app Visual Notes