Every time a neural network learns, an optimiser is doing the heavy lifting in the background — quietly nudging millions of weights, one tiny step at a time, towards a configuration that makes the model better.

Training a neural network is essentially a giant search problem. Out of trillions of possible parameter combinations, the optimiser’s job is to find the one that produces the lowest error on the data. It does this by repeatedly computing the gradient (the direction of steepest improvement) and taking small steps downhill across the so-called loss landscape.

The Loss Landscape

Picture every possible combination of model weights as a point on a map. The “height” at each point is how badly the model performs on the training data. Training the network is exactly like rolling a ball downhill on this map — the optimiser steers from a randomly chosen starting position toward a low valley where the error is small. Some valleys are deeper than others (global vs. local minima); some terrain is flat (saddle points); some is steep and narrow (ravines). The optimiser’s design determines how well it copes with each.

Gradient Descent: The Core Idea

All three optimisers in this guide are descendants of one foundational technique: gradient descent. The word “gradient” simply means “slope.” When you stand on a hill and want to walk downhill, you naturally move in the direction that goes down most steeply. Gradient descent does exactly this — but in a mathematical space with potentially millions of dimensions.

At every training step, the algorithm measures the steepness of the error surface at the current position and takes a small step in the downhill direction. The size of that step is controlled by a setting called the learning rate. Too large a learning rate and you overshoot the valley; too small and it takes forever to arrive.

Three Flavours of Gradient Descent

Most modern deep learning uses mini-batch gradient descent, processing a small group of training samples (typically 32 to 512 examples) before each weight update. This balances computational speed with the accuracy of the gradient estimate.

SGD is the grandfather of all modern optimisers. Deceptively simple on the surface, it has powered some of the most significant breakthroughs in deep learning — and with the right tuning, it can still outperform far more complex alternatives.

The “Stochastic” in SGD means “random.” Instead of computing the exact average gradient over all training data (which would be enormously expensive), SGD estimates it using a single example or a small random batch. This estimation is noisy — but that noise turns out to be surprisingly useful, as it helps the model escape shallow local minima it might otherwise get stuck in.

Why It Works — and Where It Struggles

SGD’s core strength is its simplicity and speed. Each update is cheap to compute, which means the model can update thousands of times per second on modern hardware. The built-in noise from using small batches acts as a kind of regularisation, stopping the model from over-fitting to any one pattern in the data.

However, SGD has two well-known problems. First, it uses the same learning rate for every single parameter in the network, even though different parameters may need to move at very different speeds. Second, if the loss surface has very different curvatures in different directions — a problem called pathological curvature — SGD oscillates back and forth inefficiently rather than moving straight toward the minimum.

SGD + Momentum: Adding Memory

The fix for SGD’s oscillation problem came in the form of momentum. Borrowed from physics, momentum gives the optimiser a kind of “memory” of the direction it has been moving. Instead of only looking at the current gradient to decide which direction to step, it also considers where it has been moving recently.

The parameter β (beta) controls how much of the historical direction is carried forward. A value of 0.9 means “remember 90% of where I was going, and only 10% of the current gradient.” This smooths out the noisy zigzagging and lets the optimiser accelerate in directions where gradients consistently point the same way.

# SGD with Momentum in PyTorch
import torch.optim as optim

optimizer = optim.SGD(
    model.parameters(),
    lr=0.01,         # learning rate
    momentum=0.9,    # momentum coefficient β
    weight_decay=1e-4 # optional L2 regularisation
)

RMSProp solved one of the biggest problems in neural network training — the fact that different parameters in a model should learn at different speeds. Proposed by Geoffrey Hinton in a 2012 lecture, it became an instant classic.

The key idea behind RMSProp is adaptive learning rates. Instead of using one global learning rate for every parameter, RMSProp tracks how large the recent gradients have been for each individual parameter. If a particular weight has been seeing large, consistent gradients, its effective learning rate is scaled down. If gradients have been small, the effective rate goes up. Each weight gets exactly the learning speed it needs.

How RMSProp Computes Its Adaptive Rates

RMSProp maintains a running estimate of the squared gradients for each parameter. This is computed as an exponential moving average, meaning recent gradients matter more than old ones (controlled by the decay rate ρ, typically 0.9). The step size for each parameter is then divided by the square root of this running average.

The tiny constant ε (epsilon) is added inside the square root purely as a safety measure — it prevents a division-by-zero error in the extremely unlikely case where the gradient estimate becomes exactly zero.

Where RMSProp Shines

RMSProp was specifically designed to handle non-stationary objectives — problems where the gradient landscape keeps changing during training. Recurrent neural networks (RNNs) and reinforcement learning tasks are classic examples, because the data distribution or reward signal constantly shifts. In these settings, methods that accumulate gradient information over the entire training history (like Adagrad) deteriorate over time, while RMSProp’s rolling window keeps it relevant.

# RMSProp in PyTorch
optimizer = optim.RMSprop(
    model.parameters(),
    lr=0.001,     # learning rate η
    alpha=0.9,     # decay rate ρ for running average
    eps=1e-8,     # ε for numerical stability
    momentum=0.0   # optional momentum term
)

Adam is arguably the most important optimiser in modern deep learning. It elegantly combines the best ideas from momentum and RMSProp into a single algorithm, and has become the default starting point for training almost every neural network today.

Proposed by Diederik Kingma and Jimmy Ba in 2014, Adam stands for Adaptive Moment Estimation. The name refers to the two statistical moments it tracks for each parameter: the first moment (mean of gradients — like Momentum) and the second moment (uncentred variance of gradients — like RMSProp).

The Three Steps of Adam

1. Track the first moment (m): Maintain a running exponential average of the gradients themselves. This captures direction and average magnitude — essentially a smoothed version of the gradient, similar to momentum.
2. Track the second moment (v): Maintain a running exponential average of the squared gradients. This captures how much the gradient varies — allowing per-parameter learning rate scaling like RMSProp.
3. Apply bias correction: Because both running averages start at zero, they are biased toward zero at the beginning of training. Adam corrects for this by dividing by (1 − β^t), making the early updates accurate rather than artificially small.

Understanding the Default Parameters

Adam’s original paper proposed default values that have proven remarkably robust across an enormous range of tasks. These are not magic numbers — each has a careful justification.

AdamW — The Improved Variant

Standard Adam has a subtle bug: when weight decay (L2 regularisation) is applied, it interacts incorrectly with the adaptive learning rate scaling. The result is that heavily updated parameters receive less weight penalty than they should. AdamW, proposed by Loshchilov and Hutter in 2017, decouples weight decay from the gradient-based update, fixing this issue.

# Adam in PyTorch
optimizer = optim.Adam(
    model.parameters(),
    lr=1e-3,             # learning rate η (default: 0.001)
    betas=(0.9, 0.999),    # (β₁, β₂)
    eps=1e-8              # ε for numerical stability
)

# AdamW in PyTorch (preferred for transformers)
optimizer = optim.AdamW(
    model.parameters(),
    lr=1e-4,
    weight_decay=0.01      # proper decoupled weight decay
)

Choosing an optimiser is only the first step. Getting great results means understanding the control knobs — the hyperparameters — that determine how aggressively each algorithm searches the loss landscape.

Learning Rate — The Most Important Setting

The learning rate determines how large a step the optimiser takes in the downhill direction after computing each gradient. It is almost universally the most impactful hyperparameter in training. A learning rate that is too high causes the model to overshoot and bounce chaotically. One that is too low causes glacially slow convergence — or getting stuck in a shallow local minimum.

Learning Rate Schedules

In practice, using a fixed learning rate throughout training is rarely optimal. A common strategy is to start with a higher learning rate (to take large steps through the broad landscape) and gradually decrease it as training progresses (to make fine adjustments near the minimum). Several scheduling strategies are widely used.

Training deep neural networks is treacherous terrain. The loss landscape is riddled with obstacles that can trap, slow, or mislead even the best optimisers. Understanding these challenges explains why more sophisticated algorithms were developed.

Pathological Curvature — The Shape Problem

One of the trickiest challenges is when the loss landscape has very different curvatures in different directions — technically called a poor conditioning number. Imagine a very long, narrow valley running diagonally. The correct path is along the length of the valley, but the steepest direction is across the narrow width. Plain SGD keeps sliding across the valley from side to side (the “ravine” problem), making very slow progress along the actual path to the minimum. This is precisely the problem that momentum and adaptive learning rate methods solve.

A comprehensive reference comparing the three primary optimisers across every dimension that matters for practitioners.

There is no universal winner. The best optimiser depends on your task, your architecture, and what you care about — convergence speed, final accuracy, or training stability.

Decision Framework by Task

Optimisers are not abstract mathematics — they are the invisible engines behind nearly every AI product in the world. Here is how the choice of optimiser has shaped real applications.

The story of deep learning optimisers spans nearly four decades of incremental insight, each new algorithm solving a weakness the previous one revealed.