Modern Deep Learning Optimizers

While standard Gradient Descent provides the mathematical foundation, training modern deep neural networks requires more sophisticated algorithms that can adapt to the complex, high-dimensional loss landscapes of billion-parameter models.

---

1. Stochastic Gradient Descent (SGD) with Momentum

Standard SGD often oscillates in ravines where the surface curves much more steeply in one dimension than in another. Momentum helps accelerate SGD in the relevant direction and dampens oscillations.

The Physics Intuition

Think of a ball rolling down a hill. It gains momentum as it rolls, making it harder to change direction abruptly and helping it push through small local bumps (minima).

$$\begin{aligned} v_t &= \beta v_{t-1} + (1 - \beta) \nabla_{\theta} J(\theta) \\ \theta &= \theta - \alpha v_t \end{aligned}$$

• $\beta$ (Momentum coefficient): Usually set to 0.9. It determines how much of the previous velocity to keep.

---

2. RMSProp (Root Mean Square Propagation)

Developed by Geoffrey Hinton, RMSProp addresses the problem of learning rates that are too large or too small for different parameters. It scales the learning rate by a moving average of the squared gradients.

$$\begin{aligned} s_t &= \beta_2 s_{t-1} + (1 - \beta_2) (\nabla_{\theta} J(\theta))^2 \\ \theta &= \theta - \frac{\alpha}{\sqrt{s_t + \epsilon}} \nabla_{\theta} J(\theta) \end{aligned}$$

• Effect: If a gradient is consistently large, the learning rate for that parameter is reduced. If the gradient is small, the learning rate is increased.

---

3. Adam (Adaptive Moment Estimation)

Adam is the "Swiss Army Knife" of optimizers. it combines the benefits of Momentum (first moment) and RMSProp (second moment).

The Update Rule

Adam calculates an exponential moving average of both the gradients ($m_t$) and the squared gradients ($v_t$):

1. Estimate Moments: $m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$ $v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$

2. Bias Correction: (Crucial for the early steps of training) $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$

3. Update Parameters: $\theta = \theta - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

• Standard Hyperparameters: $\alpha=0.001$, $\beta_1=0.9$, $\beta_2=0.999$.

---

4. AdamW: The Modern Standard

A common mistake in early implementations was combining L2 regularization (weight decay) directly into the gradient. AdamW decouples the weight decay from the optimization step, which significantly improves generalization.

Which Optimizer to Choose?

• Adam/AdamW: Default choice for almost all deep learning tasks (Transformers, CNNs).
• SGD with Momentum: Often preferred in Computer Vision (ResNets) for final "fine-tuning" as it can sometimes find flatter, better-generalizing minima than Adam.

---

Summary Comparison

Optimizer	Main Advantage	Key Hyperparameter
SGD + Momentum	Reduces oscillations	$\beta \approx 0.9$
RMSProp	Adapts per-parameter LR	$\beta_2 \approx 0.99$
Adam	Fast, robust, widely applicable	$\beta_1, \beta_2$
AdamW	Better regularization/generalization	Weight Decay $\lambda$

---

Python Implementation

optimizers.py

import numpy as np

class AdamOptimizer:
    def __init__(self, lr=0.001, b1=0.9, b2=0.999, eps=1e-8):
        self.lr = lr
        self.b1 = b1
        self.b2 = b2
        self.eps = eps
        self.m = None # 1st moment
        self.v = None # 2nd moment
        self.t = 0

    def update(self, w, dw):
        if self.m is None:
            self.m = np.zeros_like(dw)
            self.v = np.zeros_like(dw)
        
        self.t += 1
        # Update moments
        self.m = self.b1 * self.m + (1 - self.b1) * dw
        self.v = self.b2 * self.v + (1 - self.b2) * np.square(dw)
        
        # Bias correction
        m_hat = self.m / (1 - self.b1**self.t)
        v_hat = self.v / (1 - self.b2**self.t)
        
        # Update weights
        return w - self.lr * m_hat / (np.sqrt(v_hat) + self.eps)