Regularization: Preventing Overfitting

A powerful neural network has enough capacity to "memorize" the training data perfectly, leading to poor generalization on new data. Regularization is the set of techniques used to constrain the network's complexity, forcing it to learn meaningful patterns instead of noise.

---

1. Weight Decay (L2 Regularization)

L2 regularization adds a penalty term to the cost function proportional to the square of the weights. This discourages the network from using large weights unless they significantly reduce the loss.

$J_{reg}(\theta) = J(\theta) + \frac{\lambda}{2m} \sum \|w\|^2$

• $\lambda$ (Regularization Parameter): Controls the tradeoff between fitting the data and keeping weights small.
• Effect: It makes the weights decay towards zero, leading to a "smoother" model that is less sensitive to small fluctuations in the input.

---

2. L1 Regularization (Sparsity)

L1 adds a penalty proportional to the absolute value of the weights.

$J_{reg}(\theta) = J(\theta) + \frac{\lambda}{m} \sum |w|$

• Effect: Unlike L2, L1 can drive weights to exactly zero. This effectively performs feature selection, leaving only the most important connections active.

---

3. Dropout

Dropout is a uniquely effective technique for deep networks. During training, it randomly "drops out" (sets to zero) a fraction of neurons in a layer for each mini-batch.

Why it works:

1. Reduces Co-adaptation: A neuron cannot rely on the presence of specific other neurons to learn features, forcing it to be more robust and learn useful features independently.
2. Ensemble Effect: It's like training a massive ensemble of different sub-networks and averaging their predictions.

Implementation Detail

Dropout is only used during training. During inference (testing), all neurons are used, but their outputs are scaled by the dropout probability ($p$) to maintain the same expected activation magnitude.

---

4. Other Essential Techniques

Data Augmentation

Creating new training examples by applying transformations (rotation, cropping, flipping) to existing data. This is "free" regularization as it increases the dataset size and variety.

Early Stopping

Monitoring the validation loss and stopping training the moment it starts to increase, even if the training loss is still decreasing.

---

Summary Comparison

Technique	How it works	Primary Goal
L2 Regularization	Penalizes large weights	Generalization, smooth models
L1 Regularization	Penalizes weight magnitude	Feature selection, sparsity
Dropout	Randomly kills neurons	Robustness, reduces co-adaptation
Early Stopping	Stops training early	Prevents memorization of noise

---

Python Implementation: Dropout

regularization.py

import numpy as np

def dropout_forward(A, keep_prob):
    """
    Implements the forward pass with dropout (inverted dropout).
    """
    # Create mask: 1 if we keep the neuron, 0 if we drop it
    mask = (np.random.rand(*A.shape) < keep_prob).astype(float)
    
    # Shut down some neurons
    A *= mask
    
    # Scale A to keep the same expected value (Inverted Dropout)
    A /= keep_prob
    
    return A, mask

def dropout_backward(dA, mask, keep_prob):
    """
    Implements the backward pass with dropout.
    """
    dA *= mask
    dA /= keep_prob
    return dA