Activation Functions: The Soul of Non-Linearity

Activation functions are the mathematical engines that allow neural networks to learn complex, non-linear relationships. Without them, a neural network—no matter how deep—would collapse into a simple linear regression model, incapable of solving tasks like image recognition or natural language processing.

Why do we need Non-Linearity?

In a deep neural network, the output of each layer is a linear transformation of the input ($Z = W \cdot A + b$). If we don't apply a non-linear activation function, the composition of multiple linear layers is still just a linear transformation.

$$f(g(x)) = W_2(W_1x + b_1) + b_2 = (W_2W_1)x + (W_2b_1 + b_2) = W'x + b'$$

Non-linearity allows the network to approximate any continuous function, a property known as the Universal Approximation Theorem.

---

1. The Classics

Sigmoid (Logistic)

The Sigmoid function squashes input values into a range between 0 and 1. It is primarily used in the output layer for binary classification.

$\sigma(x) = \frac{1}{1 + e^{-x}}$

• Pros: Smooth gradient, clear probabilistic interpretation.
• Cons: Vanishing Gradient Problem (gradients become very small for high/low inputs), outputs are not zero-centered.

Hyperbolic Tangent (Tanh)

Tanh squashes values between -1 and 1. It is generally preferred over Sigmoid for hidden layers because it is zero-centered, which helps keep the gradients moving in both directions during training.

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$

• Pros: Zero-centered, stronger gradients than Sigmoid.
• Cons: Still suffers from the Vanishing Gradient problem at the extremes.

---

2. The ReLU Revolution

ReLU (Rectified Linear Unit)

ReLU is the default choice for most deep learning architectures. It is simple: if $x > 0$, output $x$; otherwise, output 0.

$f(x) = \max(0, x)$

• Pros: Computationally efficient, reduces the likelihood of vanishing gradients.
• Cons: Dying ReLU Problem—neurons can become "dead" and stop responding to variations in error if their weights lead to negative inputs.

Leaky ReLU

To fix the Dying ReLU problem, Leaky ReLU introduces a small slope for negative values (usually 0.01).

$f(x) = \max(0.01x, x)$

• Pros: Prevents dead neurons by ensuring a small gradient even for negative inputs.

---

3. Modern State-of-the-Art (SOTA)

GELU (Gaussian Error Linear Unit)

GELU is the standard in modern Transformers like BERT, GPT-3, and ViT. It weights inputs by their percentile, rather than a hard threshold like ReLU.

$GELU(x) = x \cdot \Phi(x) \approx 0.5x \left(1 + \tanh\left[\sqrt{2/\pi}(x + 0.044715x^3)\right]\right)$

• Why it's better: It provides a smoother curve and allows for small negative values, which helps with gradient flow and probabilistic modeling.

Swish (SiLU)

Discovered by Google researchers using automated search, Swish is defined as $x \cdot \sigma(x)$. It is used extensively in EfficientNet.

$f(x) = \frac{x}{1 + e^{-x}}$

• Pros: Non-monotonic, smooth, and often outperforms ReLU on deep networks.

Mish

A self-regularized non-monotonic activation function that has shown superior performance in YOLOv4 and other computer vision tasks.

$f(x) = x \cdot \tanh(\text{softplus}(x)) = x \cdot \tanh(\ln(1 + e^x))$

---

Summary Table

Function	Equation	Range	Best Use Case
Sigmoid	$1/(1+e^{-x})$	$(0, 1)$	Binary Classification Output
Tanh	$\tanh(x)$	$(-1, 1)$	Hidden Layers (legacy/specific RNNs)
ReLU	$\max(0, x)$	$[0, \infty)$	Standard Hidden Layers
GELU	$x \Phi(x)$	$[-0.17, \infty)$	Transformers / NLP
Softmax	$e^{x_i}/\sum e^{x_j}$	$(0, 1)$	Multi-class Classification Output

---

Python Implementation

activation_functions.py

import numpy as np

class ActivationFunctions:
    """
    A collection of modern and classic activation functions for deep learning.
    """
    
    @staticmethod
    def sigmoid(x: np.ndarray) -> np.ndarray:
        return 1 / (1 + np.exp(-x))
    
    @staticmethod
    def tanh(x: np.ndarray) -> np.ndarray:
        return np.tanh(x)
    
    @staticmethod
    def relu(x: np.ndarray) -> np.ndarray:
        return np.maximum(0, x)
    
    @staticmethod
    def leaky_relu(x: np.ndarray, alpha: float = 0.01) -> np.ndarray:
        return np.where(x > 0, x, alpha * x)
    
    @staticmethod
    def gelu(x: np.ndarray) -> np.ndarray:
        """Approximation of GELU activation."""
        return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * np.power(x, 3))))
    
    @staticmethod
    def swish(x: np.ndarray) -> np.ndarray:
        return x * ActivationFunctions.sigmoid(x)
    
    @staticmethod
    def softmax(x: np.ndarray) -> np.ndarray:
        # Subtract max for numerical stability (prevents overflow)
        exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
        return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

    @staticmethod
    def relu_derivative(x: np.ndarray) -> np.ndarray:
        return (x > 0).astype(float)

Pro Tip

When in doubt, start with ReLU. If you are building a Transformer-based model, use GELU. For computer vision tasks where every bit of accuracy matters, try Mish or Swish.