Key Activation Functions

Why This Matters

Activation functions are the gatekeepers of your neural network—they determine whether and how information flows from one layer to the next. Without them, even a 100-layer network would collapse into a single linear transformation, incapable of learning anything more complex than a straight line. You're being tested on understanding why different activations exist, when to use each one, and what problems they solve (or create). The core concepts here—vanishing gradients, computational efficiency, output ranges, and gradient flow—show up repeatedly in architecture design questions.

Don't just memorize that ReLU outputs zero for negative inputs. Know why that matters for training dynamics, how it compares to sigmoid's gradient behavior, and when you'd choose a variant like Leaky ReLU instead. The best exam answers connect activation choice to the specific problem: classification vs. regression, shallow vs. deep networks, training stability vs. computational cost.

---

Saturating Activations: The Classics and Their Limitations

These functions were the original workhorses of neural networks. They "saturate"—meaning their gradients approach zero for extreme inputs—which creates the infamous vanishing gradient problem in deep networks.

Sigmoid Function

• Maps outputs to (0, 1)—ideal for binary classification where you need probability-like outputs
• Vanishing gradients occur because the derivative approaches zero for large positive or negative inputs, stalling learning in deep networks
• Non-zero-centered output forces all gradients to be the same sign, slowing convergence during backpropagation

Hyperbolic Tangent (tanh) Function

• Maps outputs to (-1, 1)—zero-centered, which improves gradient dynamics compared to sigmoid
• Steeper gradients near zero mean faster learning when inputs are in the active region
• Still saturates for large inputs, so vanishing gradients remain a problem in very deep networks

---

ReLU Family: Solving Vanishing Gradients

The Rectified Linear Unit revolutionized deep learning by providing constant gradients for positive inputs. The gradient is either 0 or 1—no saturation, no vanishing. But this simplicity introduces its own failure mode.

Rectified Linear Unit (ReLU)

• $f(x) = \max(0, x)$—outputs input directly if positive, zero otherwise
• Computationally efficient with no exponentials or divisions, enabling faster training on large networks
• Dying ReLU problem occurs when neurons get stuck outputting zero, permanently stopping learning for that unit

Leaky ReLU

• Small negative slope (typically $\alpha = 0.01$) allows gradient flow even for negative inputs
• Prevents dying neurons by ensuring every unit can still update during backpropagation
• Same computational efficiency as standard ReLU—just one extra multiplication

Parametric ReLU (PReLU)

• Learnable negative slope $\alpha$ is trained alongside network weights, adapting to the data
• More flexible than Leaky ReLU since the network discovers the optimal slope
• Adds parameters to the model, which can improve performance but risks overfitting on small datasets

Exponential Linear Unit (ELU)

• $f(x) = x$ if $x > 0$, else $\alpha(e^x - 1)$—smooth transition through zero with mean activations closer to zero
• Smoother gradients than ReLU variants, which can improve learning dynamics
• Computationally expensive due to the exponential calculation, trading speed for gradient quality

---

Modern Activations: Adaptive and Smooth

Recent research has produced activations that combine benefits of multiple approaches. These functions often learn or adapt their behavior, trading simplicity for performance.

Swish Function

• $f(x) = x \cdot \sigma(x)$ where $\sigma$ is the sigmoid function—smooth and non-monotonic
• Self-gating mechanism allows the function to selectively pass information, combining linear and non-linear properties
• Outperforms ReLU on deeper networks in many benchmarks, though with higher computational cost

---

Output Layer Activations: Task-Specific Choices

These activations aren't about hidden layer dynamics—they're about matching your output to the problem type.

Softmax Function

• Converts logits to probabilities that sum to 1, making it the standard choice for multi-class classification
• Emphasizes largest values through exponentiation, increasing confidence in the top prediction
• Sensitive to outliers because extreme logits can dominate the probability distribution, potentially causing numerical instability

Linear Activation Function

• $f(x) = x$—no transformation, output can be any real value
• Required for regression tasks where predictions aren't bounded to a specific range
• Never use in hidden layers since stacking linear functions just produces another linear function, eliminating the network's ability to learn complex patterns

Step Function

• Binary output (0 or 1) based on a threshold—the original perceptron activation
• Zero gradient everywhere except at the threshold, making gradient-based optimization impossible
• Historical significance only—not used in modern deep learning, but understanding it clarifies why smooth activations matter

---

Quick Reference Table

---

Self-Check Questions

1. Both sigmoid and tanh suffer from vanishing gradients—what property of tanh makes it generally preferable for hidden layers despite this shared weakness?

2. A colleague reports that 40% of neurons in their deep ReLU network have stopped updating entirely. Which two activation functions would you recommend as replacements, and what mechanism do they use to solve this problem?

3. Compare and contrast ELU and Leaky ReLU: how does each handle negative inputs, and what trade-off does ELU make for its smoother gradient behavior?

4. You're designing a network to classify images into exactly one of 10 categories. Which output activation should you use, and why would sigmoid be inappropriate here?

5. If an FRQ asks you to explain why a 50-layer network with sigmoid activations fails to train while the same architecture with ReLU succeeds, what two key concepts should your answer address?