Neural Network Hyperparameters

Why This Matters

Neural network hyperparameters are the control knobs you set before training begins—and they fundamentally determine whether your model learns effectively or fails spectacularly. You're being tested on understanding how these choices affect gradient flow, model capacity, and generalization, not just memorizing default values. The interplay between hyperparameters like learning rate, architecture depth, and regularization strength reveals your grasp of core optimization theory and the bias-variance tradeoff.

Think of hyperparameter tuning as balancing competing forces: you need enough model complexity to capture patterns (capacity) without memorizing noise (overfitting), and you need optimization steps that are bold enough to make progress but careful enough not to overshoot. Don't just memorize what each hyperparameter does—know why certain combinations work together and what symptoms indicate when a hyperparameter is misconfigured.

---

Optimization Dynamics

These hyperparameters control how the network navigates the loss landscape during training. The core mechanism is gradient descent: iteratively adjusting weights to minimize error, where the path taken depends critically on step size and momentum.

Learning Rate

• Controls step size during gradient descent—too large causes oscillation or divergence; too small causes painfully slow convergence or getting stuck in local minima
• Adaptive methods like Adam and RMSprop automatically adjust the rate per-parameter, combining benefits of momentum with per-dimension scaling
• Most sensitive hyperparameter in practice; often the first thing to tune, typically starting around $10^{-3}$ and adjusting by factors of 10

Momentum

• Accumulates velocity from previous gradients—the update rule becomes $v_t = \gamma v_{t-1} + \eta \nabla L$ where $\gamma$ is the momentum coefficient
• Smooths noisy gradients and accelerates convergence through flat regions of the loss surface, acting like a ball rolling downhill with inertia
• Typical values range from 0.5 to 0.9; higher momentum means more influence from past updates, risking overshooting in sharp valleys

Batch Size

• Number of samples per gradient update—affects both the quality of gradient estimates and computational efficiency
• Smaller batches (32-128) introduce beneficial noise that can escape local minima but produce noisier gradient estimates; larger batches are more stable but memory-intensive
• Interacts with learning rate—larger batches often require proportionally larger learning rates to maintain training dynamics

---

Architecture Capacity

These hyperparameters determine the expressiveness of your network—how complex a function it can represent. The key tradeoff is the bias-variance dilemma: more capacity reduces underfitting but increases overfitting risk.

Number of Hidden Layers

• Depth enables hierarchical feature learning—each layer can build increasingly abstract representations from the previous layer's outputs
• Deeper networks capture more complex patterns but face challenges like vanishing gradients and require more careful initialization
• Problem-dependent choice; simple tasks may need only 1-2 layers while image recognition often requires dozens

Number of Neurons in Each Layer

• Width determines representational capacity per layer—more neurons can capture more features but increase parameter count as $O(n_{in} \times n_{out})$
• Too few neurons cause underfitting (model can't represent the target function); too many cause overfitting and computational overhead
• Common heuristic: start with layer sizes between input and output dimensions, then tune based on validation performance

Activation Functions

• Introduce non-linearity—without them, stacked layers collapse to a single linear transformation regardless of depth
• ReLU ($f(x) = \max(0, x)$) dominates modern architectures due to computational efficiency and reduced vanishing gradient issues; sigmoid and tanh still used in output layers or recurrent networks
• Choice affects gradient flow; ReLU can cause "dead neurons" (zero gradient for negative inputs), while sigmoid saturates at extremes

---

Regularization and Generalization

These hyperparameters combat overfitting by constraining model complexity or introducing noise during training. The underlying principle is that simpler models generalize better when training data is limited or noisy.

Regularization Techniques (L1, L2)

• Add penalty terms to the loss function—L2 adds $\lambda \sum w_i^2$ penalizing large weights; L1 adds $\lambda \sum |w_i|$ encouraging sparsity
• L1 produces sparse models (many weights become exactly zero, useful for feature selection); L2 produces small but non-zero weights (weight decay)
• Regularization strength $\lambda$ must be tuned; too high causes underfitting, too low provides insufficient regularization

Dropout Rate

• Randomly zeros neurons during training—each forward pass uses a different "thinned" network, preventing co-adaptation between neurons
• Acts as ensemble averaging—at test time, the full network approximates averaging predictions from many sub-networks
• Typical rates of 0.2-0.5; higher dropout for larger networks or more overfitting-prone tasks; usually not applied to output layers

---

Training Duration and Initialization

These hyperparameters affect when training starts and stops effectively. Proper initialization ensures healthy gradient flow from the first step, while epoch count determines total exposure to training data.

Number of Epochs

• One epoch = one complete pass through training data—total training iterations equals epochs × (dataset size / batch size)
• Too few epochs cause underfitting (model hasn't converged); too many cause overfitting (model memorizes training noise)
• Early stopping monitors validation loss and halts training when performance degrades, effectively making epochs a learned hyperparameter

Weight Initialization Method

• Sets starting point in weight space—poor initialization causes vanishing gradients (weights → 0) or exploding gradients (weights → ∞)
• Xavier/Glorot initialization scales weights by $\sqrt{1/n_{in}}$, designed for sigmoid/tanh; He initialization scales by $\sqrt{2/n_{in}}$, designed for ReLU
• Breaks symmetry between neurons; identical initialization would cause all neurons in a layer to learn identical features

---

Quick Reference Table

---

Self-Check Questions

1. Which two hyperparameters both affect how quickly a network traverses the loss landscape, and how do their mechanisms differ?

2. If your network shows high training accuracy but poor validation accuracy, which category of hyperparameters should you adjust first, and what specific changes would you make?

3. Compare and contrast L1 and L2 regularization: when would you prefer one over the other, and what different effects do they have on the learned weights?

4. A deep ReLU network shows near-zero gradients in early layers during training. Which hyperparameter is likely misconfigured, and what initialization method would you use instead?

5. Explain the tradeoff involved in choosing batch size: how does a batch size of 32 differ from 512 in terms of gradient quality, convergence behavior, and computational requirements?