Types of Neural Network Architectures

Why This Matters

Neural network architectures aren't just different tools in a toolbox—they represent fundamentally different approaches to how machines can learn patterns from data. You're being tested on your ability to select the right architecture for the right problem, which means understanding the underlying mechanisms: how information flows, what structures the network can capture, and where each architecture excels or fails. Interviewers and exams will probe whether you grasp concepts like spatial hierarchies, temporal dependencies, latent representations, and adversarial training.

Don't just memorize architecture names and layer counts. Know what problem each architecture was designed to solve, why its structure addresses that problem, and when you'd choose one over another. The difference between a junior and senior ML engineer often comes down to architectural intuition—understanding that a CNN's weight sharing exploits spatial invariance, or that an LSTM's gating mechanism directly combats vanishing gradients. Master the "why" behind each design, and the details will stick.

---

Architectures for Spatial Data

These networks exploit the structure of grid-like data where nearby elements share meaningful relationships. The key insight is parameter sharing and local connectivity—rather than learning separate weights for every pixel, these architectures learn filters that detect patterns regardless of position.

Convolutional Neural Networks (CNNs)

• Convolutional layers apply learnable filters that slide across input data, detecting features like edges, textures, and shapes through local receptive fields
• Pooling layers reduce spatial dimensions—max pooling or average pooling decreases computational load while providing translation invariance
• Hierarchical feature learning means early layers detect simple patterns (edges), while deeper layers combine them into complex features (faces, objects)

Radial Basis Function Networks (RBFNs)

• Gaussian activation functions measure distance from learned center points, making these networks effective for interpolation and function approximation
• Three-layer architecture consists of input, RBF hidden layer, and linear output—simpler than deep networks but powerful for specific tasks
• Fast training times compared to backpropagation-heavy architectures, though they scale poorly to high-dimensional inputs

---

Architectures for Sequential Data

Sequential architectures maintain memory of previous inputs, enabling them to model temporal dependencies and variable-length sequences. The core challenge is propagating information across time steps without gradients exploding or vanishing.

Recurrent Neural Networks (RNNs)

• Hidden state acts as memory—each time step's output depends on current input and the previous hidden state, creating a feedback loop
• Vanishing gradient problem occurs when gradients shrink exponentially during backpropagation through time, making long-range dependencies nearly impossible to learn
• Suitable for short sequences like part-of-speech tagging, but struggle with tasks requiring memory beyond ~10-20 time steps

Long Short-Term Memory Networks (LSTMs)

• Gating mechanisms control information flow—the forget gate ($f_t$), input gate ($i_t$), and output gate ($o_t$) determine what to remember, update, and expose
• Cell state provides a highway for gradients—information can flow unchanged across many time steps, directly addressing the vanishing gradient problem
• Excels at long-range dependencies in tasks like machine translation, speech recognition, and text generation where context from hundreds of steps ago matters

---

Architectures for Representation Learning

These networks learn compressed or transformed representations of data without explicit labels. The goal is discovering latent structure—reducing dimensionality, removing noise, or learning features that transfer to downstream tasks.

Autoencoders

• Encoder-decoder structure compresses input to a lower-dimensional latent space (bottleneck), then reconstructs the original input
• Reconstruction loss drives learning—the network learns to preserve only the most important information needed to rebuild the input
• Variants extend functionality—denoising autoencoders learn robust features by reconstructing from corrupted inputs; variational autoencoders (VAEs) learn probabilistic latent spaces for generation

Deep Belief Networks (DBNs)

• Stacked Restricted Boltzmann Machines (RBMs) learn hierarchical representations through unsupervised pre-training, one layer at a time
• Greedy layer-wise training followed by fine-tuning with backpropagation—historically important for training deep networks before modern initialization techniques
• Largely superseded by modern architectures but still relevant for understanding deep learning history and certain generative modeling applications

Self-Organizing Maps (SOMs)

• Competitive learning preserves topology—neurons compete to respond to inputs, and winning neurons pull their neighbors closer in weight space
• Produces interpretable 2D visualizations of high-dimensional data, clustering similar inputs in nearby map regions
• Unsupervised clustering and exploration—useful when you need to understand data structure before building supervised models

---

Architectures for Generation

Generative architectures learn to produce new data samples that resemble the training distribution. The fundamental challenge is modeling complex, high-dimensional probability distributions well enough to sample realistic outputs.

Generative Adversarial Networks (GANs)

• Adversarial training pits two networks against each other—the generator ($G$) creates fake samples while the discriminator ($D$) tries to distinguish real from fake
• Minimax game dynamics push both networks to improve: $\min_G \max_D \mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(z)))]$
• State-of-the-art image synthesis powers applications like StyleGAN for faces, pix2pix for image-to-image translation, and data augmentation for limited datasets

---

Foundational and Classical Architectures

These architectures established core concepts that modern networks build upon. Understanding them illuminates why certain design choices persist and provides fallback options for simpler problems.

Feedforward Neural Networks (Multilayer Perceptrons)

• Information flows in one direction only—from input layer through hidden layers to output, with no cycles or feedback connections
• Universal approximation theorem guarantees that a sufficiently wide network with one hidden layer can approximate any continuous function
• Activation functions introduce non-linearity—without ReLU, sigmoid, or tanh, stacked linear layers would collapse to a single linear transformation

Hopfield Networks

• Content-addressable memory stores patterns as attractors in an energy landscape—given partial or noisy input, the network converges to the nearest stored pattern
• Symmetric weights and energy function ensure convergence: $E = -\frac{1}{2}\sum_{i,j} w_{ij} s_i s_j$
• Limited storage capacity—can reliably store approximately $0.15N$ patterns for $N$ neurons, making them impractical for large-scale memory but valuable for optimization problems

---

Quick Reference Table

---

Self-Check Questions

1. Both CNNs and feedforward networks can perform image classification. What structural property of CNNs makes them dramatically more efficient for this task, and why does this matter for high-resolution images?

2. Compare LSTMs and vanilla RNNs: what specific mechanism allows LSTMs to maintain gradients over long sequences, and what would happen if you removed the forget gate?

3. You need to generate photorealistic faces for a dataset. Would you choose a GAN or a standard autoencoder? Explain the tradeoff you're making with your choice.

4. A colleague suggests using a Hopfield network to store 1 million user preference vectors. Why is this problematic, and what architecture would you recommend instead?

5. FRQ-style: Given an unlabeled dataset of sensor readings from industrial equipment, describe how you would use an autoencoder for anomaly detection. What would high reconstruction error indicate, and why is this approach preferable to supervised methods in this context?