Self-attention mechanisms revolutionize sequence modeling by allowing elements to interact dynamically. This powerful technique computes the importance of other elements for each element, enabling adaptive focus and modeling of long-range dependencies in various applications.

Scaled dot-product attention, the core of self-attention, efficiently computes similarities between query and key vectors. Multi-head attention in transformers further enhances model capacity by employing parallel attention mechanisms, each focusing on different aspects of the input.

Self-Attention Mechanisms

Concept of self-attention

Self-attention mechanism allows elements in a sequence to interact dynamically capturing contextual relationships
Computes importance of other elements for each element in the sequence enabling adaptive focus
Enables modeling of long-range dependencies overcoming limitations of recurrent neural networks (RNNs)
Key components include Query, Key, and Value vectors derived from input representations
Process involves computing similarity between query and key vectors then weighting value vectors accordingly
Widely applied in natural language processing (machine translation), computer vision (image captioning), and speech recognition (audio transcription)

Scaled dot-product attention mechanism

Formula: $Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$
Components: Q (Query matrix), K (Key matrix), V (Value matrix), $d_k$ (dimension of key vectors)
Implementation steps:
1. Compute dot product of query and key matrices
2. Scale result by $\frac{1}{\sqrt{d_k}}$ to stabilize gradients
3. Apply softmax function to obtain attention weights
4. Multiply result with value matrix to get final output
Computational complexity: Time $O(n^2d)$ for sequence length n and embedding dimension d, Space $O(n^2)$ for attention weights
Advantages include efficient matrix multiplication and stable gradients due to scaling factor

Concept of self-attention, Attention Mechanism in Neural Networks

Multi-head attention in transformers

Parallel attention mechanisms use different learned linear projections for queries, keys, and values
Typically employs 8 to 16 heads each focusing on different aspects of input (syntactic, semantic)
Process:
1. Create linear projections of input for each head
2. Apply scaled dot-product attention to each head independently
3. Concatenate outputs from all heads
4. Apply final linear transformation to produce output
Allows model to jointly attend to information from different representation subspaces enhancing model capacity
Dimension of each head: $d_{model} / h$ , where h is number of heads
Computational cost remains similar to single-head attention due to reduced dimensionality per head

Interpretation of attention weights

Higher weights indicate stronger relationships between elements reflecting importance of context for each token
Visualization techniques include heatmaps of attention weights and attention flow graphs
Analysis of attention patterns helps identify linguistic phenomena (coreference resolution) and understand model behavior
Visualization tools: BertViz for BERT models, Tensor2tensor library for Transformer visualizations
Applications include model debugging, improving interpretability, and identifying biases in the model
Limitations: Attention ≠ explanation, careful interpretation of visualizations required to avoid misleading conclusions