edge ai and computing unit 5 study guides

Key Concepts and Terminology

• Model compression reduces the size and computational requirements of deep learning models while maintaining performance
• Quantization reduces the precision of model parameters and activations to lower bit widths (8-bit, 4-bit)
• Pruning removes unimportant or redundant connections, neurons, or channels from the model
• Knowledge distillation transfers knowledge from a large, complex teacher model to a smaller, simpler student model
• Low-rank approximation decomposes weight matrices into lower-rank factors to reduce parameters
• Hardware-specific optimizations tailor models to the capabilities and constraints of target edge devices (smartphones, IoT devices)
• Latency refers to the time taken for a model to process an input and generate an output
• Memory footprint is the amount of memory required to store the model parameters and intermediate activations

Model Compression Basics

• Model compression is crucial for deploying deep learning models on resource-constrained edge devices
• Compressed models have smaller storage requirements, faster inference times, and lower energy consumption
• Model compression techniques aim to find a balance between model size, computational efficiency, and performance
• The choice of compression technique depends on the specific requirements of the application and target hardware
• Model compression can enable new applications of deep learning on edge devices (real-time object detection, speech recognition)
• Compressed models may have slightly lower accuracy compared to their uncompressed counterparts
  • The accuracy drop is often acceptable considering the benefits of reduced size and improved efficiency
• Model compression is an active area of research with new techniques and approaches constantly emerging

Quantization Techniques

• Quantization reduces the precision of model parameters and activations from 32-bit floating-point to lower bit widths
• Common quantization bit widths include 8-bit, 4-bit, and even binary (1-bit)
• Post-training quantization applies quantization to a pre-trained model without further fine-tuning
  • It is simple and fast but may result in a larger accuracy drop
• Quantization-aware training incorporates quantization during the model training process
  • It allows the model to adapt to the quantized weights and activations, resulting in better performance
• Symmetric quantization maps values to a fixed range (e.g., [-1, 1]) with a constant scaling factor
• Asymmetric quantization allows for different scaling factors for positive and negative values, providing more flexibility
• Quantization can be applied layer-wise or channel-wise for more fine-grained control

Pruning Methods

• Pruning removes unimportant or redundant connections, neurons, or channels from the model
• Weight pruning removes individual connections based on their magnitude or importance
  • Connections with small weights are considered less important and can be removed
• Neuron pruning removes entire neurons or channels that contribute little to the model's output
• Structured pruning removes entire blocks or layers of the model, resulting in a more compact architecture
• Iterative pruning gradually removes connections or neurons over multiple iterations, allowing the model to adapt
• One-shot pruning removes a significant portion of the model in a single step, which is faster but may impact accuracy more
• Pruning can be combined with fine-tuning to recover any lost accuracy after removing connections or neurons

Knowledge Distillation

• Knowledge distillation transfers knowledge from a large, complex teacher model to a smaller, simpler student model
• The teacher model is trained on the original task and provides soft targets for the student model
  • Soft targets are the teacher's predicted probabilities, which contain more information than hard one-hot labels
• The student model is trained to mimic the teacher's behavior by minimizing the difference between its predictions and the teacher's soft targets
• Knowledge distillation can be performed using different loss functions (KL divergence, mean squared error)
• Online distillation trains the teacher and student models simultaneously, allowing them to adapt to each other
• Offline distillation uses a pre-trained teacher model to guide the training of the student model
• Knowledge distillation can also be applied in a self-distillation setting, where the same model acts as both teacher and student

Low-Rank Approximation

• Low-rank approximation decomposes weight matrices into lower-rank factors to reduce parameters
• Singular Value Decomposition (SVD) is a common technique for low-rank approximation
  • SVD factorizes a matrix into the product of three matrices: $A = U \Sigma V^T$
  • The rank of the approximation can be controlled by selecting the top-k singular values and corresponding singular vectors
• Low-rank approximation can be applied to fully-connected layers, convolutional layers, and recurrent layers
• The choice of rank for the approximation depends on the desired compression ratio and acceptable accuracy loss
• Low-rank approximation can be combined with other compression techniques (quantization, pruning) for further model size reduction
• Tensor decomposition methods (CP decomposition, Tucker decomposition) can be used for higher-order tensors in convolutional layers

Hardware-Specific Optimizations

• Hardware-specific optimizations tailor models to the capabilities and constraints of target edge devices
• Optimizations can target different hardware architectures (CPUs, GPUs, FPGAs, ASICs)
• Quantization can be adapted to match the supported bit widths of the target hardware
  • For example, using 8-bit quantization for devices with efficient 8-bit arithmetic units
• Pruning can be guided by hardware constraints such as memory bandwidth and cache size
  • Structured pruning can create more hardware-friendly sparse patterns
• Model architecture can be designed or modified to leverage hardware-specific features (e.g., using depthwise separable convolutions on mobile GPUs)
• Compiler optimizations (loop unrolling, vectorization) can be applied to generated code for better performance on specific hardware
• Hardware-software co-design approaches jointly optimize the model and hardware for maximum efficiency

Practical Applications and Case Studies

• Image classification on smartphones using MobileNet models compressed with quantization and pruning
  • Achieved significant reduction in model size and latency while maintaining high accuracy
• Keyword spotting on IoT devices using compressed convolutional neural networks
  • Enabled always-on keyword detection with low power consumption
• Face recognition on edge devices using knowledge distillation to transfer knowledge from a large server-side model to a compact edge model
• Object detection on drones using YOLOv3 model compressed with low-rank approximation and quantization
  • Enabled real-time object detection with limited computational resources
• Speech recognition on smart speakers using pruned and quantized recurrent neural networks
  • Reduced model size and inference latency for faster and more responsive user interactions
• Anomaly detection in industrial settings using autoencoders compressed with low-rank approximation
  • Deployed on edge devices for real-time monitoring and fault detection
• Gesture recognition on AR/VR headsets using compressed 3D convolutional neural networks
  • Optimized for the specific hardware constraints of the headset for low-latency gesture recognition