Advanced Signal Processing Unit 7 Review: Statistical Signal Processing & Estimation

Key Concepts and Foundations

• Signal processing involves the analysis, modification, and synthesis of signals to extract information or enhance signal characteristics
• Signals can be classified as continuous-time (analog) or discrete-time (digital), depending on the nature of the independent variable (time)
• Sampling is the process of converting a continuous-time signal into a discrete-time signal by measuring its amplitude at regular intervals
  • The sampling rate, or sampling frequency ($f_s$), determines the number of samples taken per second
  • The Nyquist-Shannon sampling theorem states that the sampling rate must be at least twice the highest frequency component in the signal to avoid aliasing
• Quantization is the process of mapping a continuous range of values to a finite set of discrete values, often represented by binary numbers
  • The number of bits used for quantization determines the resolution and signal-to-quantization noise ratio (SQNR)
• Fourier analysis is a fundamental tool in signal processing, allowing the representation of signals in the frequency domain
  • The Fourier transform converts a time-domain signal into its frequency-domain representation, revealing its frequency components and their amplitudes
  • The inverse Fourier transform converts a frequency-domain representation back into the time domain
• Linear time-invariant (LTI) systems are essential building blocks in signal processing, characterized by the properties of linearity and time invariance
  • Linearity means that the system's output is proportional to its input, and the principle of superposition holds
  • Time invariance means that the system's response to an input does not depend on the absolute time, only on the relative time difference
• Convolution is a mathematical operation that describes the output of an LTI system given its input and impulse response
  • In the time domain, convolution is represented as the integral of the product of the input signal and the time-reversed, shifted impulse response
  • In the frequency domain, convolution becomes multiplication, simplifying the analysis of LTI systems

Probability Theory Review

• Probability theory provides a mathematical framework for analyzing random phenomena and forms the basis for statistical signal processing
• A random variable is a variable whose value is determined by the outcome of a random experiment
  • Random variables can be discrete (taking on a countable set of values) or continuous (taking on any value within a range)
  • The probability mass function (PMF) describes the probability distribution of a discrete random variable, while the probability density function (PDF) describes the probability distribution of a continuous random variable
• The cumulative distribution function (CDF) is the probability that a random variable takes on a value less than or equal to a given value
  • For a discrete random variable, the CDF is the sum of the PMF values up to the given value
  • For a continuous random variable, the CDF is the integral of the PDF up to the given value
• The expected value (or mean) of a random variable is the average value it takes on, weighted by the probabilities of each value
  • For a discrete random variable, the expected value is the sum of the product of each value and its probability
  • For a continuous random variable, the expected value is the integral of the product of each value and its PDF
• The variance of a random variable measures the spread of its distribution around the mean, while the standard deviation is the square root of the variance
• Joint probability distributions describe the probabilities of multiple random variables occurring together
  • The joint PMF or joint PDF can be used to calculate probabilities, expected values, and other statistics for multiple random variables
• Conditional probability is the probability of one event occurring given that another event has already occurred, denoted as $P(A|B)$
• Independence between random variables means that the occurrence of one event does not affect the probability of the other event
  • For independent events A and B, $P(A|B) = P(A)$ and $P(B|A) = P(B)$
  • The joint probability of independent events is the product of their individual probabilities

Random Processes in Signal Processing

• A random process is a collection of random variables indexed by a parameter, usually time, representing the evolution of a system or signal over time
• The mean function $\mu(t)$ of a random process is the expected value of the random variable at each time instant
• The autocorrelation function $R(t_1, t_2)$ of a random process describes the correlation between the values of the process at two different time instants
  • For a stationary process, the autocorrelation function depends only on the time difference $\tau = t_2 - t_1$, and is denoted as $R(\tau)$
• The autocovariance function $C(t_1, t_2)$ is similar to the autocorrelation function but measures the covariance between the values of the process at two different time instants
• Stationarity is a property of random processes where the statistical characteristics do not change over time
  • Strictly stationary processes have joint probability distributions that are invariant under time shifts
  • Wide-sense stationary (WSS) processes have constant mean and autocorrelation functions that depend only on the time difference
• Ergodicity is a property of random processes where the time averages of a single realization are equal to the ensemble averages across multiple realizations
  • Ergodic processes allow the estimation of statistical properties from a single, sufficiently long realization of the process
• The power spectral density (PSD) of a WSS random process is the Fourier transform of its autocorrelation function, representing the distribution of power across different frequencies
• White noise is a random process with a constant PSD across all frequencies, often used as a building block for more complex processes
  • Gaussian white noise is a white noise process with samples drawn from a Gaussian (normal) distribution
• Random processes can be used to model various phenomena in signal processing, such as noise, interference, and signal sources
  • For example, thermal noise in electronic circuits is often modeled as additive white Gaussian noise (AWGN)

Estimation Theory Basics

• Estimation theory deals with the problem of inferring the values of unknown parameters or signals based on observed data
• An estimator is a function that maps the observed data to an estimate of the unknown parameter or signal
  • The goal is to design estimators that are accurate, efficient, and robust to uncertainties in the data or model
• Point estimation involves finding a single "best" estimate of the unknown parameter based on the observed data
  • Common point estimators include the maximum likelihood estimator (MLE), which maximizes the likelihood function of the data given the parameter, and the minimum mean square error (MMSE) estimator, which minimizes the expected squared error between the estimate and the true value
• Interval estimation involves finding a range of plausible values for the unknown parameter, often in the form of a confidence interval
  • A confidence interval is a range of values that is likely to contain the true parameter value with a specified probability (confidence level)
• Bayesian estimation incorporates prior knowledge about the unknown parameter in the form of a prior probability distribution
  • The prior distribution is combined with the likelihood function of the data to obtain the posterior distribution, which represents the updated knowledge about the parameter after observing the data
  • Bayesian estimators, such as the maximum a posteriori (MAP) estimator and the minimum mean square error (MMSE) estimator, are based on the posterior distribution
• Cramér-Rao lower bound (CRLB) is a fundamental limit on the variance of any unbiased estimator
  • The CRLB provides a benchmark for evaluating the performance of estimators and can be used to assess the feasibility of estimation problems
• Sufficient statistics are functions of the observed data that contain all the information relevant to estimating the unknown parameter
  • Using sufficient statistics can simplify the estimation problem and lead to more efficient estimators
• Consistency is a desirable property of estimators, where the estimate converges to the true value of the parameter as the number of observations increases
• Efficiency is another desirable property, where an estimator achieves the lowest possible variance among all unbiased estimators (i.e., attains the CRLB)

Linear Estimation Techniques

• Linear estimation techniques are widely used in signal processing due to their simplicity and tractability
• The linear minimum mean square error (LMMSE) estimator is a linear estimator that minimizes the expected squared error between the estimate and the true value
  • The LMMSE estimator is the optimal linear estimator for Gaussian random variables and processes
  • It can be derived using the orthogonality principle, which states that the estimation error should be orthogonal (uncorrelated) with the observed data
• The Wiener filter is a linear filter that minimizes the mean square error between the filtered output and a desired signal
  • It is derived based on the LMMSE principle and requires knowledge of the signal and noise power spectral densities
  • The Wiener filter has applications in noise reduction, signal restoration, and system identification
• Kalman filtering is a recursive linear estimation technique for estimating the state of a dynamic system from noisy measurements
  • The Kalman filter combines a model of the system dynamics with the observed measurements to produce an optimal estimate of the state in the LMMSE sense
  • It consists of a prediction step, which uses the system model to predict the state at the next time step, and an update step, which incorporates the new measurement to refine the state estimate
  • The extended Kalman filter (EKF) and the unscented Kalman filter (UKF) are extensions of the Kalman filter for nonlinear systems, using linearization and deterministic sampling techniques, respectively
• Least squares estimation is a linear estimation method that minimizes the sum of squared errors between the observed data and a linear model
  • The least squares estimator is the optimal linear unbiased estimator (BLUE) under certain conditions, such as independent and identically distributed (i.i.d.) Gaussian noise
  • Recursive least squares (RLS) is an online version of least squares estimation that updates the estimate as new data becomes available, making it suitable for adaptive filtering and system identification
• Linear prediction is a technique for predicting future values of a signal based on a linear combination of its past values
  • Linear predictive coding (LPC) is a popular method for speech analysis and compression, where the speech signal is modeled as the output of a linear system excited by a periodic or noise-like input
  • The LPC coefficients, which represent the system's transfer function, are estimated using linear estimation techniques such as least squares or the Levinson-Durbin algorithm

Nonlinear Estimation Methods

• Nonlinear estimation methods are necessary when the relationship between the observed data and the unknown parameters or signals is nonlinear
• The maximum likelihood estimator (MLE) is a popular nonlinear estimator that maximizes the likelihood function of the data given the unknown parameters
  • The MLE is asymptotically unbiased, consistent, and efficient under certain regularity conditions
  • Finding the MLE often involves solving a nonlinear optimization problem, which can be computationally challenging
• The maximum a posteriori (MAP) estimator is a Bayesian estimator that maximizes the posterior probability distribution of the unknown parameters given the observed data
  • The MAP estimator incorporates prior knowledge about the parameters in the form of a prior probability distribution
  • It reduces to the MLE when the prior distribution is uniform (non-informative)
• Particle filtering is a sequential Monte Carlo method for estimating the state of a nonlinear, non-Gaussian dynamic system
  • It represents the posterior distribution of the state by a set of weighted particles, which are updated and resampled as new measurements become available
  • Particle filtering can handle complex, multimodal distributions and is more flexible than parametric methods like the extended Kalman filter
• Expectation-maximization (EM) is an iterative algorithm for finding the MLE or MAP estimate in the presence of missing or latent data
  • The EM algorithm alternates between an expectation (E) step, which computes the expected value of the log-likelihood function with respect to the latent data, and a maximization (M) step, which updates the parameter estimates to maximize the expected log-likelihood
  • The EM algorithm is particularly useful for problems involving mixture models, hidden Markov models, and other latent variable models
• Markov chain Monte Carlo (MCMC) methods are a class of algorithms for sampling from complex probability distributions, such as the posterior distribution in Bayesian estimation
  • MCMC methods, such as the Metropolis-Hastings algorithm and the Gibbs sampler, generate a Markov chain whose stationary distribution is the target distribution
  • Samples from the Markov chain, after a burn-in period, can be used to approximate the target distribution and compute various statistics of interest
• Nonlinear least squares is an extension of the least squares method for estimating the parameters of a nonlinear model
  • It involves minimizing the sum of squared errors between the observed data and the nonlinear model predictions
  • Solving nonlinear least squares problems typically requires iterative optimization algorithms, such as the Gauss-Newton method or the Levenberg-Marquardt algorithm
• Kernel density estimation is a nonparametric method for estimating the probability density function of a random variable based on a finite sample of observations
  • It involves placing a kernel function (e.g., a Gaussian kernel) centered at each observation and summing the contributions of all kernels to obtain a smooth estimate of the density
  • The choice of kernel function and bandwidth parameter can significantly affect the quality of the density estimate

Spectral Analysis and Applications

• Spectral analysis is the study of the frequency content of signals and the estimation of their power spectral density (PSD)
• The periodogram is a simple nonparametric estimator of the PSD, obtained by computing the squared magnitude of the Fourier transform of the signal
  • The periodogram is an inconsistent estimator, as its variance does not decrease with increasing data length
  • Techniques like averaging, smoothing, or tapering can be used to improve the statistical properties of the periodogram
• Welch's method is an improved PSD estimator that involves dividing the signal into overlapping segments, computing the periodogram of each segment, and averaging the results
  • Overlapping segments and the use of window functions (e.g., Hann or Hamming windows) help reduce the variance and spectral leakage of the estimate
  • The trade-off between variance reduction and frequency resolution can be controlled by the choice of segment length and overlap
• Parametric spectral estimation methods model the signal as the output of a linear system driven by white noise
  • Examples include the Yule-Walker autoregressive (AR) method, the Burg method, and the maximum entropy method (MEM)
  • Parametric methods can provide high-resolution PSD estimates with fewer data samples compared to nonparametric methods, but they rely on the accuracy of the assumed model
• Multitaper spectral estimation is a nonparametric method that uses multiple orthogonal window functions (tapers) to compute independent spectral estimates, which are then averaged
  • The tapers are designed to have good leakage properties and are often based on discrete prolate spheroidal sequences (DPSS) or Slepian sequences
  • Multitaper methods offer a balance between variance reduction and spectral resolution, and are particularly useful for short or non-stationary signals
• Time-frequency analysis involves studying the time-varying frequency content of signals
  • The short-time Fourier transform (STFT) computes the Fourier transform of the signal within a sliding window, producing a spectrogram that shows the evolution of the spectrum over time
  • The continuous wavelet transform (CWT) uses scaled and shifted versions of a mother wavelet to analyze the signal at different time-frequency resolutions
  • Other time-frequency distributions, such as the Wigner-Ville distribution and the Cohen class of distributions, aim to provide better joint time-frequency resolution than the STFT
• Spectral analysis has numerous applications in signal processing, including:
  • Speech and audio processing: speaker identification, speech enhancement, audio compression
  • Radar and sonar: target detection, Doppler estimation, clutter suppression
  • Biomedical signal processing: analysis of EEG, ECG, and other physiological signals
  • Mechanical and structural health monitoring: vibration analysis, fault detection, modal analysis
  • Geophysical signal processing: seismic data analysis, gravitational wave detection

Advanced Topics and Current Research

• Sparse signal processing exploits the sparsity of signals in some domain (e.g., time, frequency, or wavelet) to develop efficient algorithms for signal acquisition, compression, and recovery
  • Compressed sensing is a framework for acquiring and reconstructing sparse signals using fewer measurements than traditional sampling methods
  • Sparse regression techniques, such as the least absolute shrinkage and selection operator (LASSO) and matching pursuit, are used for