Theoretical Statistics Unit 5 ReviewLimit Theorems and Convergence in Statistics

Key Concepts and Definitions

• Convergence describes the behavior of a sequence of random variables as the sample size increases, with different types of convergence (convergence in probability, almost sure convergence, convergence in distribution, and convergence in mean)
• Limit theorems are fundamental results in probability theory that describe the asymptotic behavior of sequences of random variables
  • These theorems provide the theoretical foundation for many statistical methods and help justify the use of approximations in statistical inference
• Stochastic convergence refers to the convergence of random variables, which is different from the convergence of deterministic sequences
• Asymptotic properties describe the limiting behavior of estimators or test statistics as the sample size approaches infinity
• Consistency is a desirable property of an estimator, ensuring that it converges in probability to the true parameter value as the sample size increases
• Efficiency of an estimator refers to its ability to achieve the lowest possible variance among all unbiased estimators, with the Cramér-Rao lower bound providing a benchmark for efficiency
• Asymptotic normality is a property of many estimators, where their distribution approaches a normal distribution as the sample size increases, facilitating the construction of confidence intervals and hypothesis tests

Types of Convergence

• Convergence in probability occurs when the probability of the absolute difference between a sequence of random variables and a target value being greater than any positive number approaches zero as the sample size increases
• Almost sure convergence (or convergence with probability 1) is a stronger form of convergence, where the probability of the sequence of random variables converging to a target value is 1
• Convergence in distribution (or weak convergence) happens when the cumulative distribution function (CDF) of a sequence of random variables converges to the CDF of a target distribution at all continuity points
  • This type of convergence is particularly relevant for the Central Limit Theorem
• Convergence in mean (or $L^p$ convergence) occurs when the expected value of the absolute difference between a sequence of random variables and a target value, raised to the power $p$, approaches zero as the sample size increases
  • Special cases include convergence in mean square ($p=2$) and convergence in absolute mean ($p=1$)
• Relationships between types of convergence:
  • Almost sure convergence implies convergence in probability
  • Convergence in probability implies convergence in distribution
  • Convergence in $L^p$ implies convergence in $L^q$ for $p > q$

Law of Large Numbers

• The Law of Large Numbers (LLN) states that the sample mean of a sequence of independent and identically distributed (i.i.d.) random variables converges to the population mean as the sample size increases
• Weak Law of Large Numbers (WLLN):
  • The sample mean converges in probability to the population mean
  • Formally, for a sequence of i.i.d. random variables $X_1, X_2, \ldots$ with finite mean $\mu$, $\frac{1}{n} \sum_{i=1}^n X_i \xrightarrow{p} \mu$ as $n \to \infty$
• Strong Law of Large Numbers (SLLN):
  • The sample mean converges almost surely to the population mean
  • Formally, for a sequence of i.i.d. random variables $X_1, X_2, \ldots$ with finite mean $\mu$, $P(\lim_{n \to \infty} \frac{1}{n} \sum_{i=1}^n X_i = \mu) = 1$
• The LLN holds under more general conditions, such as for sequences of independent random variables with finite variances and for some dependent sequences (e.g., ergodic stationary processes)
• The LLN justifies the use of sample means as consistent estimators of population means and is fundamental to the concept of statistical inference

Central Limit Theorem

• The Central Limit Theorem (CLT) states that the standardized sum (or average) of a sequence of i.i.d. random variables with finite mean and variance converges in distribution to a standard normal distribution as the sample size increases
• Formally, for a sequence of i.i.d. random variables $X_1, X_2, \ldots$ with mean $\mu$ and variance $\sigma^2$, $\frac{\sum_{i=1}^n X_i - n\mu}{\sigma \sqrt{n}} \xrightarrow{d} N(0, 1)$ as $n \to \infty$
  • Equivalently, $\frac{\bar{X}_n - \mu}{\sigma / \sqrt{n}} \xrightarrow{d} N(0, 1)$, where $\bar{X}_n$ is the sample mean
• The CLT holds under more general conditions, such as the Lindeberg-Feller CLT for independent but not necessarily identically distributed random variables with finite variances
• The CLT is the foundation for many statistical procedures, including:
  • Construction of confidence intervals and hypothesis tests for means
  • Approximating the distribution of sample means and sums
  • Justifying the use of normal-based methods in large samples
• The rate of convergence in the CLT is $O(n^{-1/2})$, meaning that the approximation error decreases at a rate proportional to the square root of the sample size

Other Important Limit Theorems

• The Delta Method is a technique for approximating the distribution of a transformed sequence of random variables using a Taylor series expansion and the CLT
  • It is useful for deriving the asymptotic distribution of functions of estimators
• The Continuous Mapping Theorem states that if a sequence of random variables converges in distribution to a random variable and a function is continuous, then the sequence of transformed random variables converges in distribution to the transformed random variable
• Slutsky's Theorem allows for the combination of convergent sequences of random variables, stating that if $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} c$ (where $c$ is a constant), then $X_n + Y_n \xrightarrow{d} X + c$, $X_n Y_n \xrightarrow{d} cX$, and $X_n / Y_n \xrightarrow{d} X / c$ (provided $c \neq 0$)
• The Mann-Wald Theorem (or the Convergence of Random Functions Theorem) extends the continuous mapping theorem to sequences of random functions
• The Portmanteau Theorem provides several equivalent characterizations of convergence in distribution, which can be useful for proving convergence results

Applications in Statistical Inference

• Limit theorems provide the theoretical justification for many common statistical methods and help quantify the uncertainty associated with estimators and test statistics
• The LLN justifies the use of sample means, proportions, and other moments as consistent estimators of their population counterparts
  • This consistency property is crucial for the validity of statistical inference procedures
• The CLT allows for the construction of confidence intervals and hypothesis tests for means and other parameters in large samples
  • For example, a $(1-\alpha)$ confidence interval for a population mean $\mu$ can be constructed as $\bar{X}_n \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}}$, where $z_{\alpha/2}$ is the $(1-\alpha/2)$ quantile of the standard normal distribution
• The Delta Method enables the derivation of asymptotic distributions for functions of estimators, such as the asymptotic distribution of the sample variance or the ratio of two sample means
• Limit theorems are essential for the development and analysis of statistical learning methods, such as regression, classification, and clustering algorithms
  • They provide guidance on the choice of loss functions, regularization techniques, and model selection criteria
• Asymptotic efficiency results, such as the Cramér-Rao lower bound, help compare the performance of different estimators and guide the selection of optimal procedures

Common Misconceptions and Pitfalls

• The CLT does not imply that the sample itself is normally distributed, only that the distribution of the standardized sample mean (or sum) approaches a normal distribution as the sample size increases
• The CLT does not require the population distribution to be normal; it holds for any population distribution with finite mean and variance
• The convergence of the sample mean to the population mean (LLN) does not imply that the sample mean will exactly equal the population mean for a finite sample size
• The rate of convergence in the CLT is relatively slow ($O(n^{-1/2})$), so the normal approximation may not be accurate for small sample sizes or for populations with heavy tails or extreme skewness
• Limit theorems typically require independence (or weak dependence) among the random variables; violations of this assumption can lead to incorrect conclusions
  • For example, applying the CLT to dependent data (e.g., time series) without appropriate modifications may result in underestimating the variability of the sample mean
• Asymptotic efficiency results, such as the Cramér-Rao lower bound, are based on large-sample approximations and may not hold for small sample sizes
• Misinterpreting p-values and confidence intervals based on asymptotic approximations can lead to incorrect conclusions, especially when the sample size is small or the model assumptions are violated

Practice Problems and Examples

• Suppose $X_1, X_2, \ldots$ are i.i.d. random variables with mean $\mu=2$ and variance $\sigma^2=4$. Find the asymptotic distribution of $\sqrt{n}(\bar{X}_n - 2)$ using the CLT.
• Let $X_1, X_2, \ldots$ be i.i.d. random variables with a Poisson($\lambda$) distribution. Use the LLN to show that the sample mean $\bar{X}_n$ is a consistent estimator of $\lambda$.
• Suppose $X_1, X_2, \ldots, X_n$ are i.i.d. random variables with mean $\mu$ and variance $\sigma^2$. Use the Delta Method to find the asymptotic distribution of $\sqrt{n}(e^{\bar{X}_n} - e^\mu)$.
• Let $X_1, X_2, \ldots$ be i.i.d. random variables with a Bernoulli($p$) distribution. Use the CLT to construct a 95% confidence interval for $p$ when $n=100$ and the observed sample proportion is $\hat{p}=0.6$.
• Suppose $X_1, X_2, \ldots$ are i.i.d. random variables with a uniform distribution on $[0, \theta]$, where $\theta > 0$ is an unknown parameter. Show that the maximum likelihood estimator $\hat{\theta}_n = \max(X_1, \ldots, X_n)$ converges almost surely to $\theta$ using the SLLN.
• Let $X_1, X_2, \ldots$ be i.i.d. random variables with a standard normal distribution. Use the continuous mapping theorem to find the asymptotic distribution of $\sqrt{n}(\bar{X}_n^2 - 1)$.
• Suppose $X_1, X_2, \ldots$ are i.i.d. random variables with a Gamma($\alpha, \beta$) distribution, where $\alpha$ and $\beta$ are unknown parameters. Find the asymptotic distribution of the maximum likelihood estimators $\hat{\alpha}_n$ and $\hat{\beta}_n$ using the CLT and the Delta Method.