4.2 Maximum likelihood estimation

Maximum likelihood estimation is a fundamental technique in statistical inference, used to estimate parameters of probability distributions. It finds the parameter values that make observed data most probable, providing a bridge between frequentist and Bayesian approaches to statistics.

This method plays a crucial role in developing and evaluating statistical models. By maximizing the likelihood function, it yields point estimates with desirable properties like consistency and efficiency, forming the basis for many statistical techniques used in data analysis and modeling.

Concept of maximum likelihood

• Maximum likelihood estimation forms a cornerstone of frequentist statistical inference used to estimate parameters of statistical models
• Connects to Bayesian statistics through its role in parameter estimation and model selection, providing a foundation for comparing frequentist and Bayesian approaches
• Serves as a crucial tool in developing and evaluating statistical models, including those used in Bayesian analysis

Definition and purpose

Top images from around the web for Definition and purpose

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum likelihood estimation - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

Top images from around the web for Definition and purpose

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum likelihood estimation - Wikipedia View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

• Statistical method for estimating parameters of a probability distribution by maximizing the likelihood function
• Aims to find parameter values that make the observed data most probable
• Provides a principled approach to parameter estimation in various statistical models
• Yields point estimates that possess desirable statistical properties (consistency, efficiency)

Historical background

• Developed by R.A. Fisher in the 1920s as part of his work on statistical inference
• Evolved from earlier methods of moment matching and least squares estimation
• Gained widespread adoption in the mid-20th century with advances in computational power
• Influenced the development of other statistical techniques (likelihood ratio tests, information criteria)

Relationship to Bayesian inference

• Serves as a special case of Bayesian inference when using uniform prior distributions
• Provides the basis for constructing likelihood functions used in Bayesian analysis
• Differs from Bayesian methods in its treatment of parameters as fixed unknown quantities rather than random variables
• Often used as a starting point for more complex Bayesian models and analyses

Likelihood function

Mathematical formulation

• Expresses the probability of observing the data given specific parameter values
• Defined as $L(\theta|x) = f(x|\theta)$ where $\theta$ represents parameters and $x$ represents observed data
• For independent and identically distributed (i.i.d.) observations, likelihood factorizes as $L(\theta|x_1, ..., x_n) = \prod_{i=1}^n f(x_i|\theta)$
• Incorporates the probability density function (continuous data) or probability mass function (discrete data)

Properties of likelihood

• Not a probability distribution over parameters, but a function of parameters given fixed data
• Invariant under one-to-one transformations of parameters
• Allows for comparison of different parameter values within the same model
• Satisfies the likelihood principle, which states that all relevant information about parameters is contained in the likelihood function

Log-likelihood function

• Logarithm of the likelihood function, often denoted as $\ell(\theta|x) = \log L(\theta|x)$
• Simplifies calculations by converting products to sums
• Preserves the location of maxima and minima due to monotonicity of the logarithm
• Improves numerical stability in computations, especially for large datasets
• Often used in optimization algorithms for finding maximum likelihood estimates

Maximum likelihood estimators

Definition and characteristics

• Parameter values that maximize the likelihood (or log-likelihood) function
• Formally defined as $\hat{\theta}_{MLE} = \arg\max_{\theta} L(\theta|x)$
• Invariant under one-to-one transformations of parameters
• Possess desirable asymptotic properties (consistency, efficiency, normality)
• May not always exist or may not be unique in some cases

Consistency and efficiency

• Consistency ensures convergence to true parameter values as sample size increases
• Efficiency refers to achieving the lowest possible variance among unbiased estimators
• MLEs are asymptotically efficient under certain regularity conditions
• Achieve the Cramér-Rao lower bound asymptotically, making them minimum variance unbiased estimators

Asymptotic properties

• Asymptotically normally distributed with mean equal to the true parameter value
• Variance approaches the inverse of the Fisher information as sample size increases
• Convergence rate typically of order $O(1/\sqrt{n})$ where n is the sample size
• Asymptotic properties form the basis for constructing confidence intervals and hypothesis tests

Methods for finding MLEs

Analytical solutions

• Closed-form solutions exist for some simple distributions (normal, exponential, Poisson)
• Involve setting the derivative of the log-likelihood to zero and solving for parameters
• Example for normal distribution: sample mean and variance are MLEs for population parameters
• Analytical solutions provide exact results but are not available for many complex models

Numerical optimization techniques

• Used when analytical solutions are not available or difficult to derive
• Include methods like Newton-Raphson, gradient descent, and conjugate gradient
• Iteratively update parameter estimates to maximize the likelihood function
• Require careful selection of starting values and convergence criteria
• May encounter issues with local maxima or flat likelihood surfaces

EM algorithm

• Expectation-Maximization algorithm used for finding MLEs in models with latent variables or missing data
• Consists of two steps: E-step (compute expected log-likelihood) and M-step (maximize expected log-likelihood)
• Iteratively refines parameter estimates until convergence
• Widely used in mixture models, hidden Markov models, and factor analysis
• Guarantees increase in likelihood at each iteration but may converge to local maxima

Applications in statistical models

Linear regression

• MLEs for coefficients in linear regression equivalent to ordinary least squares estimates
• Assumes normally distributed errors with constant variance
• Closed-form solution exists: $\hat{\beta} = (X^TX)^{-1}X^Ty$
• MLE approach provides a probabilistic interpretation of linear regression parameters

Logistic regression

• Used for binary outcome data, models probability of success as a function of predictors
• MLEs found through iterative numerical optimization (Newton-Raphson, iteratively reweighted least squares)
• No closed-form solution exists due to non-linearity of logistic function
• Likelihood function based on Bernoulli distribution for each observation

Poisson regression

• Models count data assuming Poisson-distributed response variable
• MLEs typically found through iterative numerical methods
• Log-link function used to ensure non-negative predicted values
• Applications include modeling rare events, accident frequencies, or species counts in ecology

Likelihood ratio tests

Test statistic formulation

• Compares likelihood of data under null hypothesis to likelihood under alternative hypothesis
• Test statistic defined as $\Lambda = -2 \log\frac{L(\theta_0)}{L(\hat{\theta})}$ where $\theta_0$ is the parameter value under the null hypothesis
• Measures the evidence against the null hypothesis relative to the alternative
• Large values of $\Lambda$ indicate strong evidence against the null hypothesis

Asymptotic distribution

• Under certain regularity conditions, -2 log Λ follows a chi-square distribution asymptotically
• Degrees of freedom equal to the difference in dimensionality between alternative and null models
• Asymptotic result holds as sample size approaches infinity
• Provides basis for calculating p-values and critical values for hypothesis testing

Power of likelihood ratio tests

• Probability of correctly rejecting the null hypothesis when it is false
• Increases with sample size and effect size
• Often more powerful than other test statistics (Wald test, score test) in finite samples
• Can be calculated analytically for simple hypotheses or through simulation for complex models

Limitations and alternatives

Small sample issues

• MLEs may be biased or inefficient in small samples
• Asymptotic properties may not hold, leading to unreliable inference
• Alternatives include bias-corrected estimators or exact methods for specific distributions
• Bayesian methods with informative priors can provide more stable estimates in small samples

Regularization techniques

• Address overfitting and improve generalization in high-dimensional settings
• Include methods like ridge regression, lasso, and elastic net
• Modify likelihood function by adding penalty terms on parameter magnitudes
• Balance between model fit and complexity, often leading to sparse solutions

Bayesian vs maximum likelihood

• Bayesian methods incorporate prior information, while MLE uses only the likelihood
• Bayesian inference provides full posterior distributions rather than point estimates
• MLE can be viewed as a special case of Bayesian inference with uniform priors
• Bayesian methods handle uncertainty more naturally but require specification of priors

Computational aspects

Software implementations

• Statistical software packages (R, SAS, STATA) provide built-in functions for MLE
• Machine learning libraries (scikit-learn, TensorFlow) implement MLE for various models
• Specialized optimization libraries (scipy.optimize, nlopt) offer flexible tools for custom likelihood functions
• Bayesian software (Stan, PyMC) often include MLE as a special case or starting point

Computational complexity

• Varies widely depending on model complexity and dataset size
• Simple models with closed-form solutions have low computational cost
• Iterative methods for complex models may require many function evaluations
• High-dimensional problems can become computationally intensive or intractable

Parallel processing for MLEs

• Embarrassingly parallel nature of likelihood calculations for independent observations
• Enables efficient use of multi-core processors and distributed computing systems
• Particularly useful for bootstrap resampling and cross-validation procedures
• Implemented in modern statistical software to handle large-scale data analysis

Advanced topics

Profile likelihood

• Technique for dealing with nuisance parameters in likelihood-based inference
• Involves maximizing likelihood over nuisance parameters for each value of parameter of interest
• Useful for constructing confidence intervals and hypothesis tests in complex models
• Provides more accurate inference than methods based on asymptotic normality in some cases

Penalized maximum likelihood

• Incorporates penalty terms into likelihood function to address overfitting or enforce constraints
• Examples include L1 (lasso) and L2 (ridge) penalties in regression models
• Balances model fit with complexity or prior beliefs about parameter values
• Often results in sparse solutions, facilitating variable selection in high-dimensional settings

Empirical likelihood methods

• Nonparametric approach to likelihood-based inference
• Constructs a likelihood function without assuming a specific parametric form
• Combines flexibility of nonparametric methods with efficiency of likelihood-based inference
• Applications include constructing confidence regions and hypothesis testing in semiparametric models