2.4 Probability density functions

Probability density functions (PDFs) are essential tools in theoretical statistics for describing continuous random variables. They provide a mathematical framework to analyze and model various phenomena across different fields, forming the foundation for advanced statistical concepts and inference techniques.

PDFs describe the relative likelihood of a continuous random variable taking specific values. They're represented by non-negative functions that integrate to 1 over their domain. Understanding PDFs is crucial for grasping key statistical properties, relationships between variables, and applying statistical methods in real-world scenarios.

Definition and properties

• Probability density functions (PDFs) serve as fundamental tools in theoretical statistics for describing continuous random variables
• PDFs provide a mathematical framework to analyze and model various phenomena in fields such as physics, finance, and engineering
• Understanding PDFs forms the foundation for more advanced statistical concepts and inference techniques

Concept of PDF

• Describes the relative likelihood of a continuous random variable taking on a specific value
• Represented by a non-negative function $f(x)$ that integrates to 1 over its entire domain
• Area under the PDF curve between two points represents the probability of the random variable falling within that interval
• Cannot be used directly to calculate probabilities for exact values, unlike probability mass functions for discrete variables

Relationship to CDF

• Cumulative Distribution Function (CDF) $F(x)$ obtained by integrating the PDF from negative infinity to x
• CDF represents the probability that the random variable takes on a value less than or equal to x
• PDF can be derived from the CDF by taking its derivative: $f(x) = \frac{d}{dx}F(x)$
• CDF always ranges from 0 to 1, while PDF can take any non-negative value

Properties of PDFs

• Non-negative for all values in its domain: $f(x) \geq 0$ for all x
• Integrates to 1 over its entire domain: $\int_{-\infty}^{\infty} f(x) dx = 1$
• Continuous and smooth for most common distributions, with possible exceptions at specific points
• May have multiple modes (peaks) or be symmetric or skewed, depending on the distribution
• Determines various statistical properties of the random variable (mean, variance, quantiles)

Common probability density functions

• Theoretical statistics employs a diverse set of probability density functions to model various real-world phenomena
• Understanding common PDFs provides a foundation for selecting appropriate models in statistical analysis and hypothesis testing
• Each PDF has unique characteristics and parameters that determine its shape, location, and scale

Normal distribution

• Bell-shaped, symmetric distribution characterized by mean (μ) and standard deviation (σ)
• PDF given by $f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$
• Widely used due to the Central Limit Theorem and its occurrence in natural phenomena
• Standard normal distribution has μ = 0 and σ = 1, often denoted as N(0,1)
• Useful for modeling phenomena influenced by many small, independent factors (height, measurement errors)

Exponential distribution

• Models time between events in a Poisson process or the lifetime of certain components
• PDF given by $f(x) = \lambda e^{-\lambda x}$ for x ≥ 0, where λ is the rate parameter
• Characterized by the memoryless property, meaning the future lifetime is independent of the past
• Mean and standard deviation both equal to 1/λ
• Commonly used in reliability analysis and queueing theory

Uniform distribution

• Represents equal probability over a continuous interval [a, b]
• PDF given by $f(x) = \frac{1}{b-a}$ for a ≤ x ≤ b
• Constant probability density throughout its range
• Often used as a basis for generating random numbers and in simulation studies
• Mean is (a+b)/2, and variance is (b-a)²/12

Gamma distribution

• Generalizes the exponential distribution and models waiting times or amounts
• PDF given by $f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x}$ for x > 0, where α is the shape parameter and β is the rate parameter
• Includes exponential and chi-squared distributions as special cases
• Flexible shape allows modeling of various skewed distributions
• Mean is α/β, and variance is α/β²

Beta distribution

• Defined on the interval [0, 1] and often used to model proportions or probabilities
• PDF given by $f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$ where B(α,β) is the beta function
• Shape determined by two positive parameters, α and β
• Useful in Bayesian statistics as a conjugate prior for binomial and Bernoulli distributions
• Mean is α/(α+β), and variance is αβ/((α+β)²(α+β+1))

Multivariate density functions

• Multivariate density functions extend the concept of PDFs to random vectors in higher dimensions
• These functions play a crucial role in analyzing relationships between multiple random variables
• Understanding multivariate densities is essential for advanced statistical modeling and inference

Joint PDFs

• Describe the simultaneous behavior of two or more random variables
• Represented by a function $f(x_1, x_2, ..., x_n)$ for n random variables
• Must integrate to 1 over the entire n-dimensional space
• Capture dependencies and correlations between variables
• Allow calculation of probabilities for events involving multiple variables simultaneously

Marginal PDFs

• Derived from joint PDFs by integrating out other variables
• Represent the distribution of a single variable, ignoring others
• Obtained by integrating the joint PDF over all other variables
• For two variables: $f_X(x) = \int_{-\infty}^{\infty} f(x,y) dy$
• Useful for analyzing individual variables in a multivariate context

Conditional PDFs

• Describe the distribution of one variable given specific values of others
• Defined as the ratio of joint PDF to marginal PDF: $f_{Y|X}(y|x) = \frac{f(x,y)}{f_X(x)}$
• Capture how the distribution of one variable changes based on known values of others
• Essential for understanding dependencies and making predictions
• Form the basis for concepts like conditional expectation and regression analysis

Transformations of random variables

• Transformations of random variables are crucial in theoretical statistics for deriving new distributions
• These techniques allow statisticians to relate different probability distributions and simplify complex problems
• Understanding transformations is essential for advanced statistical modeling and inference

Change of variables technique

• Method for finding the PDF of a function of one or more random variables
• Involves transforming the original PDF using the inverse function and its derivative
• For a monotonic function Y = g(X), the PDF of Y is given by $f_Y(y) = f_X(g^{-1}(y)) \left|\frac{d}{dy}g^{-1}(y)\right|$
• Allows derivation of new distributions from known ones (log-normal from normal)
• Crucial for understanding relationships between different probability distributions

Jacobian determinant

• Generalizes the change of variables technique to multivariate transformations
• Represents the scaling factor for volumes under the transformation
• For a transformation Y = g(X) in n dimensions, the joint PDF of Y is given by $f_Y(y) = f_X(g^{-1}(y)) |J|$
• J is the Jacobian matrix of partial derivatives of the inverse transformation
• Essential for analyzing multivariate transformations and deriving multivariate distributions
• Applications include coordinate transformations in physics and economics

Moments and expectation

• Moments and expectations provide essential summary statistics for probability distributions
• These concepts allow for characterizing and comparing different distributions
• Understanding moments is crucial for parameter estimation and hypothesis testing in theoretical statistics

Expected value

• Represents the average or mean of a random variable
• Calculated as $E[X] = \int_{-\infty}^{\infty} x f(x) dx$ for continuous random variables
• Provides a measure of central tendency for the distribution
• Linear property: $E[aX + b] = aE[X] + b$ for constants a and b
• Forms the basis for many statistical estimators and decision rules

Variance and standard deviation

• Variance measures the spread or dispersion of a random variable around its mean
• Defined as $Var(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2$
• Standard deviation is the square root of variance, providing a measure in the same units as the original variable
• Important for assessing the precision of estimates and constructing confidence intervals
• Plays a crucial role in hypothesis testing and statistical inference

Higher-order moments

• Generalize the concept of expectation to higher powers of the random variable
• kth moment defined as $E[X^k] = \int_{-\infty}^{\infty} x^k f(x) dx$
• Central moments use deviations from the mean: $E[(X - E[X])^k]$
• Third central moment (skewness) measures asymmetry of the distribution
• Fourth central moment (kurtosis) measures the tailedness of the distribution
• Higher-order moments provide additional information about the shape and characteristics of distributions

Parameter estimation

• Parameter estimation forms a cornerstone of statistical inference in theoretical statistics
• These techniques allow for drawing conclusions about population parameters from sample data
• Understanding estimation methods is crucial for applying statistical theory to real-world problems

Method of moments

• Estimates parameters by equating sample moments to theoretical moments
• Involves solving a system of equations based on the first k moments for k parameters
• Simple to implement and computationally efficient
• May not always produce optimal estimates, especially for small sample sizes
• Useful for obtaining initial estimates or when maximum likelihood is computationally intensive

Maximum likelihood estimation

• Estimates parameters by maximizing the likelihood function of the observed data
• Based on finding parameter values that make the observed data most probable
• Often leads to consistent, efficient, and asymptotically normal estimators
• Involves solving $\frac{\partial}{\partial \theta} \log L(\theta; x) = 0$ where L is the likelihood function
• Widely used due to its optimal asymptotic properties and flexibility
• Can be computationally intensive for complex models or large datasets

Applications in statistics

• Probability density functions play a crucial role in various statistical applications
• These applications form the basis for statistical inference and decision-making
• Understanding these concepts is essential for applying theoretical statistics to real-world problems

Likelihood functions

• Represent the probability of observing the data given specific parameter values
• Defined as the joint PDF of the observed data, viewed as a function of the parameters
• Form the basis for maximum likelihood estimation and likelihood ratio tests
• Allow for comparing different statistical models and hypotheses
• Crucial for Bayesian inference, where they are combined with prior distributions

Hypothesis testing

• Uses probability distributions to make decisions about population parameters
• Test statistics often follow known distributions under null hypotheses (t, F, chi-squared)
• P-values calculated using the PDF or CDF of the test statistic's distribution
• Power of a test determined by the distribution of the test statistic under alternative hypotheses
• Critical in scientific research for assessing the significance of experimental results

Confidence intervals

• Provide a range of plausible values for population parameters
• Constructed using the sampling distribution of estimators, often based on normal approximations
• Interval endpoints typically involve quantiles of known distributions (t, normal)
• Confidence level determined by the area under the PDF of the sampling distribution
• Essential for quantifying uncertainty in parameter estimates and making inferences about populations

Numerical methods

• Numerical methods are essential in theoretical statistics for handling complex probability distributions
• These techniques allow for approximating integrals, generating random samples, and solving optimization problems
• Understanding numerical methods is crucial for applying statistical theory to real-world problems with intractable analytical solutions

Monte Carlo integration

• Approximates complex integrals using random sampling
• Estimates expected values by averaging over randomly generated samples
• Convergence rate proportional to $1/\sqrt{n}$, where n is the number of samples
• Particularly useful for high-dimensional integrals and complex probability distributions
• Applications include calculating probabilities, expectations, and variances for complicated distributions

Importance sampling

• Improves efficiency of Monte Carlo methods by sampling from an alternative distribution
• Reduces variance of estimates by focusing on important regions of the integration domain
• Involves using a proposal distribution q(x) and weighting samples by $w(x) = f(x)/q(x)$
• Particularly useful for rare event simulation and Bayesian computation
• Requires careful choice of proposal distribution to be effective

Relationship to other concepts

• Understanding the relationships between different probabilistic concepts is crucial in theoretical statistics
• These relationships provide a unified framework for analyzing both discrete and continuous random phenomena
• Recognizing the connections and distinctions between these concepts is essential for applying appropriate statistical methods

PDFs vs PMFs

• Probability Density Functions (PDFs) describe continuous random variables
• Probability Mass Functions (PMFs) describe discrete random variables
• PDFs integrate to 1 over their domain, while PMFs sum to 1
• PDFs can take values greater than 1, unlike PMFs which are always between 0 and 1
• Both provide a complete description of the probability distribution for their respective types of random variables

Continuous vs discrete distributions

• Continuous distributions use PDFs and are defined over intervals of real numbers
• Discrete distributions use PMFs and are defined over countable sets of values
• Continuous distributions allow for infinitely precise measurements, while discrete distributions represent countable outcomes
• Some distributions (binomial, Poisson) can approximate continuous distributions under certain conditions
• Many statistical techniques apply to both types, but specific methods may differ (integration vs summation)