Intro to Probability Unit 13 Review: Moment and Probability Generating Functions

Key Concepts and Definitions

• Moment generating function (MGF) of a random variable $X$ is defined as $M_X(t) = E[e^{tX}]$
• MGF uniquely determines the probability distribution of a random variable
• Probability generating function (PGF) of a discrete random variable $X$ is defined as $G_X(s) = E[s^X] = \sum_{x} s^x P(X = x)$
• PGF encodes the probability mass function of a discrete random variable
• Moments of a random variable can be obtained by differentiating the MGF or PGF and evaluating at $t=0$ or $s=1$, respectively
  • First moment (mean): $E[X] = M'_X(0)$ or $E[X] = G'_X(1)$
  • Second moment: $E[X^2] = M''_X(0)$ or $E[X^2] = G''_X(1) + G'_X(1)$
• Cumulants are another set of descriptors for probability distributions, related to the logarithm of the MGF

Properties of Moment Generating Functions

• Linearity: For constants $a$ and $b$, $M_{aX+b}(t) = e^{bt} M_X(at)$
• Multiplication: If $X$ and $Y$ are independent random variables, then $M_{X+Y}(t) = M_X(t) \cdot M_Y(t)$
• Uniqueness: If two random variables have the same MGF, they have the same probability distribution
• Existence: The MGF of a random variable may not always exist (e.g., Cauchy distribution)
• Derivatives: The $n$-th derivative of the MGF at $t=0$ gives the $n$-th moment of the random variable
  • $E[X^n] = M^{(n)}_X(0)$
• Continuity: If a sequence of MGFs converges pointwise to a limit, the corresponding sequence of probability distributions converges weakly to the limiting distribution

Properties of Probability Generating Functions

• Linearity: For constants $a$ and $b$, $G_{aX+b}(s) = s^b G_X(s^a)$
• Multiplication: If $X$ and $Y$ are independent random variables, then $G_{X+Y}(s) = G_X(s) \cdot G_Y(s)$
• Uniqueness: If two discrete random variables have the same PGF, they have the same probability distribution
• Derivatives: The $n$-th derivative of the PGF at $s=1$ gives the $n$-th factorial moment of the random variable
  • $E[X(X-1)\cdots(X-n+1)] = G^{(n)}_X(1)$
• Probability mass function: The probability mass function can be recovered from the PGF by taking derivatives
  • $P(X = k) = \frac{1}{k!} G^{(k)}_X(0)$
• Composition: If $N$ is a non-negative integer-valued random variable with PGF $G_N(s)$ and $X_1, X_2, \ldots$ are independent and identically distributed random variables with PGF $G_X(s)$, then the PGF of the random sum $S_N = \sum_{i=1}^N X_i$ is given by $G_{S_N}(s) = G_N(G_X(s))$

Applications in Probability Theory

• Deriving the distribution of the sum of independent random variables
  • If $X$ and $Y$ are independent, the distribution of $X+Y$ can be found using the product of their MGFs or PGFs
• Calculating moments and cumulants of probability distributions
  • MGFs and PGFs provide a convenient way to calculate moments and cumulants without directly integrating or summing
• Proving central limit theorems
  • MGFs are used in the proofs of various central limit theorems, which describe the convergence of sums of random variables to normal distributions
• Analyzing branching processes and random walks
  • PGFs are used to study the evolution of population sizes in branching processes and the distribution of positions in random walks
• Solving problems in queuing theory and reliability analysis
  • MGFs and PGFs are used to derive performance measures in queuing systems (e.g., waiting time distribution) and reliability models (e.g., time to failure)

Relationship Between MGFs and PGFs

• PGFs can be seen as a special case of MGFs for discrete random variables
  • For a discrete random variable $X$, $G_X(s) = M_X(\ln(s))$
• MGFs and PGFs share many properties due to their similar definitions
  • Linearity, multiplication for independent random variables, uniqueness, and the ability to recover moments
• Some distributions have both MGFs and PGFs (e.g., Poisson distribution), while others may have only one or neither
• The choice between using an MGF or PGF depends on the nature of the random variable (continuous or discrete) and the problem at hand

Solving Problems with Generating Functions

• Identify the type of random variable (continuous or discrete) and the corresponding generating function (MGF or PGF)
• Determine the generating function of the random variable(s) involved in the problem
  • Use known MGFs or PGFs for common distributions or derive them from the definition
• Apply the appropriate properties of generating functions to solve the problem
  • Linearity, multiplication for independent random variables, or composition for random sums
• Recover the desired probability distribution, moments, or other quantities from the resulting generating function
  • Differentiate and evaluate at $t=0$ or $s=1$ for moments, or expand the generating function and identify the coefficients for probabilities
• Interpret the results in the context of the original problem

Common Distributions and Their Generating Functions

• Normal distribution: $M_X(t) = \exp(\mu t + \frac{1}{2}\sigma^2 t^2)$
• Poisson distribution: $M_X(t) = \exp(\lambda(e^t - 1))$ and $G_X(s) = \exp(\lambda(s - 1))$
• Binomial distribution: $G_X(s) = (1 - p + ps)^n$
• Geometric distribution: $G_X(s) = \frac{ps}{1 - (1-p)s}$
• Exponential distribution: $M_X(t) = \frac{1}{1 - t/\lambda}$ for $t < \lambda$
• Gamma distribution: $M_X(t) = (1 - t/\lambda)^{-\alpha}$ for $t < \lambda$
• Negative binomial distribution: $G_X(s) = \left(\frac{p}{1 - (1-p)s}\right)^r$

Advanced Topics and Extensions

• Multivariate generating functions for joint distributions of multiple random variables
  • Joint MGF: $M_{\mathbf{X}}(\mathbf{t}) = E[\exp(\mathbf{t}^T \mathbf{X})]$
  • Joint PGF: $G_{\mathbf{X}}(\mathbf{s}) = E[\mathbf{s}^{\mathbf{X}}]$
• Conditional generating functions for studying conditional distributions
  • Conditional MGF: $M_{X|Y}(t|y) = E[\exp(tX)|Y=y]$
  • Conditional PGF: $G_{X|Y}(s|y) = E[s^X|Y=y]$
• Laplace transforms as a generalization of moment generating functions
  • Laplace transform of a non-negative random variable $X$: $\mathcal{L}_X(s) = E[\exp(-sX)]$
• Characteristic functions as a complex-valued generalization of moment generating functions
  • Characteristic function of a random variable $X$: $\phi_X(t) = E[\exp(itX)]$
• Saddlepoint approximations for density and distribution functions based on generating functions
  • Approximate the density or distribution function of a random variable using the inverse Laplace transform of the moment generating function