🎲Intro to Probability Unit 7 – Expectation and Variance of Random Variables

Key Concepts and Definitions

• Random variable is a function that maps outcomes of a random experiment to real numbers
• Expectation (mean) of a random variable $X$, denoted as $E(X)$, is the average value of the variable over many trials
• Variance of a random variable $X$, denoted as $Var(X)$ or $\sigma^2$, measures the average squared deviation from the mean
  • Formula for variance: $Var(X) = E[(X - E(X))^2]$
• Standard deviation $\sigma$ is the square root of the variance and has the same units as the random variable
• Moment generating function (MGF) of a random variable $X$ is defined as $M_X(t) = E(e^{tX})$
  • MGF uniquely determines the distribution of a random variable
• Probability mass function (PMF) for a discrete random variable $X$ gives the probability of each possible value
• Probability density function (PDF) for a continuous random variable $X$ describes the relative likelihood of the variable taking on a specific value

Understanding Expectation

• Expectation represents the long-run average value of a random variable over many independent trials
• For a discrete random variable $X$ with PMF $p(x)$, the expectation is calculated as $E(X) = \sum_{x} x \cdot p(x)$
• For a continuous random variable $X$ with PDF $f(x)$, the expectation is calculated as $E(X) = \int_{-\infty}^{\infty} x \cdot f(x) dx$
• Linearity of expectation states that for random variables $X$ and $Y$ and constants $a$ and $b$, $E(aX + bY) = aE(X) + bE(Y)$
  • This property holds even if $X$ and $Y$ are dependent
• Law of the unconscious statistician (LOTUS) allows calculating the expectation of a function $g(X)$ of a random variable $X$ as $E(g(X)) = \sum_{x} g(x) \cdot p(x)$ for discrete $X$ or $E(g(X)) = \int_{-\infty}^{\infty} g(x) \cdot f(x) dx$ for continuous $X$

Properties of Expectation

• Expectation is a linear operator, meaning $E(aX + bY) = aE(X) + bE(Y)$ for constants $a$ and $b$
• If $X$ is a constant random variable with value $c$, then $E(X) = c$
• For independent random variables $X$ and $Y$, $E(XY) = E(X)E(Y)$
  • This property does not generally hold for dependent random variables
• Expectation of a sum of random variables equals the sum of their individual expectations: $E(\sum_{i=1}^{n} X_i) = \sum_{i=1}^{n} E(X_i)$
• Monotonicity of expectation states that if $X \leq Y$ for all outcomes, then $E(X) \leq E(Y)$
• Expectation of a non-negative random variable is always non-negative: if $X \geq 0$, then $E(X) \geq 0$

Calculating Variance

• Variance measures the average squared deviation of a random variable from its mean
• Formula for variance: $Var(X) = E[(X - E(X))^2]$
  • Expanded form: $Var(X) = E(X^2) - [E(X)]^2$
• For a discrete random variable $X$ with PMF $p(x)$, the variance is calculated as $Var(X) = \sum_{x} (x - E(X))^2 \cdot p(x)$
• For a continuous random variable $X$ with PDF $f(x)$, the variance is calculated as $Var(X) = \int_{-\infty}^{\infty} (x - E(X))^2 \cdot f(x) dx$
• Properties of variance:
  • $Var(aX + b) = a^2Var(X)$ for constants $a$ and $b$
  • For independent random variables $X$ and $Y$, $Var(X + Y) = Var(X) + Var(Y)$
• Standard deviation $\sigma$ is the square root of the variance and has the same units as the random variable

Relationships Between Expectation and Variance

• Variance can be expressed in terms of expectation: $Var(X) = E(X^2) - [E(X)]^2$
• Chebyshev's inequality relates expectation and variance to provide bounds on the probability of a random variable deviating from its mean
  • For any random variable $X$ and positive constant $k$, $P(|X - E(X)| \geq k\sigma) \leq \frac{1}{k^2}$
• Markov's inequality provides an upper bound on the probability of a non-negative random variable exceeding a certain value
  • For a non-negative random variable $X$ and constant $a > 0$, $P(X \geq a) \leq \frac{E(X)}{a}$
• Jensen's inequality states that for a convex function $g$ and random variable $X$, $E(g(X)) \geq g(E(X))$
  • For a concave function $g$, the inequality is reversed: $E(g(X)) \leq g(E(X))$

Applications in Probability Problems

• Expectation and variance are used to characterize the behavior of random variables and their distributions
• In decision theory, expectation is used to calculate the expected value of different strategies or actions
  • Example: In a game with payoffs, the expected value of each strategy can be computed to determine the optimal choice
• Variance and standard deviation are used to quantify risk and uncertainty in various fields (finance, insurance)
  • Example: Portfolio theory uses variance to measure the risk of investment portfolios
• Moment generating functions (MGFs) are used to uniquely determine the distribution of a random variable and calculate its moments
  • The $n$-th moment of a random variable $X$ is defined as $E(X^n)$ and can be obtained by differentiating the MGF $n$ times and evaluating at $t=0$
• Expectation and variance are central to the study of limit theorems in probability, such as the law of large numbers and the central limit theorem

Common Distributions and Their Moments

• Bernoulli distribution (single trial with binary outcome):
  • PMF: $P(X = 1) = p$, $P(X = 0) = 1 - p$
  • Expectation: $E(X) = p$
  • Variance: $Var(X) = p(1 - p)$
• Binomial distribution (number of successes in $n$ independent Bernoulli trials):
  • PMF: $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$
  • Expectation: $E(X) = np$
  • Variance: $Var(X) = np(1-p)$
• Poisson distribution (number of events in a fixed interval):
  • PMF: $P(X = k) = \frac{e^{-\lambda}\lambda^k}{k!}$
  • Expectation: $E(X) = \lambda$
  • Variance: $Var(X) = \lambda$
• Normal (Gaussian) distribution:
  • PDF: $f(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$
  • Expectation: $E(X) = \mu$
  • Variance: $Var(X) = \sigma^2$

Practice Problems and Examples

1. A fair six-sided die is rolled. Let $X$ be the number shown on the die. Calculate the expectation and variance of $X$.

   • Solution:
     • $E(X) = \sum_{x=1}^{6} x \cdot P(X = x) = \frac{1+2+3+4+5+6}{6} = 3.5$
     • $Var(X) = E(X^2) - [E(X)]^2 = \frac{1^2+2^2+3^2+4^2+5^2+6^2}{6} - 3.5^2 = 2.92$

2. The time (in minutes) a customer spends in a store follows an exponential distribution with parameter $\lambda = 0.2$. Find the expected time spent in the store and the variance of the time spent.

   • Solution:
     • For an exponential distribution with parameter $\lambda$, the expectation is $E(X) = \frac{1}{\lambda}$ and the variance is $Var(X) = \frac{1}{\lambda^2}$.
     • $E(X) = \frac{1}{0.2} = 5$ minutes
     • $Var(X) = \frac{1}{0.2^2} = 25$ square minutes

3. Let $X$ be a random variable with $E(X) = 2$ and $Var(X) = 4$. Find $E(3X - 5)$ and $Var(3X - 5)$.

   • Solution:
     • Using the linearity of expectation, $E(3X - 5) = 3E(X) - 5 = 3 \cdot 2 - 5 = 1$
     • Using the properties of variance, $Var(3X - 5) = 3^2Var(X) = 9 \cdot 4 = 36$

4. The number of customers arriving at a store follows a Poisson distribution with a mean of 10 per hour. Calculate the probability that more than 12 customers arrive in a given hour using Markov's inequality.

   • Solution:
     • Let $X$ be the number of customers arriving in an hour. We want to find $P(X > 12)$.
     • By Markov's inequality, $P(X > 12) \leq \frac{E(X)}{12} = \frac{10}{12} \approx 0.833$
     • This provides an upper bound on the probability, but the actual probability will be lower due to the Poisson distribution's properties.