Key Concepts of Random Variables

Why This Matters

Random variables are the mathematical foundation for modeling uncertainty. Whether you're analyzing signal noise, predicting system failures, or modeling network traffic, you're working with random variables. This topic connects directly to everything else in your probability course: from basic probability axioms to statistical inference.

You're being tested on more than definitions here. Exam questions will ask you to choose the right distribution for a scenario, calculate expected values and variances, and apply limit theorems to real problems. Don't just memorize formulas. Understand when each distribution applies, how the PMF/PDF/CDF relate to each other, and why concepts like independence and the Central Limit Theorem matter. Master the underlying mechanics, and the formulas will make sense.

---

Foundations: Types of Random Variables

Before diving into specific distributions, you need to understand the fundamental distinction between discrete and continuous random variables. This classification determines which mathematical tools you'll use: summations vs. integrals, PMFs vs. PDFs.

Discrete Random Variables

A discrete random variable takes on specific, separated values (often integers). Think of things you can count: the number of defects in a batch, the number of heads in 10 coin flips.

• Probability Mass Function (PMF) assigns a probability to each possible value: $P(X = x) \geq 0$ and $\sum_x P(X = x) = 1$
• Key identifier: ask yourself "can I list all possible values?" If yes, it's discrete.

Continuous Random Variables

A continuous random variable can take any value within an interval. Voltage measurements, time between failures, and temperature readings are all continuous.

• Probability Density Function (PDF) describes likelihood, but $P(X = x) = 0$ for any single specific value. Only intervals have nonzero probability.
• Integration required: probabilities are calculated as $P(a \leq X \leq b) = \int_a^b f(x)\,dx$ where $f(x)$ is the PDF, and the total area under the PDF equals 1.

Cumulative Distribution Function (CDF)

The CDF works for both discrete and continuous variables, defined as $F(x) = P(X \leq x)$.

• Properties to know: non-decreasing, $\lim_{x \to -\infty} F(x) = 0$, and $\lim_{x \to \infty} F(x) = 1$
• PDF recovery: for continuous variables, $f(x) = \frac{dF(x)}{dx}$. The derivative of the CDF gives you the PDF.
• For discrete variables, the CDF is a staircase function with jumps at each value where the PMF is nonzero.

---

Describing Distributions: Location and Spread

Every distribution can be characterized by its moments: numerical summaries that capture where the distribution is centered and how spread out it is. These quantities are essential for comparing distributions and solving problems.

Expected Value (Mean)

The expected value is the long-run average. If you repeated the experiment infinitely many times, this is the average outcome you'd observe.

• Discrete: $E[X] = \sum_x x \cdot P(X = x)$
• Continuous: $E[X] = \int_{-\infty}^{\infty} x \cdot f(x)\,dx$
• Linearity property: $E[aX + b] = aE[X] + b$. This always holds, whether or not variables are independent. It simplifies many calculations.

Variance and Standard Deviation

• Variance measures spread around the mean: $\text{Var}(X) = E[(X - \mu)^2] = E[X^2] - (E[X])^2$. That second form (sometimes called the "computational formula") is almost always easier to use in practice.
• Standard deviation $\sigma = \sqrt{\text{Var}(X)}$ puts dispersion back in the same units as the original variable.
• Scaling property: $\text{Var}(aX + b) = a^2 \text{Var}(X)$. The constant $a$ gets squared, and the additive constant $b$ disappears entirely.

Moment Generating Functions

• Definition: $M_X(t) = E[e^{tX}]$. This function encodes all moments of a distribution.
• Moment extraction: the $n$th moment is $E[X^n] = M_X^{(n)}(0)$, meaning you take the $n$th derivative and evaluate at $t = 0$.
• Distribution identification: if two variables have the same MGF, they have the same distribution. This is a powerful tool for proofs and for identifying what distribution a transformed variable follows.

---

Discrete Distributions: Counting Events

These distributions model scenarios where you're counting occurrences. The key is matching the physical situation to the right model based on the underlying assumptions.

Bernoulli Distribution

The simplest random variable: a single trial with two outcomes. Success (1) occurs with probability $p$, failure (0) with probability $1-p$.

• Building block: every other discrete distribution in this section is built from Bernoulli trials.
• Moments: $E[X] = p$ and $\text{Var}(X) = p(1-p)$. Notice that variance is maximized at $p = 0.5$.

Binomial Distribution

Counts the number of successes in a fixed number $n$ of independent trials, each with the same success probability $p$.

• PMF: $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k = 0, 1, \ldots, n$
• Moments: $E[X] = np$ and $\text{Var}(X) = np(1-p)$
• Typical scenarios: number of defective items in a batch, number of correct answers on a multiple-choice test (if guessing randomly)

Poisson Distribution

Counts events occurring in a continuous interval of time or space, at a constant average rate $\lambda$.

• PMF: $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$
• Key property: $E[X] = \text{Var}(X) = \lambda$. When mean equals variance, think Poisson.
• Typical scenarios: number of emails per hour, number of typos per page, number of arrivals at a service counter

---

Continuous Distributions: Measuring Quantities

These distributions model measurements that can take any value in an interval. Each has distinct shapes and applications. Learn to recognize them from problem context.

Uniform Distribution

Every value in $[a, b]$ is equally likely. The PDF is flat: $f(x) = \frac{1}{b-a}$ for $a \leq x \leq b$.

• Moments: $E[X] = \frac{a+b}{2}$ (the midpoint) and $\text{Var}(X) = \frac{(b-a)^2}{12}$
• When to use it: often the right model when you have no information favoring any particular value within a range. Also commonly used for random number generation.

Normal (Gaussian) Distribution

The classic bell curve, symmetric around mean $\mu$, with spread controlled by $\sigma$.

• PDF: $f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$
• Standardization: $Z = \frac{X - \mu}{\sigma}$ transforms any normal variable to $N(0,1)$. This is essential for using Z-tables to look up probabilities.
• Central role: the Central Limit Theorem makes this distribution appear everywhere in statistics, even when the underlying data isn't normal.

Exponential Distribution

Models the time until the first event, when events occur at a constant rate $\lambda$. It's the continuous counterpart to the Poisson distribution: if events arrive at a Poisson rate, the waiting time between them is exponential.

• PDF: $f(x) = \lambda e^{-\lambda x}$ for $x \geq 0$
• CDF: $F(x) = 1 - e^{-\lambda x}$
• Moments: $E[X] = \frac{1}{\lambda}$ and $\text{Var}(X) = \frac{1}{\lambda^2}$
• Memoryless property: $P(X > s + t \mid X > s) = P(X > t)$. This means the probability of waiting another $t$ units doesn't depend on how long you've already waited. The exponential is the only continuous distribution with this property.

---

Multivariate Concepts: Multiple Random Variables

Real problems often involve multiple interacting variables. Understanding how variables relate is crucial for system analysis.

Joint Probability Distributions

A joint distribution describes the simultaneous behavior of two (or more) random variables.

• Joint PMF: $P(X = x, Y = y)$ for discrete variables; Joint PDF: $f(x, y)$ for continuous variables
• Marginal distributions are recovered by summing out (discrete) or integrating out (continuous) the other variable. For example: $f_X(x) = \int_{-\infty}^{\infty} f(x, y)\,dy$

Conditional Probability Distributions

Conditional distributions describe one variable given knowledge of another.

• Formula: $f(x \mid y) = \frac{f(x, y)}{f_Y(y)}$. Joint divided by marginal.
• This is the continuous analog of $P(A \mid B) = P(A \cap B) / P(B)$ from basic probability. Bayesian updating and signal detection both rely on conditional distributions.

Independence of Random Variables

$X$ and $Y$ are independent if knowing the value of one tells you nothing about the other.

• Formal condition: $P(X = x, Y = y) = P(X = x) \cdot P(Y = y)$ for all $x, y$ (discrete), or $f(x, y) = f_X(x) \cdot f_Y(y)$ (continuous). The joint factors into the product of the marginals.
• Why it matters: independence dramatically simplifies calculations. For instance, $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$ holds only when $X$ and $Y$ are independent (or more generally, uncorrelated).

Covariance and Correlation

• Covariance: $\text{Cov}(X, Y) = E[XY] - E[X]E[Y]$. Positive covariance means the variables tend to increase together; negative means one tends to decrease when the other increases.
• Correlation: $\rho = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$. This is standardized to $[-1, 1]$ and measures the strength of the linear relationship.
• Independence implies zero correlation, but zero correlation does not imply independence. A classic counterexample: let $X$ be uniform on $[-1, 1]$ and $Y = X^2$. Then $\rho = 0$, but $Y$ is completely determined by $X$.

---

Limit Theorems: Large-Sample Behavior

These theorems explain why probability works in practice and justify most of statistical inference. They're conceptual cornerstones. Expect them on exams.

Law of Large Numbers

As the sample size $n$ grows, the sample mean $\bar{X}_n$ converges to the true population mean $E[X]$.

• Practical meaning: averages of large samples reliably estimate population means. This is why polling works, why casinos are profitable in the long run, and why Monte Carlo simulation converges.
• Requirement: the observations must be independent and identically distributed (i.i.d.) with a finite mean.

Central Limit Theorem

For large $n$, the standardized sample mean is approximately normal:

$\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \approx N(0, 1)$

This holds regardless of the original distribution of the individual observations, as long as they're i.i.d. with finite mean $\mu$ and finite variance $\sigma^2$.

• Rule of thumb: $n \geq 30$ often suffices for a good approximation. Fewer samples are needed for symmetric distributions; more are needed for highly skewed ones.
• Applications: this justifies confidence intervals, hypothesis tests, and normal approximations to the binomial and Poisson.

---

Quick Reference Table

---

Self-Check Questions

1. A quality engineer counts defective chips in batches of 100. Which distribution applies: Binomial or Poisson? What if she instead counts defects arriving per hour at a testing station?

2. You're given that $E[X] = 5$ and $\text{Var}(X) = 5$. Which distribution might $X$ follow, and why does this moment relationship matter?

3. Compare the CDF for discrete vs. continuous random variables. How does the CDF behave at jump points for a discrete variable?

4. Two random variables have correlation $\rho = 0$. Are they necessarily independent? Provide a counterexample or explain why independence would follow.

5. You need to approximate the distribution of the sample mean from 50 independent measurements of a skewed variable. Which theorem justifies using a normal approximation, and what parameters would you use?