2.3 Joint probability distributions

Joint probability distribution definition

Joint probability distributions describe how two or more random variables behave together. Rather than looking at each variable in isolation, a joint distribution captures the probabilistic relationship between them, telling you how likely different combinations of values are to occur simultaneously. This is foundational in stochastic processes, where you're almost always dealing with multiple uncertain quantities evolving together.

Discrete vs continuous

Discrete joint distributions apply when the random variables take on a countable number of values (integers, specific categories). Continuous joint distributions apply when the variables range over uncountably many values (real numbers on an interval or the whole real line).

The distinction matters because it changes how you compute probabilities:

• Discrete case: you sum the joint PMF over the outcomes of interest
• Continuous case: you integrate the joint PDF over a region of interest

Marginal vs conditional distributions

Marginal distributions extract the behavior of a single variable from the joint distribution, ignoring the others.

• Discrete: sum the joint PMF over all values of the other variables
• Continuous: integrate the joint PDF over all values of the other variables
• The result is a univariate distribution for that one variable alone

Conditional distributions describe how one variable behaves given that you know the value of another.

• Calculated by dividing the joint PMF/PDF by the marginal of the conditioning variable:

$p(x | y) = \frac{p(x, y)}{p_Y(y)} \quad \text{or} \quad f(x | y) = \frac{f(x, y)}{f_Y(y)}$

• This tells you how the distribution of $X$ shifts once you observe a specific value of $Y$

Joint probability mass functions

A joint probability mass function (PMF) assigns a probability to each possible combination of values for discrete random variables. Formally, $p(x_1, x_2, \ldots, x_n)$ maps from the set of all possible value-tuples to probabilities in $[0, 1]$.

Two properties must hold for a valid joint PMF:

• $p(x_1, \ldots, x_n) \geq 0$ for all combinations
• $\sum_{\text{all } (x_1, \ldots, x_n)} p(x_1, \ldots, x_n) = 1$

Discrete random variables

Joint PMFs are defined over a countable sample space consisting of all possible value-tuples. Common multivariate discrete distributions include the multinomial (generalization of the binomial to multiple categories) and joint Poisson models.

Most univariate concepts extend naturally: you can compute expected values, variances, and moment generating functions from the joint PMF, just with sums over multiple indices instead of one.

Multivariate distributions

A multivariate PMF can be laid out in a table or matrix. For two variables $X$ and $Y$, you'd have a grid where each cell $(x_i, y_j)$ contains $p(x_i, y_j)$. Row sums give the marginal PMF of $Y$, and column sums give the marginal PMF of $X$.

From this table you can derive marginals, conditionals, and moments by applying the appropriate sums.

Calculating probabilities

To find the probability of an event $A$ defined by conditions on the variables, sum the PMF over all outcomes in $A$:

$P(A) = \sum_{(x_1,\ldots,x_n) \in A} p(x_1,\ldots,x_n)$

For complex events, the inclusion-exclusion principle and combinatorial counting techniques help you identify which outcomes belong to $A$.

Joint probability density functions

A joint probability density function (PDF) specifies a continuous multivariate distribution. Unlike a PMF, the value $f(x_1, \ldots, x_n)$ at a single point is not a probability. It represents relative likelihood. To get an actual probability, you integrate the PDF over a region.

Continuous random variables

Joint PDFs apply when the random variables can take any value in a continuous range. Common examples include the multivariate normal, joint exponential, and Dirichlet distributions.

Working with densities lets you model continuous quantities (time, distance, temperature) without artificially discretizing them.

Multivariate density functions

A valid joint PDF $f(x_1, \ldots, x_n)$ must satisfy:

• $f(x_1, \ldots, x_n) \geq 0$ everywhere
• $\int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} f(x_1, \ldots, x_n) \, dx_1 \cdots dx_n = 1$

Marginal PDFs are obtained by integrating out the other variables, and conditional PDFs are found by dividing the joint PDF by the appropriate marginal, just as in the discrete case.

Probability calculations with integrals

For an event $A$ defined by conditions on continuous random variables:

$P(A) = \int_{A} f(x_1,\ldots,x_n) \, dx_1 \cdots dx_n$

This often requires setting up multiple integrals with carefully chosen limits that describe the region $A$. For complex regions, changing the order of integration or switching to polar/spherical coordinates can simplify the computation.

Joint cumulative distribution functions

The joint cumulative distribution function (CDF) is defined as:

$F(x_1,\ldots,x_n) = P(X_1 \leq x_1,\ldots, X_n \leq x_n)$

It gives the probability that every variable simultaneously falls at or below its specified value. The CDF works for both discrete and continuous distributions, making it a universal way to specify any joint distribution.

CDF definition for joint distributions

For discrete variables, the CDF is a cumulative sum:

$F(x_1,\ldots,x_n) = \sum_{y_1 \leq x_1} \cdots \sum_{y_n \leq x_n} p(y_1,\ldots,y_n)$

For continuous variables, it's a cumulative integral:

$F(x_1,\ldots,x_n) = \int_{-\infty}^{x_1} \cdots \int_{-\infty}^{x_n} f(y_1,\ldots,y_n) \, dy_n \cdots dy_1$

You can recover the PDF from the CDF by taking mixed partial derivatives: $f(x_1, \ldots, x_n) = \frac{\partial^n F}{\partial x_1 \cdots \partial x_n}$.

Properties of joint CDFs

• Monotonicity: $F$ is non-decreasing in each argument. If $x_i \leq y_i$ for all $i$, then $F(x_1,\ldots,x_n) \leq F(y_1,\ldots,y_n)$.
• Boundary behavior: $F \to 1$ as all arguments go to $+\infty$, and $F \to 0$ if any single argument goes to $-\infty$.
• Marginal CDFs are obtained by sending the other arguments to $+\infty$:

$F_{X_i}(x_i) = \lim_{x_1,\ldots,x_{i-1},x_{i+1},\ldots,x_n \to \infty} F(x_1,\ldots,x_n)$

• Right-continuity: The joint CDF is right-continuous in each argument.

Relationship to probability

The CDF directly gives the probability of rectangular regions. For two variables, the probability that $X$ falls in $(a_1, b_1]$ and $Y$ falls in $(a_2, b_2]$ is:

$P(a_1 < X \leq b_1,\, a_2 < Y \leq b_2) = F(b_1, b_2) - F(a_1, b_2) - F(b_1, a_2) + F(a_1, a_2)$

This is an inclusion-exclusion over the four corners of the rectangle. For $n$ variables, the formula generalizes to $2^n$ terms with alternating signs:

$P(a_1 < X_1 \leq b_1, \ldots, a_n < X_n \leq b_n) = \sum_{\text{corners}} (-1)^{\text{number of } a_i\text{'s}} F(\text{corner})$

Independent vs dependent variables

Independence and dependence describe whether knowing the value of one variable tells you anything about the others. This distinction has major consequences for how you work with joint distributions: independence dramatically simplifies calculations, while dependence requires you to carefully track how variables interact.

Definition of independence

Random variables $X_1, \ldots, X_n$ are independent if their joint distribution factors into a product of marginals:

$p(x_1,\ldots,x_n) = p_1(x_1) \cdots p_n(x_n) \quad \text{(discrete)}$

$f(x_1,\ldots,x_n) = f_1(x_1) \cdots f_n(x_n) \quad \text{(continuous)}$

Equivalently, the joint CDF factors: $F(x_1, \ldots, x_n) = F_1(x_1) \cdots F_n(x_n)$.

The intuition: knowing the realized values of some variables doesn't change the probability distribution of the others.

Factoring joint distributions

When variables are independent, the product structure lets you:

• Compute joint probabilities by simply multiplying marginal probabilities
• Treat each variable separately for expectations: $E[g(X)h(Y)] = E[g(X)] \cdot E[h(Y)]$
• Derive that $\text{Cov}(X, Y) = 0$ (though the converse is not generally true)

Many results about sums, maxima, and other functions of random variables become tractable specifically because of this factorization.

Conditional distributions for dependence

When variables are not independent, conditional distributions are the primary tool for describing their relationship:

$p(x_1,\ldots,x_k \mid x_{k+1},\ldots,x_n) = \frac{p(x_1,\ldots,x_n)}{p(x_{k+1},\ldots,x_n)}$

$f(x_1,\ldots,x_k \mid x_{k+1},\ldots,x_n) = \frac{f(x_1,\ldots,x_n)}{f(x_{k+1},\ldots,x_n)}$

The denominator must be nonzero (positive marginal probability or density at the conditioning point). Conditional distributions are central to Bayesian inference, where you update beliefs about unknown quantities after observing data.

Covariance and correlation

Covariance and correlation quantify the linear dependence between two random variables. They appear throughout probability and statistics, from the variance of sums to the multivariate normal distribution.

Measures of dependence

Covariance between $X$ and $Y$:

$\text{Cov}(X,Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y]$

• Positive covariance: $X$ and $Y$ tend to be large (or small) together
• Negative covariance: when one is large, the other tends to be small
• Zero covariance: no linear association (but nonlinear dependence can still exist)

Correlation (Pearson correlation coefficient):

$\rho(X,Y) = \frac{\text{Cov}(X,Y)}{\sqrt{\text{Var}(X)\,\text{Var}(Y)}}$

• Always between $-1$ and $1$
• $\rho = \pm 1$ means $Y$ is an exact linear function of $X$
• $\rho = 0$ means no linear relationship, but a nonlinear one may still exist

Covariance matrix

For a random vector $\mathbf{X} = (X_1, \ldots, X_n)$, the covariance matrix $\Sigma$ is the $n \times n$ matrix with $(i,j)$ entry $\text{Cov}(X_i, X_j)$.

Key properties:

• Symmetric: $\Sigma_{ij} = \Sigma_{ji}$ since $\text{Cov}(X_i, X_j) = \text{Cov}(X_j, X_i)$
• Positive semi-definite: $\mathbf{a}^T \Sigma \mathbf{a} \geq 0$ for any vector $\mathbf{a}$
• Diagonal entries are the variances: $\Sigma_{ii} = \text{Var}(X_i)$

The covariance matrix appears in the multivariate normal PDF, in the multivariate CLT, and in linear transformations of random vectors.

Correlation coefficient

The correlation matrix $\mathbf{R}$ has $(i,j)$ entry $\rho(X_i, X_j)$. It's the covariance matrix of the standardized variables $Z_i = (X_i - \mu_i)/\sigma_i$.

• Diagonal entries are all 1
• Off-diagonal entries lie in $[-1, 1]$
• Often easier to interpret than the covariance matrix because it's scale-free

Transformations of random vectors

Transformations create new random variables or vectors from existing ones. You might standardize variables, rotate coordinate systems, or apply nonlinear mappings. The goal is to find the distribution of the transformed vector given the distribution of the original.

Linear transformations

A linear transformation takes the form $\mathbf{Y} = \mathbf{A}\mathbf{X} + \mathbf{b}$, where $\mathbf{A}$ is an $m \times n$ matrix and $\mathbf{b}$ is an $m \times 1$ vector.

The mean and covariance transform as:

• $E[\mathbf{Y}] = \mathbf{A}\,E[\mathbf{X}] + \mathbf{b}$
• $\text{Cov}(\mathbf{Y}) = \mathbf{A}\,\text{Cov}(\mathbf{X})\,\mathbf{A}^T$

These formulas are used constantly in statistics and signal processing (PCA, Kalman filtering, whitening transforms, etc.).

Jacobian matrix

For a general (possibly nonlinear) transformation $\mathbf{Y} = g(\mathbf{X})$ where $g$ is one-to-one and differentiable, the joint PDF of $\mathbf{Y}$ is:

$f_{\mathbf{Y}}(\mathbf{y}) = f_{\mathbf{X}}(g^{-1}(\mathbf{y})) \cdot |\det(J_{g^{-1}}(\mathbf{y}))|$

Here $J_{g^{-1}}(\mathbf{y})$ is the Jacobian matrix of the inverse transformation, with $(i,j)$ entry $\frac{\partial x_i}{\partial y_j}$.

The absolute value of the Jacobian determinant accounts for how the transformation stretches or compresses volume in the sample space. If the transformation expands a region, the density decreases proportionally, and vice versa.

Distribution of transformed variables

Continuous case: The joint CDF of $\mathbf{Y} = g(\mathbf{X})$ is:

$F_{\mathbf{Y}}(\mathbf{y}) = P(g(\mathbf{X}) \leq \mathbf{y}) = \int_{g(\mathbf{x}) \leq \mathbf{y}} f_{\mathbf{X}}(\mathbf{x}) \, d\mathbf{x}$

For linear transformations with invertible $\mathbf{A}$, the Jacobian simplifies to $J_{g^{-1}} = \mathbf{A}^{-1}$, so $|\det(J)| = 1/|\det(\mathbf{A})|$.

Discrete case: The PMF of $\mathbf{Y}$ is found by summing over all pre-images:

$p_{\mathbf{Y}}(\mathbf{y}) = \sum_{\mathbf{x}:\, g(\mathbf{x}) = \mathbf{y}} p_{\mathbf{X}}(\mathbf{x})$

Sums of random variables

Sums of random variables come up everywhere: total measurement error, aggregate demand, cumulative waiting times. The distribution of a sum depends on the joint distribution of the summands, and when the variables are independent, convolution gives you a clean formula.

Convolution for discrete variables

For independent discrete random variables $X$ and $Y$, the PMF of $Z = X + Y$ is the convolution of their individual PMFs:

$p_Z(z) = \sum_k p_X(k)\, p_Y(z - k)$

Each term in the sum represents one way to split the total $z$ between $X = k$ and $Y = z - k$. For more than two variables:

$p_{X_1 + \cdots + X_n}(z) = \sum_{k_1 + \cdots + k_n = z} p_{X_1}(k_1) \cdots p_{X_n}(k_n)$

These sums can become computationally expensive, but probability generating functions and the discrete Fourier transform (via the convolution theorem) make them more tractable.

Convolution integral for continuous variables

For independent continuous random variables $X$ and $Y$ with PDFs $f_X$ and $f_Y$, the PDF of $Z = X + Y$ is:

$f_Z(z) = \int_{-\infty}^{\infty} f_X(t)\, f_Y(z - t)\, dt$

The logic is the same as the discrete case, but with integration replacing summation. Moment generating functions (or characteristic functions) often provide a faster route: if $X$ and $Y$ are independent, then $M_Z(s) = M_X(s) \cdot M_Y(s)$, and you can identify the distribution of $Z$ from its MGF.