Covariance Properties

Why This Matters

Covariance sits at the heart of probabilistic modeling—it's the mathematical tool that captures how random variables move together. When you're analyzing portfolio risk, modeling physical systems with multiple interacting components, or building any statistical model with more than one variable, you're relying on covariance properties. The exam will test your ability to manipulate covariance expressions, recognize when properties apply, and connect covariance to related concepts like variance, correlation, and independence.

Here's the key insight: covariance properties aren't arbitrary rules to memorize. They flow from the definition $\text{Cov}(X, Y) = E[XY] - E[X]E[Y]$ and the linearity of expectation. If you understand why each property holds, you can derive them on the spot and apply them flexibly to novel problems. Don't just memorize formulas—know what structural feature of covariance each property demonstrates.

---

Foundational Definition and Structure

These properties establish what covariance is and how it connects to other fundamental concepts. The definition determines everything else.

Definition of Covariance

• $\text{Cov}(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]$—measures how two variables deviate from their means together
• Equivalent computational form: $\text{Cov}(X, Y) = E[XY] - E[X]E[Y]$, often easier for calculations
• Sign interpretation: positive covariance means variables tend to increase together; negative means one increases as the other decreases

Variance as a Special Case

• $\text{Var}(X) = \text{Cov}(X, X)$—variance is just covariance of a variable with itself
• This guarantees $\text{Var}(X) \geq 0$ since $\text{Cov}(X, X) = E[(X - \mu_X)^2]$
• Conceptual bridge: understanding this connection lets you apply covariance properties directly to variance problems

Symmetry Property

• $\text{Cov}(X, Y) = \text{Cov}(Y, X)$—order of arguments doesn't matter
• Follows directly from the definition: $(X - \mu_X)(Y - \mu_Y) = (Y - \mu_Y)(X - \mu_X)$
• Practical implication: covariance matrices are always symmetric, which has major computational advantages

---

Linearity Properties

Covariance inherits linearity from expectation. These properties are your primary tools for simplifying complex expressions.

Linearity in Each Argument

• $\text{Cov}(aX + b, Y) = a \cdot \text{Cov}(X, Y)$—constants added (like $b$) vanish; multipliers scale
• Covariance with a constant is zero: $\text{Cov}(c, Y) = 0$ for any constant $c$
• Why it works: constants have zero variance and don't co-vary with anything

Effect of Linear Transformations

• Full transformation rule: $\text{Cov}(aX + b, cY + d) = ac \cdot \text{Cov}(X, Y)$
• Only multipliers matter—the additive constants $b$ and $d$ disappear entirely
• Application: when standardizing variables (subtracting mean, dividing by standard deviation), this tells you exactly how covariance changes

Covariance of Sums

• $\text{Cov}(X + Y, Z) = \text{Cov}(X, Z) + \text{Cov}(Y, Z)$—covariance distributes over addition
• Extends to variance of sums: $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)$
• Powerful for decomposition: break complex random variables into simpler components

---

Independence and Correlation

These properties connect covariance to deeper probabilistic concepts. Understanding the gap between zero covariance and independence is exam-critical.

Independence and Zero Covariance

• Independence implies zero covariance: if $X \perp Y$, then $\text{Cov}(X, Y) = 0$
• The converse is FALSE—zero covariance only rules out linear relationships, not all dependence
• Classic counterexample: $X \sim \text{Uniform}(-1, 1)$ and $Y = X^2$ have $\text{Cov}(X, Y) = 0$ but are clearly dependent

Relationship to Correlation

• Correlation standardizes covariance: $\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$
• Bounded range: $-1 \leq \rho \leq 1$, with $|\rho| = 1$ only for perfect linear relationships
• Dimensionless: unlike covariance, correlation allows comparison across different variable pairs and scales

---

Matrix Formulation

For multivariate problems, covariance properties extend naturally to matrix form. This is essential for working with joint distributions of multiple variables.

Covariance Matrix Structure

• Definition: for random vector $\mathbf{X} = (X_1, \ldots, X_n)^T$, the covariance matrix $\Sigma$ has entries $\Sigma_{ij} = \text{Cov}(X_i, X_j)$
• Diagonal entries are variances: $\Sigma_{ii} = \text{Var}(X_i)$; off-diagonal entries are covariances
• Always symmetric and positive semi-definite—these properties follow from covariance symmetry and non-negativity of variance

Expected Value Formulation

• Matrix form: $\Sigma = E[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^T]$
• Computational form: $\Sigma = E[\mathbf{X}\mathbf{X}^T] - \boldsymbol{\mu}\boldsymbol{\mu}^T$
• Linear transformation rule: if $\mathbf{Y} = A\mathbf{X} + \mathbf{b}$, then $\text{Cov}(\mathbf{Y}) = A\Sigma A^T$

---

Quick Reference Table

---

Self-Check Questions

1. Using linearity properties, expand $\text{Var}(2X - 3Y + 5)$ in terms of $\text{Var}(X)$, $\text{Var}(Y)$, and $\text{Cov}(X, Y)$.

2. Give an example of two random variables with $\text{Cov}(X, Y) = 0$ that are not independent. What does this tell you about what covariance measures?

3. Compare and contrast covariance and correlation: which properties do they share, and why is correlation often preferred for interpretation?

4. If $X_1, X_2, X_3$ are random variables, express $\text{Cov}(X_1 + X_2, X_2 + X_3)$ in terms of variances and pairwise covariances.

5. Explain why the covariance matrix must be symmetric and positive semi-definite. Which covariance properties guarantee these matrix properties?