2.2 Properties of probability

Properties of Probability

Probability properties give you the rules for how probabilities behave. They tell you what values probabilities can take, how they relate to each other, and how to combine them. Everything else in this course builds on these properties, so understanding them well now pays off later.

Fundamental Properties

Probability measures the likelihood of an event occurring, expressed as a number between 0 and 1.

There are a few core properties you need to know:

• Non-negativity: The probability of any event is greater than or equal to zero. You can't have a negative chance of something happening.
• Boundedness: Probabilities can't exceed 1. Combined with non-negativity, this means every probability lives in the range [0, 1].
• Normalization: The probabilities of all possible outcomes in the sample space must add up to 1. Something has to happen.
• Probability of the empty set: $P(\emptyset) = 0$. An impossible event has zero probability.
• Probability of the sample space: $P(S) = 1$. The certain event has probability 1.

Complement rule: For any event $A$, the probability of $A$ not happening is $P(A') = 1 - P(A)$. This works because either $A$ happens or it doesn't, and those two probabilities must sum to 1.

Monotonicity: If event $A$ is a subset of event $B$ (meaning every outcome in $A$ is also in $B$), then $P(A) \leq P(B)$. A smaller collection of outcomes can't be more likely than a larger one that contains it.

Kolmogorov's Axioms and Proofs

All probability properties can be derived from just three axioms, laid out by Kolmogorov:

• Axiom 1 (Non-negativity): $P(A) \geq 0$ for any event $A$
• Axiom 2 (Normalization): $P(S) = 1$, where $S$ is the sample space
• Axiom 3 (Additivity): For mutually exclusive events $A$ and $B$, $P(A \cup B) = P(A) + P(B)$

These three axioms are the foundation. Every other property is a consequence of them.

Proof of the Complement Rule

$P(A') = 1 - P(A)$

1. $A$ and $A'$ are mutually exclusive and together cover the entire sample space, so $A \cup A' = S$.
2. By Axiom 3: $P(A) + P(A') = P(S)$.
3. By Axiom 2: $P(S) = 1$.
4. Rearranging: $P(A') = 1 - P(A)$.

Proof of the Inclusion-Exclusion Principle

For any two events $A$ and $B$:

$P(A \cup B) = P(A) + P(B) - P(A \cap B)$

1. You can split $A \cup B$ into two mutually exclusive parts: $A$ and $B \setminus A$ (the part of $B$ not in $A$).

2. By Axiom 3: $P(A \cup B) = P(A) + P(B \setminus A)$.

3. Similarly, $B$ can be split into $A \cap B$ and $B \setminus A$, so $P(B) = P(A \cap B) + P(B \setminus A)$.

4. Rearranging step 3: $P(B \setminus A) = P(B) - P(A \cap B)$.

5. Substituting into step 2: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$.

The subtraction of $P(A \cap B)$ corrects for double-counting the overlap between $A$ and $B$.

Proof of Monotonicity

If $A \subseteq B$, then $P(A) \leq P(B)$.

1. Write $B = A \cup (B \setminus A)$, where $A$ and $B \setminus A$ are mutually exclusive.
2. By Axiom 3: $P(B) = P(A) + P(B \setminus A)$.
3. By Axiom 1: $P(B \setminus A) \geq 0$.
4. Therefore: $P(B) \geq P(A)$.

Law of Total Probability

If events $B_1, B_2, \ldots, B_n$ form a partition of the sample space (they're mutually exclusive and together cover all of $S$), then for any event $A$:

$P(A) = \sum_{i} P(A \mid B_i) \cdot P(B_i)$

This is useful when you can't calculate $P(A)$ directly but you can figure it out by breaking it into cases.

Bayes' Theorem

$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$

The proof follows from the definition of conditional probability:

1. By definition: $P(A \cap B) = P(A \mid B) \cdot P(B)$.
2. Also by definition: $P(A \cap B) = P(B \mid A) \cdot P(A)$.
3. Setting these equal and solving for $P(A \mid B)$ gives Bayes' theorem.

Bayes' theorem lets you "reverse" a conditional probability. If you know $P(B \mid A)$, you can find $P(A \mid B)$.

Applying Probability Properties

These properties aren't just abstract rules. Here's how they show up in practice:

Complement rule in medical testing: If the probability of having a certain condition is $P = 0.05$, then the probability of not having it is $1 - 0.05 = 0.95$. This is often easier than computing the "not having it" probability from scratch.

Normalization in weather forecasting: A forecast might assign probabilities of 60% sunny, 30% cloudy, and 10% rainy. These must sum to 100%. If they didn't, the model would be broken.

Additivity for system reliability: If a system has three independent failure modes with probabilities 0.01, 0.02, and 0.005, and these failures are mutually exclusive, the total failure probability is $0.01 + 0.02 + 0.005 = 0.035$. (Note: for non-mutually-exclusive failures, you'd need inclusion-exclusion.)

Monotonicity in quality control: If an automated assembly line has a defect rate of 1.5% and a manual line has a defect rate of 3%, and the set of conditions causing defects in the automated line is a subset of those in the manual line, monotonicity confirms the automated line's probability should be lower.

Boundedness as a sanity check: Whenever you calculate a probability, check that your answer falls between 0 and 1. If you get $P = -0.3$ or $P = 1.4$, something went wrong. This is one of the quickest ways to catch errors.