11.1 Evolutionary stable strategies (ESS)

Evolutionary stable strategies (ESS) explain how certain behaviors persist in populations over time. An ESS is a strategy that, once widely adopted, can't be displaced by any rare alternative. This concept connects game theory to biology and social science, helping us understand why specific behaviors and social norms are so durable.

Evolutionary Stable Strategies

Definition and Role in Evolutionary Game Theory

An evolutionary stable strategy (ESS) is a strategy that, if adopted by a population, cannot be invaded by any alternative strategy that starts out rare. The idea was introduced by John Maynard Smith and George R. Price in 1973 to explain evolved behavioral patterns in animals, such as fighting, mating, and parental care.

ESS sits at the intersection of game theory and evolutionary biology. In classical game theory, you ask "what's the rational choice?" In evolutionary game theory, you ask "which strategy survives and spreads?" An ESS answers that second question: it's the strategy that natural selection will maintain once it's established.

An ESS is a special kind of Nash equilibrium. Like any Nash equilibrium, no player benefits from unilaterally switching. But an ESS goes further: it must also resist invasion by rare mutant strategies that enter the population through mutation or migration. Not every Nash equilibrium is an ESS, but every ESS is a Nash equilibrium.

The core logic is about fitness. When the vast majority of the population uses strategy S, any individual using S must do at least as well as an individual using some alternative strategy T. If a mutant strategy can't gain a foothold, the population stays locked in on S.

Fitness and Resistance to Invasion

For a strategy to qualify as an ESS, two things must hold:

• Higher fitness when dominant. When almost everyone plays the ESS, individuals using it must earn fitness equal to or greater than individuals using any alternative.
• Resistance to rare invaders. If a small fraction of mutants enters the population playing a different strategy, those mutants must do worse on average than the residents playing the ESS.

This resistance is what separates an ESS from an ordinary equilibrium. A population at a Nash equilibrium might be vulnerable to a trickle of mutants who do equally well against the majority but do better against each other. An ESS rules that out.

The result is a stable population state: the ESS persists over evolutionary time scales because no alternative can get enough of a toehold to spread.

Conditions for ESS

Equilibrium Condition

The first formal requirement is the equilibrium condition: the strategy must be a best response to itself.

If every individual in the population plays strategy S, then no individual can improve their payoff by switching to some other strategy T. In notation:

$E(S, S) \geq E(T, S) \quad \text{for all alternative strategies } T$

where $E(X, Y)$ is the payoff to strategy X when matched against strategy Y.

This is exactly the Nash equilibrium requirement applied to a symmetric game where you're effectively playing against a copy of the population.

Stability Condition

The equilibrium condition alone isn't enough. You also need the stability condition, which handles the case where a mutant strategy T ties with S against the resident population.

If $E(S, S) = E(T, S)$, then the tiebreaker is how each strategy performs against the mutant:

$E(S, T) > E(T, T)$

This means that even when a mutant does equally well against the majority, the ESS must outperform the mutant in head-to-head matchups with other mutants.

Putting both conditions together, a strategy S is an ESS if and only if, for every alternative strategy $T \neq S$:

1. $E(S, S) > E(T, S)$, or
2. $E(S, S) = E(T, S)$ and $E(S, T) > E(T, T)$

The first condition alone is sufficient (strict best response to itself). The second condition kicks in only when there's a tie. Together, they guarantee that no mutant can invade, even in small numbers.

ESS Applications

Animal Behavior

The hawk-dove game is the classic ESS example. Two animals compete for a resource worth $V$:

• Hawks always escalate and fight.
• Doves display but retreat if the opponent escalates.

If two hawks meet, they fight, and each pays a cost $C$ (injury risk). The winner gets $V$, so the expected payoff to each hawk in a hawk-hawk encounter is $\frac{V - C}{2}$. If a hawk meets a dove, the hawk takes the full resource $V$. Two doves split the resource, each getting $\frac{V}{2}$.

When $C > V$ (fighting costs more than the resource is worth), neither pure hawk nor pure dove is an ESS. Instead, the ESS is a mixed strategy where individuals play hawk with probability $\frac{V}{C}$. This predicts that animals should escalate more often when resources are valuable relative to fighting costs, which matches observed behavior in many species.

ESS has also been applied to cooperation. In the iterated prisoner's dilemma, strategies like tit-for-tat (cooperate first, then copy your opponent's last move) can be evolutionarily stable under certain conditions. This framework helps explain reciprocal altruism, such as blood-sharing in vampire bats, where individuals share food with hungry roostmates who have shared with them in the past.

Human Social Interactions

ESS extends beyond biology into human social behavior:

• Social norms and conventions. Driving on the right (or left) side of the road is an ESS in a coordination game. Once a convention is established, no individual benefits from deviating.
• Property rights. The "bourgeois" strategy (defer to the current possessor) can be an ESS in territorial games, offering a game-theoretic foundation for how property norms emerge without central authority.
• Language. Linguistic conventions persist because switching to a different vocabulary or grammar unilaterally makes communication harder. ESS analysis helps explain why languages are stable yet can shift when a critical mass adopts new forms.
• Cultural traits. Religious practices, moral norms, and cooperative institutions can be analyzed as ESS when they provide fitness advantages (or at least no disadvantage) to individuals who follow them in a population of followers.

These applications show that "evolutionary" doesn't require biological evolution. Any system where more successful strategies spread (through imitation, learning, or cultural transmission) can be analyzed with ESS.

Strategy Stability in Games

Payoff Matrix Analysis

To check whether a strategy is an ESS, start with the payoff matrix of the symmetric two-player game. Here's the process:

1. Write out the payoff matrix. List all strategies and the payoff each earns against every other strategy (including itself).
2. Check the equilibrium condition. For a candidate strategy S, verify that $E(S, S) \geq E(T, S)$ for every alternative T. If S isn't a best response to itself, it's not an ESS.
3. Check for strict inequality. If $E(S, S) > E(T, S)$ for all T, you're done. S is an ESS.
4. Apply the stability condition for ties. For any T where $E(S, S) = E(T, S)$, check whether $E(S, T) > E(T, T)$. If this holds for every such T, S is an ESS. If it fails for any T, S is not an ESS.

In the hawk-dove game, you'd plug in the payoffs using $V$ and $C$, then test whether hawk, dove, or a mixed strategy satisfies both conditions.

Replicator Dynamics

For games with multiple strategies, replicator dynamics describe how strategy frequencies shift over time. The core idea: strategies that earn above-average fitness grow in frequency, while below-average strategies shrink.

The replicator equation for strategy $i$ is:

$\dot{x}_i = x_i \left[ f_i(\mathbf{x}) - \bar{f}(\mathbf{x}) \right]$

where $x_i$ is the frequency of strategy $i$, $f_i$ is its fitness, and $\bar{f}$ is the population's average fitness.

An ESS corresponds to an asymptotically stable rest point of the replicator dynamics. If the population starts near an ESS, the dynamics push it back toward that ESS. This connection between the static ESS concept and the dynamic replicator equation is one of the most important results in evolutionary game theory.

In games with multiple ESS, the population's initial conditions determine which ESS is reached. Each ESS has a basin of attraction: the set of starting frequencies from which the replicator dynamics converge to that ESS.

Factors Influencing ESS Stability

Several real-world factors can affect whether an ESS holds up in practice:

• Population structure. If individuals interact only with neighbors (on a spatial grid or social network) rather than mixing randomly, strategies that aren't ESS in well-mixed populations can sometimes persist, and vice versa.
• Mutation rate. Higher mutation rates introduce more mutants, which can destabilize an ESS that only barely resists invasion. Very high mutation rates can prevent any strategy from fully dominating.
• Selection intensity. This measures how strongly payoff differences translate into reproductive success. Under weak selection, drift and randomness play a larger role, and ESS predictions may not hold as cleanly. Under strong selection, the ESS framework applies more directly.
• Finite population size. The standard ESS definition assumes an infinite population. In finite populations, random drift can allow mutants to invade even against an ESS, especially in small groups.

These factors don't invalidate the ESS concept, but they define the boundaries of when the basic model's predictions will match what you actually observe.