10.4 Information-theoretic interpretation of thermodynamics

Foundations of information theory

Information theory gives us a precise mathematical language for talking about uncertainty, and that turns out to be exactly what statistical mechanics needs. The core idea is that thermodynamic entropy is a measure of missing information about a system's microstate. Once you see that connection, the laws of thermodynamics start to look like statements about how information behaves.

This section covers the key information-theoretic quantities you'll need before connecting them to physics.

Shannon entropy

Shannon entropy measures the average uncertainty associated with a random variable. If you have a set of outcomes $x_i$ with probabilities $p(x_i)$, Shannon entropy is:

$H(X) = -\sum_{i} p(x_i) \log p(x_i)$

The higher $H(X)$ is, the more "surprised" you'd be on average by a given outcome. A fair coin has higher Shannon entropy than a biased one because the outcome is harder to predict.

• The logarithm base matters: base 2 gives bits, base $e$ gives nats. In physics, we typically use the natural log.
• $H(X)$ is maximized when all outcomes are equally likely (uniform distribution).
• $H(X) = 0$ only when one outcome has probability 1, meaning there's no uncertainty at all.

Shannon entropy is the information-theoretic ancestor of Gibbs entropy, and recognizing their structural similarity is the whole point of this unit.

Kullback-Leibler divergence

The Kullback-Leibler (KL) divergence measures how much one probability distribution $P$ differs from a reference distribution $Q$:

$D_{KL}(P \| Q) = \sum_{i} P(i) \log \frac{P(i)}{Q(i)}$

Think of it as the "information cost" of using $Q$ to approximate $P$. A few properties to keep straight:

• $D_{KL} \geq 0$ always, with equality only when $P = Q$ (Gibbs' inequality).
• It is not symmetric: $D_{KL}(P \| Q) \neq D_{KL}(Q \| P)$ in general, so it's not a true distance metric.
• If $Q(i) = 0$ for some state where $P(i) \neq 0$, the divergence is infinite.

In statistical mechanics, KL divergence shows up when you want to quantify how far a system is from equilibrium. The non-equilibrium distribution $P$ compared against the equilibrium distribution $Q$ gives a measure of the "extra" entropy production or free energy difference.

Mutual information

Mutual information captures how much knowing one variable tells you about another:

$I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x)p(y)}$

• If $X$ and $Y$ are independent, $p(x,y) = p(x)p(y)$, so $I(X;Y) = 0$.
• $I(X;Y)$ is symmetric and always non-negative.
• It can be rewritten as $I(X;Y) = H(X) + H(Y) - H(X,Y)$, which shows it measures the "overlap" in uncertainty between the two variables.

In thermodynamic contexts, mutual information quantifies correlations between subsystems. Near a phase transition, for example, distant parts of a system become strongly correlated, and mutual information captures this even when simple two-point correlation functions might miss nonlinear dependencies.

Thermodynamic entropy

The central claim of this unit is that thermodynamic entropy and information-theoretic entropy are not just analogous; they're the same quantity (up to a constant). This section traces that connection through the major entropy formulas of statistical mechanics.

Boltzmann's entropy formula

For an isolated system where all $W$ accessible microstates are equally probable:

$S = k_B \ln W$

Here $k_B$ is Boltzmann's constant ($1.381 \times 10^{-23}$ J/K) and $W$ is the number of microstates consistent with the macroscopic constraints (energy, volume, particle number).

This formula applies specifically to the microcanonical ensemble, where every microstate has probability $1/W$. The logarithm ensures entropy is additive: if you combine two independent systems with $W_1$ and $W_2$ microstates, the total entropy is $S = k_B \ln(W_1 W_2) = k_B \ln W_1 + k_B \ln W_2$.

Gibbs entropy

Gibbs entropy generalizes Boltzmann's formula to situations where microstates are not equally probable:

$S = -k_B \sum_i p_i \ln p_i$

Compare this directly to Shannon entropy: $S = k_B \cdot H(X)$ when using the natural log. The only difference is the dimensional prefactor $k_B$, which converts from nats to J/K.

• Gibbs entropy reduces to Boltzmann's formula when all $p_i = 1/W$.
• It applies to any ensemble (microcanonical, canonical, grand canonical) and to non-equilibrium distributions.
• It's the formula you should default to when thinking about the information-theoretic meaning of thermodynamic entropy.

Entropy vs. information

The relationship between entropy and information is essentially inverse: high entropy means low information about the system's microstate, and vice versa.

If you know exactly which microstate a system occupies ($p_i = 1$ for one state), the entropy is zero and your information is maximal. If the system could be in any of a huge number of microstates with equal probability, entropy is large and you know very little.

Maxwell's demon illustrates this beautifully. The demon appears to violate the second law by sorting fast and slow molecules, decreasing entropy. The resolution (due to Bennett, building on Landauer) is that the demon must store information about each molecule, and erasing that information later dissipates at least $k_B T \ln 2$ of heat per bit. The total entropy of system + demon never decreases.

Negentropy (negative entropy) is sometimes used to describe the information content or "orderliness" a system possesses. Schrödinger popularized this idea in What is Life?, arguing that living organisms maintain low entropy by exporting entropy to their environment.

Statistical mechanics and information

The standard ensembles of statistical mechanics can each be derived from the maximum entropy principle by choosing different constraints. This reframes ensemble theory as an exercise in inference rather than a set of physical postulates.

Microcanonical ensemble

The microcanonical ensemble describes an isolated system with fixed energy $E$, volume $V$, and particle number $N$.

• All microstates with energy $E$ are equally probable: $p_i = 1/\Omega(E)$.
• Entropy is $S = k_B \ln \Omega(E)$.
• Temperature emerges from the entropy: $\frac{1}{T} = \frac{\partial S}{\partial E}\bigg|_{V,N}$.

From an information-theoretic perspective, the uniform distribution over microstates is the maximum entropy distribution when the only constraint is a fixed total energy. You're assuming as little as possible beyond what you know.

Canonical ensemble

The canonical ensemble describes a system in thermal contact with a heat bath at temperature $T$. Energy fluctuates, but the average energy $\langle E \rangle$ is fixed as a constraint.

The probability of microstate $i$ follows the Boltzmann distribution:

$p_i = \frac{1}{Z} e^{-\beta E_i}$

where $\beta = 1/(k_B T)$ and the partition function is:

$Z = \sum_i e^{-\beta E_i}$

This distribution is exactly what you get by maximizing Gibbs entropy subject to a fixed average energy. The Lagrange multiplier enforcing that constraint turns out to be $\beta$, which is why temperature has a natural information-theoretic meaning: it controls how sharply the distribution is peaked around low-energy states.

Grand canonical ensemble

The grand canonical ensemble allows exchange of both energy and particles with a reservoir at temperature $T$ and chemical potential $\mu$.

$p_i = \frac{1}{\Xi} e^{-\beta(E_i - \mu N_i)}$

The grand partition function is $\Xi = \sum_i e^{-\beta(E_i - \mu N_i)}$.

Again, this is the maximum entropy distribution when you constrain both $\langle E \rangle$ and $\langle N \rangle$. The chemical potential $\mu$ appears as the Lagrange multiplier for the particle number constraint, just as $\beta$ appears for the energy constraint.

Applications include open systems like gases in contact with a particle reservoir, adsorption on surfaces, and systems near phase transitions where particle number fluctuations become important.

Information-theoretic approach to thermodynamics

This is the conceptual heart of the unit. Rather than treating statistical mechanics as a physical theory that happens to use probability, Jaynes argued it's fundamentally a theory of inference about physical systems.

Maximum entropy principle

The maximum entropy principle (MaxEnt) states: given a set of constraints (like fixed average energy), the least biased probability distribution is the one that maximizes entropy subject to those constraints.

The procedure works as follows:

1. Identify your constraints (e.g., $\langle E \rangle = U$, normalization $\sum_i p_i = 1$).
2. Write the entropy functional $S = -k_B \sum_i p_i \ln p_i$.
3. Introduce a Lagrange multiplier for each constraint.
4. Maximize $S$ with respect to each $p_i$.
5. Solve for the $p_i$ and identify the Lagrange multipliers with physical quantities.

For the canonical ensemble, this procedure yields the Boltzmann distribution, with $\beta$ as the multiplier for the energy constraint. For quantum systems, the same approach with the constraint of fixed average particle number and the correct quantum statistics gives the Fermi-Dirac distribution (fermions) or Bose-Einstein distribution (bosons).

Jaynes' interpretation

E. T. Jaynes proposed in the 1950s that the probability distributions of statistical mechanics are not statements about physical randomness but about our state of knowledge. Entropy measures how much we don't know about the microstate.

Key consequences of this view:

• The ensembles aren't different physical situations; they reflect different information available to the observer.
• The second law becomes a statement about inference: as systems evolve and we lose track of microscopic details, our uncertainty (entropy) increases.
• This framework extends naturally to non-equilibrium situations. You can apply MaxEnt whenever you have partial information, not just at equilibrium.

Jaynes' interpretation doesn't change any predictions of statistical mechanics. It changes the meaning of those predictions, grounding them in probability theory and information rather than in assumptions about ergodicity or equal a priori probabilities.

Thermodynamic potentials

Thermodynamic potentials (internal energy $U$, Helmholtz free energy $F$, Gibbs free energy $G$, enthalpy $H$) each correspond to a different set of controlled variables, and information theory clarifies why.

• Helmholtz free energy: $F = U - TS = -k_B T \ln Z$. This is the quantity minimized at constant $T$ and $V$. The $-TS$ term represents the "information cost" of thermal fluctuations.
• Gibbs free energy: $G = F + PV$, minimized at constant $T$ and $P$. The additional $PV$ term accounts for the constraint of fixed pressure rather than fixed volume.

The Maxwell relations (e.g., $\left(\frac{\partial S}{\partial V}\right)_T = \left(\frac{\partial P}{\partial T}\right)_V$) follow from the mathematical structure of these potentials and can be understood as consistency conditions on the information content of the system.

Different potentials correspond to different Legendre transforms, which in information-theoretic language means different choices of which variables you treat as constraints versus which you allow to fluctuate.

Connections to statistical physics

Partition function

The partition function is the single most important object in equilibrium statistical mechanics. For a discrete system at temperature $T$:

$Z = \sum_i e^{-\beta E_i}$

For continuous degrees of freedom:

$Z = \int e^{-\beta E(x)} \, dx$

Every equilibrium thermodynamic quantity can be extracted from $Z$ or its derivatives:

• Free energy: $F = -k_B T \ln Z$
• Average energy: $\langle E \rangle = -\frac{\partial \ln Z}{\partial \beta}$
• Entropy: $S = k_B \left( \ln Z + \beta \langle E \rangle \right)$
• Heat capacity: $C_V = \frac{\partial \langle E \rangle}{\partial T}$

From an information-theoretic standpoint, $\ln Z$ is the cumulant generating function of the energy distribution. It encodes all the statistical information about energy fluctuations.

Free energy

The Helmholtz free energy $F = -k_B T \ln Z$ has a clean information-theoretic interpretation. For any distribution $P$ over microstates (not necessarily the equilibrium Boltzmann distribution $P^*$):

$F[P] = \langle E \rangle_P - T S[P]$

The equilibrium distribution $P^*$ minimizes $F[P]$. The difference between $F[P]$ and $F[P^*]$ is directly proportional to the KL divergence:

$F[P] - F[P^*] = k_B T \, D_{KL}(P \| P^*)$

This is a powerful result. It means the KL divergence between a non-equilibrium distribution and the Boltzmann distribution measures the excess free energy, which is exactly the maximum work you could extract by relaxing the system to equilibrium.

Fluctuations and correlations

Thermodynamic quantities aren't perfectly sharp; they fluctuate. Information theory provides tools to characterize these fluctuations.

• The fluctuation-dissipation theorem connects the variance of energy fluctuations to the heat capacity: $\langle (\Delta E)^2 \rangle = k_B T^2 C_V$.
• Mutual information between subsystems captures all correlations (linear and nonlinear), unlike standard correlation functions which only capture linear relationships.
• Near critical points, correlations extend over long distances and mutual information between distant regions diverges, reflecting the system's scale invariance.

Transfer entropy (a directed version of mutual information) can identify causal relationships in time-series data from simulations, distinguishing which subsystem is driving which.

Applications in thermodynamics

Second law of thermodynamics

The second law states that the total entropy of an isolated system never decreases. Information theory reframes this: irreversible processes destroy information about the system's microstate, and that lost information shows up as increased entropy.

Landauer's principle makes this quantitative. Erasing one bit of information requires dissipating at least:

$Q \geq k_B T \ln 2$

of heat into the environment. This has been experimentally verified (Bérut et al., 2012, measured the heat dissipated when erasing a single bit in a colloidal particle system).

The second law, in this view, is not a dynamical law but a consequence of the fact that macroscopic descriptions inevitably lose microscopic information. Entropy increases because we can't track every degree of freedom.

Irreversibility and information loss

Why do macroscopic processes appear irreversible when the underlying microscopic laws are time-reversible? This is Loschmidt's paradox, and information theory offers a clear answer.

• Coarse-graining is the key mechanism. When you describe a system using macroscopic variables (temperature, pressure), you're grouping many microstates into one macrostate. This grouping discards information.
• The KL divergence $D_{KL}(P(t) \| P_{eq})$ decreases monotonically as the system relaxes toward equilibrium, quantifying the irreversible loss of information.
• Microscopic reversibility is never violated. If you could track every particle, you'd see no information loss. Irreversibility is a feature of our description, not of the dynamics themselves.

This perspective connects to the H-theorem: Boltzmann's $H$-function decreases over time precisely because the Boltzmann equation already contains a coarse-graining assumption (molecular chaos).

Heat engines and efficiency

The Carnot efficiency $\eta = 1 - T_C/T_H$ sets the maximum efficiency for any heat engine operating between temperatures $T_H$ and $T_C$. Information theory illuminates why this limit exists.

Maxwell's demon and the Szilard engine:

1. A Szilard engine is a single-molecule heat engine. A demon measures which half of a box the molecule occupies (gaining 1 bit of information).
2. The demon uses this information to extract $k_B T \ln 2$ of work via isothermal expansion.
3. To complete the cycle, the demon must erase its memory, which by Landauer's principle costs at least $k_B T \ln 2$ of heat.
4. Net work extracted over a full cycle: zero. The second law is preserved.

More generally, any heat engine that uses feedback control (measuring the system and adjusting accordingly) can extract additional work proportional to the mutual information gained, but the cost of processing that information always balances the books.

Information in quantum systems

Von Neumann entropy

The von Neumann entropy extends Shannon/Gibbs entropy to quantum mechanics. For a system described by density matrix $\rho$:

$S(\rho) = -\text{Tr}(\rho \ln \rho)$

• For a pure state ($\rho^2 = \rho$), $S = 0$. Pure states carry no uncertainty.
• For a maximally mixed state ($\rho = I/d$ in $d$ dimensions), $S = k_B \ln d$, the maximum possible value.
• Von Neumann entropy is the quantum analog of Gibbs entropy and reduces to it when $\rho$ is diagonal in the energy eigenbasis.

A particularly important application: the entanglement entropy of a subsystem. If you have a composite system $AB$ in a pure state, the von Neumann entropy of the reduced density matrix $\rho_A = \text{Tr}_B(\rho_{AB})$ measures the entanglement between $A$ and $B$.

Quantum entanglement

Entanglement is a uniquely quantum form of correlation with no classical analog. Two entangled subsystems have $I(A;B) > 0$ even though neither subsystem individually carries information about the other's state in a classical sense.

• Entanglement entropy (von Neumann entropy of a subsystem) is the standard measure for pure bipartite states.
• In many-body physics, the scaling of entanglement entropy with subsystem size distinguishes different phases of matter. Area-law scaling ($S \propto L^{d-1}$) is typical for gapped ground states; volume-law scaling ($S \propto L^d$) appears in thermal states and after quantum quenches.
• Entanglement plays a role in thermalization of closed quantum systems: subsystems appear thermal because they're entangled with the rest of the system, even though the global state evolves unitarily.

Quantum thermodynamics

Quantum thermodynamics applies thermodynamic reasoning to systems where quantum effects are significant.

• Quantum heat engines (e.g., the quantum Otto cycle) use quantum working substances like two-level systems or harmonic oscillators. Their efficiency is still bounded by Carnot, but quantum coherence can affect power output.
• Quantum fluctuation theorems (Jarzynski equality, Crooks relation) extend to the quantum regime, with work defined through two-point energy measurements.
• Measurement and decoherence play thermodynamic roles: quantum measurement collapses superpositions, which is an irreversible process that increases entropy. Decoherence transfers quantum information from the system into correlations with the environment.

Computational aspects

Monte Carlo methods

Monte Carlo methods use random sampling to estimate thermodynamic properties of systems too complex for analytical treatment.

The Metropolis algorithm is the workhorse:

1. Start from a configuration with energy $E_1$.

2. Propose a random change to get a new configuration with energy $E_2$.

3. If $E_2 < E_1$, accept the move.

4. If $E_2 > E_1$, accept with probability $e^{-\beta(E_2 - E_1)}$.

5. Repeat. After many steps, the sampled configurations follow the Boltzmann distribution.

Information theory helps optimize these simulations. For example, you can use entropy estimates to assess whether the sampler has adequately explored the configuration space, or use KL divergence to compare the sampled distribution against a known reference.

Applications include Ising models, lattice gauge theories, and protein folding energy landscapes.

Molecular dynamics simulations

Molecular dynamics (MD) integrates Newton's equations for a system of interacting particles, generating trajectories that reveal both equilibrium and dynamical properties.

• Thermostats (Nosé-Hoover, Langevin) couple the system to a heat bath to sample the canonical ensemble.
• Transfer entropy applied to MD trajectories can reveal causal information flow between different parts of a molecule or between solute and solvent.
• Mutual information between atomic positions at different times quantifies how predictable the dynamics are, which connects to concepts like Lyapunov exponents and chaos.

Information-based algorithms

Several computational methods draw directly on information-theoretic principles:

• Maximum entropy methods reconstruct probability distributions from limited data (e.g., inferring the structure of a protein from sparse experimental constraints).
• Relative entropy minimization calibrates force field parameters by minimizing the KL divergence between simulated and experimental observables.
• Information geometry treats the space of probability distributions as a Riemannian manifold, where the Fisher information metric defines distances. This geometric perspective can improve optimization algorithms for high-dimensional parameter spaces.

Interdisciplinary connections

Information in biology

Biological systems process and store information at every scale, and thermodynamics constrains how they do it.

• DNA stores genetic information at roughly 2 bits per base pair. The thermodynamic cost of copying and error-correcting this information connects to Landauer's principle.
• Neural networks (biological ones) process sensory information with remarkable energy efficiency. The brain dissipates about 20 W while performing computations that would require orders of magnitude more power on conventional hardware.
• Molecular motors (kinesin, ATP synthase) operate near the thermodynamic limits set by information theory, converting chemical free energy to mechanical work with efficiencies approaching the Landauer bound per step.

Economics and information theory

Thermodynamic and information-theoretic analogies appear throughout economics:

• Market efficiency can be analyzed through the lens of information: in an efficient market, prices already reflect all available information, analogous to a system at maximum entropy.
• Maximum entropy methods are used in finance to construct least-biased probability distributions for asset returns given known constraints (mean, variance).
• Entropy-based measures like the Theil index quantify economic inequality in a way that has natural information-theoretic meaning: it measures the "surprise" associated with the distribution of income relative to a uniform distribution.

Complex systems analysis

Information theory provides some of the most general tools for studying emergent behavior:

• Transfer entropy identifies directed information flow in complex networks, distinguishing correlation from causation in time-series data.
• Near critical points in networks (social, biological, technological), information-theoretic measures like mutual information diverge, signaling the onset of long-range order.
• The relationship between complexity and entropy is subtle: maximum entropy (random) systems and minimum entropy (perfectly ordered) systems are both simple. Complexity peaks at intermediate entropy, which connects to ideas about the "edge of chaos" in complex systems theory.