10.5 Jaynes' formulation of statistical mechanics

Foundations of Jaynes' Formulation

Edwin Jaynes proposed a radical reframing of statistical mechanics in the 1950s: instead of treating probability distributions as objective properties of physical systems, he argued they represent our state of knowledge about those systems. The central tool is the maximum entropy principle (MaxEnt), which says you should choose the probability distribution that maximizes Shannon entropy subject to whatever constraints you actually know. This gives you the least biased estimate possible, meaning you aren't smuggling in assumptions beyond your data.

Maximum Entropy Principle

Shannon entropy for a discrete probability distribution is:

$S = -\sum_i p_i \ln p_i$

MaxEnt says: among all distributions $\{p_i\}$ that satisfy your known constraints (like a fixed average energy), pick the one that maximizes $S$. Why? Because any distribution with lower entropy would implicitly assume information you don't have. Maximizing entropy is the mathematically principled way to be "maximally honest" about your ignorance.

This same principle shows up outside physics (image reconstruction, natural language processing), but its deepest application is in deriving the standard ensembles of statistical mechanics.

Information Theory Connection

Jaynes identified thermodynamic entropy with Shannon's information-theoretic entropy. In this view:

• Shannon entropy quantifies how much uncertainty a probability distribution contains
• Thermodynamic entropy (up to Boltzmann's constant $k_B$) is the Shannon entropy of the distribution over microstates
• The relationship is $S_{\text{thermo}} = k_B \, S_{\text{Shannon}}$

This identification isn't just an analogy. Jaynes argued it's the reason statistical mechanics works: thermodynamic entropy measures how much we don't know about the microstate, given the macroscopic constraints.

Probability vs. Entropy

These two concepts are related but distinct, and confusing them is a common mistake.

• A probability distribution $\{p_i\}$ assigns a likelihood to each microstate $i$
• Entropy is a single number that characterizes the shape of that distribution: how spread out or concentrated it is

A sharply peaked distribution (most probability on a few states) has low entropy. A flat distribution (equal probability across many states) has high entropy. MaxEnt selects the flattest distribution compatible with your constraints, which corresponds to the most probable macrostate.

Probability Distributions

Canonical Ensemble Derivation

The canonical ensemble follows directly from MaxEnt with a single constraint: the average energy $\langle E \rangle$ is fixed.

1. Write the Shannon entropy: $S = -\sum_i p_i \ln p_i$

2. Impose normalization: $\sum_i p_i = 1$

3. Impose the energy constraint: $\sum_i p_i E_i = \langle E \rangle$

4. Maximize $S$ using Lagrange multipliers $\alpha$ (for normalization) and $\beta$ (for energy)

5. Set $\frac{\partial}{\partial p_i}\left[-\sum_i p_i \ln p_i - \alpha \sum_i p_i - \beta \sum_i p_i E_i\right] = 0$

6. Solve to get: $p_i = \frac{1}{Z} e^{-\beta E_i}$

The partition function is $Z = \sum_i e^{-\beta E_i}$, and the Lagrange multiplier $\beta$ turns out to be $1/(k_B T)$. Temperature isn't assumed here; it emerges from the constraint structure.

Microcanonical Ensemble Revisited

The microcanonical ensemble is the simplest MaxEnt case. The only constraint is that the system has a definite energy $E$, so only microstates with that energy are accessible.

Maximizing entropy with no other constraints yields a uniform distribution over all accessible microstates:

$p_i = \frac{1}{\Omega(E)}$

where $\Omega(E)$ is the number of microstates at energy $E$. This is the equal a priori probability postulate, which traditional statistical mechanics takes as an axiom. In Jaynes' formulation, it's a derived result of MaxEnt, not an assumption.

Grand Canonical Ensemble Extension

When the system can exchange both energy and particles with a reservoir, you add a second constraint: the average particle number $\langle N \rangle$ is fixed.

Following the same Lagrange multiplier procedure with constraints on both $\langle E \rangle$ and $\langle N \rangle$:

$p_i = \frac{1}{\mathcal{Z}} e^{-\beta(E_i - \mu N_i)}$

Here $\mu$ is the chemical potential, which emerges as the Lagrange multiplier conjugate to the particle number constraint, and $\mathcal{Z}$ is the grand partition function. The pattern is clear: each conserved quantity you constrain introduces a corresponding Lagrange multiplier, which maps onto a familiar thermodynamic variable.

Statistical Inference

Bayesian Approach

Jaynes was a committed Bayesian, and his formulation naturally incorporates Bayesian reasoning. In this framework:

• Prior probabilities encode what you know about the system before making measurements
• Likelihood functions describe how probable your observed data is, given a particular microstate or parameter
• Posterior probabilities are the updated distributions after combining prior knowledge with new data, via Bayes' theorem

This means statistical mechanics becomes a form of inference: you're reasoning about what the microstate probably is, given your macroscopic information.

Prior Information Incorporation

One strength of Jaynes' approach is that it formalizes how to include background knowledge. Known physical laws (symmetries, conservation laws, boundary conditions) enter as constraints on the MaxEnt optimization. Different choices of prior information lead to different resulting distributions.

For example, if you know a system conserves both energy and angular momentum, you include both as constraints and get a distribution that reflects exactly that knowledge. If you later learn the system also has a fixed magnetization, you add that constraint and get a sharper (lower-entropy) distribution.

Posterior Probability Distributions

As new measurements come in, you update your probability distribution. The posterior distribution after incorporating data $D$ is:

$P(\text{microstate} \mid D) \propto P(D \mid \text{microstate}) \cdot P(\text{microstate})$

This enables continuous refinement: your statistical mechanical model improves as you gather more information. The MaxEnt distribution serves as the optimal prior when you have only constraint-level information (averages), and Bayesian updating handles the incorporation of specific measurement outcomes.

Constraints in Jaynes' Formulation

Each macroscopic constraint you impose corresponds to a Lagrange multiplier, and each multiplier maps onto a thermodynamic intensive variable. This is one of the most elegant features of the formulation.

Energy Conservation Constraint

The most common constraint is fixed average energy: $\sum_i p_i E_i = \langle E \rangle$. This yields:

• Lagrange multiplier: $\beta = 1/(k_B T)$
• Resulting distribution: canonical (Boltzmann) distribution
• Physical meaning: the system is in thermal contact with a heat bath at temperature $T$

Particle Number Constraint

For open systems, you fix the average particle number: $\sum_i p_i N_i = \langle N \rangle$. This introduces:

• Lagrange multiplier: $-\beta \mu$, where $\mu$ is the chemical potential
• Resulting distribution: grand canonical distribution
• Physical meaning: the system exchanges particles with a reservoir at chemical potential $\mu$

Volume Constraint

When volume can fluctuate (as in an isobaric ensemble), constraining the average volume $\langle V \rangle$ introduces:

• Lagrange multiplier: $\beta P$, where $P$ is the pressure
• Resulting distribution: isothermal-isobaric ensemble
• Physical meaning: the system is in mechanical contact with a pressure reservoir

Applications of Jaynes' Method

Equilibrium Thermodynamics

Jaynes' formulation reproduces all standard results of equilibrium statistical mechanics: the ideal gas law, heat capacities, equations of state, and phase equilibrium conditions. The difference is that these results now follow from a single principle (MaxEnt) rather than from separate postulates for each ensemble.

This unification is more than aesthetic. It clarifies why the ensembles are equivalent in the thermodynamic limit and provides a systematic recipe for constructing new ensembles when you have unusual combinations of constraints.

Non-Equilibrium Systems

Extending MaxEnt to non-equilibrium situations is an active area of research. The basic idea is to maximize entropy subject to time-dependent constraints (like a known current or flux). This can describe:

• Relaxation processes toward equilibrium
• Transport phenomena (diffusion, heat conduction)
• Steady-state systems driven by external forces

The results are less universal than in equilibrium, and the choice of constraints becomes more subtle, but the framework provides a principled starting point.

Quantum Statistical Mechanics

Jaynes' method adapts to quantum systems by replacing Shannon entropy with von Neumann entropy:

$S = -\text{Tr}(\hat{\rho} \ln \hat{\rho})$

where $\hat{\rho}$ is the density matrix. Maximizing this subject to constraints like $\text{Tr}(\hat{\rho} \hat{H}) = \langle E \rangle$ yields the quantum canonical ensemble $\hat{\rho} = e^{-\beta \hat{H}} / Z$. The formalism also connects naturally to quantum entanglement and decoherence, since the von Neumann entropy measures the mixedness of quantum states.

Advantages over Traditional Approaches

Generality of Formulation

Traditional derivations of statistical ensembles rely on specific physical arguments (e.g., a system weakly coupled to a large heat bath). Jaynes' approach is more general: you only need to specify what you know, and MaxEnt gives you the distribution. This works for classical systems, quantum systems, small systems, and situations where the traditional "large bath" argument doesn't cleanly apply.

Handling Incomplete Information

Real experiments rarely give you complete knowledge of a system. Jaynes' formulation explicitly handles this: you maximize entropy given whatever information you have. If you know only the average energy, you get the canonical ensemble. If you also know the average magnetization, you get a more constrained distribution. The framework tells you exactly how to proceed with partial information.

Consistency with Thermodynamics

The laws of thermodynamics emerge naturally from MaxEnt:

• The second law follows because entropy is maximized by construction
• Temperature, pressure, and chemical potential appear as Lagrange multipliers
• The Gibbs paradox (the apparent entropy of mixing for identical particles) is resolved because the MaxEnt distribution correctly accounts for indistinguishability when that information is included as a constraint

Criticisms and Limitations

Subjectivity Concerns

The most persistent criticism is that Jaynes makes entropy depend on the observer's knowledge, which seems to inject subjectivity into physics. Critics argue that thermodynamic entropy should be an objective property of a system, not a measure of someone's ignorance. Jaynes' response was that the "subjectivity" is a feature, not a bug: different observers with different information should assign different entropies, and the formalism handles this consistently. The debate remains unresolved and touches on deep questions about the interpretation of probability.

Ergodicity Assumptions

Traditional statistical mechanics often invokes ergodicity (the assumption that time averages equal ensemble averages) to justify the use of ensembles. Jaynes' formulation sidesteps this by treating ensembles as inference tools rather than physical claims about time evolution. However, this raises its own questions: if a system is non-ergodic (like a glass or a system with broken ergodicity), does the MaxEnt distribution still give physically correct predictions? In many non-ergodic cases, additional constraints beyond the standard ones are needed.

Computational Challenges

For systems with many constraints or complex state spaces, the MaxEnt optimization can be computationally demanding. Finding the Lagrange multipliers requires solving a system of nonlinear equations, which may not have closed-form solutions. Numerical methods and approximation techniques (such as iterative scaling or variational approaches) are often necessary for practical applications.

Extensions and Modern Developments

Maximum Caliber Principle

Maximum caliber extends MaxEnt from static distributions to trajectories. Instead of maximizing the entropy of a probability distribution over states, you maximize the "caliber" (path entropy) of a distribution over dynamical trajectories, subject to constraints on time-averaged quantities like fluxes or currents.

This provides a variational principle for non-equilibrium dynamics: the most probable trajectory of a system is the one that maximizes caliber. It connects to fluctuation theorems and provides a unified framework for deriving transport equations.

Non-Equilibrium Steady States

Applying Jaynes' ideas to steady-state systems maintained away from equilibrium is a major area of current research. Key questions include:

• What constraints characterize a non-equilibrium steady state?
• How does entropy production relate to the MaxEnt distribution?
• Can information-theoretic principles predict the stability and fluctuations of driven systems?

These questions don't yet have complete answers, but the Jaynes framework provides a natural language for asking them.

Quantum Information Theory

The intersection of quantum information and statistical mechanics has become one of the most active areas in theoretical physics. Jaynes' ideas connect to:

• Quantum MaxEnt: using von Neumann entropy to derive quantum thermal states
• Entanglement entropy: understanding how subsystem entropy relates to quantum correlations
• Quantum thermodynamics: defining work, heat, and efficiency for quantum systems using information-theoretic tools

These developments are reshaping our understanding of thermalization in closed quantum systems and the emergence of thermal behavior from unitary quantum dynamics.