🎲Statistical Mechanics Unit 2 Review

Entropy quantifies how many different microscopic arrangements are available to a system for a given macroscopic state. It connects the microscopic world of individual particles to the macroscopic quantities you measure in the lab, like temperature and pressure. Understanding entropy is essential for predicting which processes happen spontaneously and why certain transformations are irreversible.

This topic covers the statistical and thermodynamic definitions of entropy, how it behaves across different ensembles, its role in information theory and quantum mechanics, and its deep connection to the arrow of time.

Statistical vs thermodynamic entropy

These two formulations of entropy come from very different starting points but converge on the same physical quantity.

Statistical entropy starts from the microscopic picture. You count (or weight) the number of microstates available to a system and use that to define entropy. It's computed from probability distributions over microstates, and it tells you how "spread out" the system is across its possible configurations.

Thermodynamic entropy is defined through macroscopic measurements. For a reversible process, the entropy change is:

$\Delta S = \int \frac{\delta Q_{rev}}{T}$

This captures how much energy enters or leaves a system as heat, scaled by the temperature at which the transfer occurs.

The two are linked by the Boltzmann constant $k_B$ , which converts between the microscopic (dimensionless count of states) and macroscopic (joules per kelvin) descriptions. That single constant bridges statistical mechanics and classical thermodynamics.

Second law of thermodynamics

The second law states that the total entropy of an isolated system never decreases:

$\Delta S_{total} \geq 0$

For any spontaneous process, the combined entropy of the system plus its surroundings either stays the same (reversible process) or increases (irreversible process). Equality holds only in the idealized reversible limit.

This law imposes a fundamental ceiling on the efficiency of heat engines and refrigerators.
It explains why heat flows spontaneously from hot to cold, never the reverse.
It accounts for the irreversibility you observe in nature: mixed gases don't spontaneously unmix, and broken eggs don't reassemble.
Taken to its logical extreme, it predicts the "heat death" of the universe, a state of maximum entropy where no further work can be extracted.

Entropy as disorder

The common shorthand "entropy = disorder" is useful but can mislead you. More precisely, entropy measures the number of ways a system can be arranged while still looking the same macroscopically. Systems tend to evolve toward macrostates that correspond to the largest number of microstates, simply because those states are overwhelmingly more probable.

Everyday examples include the mixing of two gases (many more mixed configurations than separated ones) and the melting of ice (liquid water molecules can arrange themselves in far more ways than a crystal lattice allows).

However, "disorder" doesn't always match your visual intuition. Crystallization from a supersaturated solution decreases the order of the solute arrangement, but the heat released into the surroundings increases the surroundings' entropy enough to make the total entropy increase. Always check the total entropy, not just the system's.

Microscopic interpretation

Boltzmann's entropy formula

The cornerstone equation of statistical mechanics is:

$S = k_B \ln W$

$S$ is the entropy of the macrostate.
$k_B \approx 1.381 \times 10^{-23} \, \text{J/K}$ is Boltzmann's constant.
$W$ is the number of microstates consistent with the macrostate.

The logarithm is what makes entropy an extensive quantity: if you combine two independent systems with $W_1$ and $W_2$ microstates, the total number of microstates is $W_1 \times W_2$ , and $\ln(W_1 W_2) = \ln W_1 + \ln W_2$ , so entropies add. This formula also reveals the probabilistic nature of entropy: a macrostate with more microstates is exponentially more likely to be observed.

Entropy and microstates

A microstate is one specific arrangement of all the particles in a system (their positions, momenta, quantum numbers, etc.). A macrostate is defined by macroscopic variables like total energy, volume, and particle number. Many different microstates can correspond to the same macrostate.

The probability of observing a particular macrostate is proportional to the number of microstates it contains. Because the numbers involved are astronomically large (on the order of $10^{10^{23}}$ for a mole of gas), the most probable macrostate is so overwhelmingly dominant that deviations from it are essentially never observed. This is why systems evolve toward higher entropy: they're simply moving toward the macrostate with the most microstates.

Configuration entropy

Configuration entropy (also called combinatorial entropy) comes specifically from the number of spatial or structural arrangements available to particles. You calculate it using combinatorics.

For example, if you place $N$ distinguishable particles into $M$ boxes, the number of arrangements is $M^N$ , and the entropy scales with $N \ln M$ . For indistinguishable particles, you divide by $N!$ to avoid overcounting.

Configuration entropy increases with both the number of particles and the number of available states. It's the dominant contribution to the entropy of mixing: when two ideal gases mix, the increase in entropy is purely configurational, given by:

$\Delta S_{mix} = -N k_B \sum_i x_i \ln x_i$

where $x_i$ are the mole fractions. This same idea applies to alloys, polymer solutions, and other mixtures.

Entropy in statistical mechanics

Canonical ensemble

The canonical ensemble describes a system at fixed temperature $T$ , in thermal contact with a large heat bath. The probability of finding the system in microstate $i$ with energy $E_i$ is given by the Boltzmann distribution:

$P_i = \frac{e^{-E_i / k_B T}}{Z}$

The partition function $Z = \sum_i e^{-E_i / k_B T}$ normalizes these probabilities and serves as the generating function for thermodynamic quantities. Once you know $Z$ , you can extract the free energy, average energy, and entropy.

The entropy in the canonical ensemble is:

$S = -k_B \sum_i P_i \ln P_i$

This is the Gibbs entropy formula. It reduces to Boltzmann's formula $S = k_B \ln W$ in the special case where all accessible microstates are equally probable.

Microcanonical ensemble

The microcanonical ensemble describes a completely isolated system with fixed energy $E$ , volume $V$ , and particle number $N$ . The fundamental assumption is the equal a priori probability postulate: every microstate with energy $E$ is equally likely.

Entropy is then:

$S = k_B \ln \Omega(E)$

where $\Omega(E)$ is the number of microstates at energy $E$ . This is the most direct application of Boltzmann's formula. Temperature, pressure, and chemical potential are all derived quantities in this ensemble, obtained by taking appropriate derivatives of $S(E, V, N)$ . For instance, temperature is defined by:

$\frac{1}{T} = \frac{\partial S}{\partial E}\bigg|_{V,N}$

Grand canonical ensemble

The grand canonical ensemble models an open system that exchanges both energy and particles with a reservoir at temperature $T$ and chemical potential $\mu$ . The probability of a microstate now depends on both its energy and particle number.

The grand partition function $\mathcal{Z} = \sum_{N} \sum_i e^{-(E_i - \mu N)/k_B T}$ encodes all thermodynamic information.
Entropy includes contributions from fluctuations in both energy and particle number.
This ensemble is especially useful for studying phase transitions, adsorption, and chemical equilibria, where particle number isn't fixed.

Entropy and information theory

Shannon entropy

Claude Shannon defined a measure of uncertainty for discrete probability distributions:

$H = -\sum_i p_i \log_2 p_i$

This quantifies the average number of bits needed to encode an outcome drawn from the distribution. A fair coin has $H = 1$ bit; a loaded coin that always lands heads has $H = 0$ .

The mathematical form is identical to the Gibbs entropy formula (up to the choice of logarithm base and the constant $k_B$ ). This isn't a coincidence: both measure how "spread out" a probability distribution is. Shannon entropy is foundational in data compression, cryptography, and communication theory.

Statistical vs thermodynamic entropy, Entropy | Chemistry: Atoms First

Kullback-Leibler divergence

The KL divergence measures how one probability distribution $P$ differs from a reference distribution $Q$ :

$D_{KL}(P \| Q) = \sum_i P(i) \ln \frac{P(i)}{Q(i)}$

It's always non-negative and equals zero only when $P = Q$ . Note that it's not symmetric: $D_{KL}(P \| Q) \neq D_{KL}(Q \| P)$ , so it's not a true distance metric.

KL divergence quantifies the information lost when you approximate $P$ with $Q$ . It appears throughout machine learning (as a loss function for training models), coding theory (measuring coding inefficiency), and statistical mechanics (connecting to free energy differences).

Maximum entropy principle

When you have incomplete information about a system, the maximum entropy principle says you should choose the probability distribution that maximizes entropy subject to whatever constraints you do know (like a known mean or variance).

This avoids injecting assumptions you don't have evidence for. The resulting distributions are often familiar:

If you only know the mean and variance, you get the Gaussian distribution.
If you only know the mean of a positive quantity, you get the exponential distribution.
If you know nothing at all, you get the uniform distribution.

In statistical mechanics, this principle justifies the Boltzmann distribution: given a fixed average energy, the distribution that maximizes entropy is exactly $P_i \propto e^{-\beta E_i}$ .

Entropy in thermodynamic processes

Reversible vs irreversible processes

A reversible process proceeds through a continuous sequence of equilibrium states. At every step, the system is infinitesimally close to equilibrium, so the process can be reversed by an infinitesimal change in conditions. For a reversible process, the total entropy change of system plus surroundings is exactly zero.

An irreversible process involves departures from equilibrium (friction, turbulence, free expansion, heat flow across a finite temperature difference). These always produce entropy:

$\Delta S_{total} > 0$

All real processes are irreversible to some degree. Reversible processes are idealizations that set the upper bound on efficiency. The entropy produced during an irreversible process quantifies just how far from that ideal you are.

Entropy changes in phase transitions

First-order phase transitions (melting, boiling, sublimation) involve a discontinuous jump in entropy. The entropy change is directly related to the latent heat $L$ :

$\Delta S = \frac{L}{T}$

where $T$ is the transition temperature. For example, the entropy of vaporization of water at 100°C and 1 atm is $\Delta S = \frac{2260 \, \text{J/g} \times 18 \, \text{g/mol}}{373 \, \text{K}} \approx 109 \, \text{J/(mol·K)}$ .

Second-order phase transitions (ferromagnetic-to-paramagnetic, superfluid transition) have no latent heat and no entropy discontinuity. Instead, the entropy is continuous but its derivatives (like heat capacity) can diverge at the critical point. These transitions are characterized by critical exponents and scaling behavior.

Entropy production

In a general process, the entropy of a system changes due to two contributions:

$\frac{dS}{dt} = \frac{dS_e}{dt} + \frac{dS_i}{dt}$

$\frac{dS_e}{dt}$ is the entropy flow due to exchange of heat (or matter) with the surroundings. This can be positive, negative, or zero.
$\frac{dS_i}{dt}$ is the internal entropy production due to irreversible processes within the system. The second law requires $\frac{dS_i}{dt} \geq 0$ .

This decomposition is central to non-equilibrium thermodynamics. It lets you separately track what's coming from the environment versus what's being generated internally by dissipation.

Applications of entropy

Black hole entropy

One of the most striking results in theoretical physics is that black holes have entropy, and it's proportional to the area of the event horizon, not the volume:

$S_{BH} = \frac{k_B c^3 A}{4 G \hbar}$

This is the Bekenstein-Hawking formula. For a solar-mass black hole, this gives an entropy of roughly $10^{77} k_B$ , vastly exceeding the entropy of the star that collapsed to form it.

This area scaling challenges the intuition that entropy should be an extensive, volume-dependent quantity. It led to the holographic principle, which suggests that all the information contained in a volume of space can be encoded on its boundary. The AdS/CFT correspondence in string theory provides a concrete realization of this idea.

Entropy in biological systems

Living organisms are open systems that maintain low internal entropy by consuming free energy (food, sunlight) and exporting entropy to their surroundings as heat and waste. This doesn't violate the second law: the total entropy of organism plus environment still increases.

The rate of entropy production serves as a measure of metabolic activity. Rapidly metabolizing cells produce entropy faster than quiescent ones. At larger scales, entropy concepts help explain the emergence of complex, ordered structures (protein folding, cellular organization) as thermodynamically favorable when the entropy increase of the surroundings is accounted for.

Entropy in computational physics

Entropy plays a practical role in simulation methodology:

Monte Carlo methods use entropy-based importance sampling to efficiently explore configuration space. Techniques like Wang-Landau sampling directly estimate the density of states $\Omega(E)$ .
Molecular dynamics simulations extract entropy from particle trajectories using thermodynamic integration or the two-phase thermodynamic model.
Entropic forces (like depletion forces in colloidal systems) drive self-assembly in simulations and can be understood purely from entropy maximization.
Information-theoretic entropy measures help characterize quantum many-body states in tensor network and DMRG calculations.

Entropy and the arrow of time

Time-reversal symmetry

The fundamental equations of motion in classical mechanics, electromagnetism, and even quantum mechanics are symmetric under time reversal: if you reverse all velocities, the system retraces its trajectory. Yet macroscopic processes clearly have a preferred direction. Eggs break but don't unbreak. Gas expands to fill a room but doesn't spontaneously compress into a corner.

Entropy increase is what breaks this symmetry at the macroscopic level. The second law picks out a direction of time, connecting thermodynamics to our everyday experience of past and future.

Loschmidt's paradox

Loschmidt's paradox asks: if every microscopic collision is individually reversible, how can the macroscopic behavior be irreversible? If you could perfectly reverse every particle's velocity, wouldn't the system retrace its steps and decrease in entropy?

In principle, yes. But the resolution is statistical. The set of initial conditions that lead to entropy decrease is vanishingly small compared to those that lead to entropy increase. You'd need to specify particle velocities to absurd precision to achieve it. Furthermore, coarse-graining (the fact that we only track macroscopic variables, not every particle) means we inevitably lose information about correlations, and this loss of information manifests as entropy increase.

Statistical vs thermodynamic entropy, Statistical Interpretation of Entropy and the Second Law of Thermodynamics: The Underlying ...

Fluctuation theorem

The fluctuation theorem extends the second law to small systems and short timescales, where thermal fluctuations matter. It states:

$\frac{P(\Delta S = +\sigma)}{P(\Delta S = -\sigma)} = e^{\sigma / k_B}$

This means entropy-decreasing fluctuations can occur, but they become exponentially less probable as the magnitude of the decrease grows. For macroscopic systems (where $\sigma / k_B$ is enormous), the probability of observing a net entropy decrease is effectively zero, recovering the classical second law.

The fluctuation theorem provides a rigorous foundation for understanding non-equilibrium processes at the nanoscale, where systems are small enough that fluctuations are significant.

Entropy and quantum mechanics

von Neumann entropy

The quantum analog of the Gibbs/Shannon entropy is the von Neumann entropy:

$S = -\text{Tr}(\rho \ln \rho)$

where $\rho$ is the density matrix of the quantum state. For a pure state, $S = 0$ . For a maximally mixed state of dimension $d$ , $S = \ln d$ .

If the density matrix is diagonal in some basis, the von Neumann entropy reduces to the classical Shannon entropy of the diagonal elements. This formula is central to quantum information theory, where it quantifies the information content of quantum states and sets bounds on quantum communication and computation.

Entanglement entropy

When a quantum system is divided into subsystems A and B, the entanglement entropy of subsystem A is the von Neumann entropy of its reduced density matrix:

$S_A = -\text{Tr}(\rho_A \ln \rho_A)$

where $\rho_A = \text{Tr}_B(\rho_{AB})$ .

If the total state is a product state (no entanglement), $S_A = 0$ . If A and B are entangled, $S_A > 0$ . For many-body ground states, entanglement entropy often obeys an area law: it scales with the boundary area between A and B, not the volume. Violations of the area law (logarithmic scaling, for instance) signal critical behavior or topological order.

Entanglement entropy has become a primary tool for characterizing quantum phase transitions and classifying topological phases of matter.

Quantum statistical mechanics

Quantum statistical mechanics extends the classical framework by replacing phase space integrals with traces over Hilbert space and using density matrices instead of classical probability distributions.

Key quantum effects that modify entropy calculations include:

Indistinguishability: Bosons and fermions have fundamentally different state-counting rules, leading to Bose-Einstein and Fermi-Dirac statistics rather than Maxwell-Boltzmann.
Zero-point energy: Quantum systems have a minimum energy even at $T = 0$ , affecting the entropy at low temperatures.
The third law of thermodynamics: The entropy of a perfect crystal approaches zero as $T \to 0$ , consistent with the system settling into its unique quantum ground state ( $W = 1$ , so $S = k_B \ln 1 = 0$ ).

These quantum corrections are essential for explaining phenomena like Bose-Einstein condensation, superfluidity, and superconductivity.

Measuring and calculating entropy

Experimental techniques

Calorimetry is the most direct method. By measuring heat flow $\delta Q$ at known temperature $T$ , you integrate $\int \delta Q / T$ to get entropy changes. This works for chemical reactions, phase transitions, and heating/cooling processes.
Spectroscopy probes the energy level structure and degeneracies of a system, from which you can compute the partition function and then the entropy.
Magnetic susceptibility measurements reveal entropy changes associated with magnetic ordering transitions.
PVT data (pressure-volume-temperature measurements) for gases and fluids allow entropy calculation through Maxwell relations.
Electrochemical measurements determine entropy changes in redox reactions via the temperature dependence of cell voltage: $\Delta S = nF \frac{dE}{dT}$ .

Computational methods

Molecular dynamics simulations track particle trajectories and extract entropy through thermodynamic integration or velocity autocorrelation functions.
Monte Carlo methods estimate the density of states $\Omega(E)$ using techniques like Wang-Landau sampling or multicanonical methods.
Density functional theory (DFT) computes electronic contributions to entropy in materials from the electronic density of states.
Machine learning approaches are increasingly used to predict thermodynamic properties, including entropy, from structural descriptors.
Quantum Monte Carlo methods handle strongly correlated systems where mean-field approaches fail.

Approximation schemes

The harmonic approximation treats atomic vibrations as independent harmonic oscillators, giving a simple analytical expression for vibrational entropy.
The quasi-harmonic approximation improves on this by allowing vibrational frequencies to depend on volume, capturing thermal expansion effects.
Mean-field theories (like Weiss mean-field for magnets) approximate interacting systems by replacing interactions with an effective average field, yielding tractable entropy expressions.
Perturbation theory calculates entropy corrections for weakly non-ideal systems (e.g., virial expansions for real gases).
Renormalization group techniques handle the diverging fluctuations near critical points, where simpler approximations break down.

Entropy in non-equilibrium systems

Steady-state entropy production

A system driven out of equilibrium by external forces (a temperature gradient, a chemical potential difference, an applied voltage) can reach a steady state where macroscopic properties don't change in time, but entropy is continuously being produced internally and exported to the surroundings.

The steady-state entropy production rate quantifies how far the system is from equilibrium. It's related to the dissipation of free energy and the maintenance of gradients. Examples include heat conduction through a rod with fixed endpoint temperatures, and the steady metabolic activity of a living cell.

Fluctuation theorems

Fluctuation theorems are a family of exact results that generalize the second law to regimes where fluctuations matter:

The Jarzynski equality relates the work done during a non-equilibrium process to the equilibrium free energy difference:

$\langle e^{-\beta W} \rangle = e^{-\beta \Delta F}$

This holds regardless of how far from equilibrium the process is driven.

The Crooks fluctuation theorem relates the probability of observing a given work value in a forward process to the probability of observing the negative of that work in the reverse process.

Both results are remarkable because they let you extract equilibrium thermodynamic information from non-equilibrium measurements.

Non-equilibrium work relations

The Jarzynski equality deserves special attention because of its practical applications. It says that even if you drive a system rapidly and irreversibly, the exponential average of the work over many repetitions gives you the equilibrium free energy difference: