14.1 Sampling and quantization principles

Sampling Principles

Sampling and quantization are the two core steps in analog-to-digital conversion. Sampling captures the signal at discrete moments in time, and quantization maps each captured value to a finite set of numerical levels. Together, they let digital systems store, process, and transmit real-world analog signals.

Getting these steps right determines whether your digital representation is faithful to the original or hopelessly distorted. The Nyquist-Shannon theorem tells you how fast to sample, and bit depth tells you how finely to quantize. This section covers both, along with the errors and trade-offs that come with each.

Nyquist-Shannon Sampling Theorem and Sampling Rate

The Nyquist-Shannon sampling theorem states that a continuous-time signal can be perfectly reconstructed from its samples if and only if the sampling rate is greater than twice the highest frequency component in the signal.

That critical threshold is called the Nyquist rate:

$f_s > 2 f_{max}$

where $f_s$ is the sampling frequency and $f_{max}$ is the highest frequency present in the signal.

• Sampling rate is the number of samples taken per second, measured in Hertz (Hz).
• Higher sampling rates capture more detail about the original waveform, improving reconstruction quality.
• Common examples: 44.1 kHz for audio CDs (designed to cover the ~20 kHz range of human hearing), 48 kHz for digital video audio tracks.

The theorem assumes the signal is band-limited, meaning it contains no energy above $f_{max}$. In practice, real signals are never perfectly band-limited, which is why anti-aliasing filters matter (see below).

Aliasing and Oversampling

Aliasing happens when you sample below the Nyquist rate. Frequency components above $f_s / 2$ don't just disappear; they fold back into the spectrum and masquerade as lower-frequency components that weren't in the original signal. This distortion is irreversible once the signal has been sampled.

To prevent aliasing:

1. Apply an anti-aliasing filter (a low-pass filter) before sampling. This removes any frequency content above the Nyquist frequency $f_s / 2$.
2. Choose a sampling rate that comfortably exceeds $2 f_{max}$ for the signal of interest.

Oversampling means sampling at a rate significantly higher than the Nyquist rate. This provides several practical benefits:

• Spreads quantization noise across a wider frequency band, reducing the noise power density in the band of interest.
• Relaxes the design requirements for the anti-aliasing filter. A gentle roll-off filter works when you have a large gap between $f_{max}$ and $f_s / 2$, whereas sampling right at the Nyquist rate demands a very steep (and expensive) filter.
• Commonly used in high-fidelity audio systems and precision measurement instruments.

Quantization Fundamentals

Quantization Process and Error

Once you have discrete-time samples, quantization maps each sample's continuous amplitude to the nearest value in a finite set of levels. This is where the signal goes from analog-valued to truly digital.

The mapping introduces quantization error, defined as the difference between the actual sample value and the quantized value:

$e_q = x(nT) - x_q(nT)$

For a uniform quantizer with step size $\Delta$, this error is bounded:

$-\frac{\Delta}{2} \leq e_q \leq \frac{\Delta}{2}$

• Quantization error shows up as quantization noise in the reconstructed signal.
• Larger step sizes (fewer levels) mean more noise; smaller step sizes (more levels) mean less noise.
• Dithering is a technique where a small amount of random noise is intentionally added before quantization. This decorrelates the quantization error from the signal, converting harsh distortion artifacts into a low-level, less perceptible broadband noise floor.

Resolution and Bit Depth

Bit depth ($n$ bits) determines how many quantization levels are available:

$L = 2^n$

where $L$ is the number of discrete levels.

• 16-bit quantization gives $2^{16} = 65{,}536$ levels (standard for audio CDs).
• 24-bit quantization gives $2^{24} = 16{,}777{,}216$ levels (professional audio and studio recording).

Higher bit depth means each step $\Delta$ is smaller, so the signal is represented more precisely. The trade-off is straightforward: more bits per sample means larger file sizes and higher data throughput requirements.

Resolution and bit depth are closely related but not identical. Resolution refers to the smallest amplitude change the system can distinguish, while bit depth is the number of binary digits used to encode each sample. In an ideal converter, they correspond directly; in real hardware, noise and nonlinearities reduce the effective resolution below what the bit depth alone would suggest.

Signal Quality

Signal-to-Noise Ratio (SNR) and Related Metrics

Signal-to-noise ratio (SNR) quantifies how much the desired signal stands above the noise floor, expressed in decibels:

$\text{SNR (dB)} = 10 \log_{10} \left( \frac{P_{\text{signal}}}{P_{\text{noise}}} \right)$

Higher SNR means a cleaner signal. In digital systems, quantization noise is often the dominant noise source, so SNR is tightly linked to bit depth.

For a full-scale sinusoidal input and a uniform quantizer with $n$ bits, the theoretical maximum SNR is:

$\text{SNR}_{\max} \approx 6.02n + 1.76 \text{ dB}$

This is where the well-known 6 dB per bit rule comes from. Each additional bit of resolution roughly doubles the number of quantization levels and adds about 6 dB of dynamic range.

Two related metrics worth knowing:

• SQNR (Signal-to-Quantization-Noise Ratio) isolates quantization noise specifically, rather than lumping in all noise sources. For an ideal converter, SQNR equals the theoretical SNR formula above.
• ENOB (Effective Number of Bits) measures the actual resolution of a real analog-to-digital converter by accounting for all noise sources (thermal noise, clock jitter, nonlinearity). You can calculate it from a measured SINAD (signal-to-noise-and-distortion ratio):

$\text{ENOB} = \frac{\text{SINAD (dB)} - 1.76}{6.02}$

A converter advertised as 16-bit might have an ENOB of only 14 bits once real-world imperfections are included. ENOB gives you a more honest picture of converter performance than the nominal bit depth alone.