8.1 Stereoscopic vision

Stereoscopic vision is a core technique in computer vision that mimics how human eyes perceive depth. By using two cameras to capture slightly different views of the same scene, you can estimate depth and reconstruct 3D geometry. This topic covers binocular disparity, camera calibration, correspondence matching, depth map generation, and newer approaches like multi-view stereo and deep learning methods.

Fundamentals of stereoscopic vision

Stereoscopic vision enables depth perception by exploiting the slight differences between two images of the same scene, captured from slightly different viewpoints. This is the same principle your brain uses when combining input from your left and right eyes. The technique is foundational for applications in robotics, autonomous vehicles, and virtual reality.

Binocular disparity concept

Binocular disparity is the difference in horizontal position of the same object as seen in the left and right images. If you look at a nearby object, it appears shifted more between the two views than a distant object does.

• Disparity is inversely proportional to depth: closer objects produce larger disparity, farther objects produce smaller disparity
• Measured as a pixel offset (in stereo imaging) or in units of visual angle (degrees or arc minutes) in human vision
• The brain (or a stereo algorithm) uses these disparity values to estimate relative depths across a scene

Depth perception mechanisms

Depth perception doesn't rely on binocular disparity alone. Multiple cues work together:

• Stereopsis extracts depth directly from binocular disparity and is the primary mechanism in stereo vision systems
• Monocular cues also contribute: motion parallax (objects at different depths move at different apparent speeds), occlusion (nearer objects block farther ones), and linear perspective
• Accommodation (lens focusing) and convergence (eyes rotating inward for near objects) provide additional depth signals at close range
• The visual cortex integrates all of these cues, and depth perception accuracy decreases with distance from the viewer

Parallax and stereopsis

Parallax is the apparent shift of an object's position when viewed from two different locations. Stereoscopic vision is built on this principle.

• Motion parallax occurs when you move and objects at different distances shift by different amounts in your visual field
• Stereopsis specifically refers to depth perception that arises from binocular disparity, requiring the brain (or algorithm) to fuse the left and right images
• Stereopsis provides the finest depth discrimination for nearby objects, but its effectiveness drops off at greater distances

Stereo camera systems

A stereo camera system uses two cameras separated by a known distance called the baseline. This setup mimics human binocular vision and captures the image pairs needed for depth estimation.

Camera calibration techniques

Before you can extract accurate depth, you need to know your cameras' parameters. Calibration happens in two stages:

1. Intrinsic calibration determines each camera's internal parameters: focal length, principal point (optical center in the image), and lens distortion coefficients
2. Extrinsic calibration finds the relative position and orientation (rotation and translation) between the two cameras
3. Stereo calibration combines both to establish the full geometric relationship between the camera pair

Zhang's method is the most widely used approach: you capture several images of a planar checkerboard pattern from different angles, and the algorithm solves for all parameters. Bundle adjustment can then refine these parameters globally by minimizing reprojection error across all calibration images.

Epipolar geometry basics

Epipolar geometry describes the geometric relationship between two camera views of the same 3D scene. It's what makes efficient stereo matching possible.

• For any point in one image, its corresponding point in the other image must lie along a specific line called the epipolar line. This constrains the search from 2D to 1D.
• The fundamental matrix $F$ encapsulates this geometry for uncalibrated cameras. For a point $p$ in the left image and its match $p'$ in the right image: $p'^T F p = 0$
• The essential matrix $E$ is the calibrated version, relating normalized image coordinates (it equals $K'^T F K$, where $K$ and $K'$ are the intrinsic matrices)
• Epipoles are the points where the line connecting the two camera centers intersects each image plane (the projection of one camera's center onto the other's image)

Rectification process

Rectification transforms a stereo image pair so that epipolar lines become horizontal and aligned across both images. This is a critical preprocessing step:

1. Compute the rotation and reprojection needed to align both images onto a common virtual plane
2. Warp both images so that corresponding rows in the left and right images share the same vertical coordinate
3. After rectification, matching becomes a 1D search along each scanline rather than a 2D search across the whole image

This dramatically speeds up correspondence matching and reduces the disparity search range. One trade-off: rectification can introduce distortion near image borders, and some pixels may fall outside the warped image boundaries.

Correspondence problem

The correspondence problem is about finding which pixel in the left image matches which pixel in the right image. It's the hardest part of stereo vision, and the quality of your depth map depends entirely on getting this right. The main challenges are occlusions (regions visible in only one image), repetitive patterns (which create ambiguous matches), and textureless regions (where there's nothing distinctive to match on).

Feature matching algorithms

Several algorithms detect and describe distinctive image features for matching:

• SIFT (Scale-Invariant Feature Transform) detects keypoints and builds 128-dimensional descriptors that are robust to scale and rotation changes
• SURF (Speeded Up Robust Features) uses integral images for faster computation while maintaining similar robustness to SIFT
• ORB (Oriented FAST and Rotated BRIEF) produces compact binary descriptors and runs significantly faster than SIFT or SURF, making it popular for real-time applications
• Template matching slides a small image patch across the other image and uses correlation to find the best match
• Deep learning methods train neural networks to learn feature representations directly from data, often outperforming handcrafted descriptors

Dense vs sparse correspondence

• Sparse correspondence matches only a subset of distinctive points (corners, edges, blobs). It's fast but gives you depth at only a few locations.
• Dense correspondence attempts to match every pixel, producing a complete depth map. It's much more computationally expensive.
• Hybrid approaches use sparse matches as anchor points and then interpolate or propagate to fill in the rest, balancing speed and completeness.

Occlusion handling

Occlusions happen when part of the scene is visible to one camera but blocked in the other. Unhandled occlusions cause incorrect matches and depth errors.

• Left-right consistency check: compute disparity maps from both directions (left-to-right and right-to-left), then flag pixels where the two maps disagree
• Ordering constraint: assumes that the left-to-right order of objects along an epipolar line is preserved in both images (this holds for most scenes but breaks with thin foreground objects)
• Uniqueness constraint: each pixel in one image should match at most one pixel in the other
• Global optimization methods can include occlusion-aware cost terms that explicitly model the possibility of no match

Disparity computation

Disparity is the horizontal pixel offset between corresponding points in the left and right images. Larger disparity means the object is closer to the cameras. Computing disparity for every pixel gives you a disparity map, which you can then convert to depth.

Block matching methods

Block matching is the simplest approach: compare a small window of pixels around each point in the left image against candidate windows in the right image along the same scanline.

• SAD (Sum of Absolute Differences): sums the absolute intensity differences across the window. Fast and simple.
• NCC (Normalized Cross-Correlation): normalizes for mean intensity and contrast, making it more robust to illumination differences between cameras.
• Census transform: encodes each pixel's neighborhood as a binary string (is each neighbor brighter or darker than the center?), then compares strings using Hamming distance. Very robust to radiometric changes.
• Choosing window size involves a trade-off: larger windows give more stable matches in textureless areas but blur depth boundaries. Adaptive window sizes can help near depth discontinuities.

Dynamic programming approaches

Dynamic programming (DP) treats disparity estimation as an optimization problem along each epipolar line (scanline):

1. Define a cost for matching each pixel pair plus a smoothness penalty for disparity changes between adjacent pixels
2. Solve for the optimal disparity assignment along the entire scanline using DP
3. Allow "skip" states to handle occluded pixels

DP is efficient enough for real-time use, but because each scanline is optimized independently, it can produce horizontal streaking artifacts where adjacent rows have inconsistent disparities.

Global optimization techniques

Global methods minimize an energy function over the entire disparity map simultaneously, combining a data term (matching cost) with a smoothness term (penalizing abrupt disparity changes):

• Graph cuts find a global or near-global minimum for certain classes of energy functions by solving a min-cut problem on a graph
• Belief propagation uses iterative message passing between neighboring pixels to approximate the optimal solution
• Variational methods formulate disparity as a continuous optimization problem and solve using calculus of variations
• Semi-global matching (SGM) is a practical compromise: it aggregates matching costs along multiple directions (typically 8 or 16 paths) across the image, approximating global optimization at much lower computational cost. SGM is one of the most widely used methods in practice.

Depth map generation

Once you have a disparity map, you convert it into metric depth values to get a 3D representation of the scene. This depth map is the foundation for 3D reconstruction, obstacle detection, and scene understanding.

Disparity to depth conversion

The conversion uses the triangulation principle:

$Z = \frac{f \cdot B}{d}$

where $Z$ is depth, $f$ is focal length (in pixels), $B$ is the baseline distance between cameras, and $d$ is disparity (in pixels).

A few things to note:

• This relationship is nonlinear: depth resolution degrades quadratically with distance. At close range you get fine depth detail; at long range, small disparity differences correspond to large depth changes.
• Accurate calibration is essential. Errors in $f$ or $B$ directly scale your depth estimates.
• Sub-pixel disparity estimation (fitting a parabola or other curve to the matching cost around the integer minimum) improves depth precision, especially for distant objects.

Depth map refinement

Raw depth maps are noisy and contain holes. Several post-processing techniques clean them up:

• Bilateral filtering smooths depth values while preserving sharp edges at object boundaries by weighting neighbors based on both spatial distance and depth similarity
• Guided filtering uses the corresponding color image as a guide, assuming that depth edges tend to align with color edges
• Hole filling interpolates missing depth values (from occlusions or failed matches) using neighboring valid pixels
• Temporal consistency in video applications enforces smooth depth transitions between frames to reduce flickering
• Super-resolution techniques upsample low-resolution depth maps using high-resolution color images

Handling of ambiguities

Some regions produce unreliable depth estimates. Strategies for dealing with this include:

• Computing confidence measures for each disparity value (e.g., how distinct the matching cost minimum is compared to other candidates)
• Maintaining multiple depth hypotheses in uncertain regions and resolving them with additional evidence
• Sensor fusion: combining stereo depth with LiDAR or time-of-flight data to fill in where stereo struggles
• Using semantic segmentation to identify object classes and apply class-specific depth priors (e.g., knowing a wall should be planar)

Applications of stereoscopic vision

3D reconstruction

• Structure from Motion (SfM) reconstructs 3D geometry from unordered image collections by simultaneously estimating camera poses and 3D point positions
• Multi-view stereo (MVS) takes SfM output and generates dense 3D point clouds or meshes
• Photogrammetry applies stereo principles for precise measurements in surveying, mapping, and construction
• Cultural heritage preservation and reverse engineering use 3D scanning pipelines built on stereo techniques

Autonomous navigation

• Self-driving cars use stereo cameras for obstacle detection and distance estimation
• Drones rely on stereo depth for collision-free path planning in GPS-denied environments
• SLAM (Simultaneous Localization and Mapping) builds a map of the environment while tracking the camera's position within it
• Visual odometry estimates camera motion frame-to-frame using stereo correspondences
• Advanced driver assistance systems (ADAS) use stereo for lane detection, pedestrian detection, and adaptive cruise control

Virtual and augmented reality

• VR headsets present slightly different images to each eye to create stereoscopic depth, mimicking natural binocular vision
• AR systems use depth estimation for proper occlusion handling (virtual objects should appear behind real ones when appropriate)
• Autostereoscopic (glasses-free) 3D displays use stereo rendering to present depth without special eyewear
• Gesture recognition and hand tracking in interactive systems rely on real-time depth maps from stereo or structured light cameras

Challenges in stereoscopic vision

Illumination variations

Differences in lighting between the two camera views degrade matching accuracy. This can happen due to different exposure settings, reflections, or shadows falling differently in each view.

• Normalized correlation measures (like NCC and Census transform) are inherently more robust to global brightness changes than raw intensity comparisons
• Local illumination variations (specular highlights, shadows) require more sophisticated handling, such as gradient-based matching or shadow detection and removal
• HDR stereo captures multiple exposures and merges them to handle high-contrast scenes

Textureless regions

Flat, uniform surfaces (white walls, clear skies) lack distinctive features, making correspondence matching unreliable or impossible.

• Global optimization methods help by propagating reliable disparity values from textured regions into textureless ones via smoothness constraints
• Using larger matching windows in homogeneous areas captures more context but risks blurring depth edges
• Semantic information can guide estimation: if you know a region is a wall, you can enforce planarity

Real-time processing constraints

Stereo algorithms are computationally demanding, especially dense methods on high-resolution images.

• GPU acceleration enables parallel processing of matching costs across all pixels simultaneously
• Hierarchical (coarse-to-fine) approaches compute disparity at low resolution first, then refine at higher resolutions
• There's always a trade-off between accuracy and speed in algorithm design
• Dedicated hardware (FPGAs, ASICs) achieves low-latency stereo processing for embedded systems like automotive platforms

Advanced techniques

Multi-view stereo

Multi-view stereo extends beyond two cameras to use many viewpoints, improving coverage and accuracy:

• Patch-based MVS (PMVS) grows and filters oriented patches across multiple views for dense reconstruction
• Volumetric approaches divide space into voxels and determine occupancy by fusing photo-consistency scores from all views
• Photometric stereo is a related but distinct technique: it uses varying illumination (not viewpoint) to estimate surface normals
• Light field cameras (like the Lytro) capture multiple viewpoints in a single exposure using a microlens array, enabling post-capture depth estimation

Active stereo systems

Active stereo projects known patterns onto the scene, which solves the correspondence problem in textureless regions:

• Structured light systems project coded patterns (stripes, grids, or dot patterns) and decode them to establish correspondences. Microsoft's Kinect v1 used this approach.
• Time-of-flight (ToF) cameras measure depth by timing how long modulated light takes to return from the scene. These aren't strictly stereo but are often grouped with active depth sensors.
• Laser scanners combine precise rangefinding with camera imagery for high-accuracy 3D capture

Machine learning in stereo vision

Deep learning has significantly advanced stereo matching in recent years:

• End-to-end networks like DispNet and PSMNet take a rectified stereo pair as input and directly output a disparity map, learning feature extraction, matching, and regularization jointly
• PSMNet uses a spatial pyramid pooling module to capture context at multiple scales, which helps in textureless and occluded regions
• Unsupervised approaches train without ground truth disparity by using photometric consistency between warped views as the training signal
• Transfer learning adapts models trained on synthetic data (where ground truth is free) to real-world domains with limited labeled data
• GANs have been used to refine depth maps by learning to produce outputs that look like realistic, sharp depth maps

Evaluation metrics

Accuracy vs computational efficiency

Stereo algorithms are typically evaluated on both accuracy and speed, and there's an inherent trade-off between the two:

• Mean Absolute Error (MAE): average absolute difference between estimated and ground truth disparity. Treats all errors equally.
• Root Mean Square Error (RMSE): penalizes large errors more heavily than MAE due to the squaring operation.
• Bad pixel percentage: the fraction of pixels where disparity error exceeds a threshold (commonly 1, 2, or 3 pixels). This is the most commonly reported metric on benchmarks.
• Runtime and throughput (frames per second) measure computational efficiency.

Quantitative assessment methods

• Disparity error maps visualize where the algorithm succeeds and fails
• 3D reconstruction metrics compare estimated point clouds against reference models using metrics like Chamfer distance
• Separate evaluation in occluded vs non-occluded regions reveals how well the algorithm handles occlusions
• Robustness testing evaluates performance under varying illumination, noise levels, and scene types

Benchmarking datasets

• Middlebury Stereo Dataset: high-resolution indoor scenes with precise structured-light ground truth. The standard benchmark for evaluating matching accuracy.
• KITTI: real-world driving scenarios with sparse LiDAR ground truth. The primary benchmark for automotive stereo.
• ETH3D: includes both indoor and outdoor scenes at varying difficulty levels with laser-scanned ground truth.
• Scene Flow datasets (e.g., FlyingThings3D): large-scale synthetic data providing dense ground truth for training deep learning models.
• Tanks and Temples: focuses on multi-view 3D reconstruction evaluation with diverse real-world scenes.