👓AR and VR Engineering Unit 5 – AR/VR Displays and Devices

Key Concepts and Terminology

• Augmented Reality (AR) overlays digital information onto the real world, enhancing the user's perception and interaction with the environment
• Virtual Reality (VR) immerses users in a completely digital environment, replacing the real world with a simulated one
• Mixed Reality (MR) blends real and virtual worlds, allowing digital objects to interact with the physical environment
• Head-Mounted Displays (HMDs) are wearable devices that provide visual and auditory feedback to the user
• Field of View (FOV) refers to the extent of the observable world seen at any given moment, typically measured in degrees
• Latency is the delay between a user's action and the system's response, which can impact the sense of presence and immersion
• Haptic feedback provides tactile sensations to the user, enhancing the sense of touch and interaction with virtual objects
• Inside-out tracking uses sensors and cameras mounted on the HMD to track the user's position and orientation in the environment

Evolution of AR/VR Display Technology

• Early VR systems in the 1960s, such as the Sensorama and the Ultimate Display, laid the foundation for immersive experiences
• In the 1990s, VR gained popularity with the introduction of devices like the Virtual Boy and the CAVE (Cave Automatic Virtual Environment)
• Smartphone-based VR, such as Google Cardboard and Samsung Gear VR, made VR more accessible to the masses in the 2010s
• High-end VR systems, including the Oculus Rift and HTC Vive, offered improved resolution, refresh rates, and tracking capabilities
• AR devices, such as Google Glass and Microsoft HoloLens, introduced wearable AR experiences
• The convergence of AR and VR led to the development of Mixed Reality devices, blurring the line between real and virtual worlds
• Advancements in display technology, such as higher resolution screens and foveated rendering, continue to enhance the visual quality of AR/VR experiences

Types of AR/VR Displays

• Tethered VR displays, such as the Oculus Rift and HTC Vive, connect to a powerful computer to render high-quality graphics
  • These displays offer the best visual fidelity and performance but require a dedicated setup and limit user mobility
• Standalone VR displays, like the Oculus Quest and Pico Neo, integrate all the necessary components into a single device
  • They provide a more portable and convenient VR experience but may have lower performance compared to tethered displays
• Smartphone-based VR displays use a smartphone as the screen and processing unit, making VR accessible to a wider audience (Google Cardboard, Samsung Gear VR)
• AR displays can be categorized into two main types: see-through and pass-through displays
  • See-through displays, such as the Microsoft HoloLens and Magic Leap, use transparent waveguides or combiners to overlay digital content onto the real world
  • Pass-through displays, like the Varjo XR-3 and Lynx-R1, capture the real world using cameras and display it on an opaque screen, allowing for mixed reality experiences
• Projection-based AR systems use projectors to display digital content directly onto real-world surfaces, enabling shared experiences without the need for individual devices

Optical Systems and Components

• Display panels, such as LCD, OLED, and micro-LED, are the primary components responsible for generating images in AR/VR displays
  • LCD (Liquid Crystal Display) panels use liquid crystals to modulate light and create images
  • OLED (Organic Light-Emitting Diode) panels offer high contrast ratios and deep blacks by self-emitting light
  • Micro-LED displays consist of tiny, self-emitting LED pixels, providing high brightness and efficiency
• Lenses in AR/VR displays help focus the image and create a wide field of view
  • Fresnel lenses are commonly used to reduce the size and weight of the display while maintaining a large FOV
  • Pancake lenses use a combination of polarizers and quarter-wave plates to fold the optical path, resulting in a compact design
• Waveguides are thin, transparent structures that guide light from a source to the user's eyes, enabling the creation of sleek and lightweight AR displays
  • Diffractive waveguides use gratings to couple light in and out of the waveguide, allowing for the display of digital content
  • Holographic waveguides employ holographic optical elements (HOEs) to control the direction and focus of light
• Eye tracking systems monitor the user's gaze direction and eye movements, enabling foveated rendering and improved user interaction
  • Foveated rendering reduces the rendering workload by decreasing the image quality in the peripheral vision while maintaining high quality in the area of focus

Display Performance Metrics

• Resolution refers to the number of pixels in a display, typically expressed as the number of pixels per inch (PPI) or the total number of pixels (e.g., 1920x1080)
  • Higher resolutions provide sharper and more detailed images, enhancing the visual quality of the AR/VR experience
• Refresh rate is the number of times per second that the display updates the image, measured in Hertz (Hz)
  • Higher refresh rates (e.g., 90Hz, 120Hz) result in smoother motion and reduced motion blur, which is crucial for maintaining immersion and preventing motion sickness
• Latency, or motion-to-photon latency, is the time delay between a user's movement and the corresponding update in the display
  • Lower latency is essential for maintaining a sense of presence and avoiding disorientation or nausea
• Contrast ratio is the difference in luminance between the brightest white and the darkest black that a display can produce
  • Higher contrast ratios provide deeper blacks and more vivid colors, enhancing the overall visual quality
• Color accuracy refers to how closely the colors displayed match the intended colors, often measured using color spaces like sRGB or DCI-P3
  • Accurate color reproduction is important for creating realistic and visually appealing AR/VR experiences
• Brightness, measured in nits (cd/m²), determines the maximum luminance of the display
  • Higher brightness levels are necessary for AR displays to ensure visibility in various lighting conditions, including outdoor environments

Input and Tracking Devices

• Motion controllers, such as the Oculus Touch and HTC Vive controllers, allow users to interact with virtual objects and navigate the environment
  • These controllers typically include buttons, joysticks, and touchpads for input, as well as sensors for tracking position and orientation
• Gesture recognition systems use cameras or depth sensors to track hand and finger movements, enabling natural and intuitive interaction with virtual content (Leap Motion, Microsoft HoloLens 2)
• Eye tracking devices, such as the Tobii Eye Tracker and Pupil Labs, monitor the user's gaze direction and eye movements
  • This information can be used for foveated rendering, attention analysis, and intuitive user interfaces
• Haptic devices provide tactile feedback to the user, simulating the sense of touch in virtual environments
  • Haptic gloves, such as the HaptX Gloves and the Teslasuit Glove, use actuators and force feedback to create realistic tactile sensations
  • Haptic vests and suits, like the bHaptics TactSuit and the Teslasuit, provide full-body haptic feedback for enhanced immersion
• Omnidirectional treadmills, such as the Virtuix Omni and the KatVR, allow users to walk and run in any direction within a virtual environment
  • These devices enable more natural and intuitive locomotion, reducing the risk of motion sickness and enhancing immersion

Challenges and Limitations

• Vergence-accommodation conflict occurs when the eyes focus on a screen at a fixed distance while converging at a different virtual distance, leading to visual discomfort and fatigue
  • Solutions include varifocal displays, which adjust the focal distance based on the user's gaze, and light field displays, which provide multiple focal planes
• Limited field of view in current AR/VR displays can break immersion and hinder the sense of presence
  • Researchers are developing displays with wider FOVs, such as the StarVR One (210° horizontal FOV) and the Pimax 8K X (200° diagonal FOV)
• Motion sickness can occur when there is a mismatch between the visual and vestibular systems, often due to latency or inconsistent tracking
  • Reducing latency, improving tracking accuracy, and implementing comfort features like snap turning and vignetting can help mitigate motion sickness
• Bulky and uncomfortable headsets can cause physical discomfort and hinder long-term use
  • Advancements in display technology, such as micro-displays and pancake lenses, aim to create more compact and lightweight designs
• Social acceptability and privacy concerns surrounding the use of AR/VR devices in public spaces
  • Developing socially acceptable form factors and implementing privacy-preserving measures, such as camera shutters and data encryption, can help address these concerns

Future Trends and Innovations

• Foveated rendering techniques, which reduce the rendering workload by decreasing image quality in the peripheral vision, will become more prevalent as eye tracking technology improves
• Varifocal and multifocal displays will address the vergence-accommodation conflict, providing more comfortable and visually accurate experiences
• Haptic feedback will become more advanced and realistic, with the development of high-resolution tactile displays and full-body haptic suits
• 5G networks and edge computing will enable low-latency streaming of high-quality AR/VR content, making it accessible on a wider range of devices
• Photorealistic virtual humans and avatars will enhance social presence and enable more engaging interactions in virtual environments
• Brain-computer interfaces (BCIs) will allow for direct communication between the brain and AR/VR devices, enabling intuitive control and potentially reducing the need for physical input devices
• Advancements in computer vision and machine learning will enable more accurate and reliable tracking, as well as improved understanding of the user's environment and context
• AR cloud platforms, such as Niantic's Real World Platform and Microsoft's Azure Spatial Anchors, will enable persistent and shared AR experiences across devices and users