4.4 Mixed reality and holographic displays
13 min read•august 19, 2024
Mixed reality blends physical and virtual worlds, creating immersive experiences where digital content interacts with real environments. From to fully virtual experiences, MR technologies enable users to perceive and interact with that appear to coexist within the real world.
Holographic displays create 3D images that float in mid-air, providing more realistic viewing experiences than traditional 2D displays. These displays use various techniques to manipulate light, enabling depth and parallax without special glasses or headsets. Advances in this technology are crucial for creating engaging MR experiences.
Mixed reality fundamentals
- Mixed reality (MR) combines elements of both the physical and virtual worlds, creating experiences where digital content interacts with and responds to the real environment
- MR technologies enable users to perceive and interact with virtual objects that appear to coexist within the real world, blurring the line between reality and virtuality
- Understanding the fundamentals of mixed reality is crucial for creating immersive and engaging experiences in virtual and augmented reality art
Merging physical and virtual
Top images from around the web for Merging physical and virtual
This is What the Future of Mixed Reality Looks Like View original
Is this image relevant?
Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original
Is this image relevant?
Building Industry 4.0 With Mixed Reality – Alice Bonasio – Medium View original
Is this image relevant?
This is What the Future of Mixed Reality Looks Like View original
Is this image relevant?
Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original
Is this image relevant?
1 of 3
Top images from around the web for Merging physical and virtual
This is What the Future of Mixed Reality Looks Like View original
Is this image relevant?
Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original
Is this image relevant?
Building Industry 4.0 With Mixed Reality – Alice Bonasio – Medium View original
Is this image relevant?
This is What the Future of Mixed Reality Looks Like View original
Is this image relevant?
Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original
Is this image relevant?
1 of 3
- Mixed reality seamlessly blends the physical and virtual worlds, allowing digital content to interact with and respond to the real environment in real-time
- Virtual objects can appear to be anchored to specific locations in the physical world, maintaining their position and orientation relative to real-world objects
- Users can interact with virtual content using natural gestures, voice commands, and even physical objects, creating a more intuitive and
Spectrum of mixed reality
- The mixed reality spectrum encompasses a range of experiences, from entirely virtual environments (virtual reality) to the integration of digital content into the real world (augmented reality)
- Augmented reality (AR) overlays digital information onto the real world, typically through a smartphone or tablet camera (Pokémon Go, Google Lens)
- Augmented virtuality (AV) brings real-world elements into a primarily virtual environment, such as incorporating live video feeds or physical objects into a VR experience
- True mixed reality experiences sit in the middle of the spectrum, allowing virtual and real-world content to interact and respond to each other in real-time (Microsoft HoloLens, Magic Leap)
Key characteristics of MR
- : MR devices can understand and map the physical environment, allowing virtual content to interact with and respond to real-world objects and surfaces
- Real-time interaction: Users can interact with virtual content using natural gestures, voice commands, and even physical objects, with the virtual content responding in real-time
- Persistence: Virtual objects can maintain their position and orientation in the real world, even when not actively being viewed by the user
- Occlusion: Virtual objects can be occluded by real-world objects, enhancing the illusion that they exist within the physical environment
Holographic display technology
- Holographic displays create three-dimensional images that appear to float in mid-air, providing a more immersive and realistic viewing experience compared to traditional 2D displays
- These displays use various techniques to manipulate light, enabling the creation of depth and parallax without the need for special glasses or headsets
- Advances in holographic display technology are crucial for creating more realistic and engaging mixed reality experiences in virtual and augmented reality art
Light field displays
- Light field displays capture and reproduce the light rays emanating from a scene, allowing for more realistic and natural-looking 3D images
- These displays use an array of micro-lenses or other optical elements to create the illusion of depth and parallax, enabling users to view the image from different angles
- Examples of light field display technology include the Looking Glass and the Leia Display, which provide glasses-free 3D viewing experiences
Volumetric displays
- Volumetric displays create three-dimensional images by illuminating points in a volume of space, allowing users to view the image from any angle without the need for special glasses
- These displays use various techniques, such as rotating LED arrays, laser projection, or stacked liquid crystal panels, to create the illusion of a solid 3D object floating in mid-air
- Examples of volumetric display technology include the Voxon VX1 and the Holoxica 3D Display, which provide true 3D viewing experiences
Holographic optical elements
- Holographic optical elements (HOEs) are thin, transparent films that can manipulate light to create the illusion of three-dimensional images
- HOEs use diffraction patterns recorded on a photosensitive material to bend and shape light, enabling the creation of 3D images that appear to float in mid-air
- Examples of HOE technology include the HoloLens 2 and the WaveOptics waveguides, which use HOEs to project virtual images onto the user's retina
Challenges in holographic displays
- Creating high-resolution, full-color holographic displays with a wide field of view remains a significant challenge
- Current holographic display technologies often suffer from limited viewing angles, low brightness, and high power consumption
- Scaling holographic displays to larger sizes while maintaining image quality and reducing costs is another major hurdle
- Developing efficient and accurate methods for capturing and rendering real-world scenes as holographic content is an ongoing area of research
Interaction in mixed reality
- Interaction in mixed reality involves using natural and intuitive methods to manipulate and engage with virtual content seamlessly integrated into the real world
- MR systems must be able to understand and respond to user input in real-time, providing a sense of presence and agency within the mixed reality environment
- Developing effective interaction techniques is crucial for creating engaging and immersive mixed reality experiences in virtual and augmented reality art
Gaze, gesture, and voice input
- Gaze tracking allows MR systems to determine where the user is looking, enabling more natural and intuitive interaction with virtual content (eye tracking in HoloLens 2)
- Gesture recognition enables users to interact with virtual objects using hand and body movements, providing a more immersive and tactile experience (hand tracking in Oculus Quest)
- Voice input allows users to control and interact with virtual content using natural language commands, reducing the need for physical controllers or gestures (voice commands in Magic Leap)
- Combining gaze, gesture, and voice input can create more seamless and intuitive interaction experiences in mixed reality environments
Haptic feedback in MR
- Haptic feedback provides tactile sensations to users, enhancing the sense of presence and immersion in mixed reality experiences
- Haptic devices can simulate the feeling of touching, grasping, or manipulating virtual objects, providing a more realistic and engaging interaction experience
- Examples of haptic feedback in MR include the HaptX Gloves, which use microfluidic actuators to provide realistic tactile sensations, and the Teslasuit, which provides full-body haptic feedback
Spatial mapping and understanding
- Spatial mapping involves creating a digital representation of the physical environment, allowing MR systems to understand and interact with the real world
- MR devices use cameras, depth sensors, and computer vision algorithms to create 3D maps of the environment in real-time, enabling virtual content to be accurately placed and anchored within the real world
- Spatial understanding allows MR systems to recognize and respond to real-world objects and surfaces, enabling more natural and intuitive interactions between virtual and real content
Collaborative experiences in MR
- Mixed reality enables multiple users to interact with shared virtual content in real-time, fostering collaboration and social interaction
- Collaborative MR experiences can range from simple shared viewing of virtual content to complex multi-user interactions and co-creation of virtual environments
- Examples of collaborative MR experiences include Microsoft Mesh, which enables users to interact with shared 3D content in real-time, and Spatial, which provides a platform for virtual meetings and collaboration in MR
Mixed reality devices
- Mixed reality devices combine hardware and software components to create immersive experiences that blend virtual content with the real world
- These devices use various display technologies, sensors, and input methods to enable users to perceive and interact with virtual content seamlessly integrated into their physical environment
- Understanding the capabilities and limitations of current MR devices is essential for creating effective and engaging virtual and augmented reality art experiences
Microsoft HoloLens
- The Microsoft HoloLens is a self-contained mixed reality headset that uses holographic waveguide displays to project virtual images onto the user's retina
- The device features an array of sensors, including depth cameras, eye trackers, and inertial measurement units (IMUs), which enable spatial mapping, gaze tracking, and gesture recognition
- The HoloLens runs on the Windows Mixed Reality platform, which provides a framework for developing and deploying MR applications using tools like and Unreal Engine
Magic Leap
- Magic Leap is a mixed reality headset that uses a proprietary lightfield display technology to create highly realistic and immersive 3D images
- The device features an array of cameras and sensors for spatial mapping and hand tracking, as well as a handheld controller for more precise input
- Magic Leap runs on the Lumin OS, a custom operating system designed specifically for mixed reality applications, and provides a suite of development tools for creating MR content
Differences vs VR and AR devices
- Mixed reality devices like the HoloLens and Magic Leap provide a more immersive and interactive experience compared to traditional AR devices, which typically rely on smartphone or tablet displays
- Unlike VR devices, which fully immerse users in a virtual environment, MR devices allow users to see and interact with the real world while simultaneously engaging with virtual content
- MR devices often feature more advanced spatial mapping and understanding capabilities compared to AR devices, enabling more seamless integration of virtual content with the real world
Limitations and future developments
- Current MR devices are often limited by their display resolution, field of view, and battery life, which can impact the quality and duration of MR experiences
- The high cost of MR devices like the HoloLens and Magic Leap can be a barrier to widespread adoption, particularly in consumer markets
- Future developments in MR technology are likely to focus on improving display quality, increasing field of view, and reducing the size and weight of MR devices
- Advancements in 5G networks and edge computing could enable more sophisticated and responsive MR experiences, particularly for multi-user and collaborative applications
Applications of mixed reality
- Mixed reality has the potential to revolutionize a wide range of industries and domains, from education and entertainment to healthcare and industrial design
- By blending virtual content with the real world, MR can provide more engaging, interactive, and effective experiences compared to traditional media and technologies
- Exploring the diverse applications of mixed reality is crucial for understanding its potential impact and identifying new opportunities for virtual and augmented reality art
Education and training
- Mixed reality can enhance learning experiences by allowing students to interact with 3D models, simulations, and virtual environments in a more immersive and hands-on way (HoloAnatomy for medical education)
- MR training applications can provide realistic and safe simulations for high-risk or complex tasks, such as surgery, equipment maintenance, or emergency response (HoloLens for military training)
- Collaborative MR experiences can enable remote learning and training, allowing students and instructors to interact with shared virtual content in real-time (Prisms for math education)
Entertainment and gaming
- Mixed reality gaming experiences can blend virtual content with the real world, creating more immersive and interactive gameplay (Minecraft Earth, Pokémon Go)
- MR can enhance traditional entertainment experiences, such as movies and theme parks, by providing interactive and personalized content that responds to the user's actions and environment (The Void, Dreamscape Immersive)
- Collaborative MR experiences can enable social gaming and entertainment, allowing users to interact with shared virtual content in real-time (Spatial for virtual concerts)
Industrial design and visualization
- Mixed reality can streamline the product design and development process by allowing designers and engineers to visualize and interact with 3D models in real-time (HoloLens for automotive design)
- MR can enable remote collaboration and design reviews, allowing teams to work together on shared virtual models and prototypes from different locations (PiXYZ Review for design collaboration)
- MR visualization tools can help communicate complex designs and concepts to stakeholders and customers, providing a more engaging and interactive experience compared to traditional media (Magic Leap for architecture visualization)
Healthcare and medical imaging
- Mixed reality can enhance medical imaging by allowing healthcare professionals to visualize and interact with 3D patient data, such as CT scans and MRIs, in a more intuitive and immersive way (HoloLens for surgical planning)
- MR can enable remote collaboration and consultation between medical experts, allowing them to share and discuss patient data and treatment plans in real-time (Medivis for surgical collaboration)
- MR training applications can provide realistic simulations for medical procedures and emergency response, allowing healthcare professionals to practice and refine their skills in a safe and controlled environment (CAE VimedixAR for ultrasound training)
Developing for mixed reality
- Developing for mixed reality involves creating applications and experiences that blend virtual content with the real world, taking into account the unique capabilities and constraints of MR devices and platforms
- MR development requires a combination of skills and tools, including 3D modeling, game engine programming, spatial computing, and user experience design
- Understanding the key concepts and best practices of MR development is essential for creating effective and engaging virtual and augmented reality art experiences
Unity and Unreal Engine for MR
- Unity and Unreal Engine are popular game engines that support mixed reality development, providing tools and frameworks for creating MR applications and experiences
- Both engines offer built-in support for MR devices like the HoloLens and Magic Leap, as well as plugins and extensions for additional functionality (Mixed Reality Toolkit for Unity, Unreal Engine 4 XR Plugin)
- Unity and Unreal Engine provide visual scripting tools (Bolt for Unity, Blueprints for Unreal) that can make MR development more accessible to non-programmers and artists
Spatial anchors and world locking
- Spatial anchors are virtual objects that are anchored to specific locations in the real world, allowing MR content to maintain its position and orientation relative to the physical environment
- World locking techniques ensure that virtual content remains stable and aligned with the real world, even as the user moves around or the environment changes
- Platforms like Azure Spatial Anchors and ARCore Cloud Anchors provide cloud-based services for creating and managing spatial anchors across multiple devices and sessions
Performance optimization techniques
- Optimizing performance is critical for creating smooth and responsive MR experiences, particularly on mobile devices with limited processing power and battery life
- Techniques for optimizing MR performance include reducing polygon count and texture resolution, using efficient shading techniques, and minimizing the use of physics and particle effects
- Asynchronous loading and streaming can help manage the memory footprint of MR applications, while spatial partitioning techniques like occlusion culling can reduce the rendering overhead
Best practices for MR design
- Designing for mixed reality requires a different approach compared to traditional 2D or even VR interfaces, taking into account the unique characteristics of MR devices and the user's physical environment
- Best practices for MR design include creating content at an appropriate scale and distance from the user, providing clear visual and audio feedback, and designing for 360-degree interaction
- Usability testing and user feedback are essential for refining MR designs and ensuring a smooth and intuitive user experience
- Accessibility considerations, such as providing alternative input methods and accommodating different user abilities and preferences, are important for creating inclusive MR experiences
Perceptual considerations in MR
- Perceptual considerations in mixed reality involve understanding how the human visual system processes and interprets virtual content blended with the real world
- MR experiences must take into account factors such as depth perception, stereoscopy, and the accommodation-convergence conflict to create a comfortable and convincing illusion of virtual objects coexisting with the real environment
- Addressing perceptual issues and ensuring user comfort and safety are critical for creating effective and engaging virtual and augmented reality art experiences
Depth cues and stereoscopy
- Depth cues are visual and non-visual signals that help the brain perceive the relative distance and position of objects in the environment, such as occlusion, perspective, and motion parallax
- Stereoscopy is the technique of presenting slightly different images to each eye to create the illusion of depth and 3D space, mimicking the way the human visual system naturally works
- MR devices like the HoloLens and Magic Leap use stereoscopic displays to create a convincing illusion of virtual objects existing in the real world, enhancing the sense of presence and immersion
Accommodation-convergence conflict
- The accommodation-convergence conflict occurs when the focus distance (accommodation) and the vergence distance (convergence) of the eyes do not match, leading to visual discomfort and fatigue
- In MR displays, the accommodation distance is fixed at the display plane, while the convergence distance varies depending on the virtual content, potentially causing a mismatch and visual strain
- Techniques for minimizing the accommodation-convergence conflict include using multiple focal planes, adjusting the virtual content to match the display plane, and providing user controls for adjusting the focal distance
Perceptual illusions in MR
- Perceptual illusions can occur in mixed reality when the virtual content and the real world do not align or interact in a convincing or consistent manner, breaking the illusion of presence
- Examples of perceptual illusions in MR include occlusion violations (virtual objects appearing to pass through real objects), incorrect shadows or lighting, and inconsistent scale or perspective
- Minimizing perceptual illusions requires careful design and calibration of the MR experience, taking into account factors such as the user's position, the lighting conditions, and the characteristics of the physical environment
User comfort and safety
- Ensuring user comfort and safety is essential for creating successful and enjoyable MR experiences, particularly for extended or repeated use
- Factors that can impact user comfort in MR include visual fatigue, motion sickness, and physical strain from prolonged use or awkward postures
- Safety considerations in MR include avoiding virtual content that could cause users to collide with real-world objects, providing clear boundaries and warnings, and allowing users to easily exit the experience if needed
- Designing for user comfort and safety requires testing and feedback from a diverse range of users, as well as adherence to established guidelines and best practices for MR development
Future of mixed reality
- The future of mixed reality holds tremendous potential for transforming the way we interact with digital content and the world around us, from education and entertainment to healthcare and industrial applications
- Advancements in display technology, spatial computing, and artificial intelligence are likely to drive the development of more sophisticated and immersive MR experiences in the coming years
- As MR
Key Terms to Review (18)
3D Rendering: 3D rendering is the process of generating a two-dimensional image from a three-dimensional model using computer software. This technique transforms the model's data into visual representations that can include color, texture, lighting, and shadows. It's essential for creating realistic visuals in various applications, including mixed reality and holographic displays, which blend digital content with the real world to enhance user experiences.
Augmented reality: Augmented reality (AR) is a technology that overlays digital information, such as images or sounds, onto the real world through devices like smartphones or AR glasses. This blend of digital and physical environments enhances user interaction by allowing them to see and interact with virtual elements in real time while still being aware of their physical surroundings. It connects to various aspects of immersive experiences, including how mixed reality combines both virtual and real-world elements, and how narrative structures can be enriched by integrating AR into storytelling.
Blender: Blender is a powerful, open-source 3D creation software used for modeling, animating, rendering, and texturing. It serves as an all-in-one tool that integrates various aspects of 3D art production, making it essential for tasks like UV mapping, rigging, and skinning. Its versatility also extends to game development and content creation, making it suitable for designing avatars and environments in mixed reality applications.
Digital Overlay: A digital overlay is a virtual layer of information or imagery that is superimposed onto the real-world environment, enhancing the user's perception and interaction with that environment. This technology allows users to see additional contextual information, graphics, or animations seamlessly integrated with their physical surroundings, creating a richer experience. Digital overlays are key components in mixed reality and holographic displays, as they enable the blending of digital content with real-world elements in an immersive way.
Disruption: Disruption refers to a significant change or interruption in an existing process, system, or industry, often due to the introduction of innovative technologies or ideas. This concept is particularly relevant in the realm of mixed reality and holographic displays, where traditional ways of interacting with digital content and the physical world are transformed. The emergence of new technologies can lead to novel experiences that challenge existing norms and require users to adapt to new paradigms of interaction.
Holography: Holography is a photographic technique that captures the light field of an object, creating a three-dimensional image called a hologram. This method relies on the interference of light beams from a laser or another coherent light source, allowing for the recording and reconstruction of an object's light pattern. Holography plays a crucial role in mixed reality and holographic displays by enabling the creation of lifelike, interactive visual experiences that blend the physical and digital worlds.
Immersive Experience: An immersive experience refers to a deep engagement in a virtual environment that captivates users, making them feel as though they are part of the action. This type of experience often leverages advanced technologies, such as mixed reality and holographic displays, to create realistic and interactive scenarios that blur the lines between the digital and physical worlds. The goal is to provide a sense of presence, where users can interact meaningfully with the environment and objects within it.
Interactive installation: An interactive installation is a type of art that encourages audience participation and engagement, often blending physical and digital elements to create immersive experiences. These installations invite viewers to interact with the artwork, making them an integral part of the experience, rather than passive observers. This interactivity can involve technology such as sensors, projections, or virtual environments, allowing for a dynamic relationship between the art and the audience.
Jaron Lanier: Jaron Lanier is a computer scientist, author, and musician known for his pioneering work in virtual reality (VR) and immersive technology. He played a crucial role in developing early VR systems in the 1980s and is also recognized for his critical perspective on technology's impact on society and culture.
Light Field Technology: Light field technology is a method of capturing and displaying visual information in a way that allows for the reproduction of light rays coming from a scene, providing depth perception and allowing viewers to see different perspectives without the need for special glasses. This technology enables the creation of realistic, three-dimensional images and has applications in mixed reality and holographic displays, enhancing the immersive experience by offering a more natural interaction with virtual objects.
Marina Abramović: Marina Abramović is a Serbian performance artist known for her pioneering work in the field of performance art, often exploring the relationship between artist and audience, the limits of the body, and the concept of presence. Her innovative approaches have significantly influenced the development of immersive art experiences, particularly within virtual and mixed reality contexts.
Post-digital art: Post-digital art refers to a contemporary artistic movement that acknowledges and critiques the influence of digital technology on the creation, distribution, and experience of art. This movement emphasizes the integration of digital tools while also exploring the physicality and materiality of artworks, often blurring the lines between traditional and digital forms. Artists in this realm are keen to reflect on how technology affects human experience and culture, creating works that engage with both the virtual and tangible worlds.
Spatial Awareness: Spatial awareness is the ability to understand and interact with the space around us, recognizing how objects relate to one another in a given environment. It plays a crucial role in how we perceive dimensions, distances, and the layout of objects within both real and virtual settings. This skill is particularly important in creating immersive experiences, where users must navigate and manipulate virtual environments effectively.
Transformation: Transformation refers to the process of changing the form, appearance, or nature of something. In the realm of mixed reality and holographic displays, transformation plays a crucial role as it enables digital content to be integrated seamlessly with the physical world. This ability to manipulate and alter virtual objects or environments enhances user experience and interactivity, making digital art more immersive and dynamic.
Transmedia Art: Transmedia art is a storytelling technique that combines multiple media platforms to create a cohesive narrative experience. This approach allows artists to expand their work across different formats, such as video, digital media, installations, and live performances, engaging audiences in various ways. By integrating elements of mixed reality and holographic displays, transmedia art can create immersive experiences that blend the real and virtual worlds, enriching the viewer's interaction with the artwork.
Unity: Unity refers to the cohesion and harmony among different elements within immersive environments, ensuring that all components work together seamlessly to create an engaging experience. This concept is crucial for achieving a balanced interaction between visuals, audio, and user input, enhancing overall immersion and user satisfaction.
User Agency: User agency refers to the capacity of individuals to act independently and make their own choices within a virtual environment. This concept is crucial in immersive and virtual experiences, as it impacts how users engage with the content and interact with their surroundings, ultimately influencing the overall experience and narrative.
Virtual Objects: Virtual objects are computer-generated entities that exist within a digital environment, enabling users to interact with them as if they were real. These objects can be displayed in various forms, such as 3D models or animations, and are essential for creating immersive experiences in mixed reality and holographic displays. By blending digital content with the real world, virtual objects enhance user engagement and provide dynamic interactions that are crucial for applications in gaming, education, and design.