12.3 Virtual and augmented reality

Virtual and augmented reality are reshaping how TV content gets made and consumed. These technologies create immersive experiences that go well beyond a traditional screen, and TV writers need to understand them to stay relevant as the industry evolves.

This section covers the core technology, how it's being used in production and storytelling, what it means for writers, and where things are headed.

Basics of VR and AR

VR and AR are distinct technologies that often get lumped together. Knowing the differences matters because each one creates a fundamentally different experience for the viewer, and that changes how you write for it.

Definitions and Distinctions

Virtual Reality (VR) immerses users in a completely digital environment, blocking out the physical world entirely. You put on a headset and you're somewhere else.

Augmented Reality (AR) overlays digital content onto the real world. You're still seeing your actual surroundings, but with digital elements layered on top. Think smartphone camera filters or heads-up displays.

Mixed Reality (MR) combines elements of both, allowing digital and physical objects to interact in real time. A virtual character could "sit" on your real couch, for example.

The hardware differs too. VR typically requires a dedicated headset (like the Meta Quest), while AR can run on smartphones or smart glasses.

Historical Development

• 1960s: Ivan Sutherland creates the first head-mounted display system, laying the groundwork for VR
• 1990s: VR gains traction in arcade games and military training simulations
• 2010s: The Oculus Rift Kickstarter campaign sparks a VR resurgence; Facebook acquires Oculus in 2014
• 2016: Pokémon GO demonstrates the mass-market potential of AR, reaching over 500 million downloads in its first year
• 2020s: AR development accelerates as smartphone computing power increases and Apple, Google, and Meta invest heavily in spatial computing

Key Technologies Involved

• Head-mounted displays (HMDs) provide immersive visual experiences in VR
• Motion tracking systems capture user movements and translate them into the virtual environment
• Haptic feedback devices simulate touch sensations for enhanced immersion
• Computer vision algorithms enable AR systems to recognize and track real-world objects
• Spatial audio creates 3D soundscapes that respond to user movements, so sound actually feels like it's coming from a specific direction

VR/AR in TV Production

These tools are already changing how TV gets made, from early planning stages through live broadcasts. For writers, the key takeaway is that VR/AR affects not just the final product but the entire production pipeline.

Pre-visualization Techniques

VR lets directors and cinematographers scout virtual locations before any physical production begins. Instead of flying a team to a location, you can walk through a digital version of it.

• Storyboards and animatics created in VR give a far more immersive sense of planned shots than flat drawings
• Virtual camera systems let filmmakers frame shots within a 3D environment, previewing complex visual effects before committing resources
• AR apps assist with blocking actors and planning camera movements on real-world sets

Virtual Set Design

• VR tools let set designers create and iterate on virtual environments quickly and at lower cost than building physical sets
• Green screen studios combined with real-time rendering engines (like Unreal Engine) create hybrid physical-virtual sets. This is the technology behind "The Volume," the LED wall stage used on The Mandalorian
• Virtual set extensions expand the scope of productions without large physical constructions
• AR markers on set allow actors to interact with virtual elements that get added in post-production

Live Broadcast Applications

AR graphics are already standard in many live TV contexts:

• Sports broadcasts overlay real-time stats, first-down lines, and player information directly on the field
• Virtual studios use AR to create dynamic backgrounds for news and weather reports
• Interactive AR elements let viewers engage with live content through second-screen experiences on their phones
• VR cameras provide 360-degree views of live events, letting viewers choose their own perspective

Storytelling in VR/AR

This is where things get most relevant for writers. VR and AR don't just change the medium; they change the fundamental relationship between the story and the audience. The viewer is no longer watching from outside. They're inside the narrative space.

Immersive Narrative Techniques

• Environmental storytelling uses the virtual space itself to convey information. A cluttered desk, a cracked window, graffiti on a wall: these details do narrative work without dialogue or exposition
• Non-linear narratives let users explore story elements in whatever order they choose
• Branching storylines provide multiple paths through the narrative based on user choices
• Spatial audio cues guide users' attention. A whisper from behind you, footsteps approaching from the left: sound becomes a directing tool
• Haptic feedback adds physical sensation to narrative moments, like a rumble during an explosion

360-Degree Video Production

Shooting in 360 degrees changes almost every production convention:

1. Blocking and staging must ensure important elements are visible from multiple angles, since you can't control where the viewer looks
2. Stitching techniques combine footage from multiple cameras to create seamless 360-degree environments
3. Directors must guide viewer attention without traditional framing. You can't cut to a close-up, so you rely on sound, light, movement, and character eyelines instead
4. Post-production workflows require specialized software different from standard editing tools
5. Sound design becomes more complex, requiring spatialized audio that shifts as the viewer turns their head

Interactive vs. Linear Storytelling

These represent two ends of a spectrum, and most VR/AR projects fall somewhere in between.

• Interactive narratives let users influence the story through choices and actions. The viewer has real agency.
• Linear VR/AR storytelling keeps a predetermined narrative but places the viewer inside an immersive environment. You experience the story, but you don't change it.
• Hybrid approaches offer limited interactivity within a structured story. You might choose which room to explore, but the main plot beats stay the same.

The core tension for writers: the more agency you give the user, the harder it is to control pacing, dramatic tension, and narrative payoff.

Writing for VR/AR Experiences

Writing for VR/AR requires a fundamentally different approach compared to traditional TV scripts. You're no longer writing for a rectangular frame. You're writing for a space that the viewer inhabits.

Spatial Storytelling Considerations

• You have to think in three dimensions. Where is the viewer standing? What's behind them? What happens if they look up?
• Narrative cues can be embedded in the environment rather than delivered through dialogue or exposition
• Character placement and movement become primary storytelling tools. A character walking toward you in VR feels very different from watching that same action on a flat screen
• You need to anticipate multiple viewpoints and exploration patterns. Some viewers will follow the obvious path; others will wander
• Spatial audio design plays a critical role in guiding users through the experience

User Interaction and Agency

• Scripts must allow for user choice while still maintaining narrative coherence. Every branching path needs to feel intentional, not random
• Interactive objects and characters that respond to user actions create a sense of agency
• Dialogue systems may need to account for multiple conversation paths
• Writers collaborate closely with game designers to create interactions that genuinely drive the story forward, not just serve as gimmicks
• Balancing user freedom with narrative progression requires extensive planning and iteration

Pacing and Timing Challenges

Traditional pacing tools don't always translate to immersive environments:

• Standard three-act structure may need to be adapted for non-linear or interactive narratives
• You have to account for variable exploration times. One viewer might spend 30 seconds in a room; another might spend five minutes
• Techniques for building tension and suspense work differently when the viewer controls their own perspective
• Montages and time jumps require creative reimagining since you can't just "cut to" the next scene the same way
• Cognitive load and motion sickness are real constraints. Overstimulating the viewer or forcing rapid movement can cause physical discomfort, which kills immersion fast

VR/AR Platforms and Distribution

The VR/AR landscape includes a range of hardware platforms and content delivery methods. Writers and producers need to understand what each platform can and can't do, because those constraints directly shape what stories are possible.

Major VR/AR Hardware

• Tethered VR headsets (Valve Index, PlayStation VR2) offer high-end graphics but require a connected PC or console
• Standalone VR headsets (Meta Quest 3) are self-contained and more accessible, with no wires or external hardware
• AR smart glasses (Microsoft HoloLens, Apple Vision Pro) offer hands-free augmented experiences
• Smartphone-based AR uses existing mobile devices, making it the most accessible entry point for audiences
• Haptic suits and gloves provide full-body immersion for advanced VR experiences
• CAVE systems (Cave Automatic Virtual Environments) create room-scale VR using projected walls instead of headsets

Content Delivery Methods

• App stores (Meta Quest Store, SteamVR) serve as primary distribution channels for VR content
• Web-based VR uses WebXR standards for cross-platform compatibility without requiring app downloads
• AR platforms often integrate with existing mobile apps or use dedicated frameworks (Apple ARKit, Google ARCore)
• Location-based installations offer high-end VR experiences in physical venues, similar to a theme park attraction
• Streaming services are experimenting with VR content delivery (Netflix VR app, Apple's spatial video)

Audience Engagement Metrics

Measuring success in VR/AR differs from traditional TV ratings:

• Time spent in VR/AR experiences indicates engagement levels
• Heat maps track user gaze and movement patterns within virtual environments, showing what actually holds attention
• Interaction rates measure how often users engage with interactive elements
• Completion rates for narrative experiences indicate whether the story holds up
• Social sharing metrics track how users share their VR/AR experiences with others

Ethical Considerations

The immersive nature of VR/AR raises ethical questions that don't exist with traditional TV. Because these technologies can feel so real, the potential for harm is different in kind, not just degree.

Privacy Concerns

• VR/AR systems collect extensive user data, including physical movements, eye tracking, and even pupil dilation
• Biometric information gathered by these devices raises serious questions about data ownership and use
• AR applications in public spaces may capture bystanders without their consent
• Virtual environments can potentially be used for surveillance or behavioral monitoring
• Data protection regulations like GDPR apply to VR/AR platforms, but enforcement is still catching up

Psychological Effects

• Prolonged VR use may lead to dissociation or difficulty distinguishing virtual from real experiences
• Intense VR content can cause lasting emotional responses. The sense of "presence" means disturbing content hits harder than it would on a flat screen
• AR applications that alter perception of the real world can create safety issues (think distracted walking or driving)
• Addiction to immersive experiences is a growing concern among mental health professionals
• VR/AR may exacerbate existing mental health conditions in ways that aren't yet well understood

Content Moderation Challenges

• Virtual environments enable new forms of harassment and abuse that are difficult to monitor. Personal space violations in VR, for instance, can feel genuinely threatening
• Real-time interaction in social VR spaces requires novel approaches to moderation
• AR content overlaid on the real world may need context-aware filtering systems
• User-generated content in VR/AR raises questions of ownership and liability
• Balancing free expression with user safety becomes more complex when the experience feels physically real

Future of VR/AR in TV

These technologies are evolving fast, and several developments on the horizon could significantly change TV production and consumption.

Emerging Technologies

• Brain-computer interfaces may eventually allow direct neural interaction with virtual environments
• Holographic displays could eliminate the need for headsets in some AR applications
• Advanced haptics are becoming more sophisticated, enabling more realistic touch sensations
• 5G and future network standards will enable more powerful cloud-based VR/AR experiences with lower latency
• AI-driven rendering is enhancing real-time graphics and procedural content generation, making VR/AR production faster and cheaper

Potential Industry Disruptions

• Virtual production techniques (LED volumes, real-time rendering) may replace traditional on-location filming for many productions
• Social VR platforms could compete with traditional TV for audience attention and advertising revenue
• AR-enhanced second-screen experiences may become standard for live TV events
• Increasingly accessible VR/AR creation tools could democratize production, enabling more independent creators
• Immersive storytelling could give rise to entirely new genres and formats that don't fit traditional TV categories

Integration with Traditional Media

Rather than replacing traditional TV, VR/AR is more likely to complement it:

• Hybrid viewing experiences combine traditional TV with AR elements viewed on a mobile device
• VR companion pieces extend TV show universes, offering deeper engagement for dedicated fans
• Interactive AR elements during live broadcasts enable real-time audience participation
• VR social viewing platforms recreate the communal TV watching experience in virtual spaces
• Transmedia storytelling strategies incorporate VR/AR elements alongside traditional TV content

Case Studies

Looking at real projects reveals what works, what doesn't, and why.

Successful VR/AR TV Projects

• "The Void" Star Wars experience combined physical sets with VR for a highly immersive walkthrough narrative. You could feel walls, smell smoke, and see stormtroopers in VR simultaneously
• "Bandersnatch" (Black Mirror) pioneered interactive storytelling on Netflix, letting viewers make choices that affected the plot. It proved mainstream audiences would engage with branching narratives
• "The Walking Dead: Our World" AR game successfully extended the TV show's universe to mobile devices, placing zombies in players' real-world surroundings
• BBC's "Civilisations AR" app brought historical artifacts to life using smartphone AR, letting users examine 3D models of museum objects in their own homes
• "Wolves in the Walls" adapted a children's book into an interactive VR narrative that won an Emmy for Outstanding Innovation in Interactive Media

Failed Attempts and Lessons

• Google Glass faced intense privacy backlash and social stigma ("Glassholes"), leading to its discontinuation as a consumer product. Lesson: social acceptance matters as much as technology
• "Defiance" attempted a TV show and video game crossover but struggled with meaningful integration between the two. Lesson: transmedia storytelling requires genuine narrative connection, not just shared branding
• Early VR chat shows (AltspaceVR) faced technical limitations and low user adoption. The audience simply wasn't there yet
• 3D TV failed to gain widespread adoption despite heavy industry investment. Lesson: requiring special hardware for a marginal improvement in experience is a tough sell
• "The Void" closures highlight that even successful VR experiences can fail as businesses due to high operational costs and external disruptions like COVID-19

Innovative Uses in Broadcasting

• Weather reports use AR to visualize storm patterns, flooding levels, and temperature data in ways flat graphics can't match
• Sports broadcasts implement AR for real-time stats, player tracking, and virtual advertising placed on the field
• News programs use virtual sets to create flexible studio environments that can change instantly
• Nature documentaries employ AR overlays to display information about animals and ecosystems
• Election night coverage uses AR for data visualization and real-time results mapping, turning abstract numbers into spatial graphics

Production Challenges

Producing VR/AR content involves obstacles that traditional TV production doesn't face. Writers benefit from understanding these constraints because they directly affect what's feasible to put on the page.

Technical Limitations

• High-end VR requires powerful hardware, which limits your potential audience
• AR tracking can be inconsistent in varying lighting conditions or complex environments
• Rendering realistic graphics in real time remains challenging, especially on mobile devices
• Motion sickness in VR requires careful design. Rapid camera movements, low frame rates, and certain locomotion methods can cause nausea
• File sizes for VR content can be enormous, creating distribution and storage challenges

Budget Considerations

• VR/AR production requires specialized equipment and software, increasing costs significantly
• Development time for interactive experiences typically runs longer than traditional linear content
• Testing and iteration cycles are essential for VR/AR (you can't just watch a rough cut; you have to experience it), which extends production schedules
• Training crew members in new technologies adds to expenses
• Marketing VR/AR content often requires unique strategies since you can't easily convey the experience in a traditional trailer

Talent and Crew Adaptation

• Actors must adapt to performing in virtual environments without traditional visual references. Acting against a green void is disorienting
• Directors learn new techniques for staging in 360-degree environments where there's no "off-camera"
• Cinematographers explore new tools for framing and movement in virtual spaces
• Editors develop skills in cutting immersive content while maintaining spatial continuity (jarring cuts can cause disorientation)
• Writers collaborate more closely with designers and developers throughout production, since the script and the interactive design are deeply intertwined

VR/AR vs. Traditional TV

Understanding how VR/AR differs from traditional TV helps writers make informed decisions about which medium best serves a given story.

Viewer Experience Comparison

• VR provides full immersion; traditional TV offers a window into another world. Both are valid, but they create very different emotional relationships with the content
• AR enhances the viewer's real environment, while TV content stays confined to a screen
• Interactivity in VR/AR gives users agency, contrasting with the passive nature of traditional viewing
• VR can create a sense of presence and embodiment that 2D screens simply can't replicate
• Traditional TV benefits from established viewing habits. People know how to watch TV; VR/AR still has a learning curve

Production Workflow Differences

• VR/AR production often uses real-time game engines (Unity, Unreal Engine) instead of traditional editing software like Avid or Premiere
• 360-degree filming requires different camera rigs and consideration of the entire environment, including what's behind the crew
• Post-production for VR includes stitching footage, spatial audio mixing, and integrating interactive elements
• AR production involves creating digital assets that blend seamlessly with real-world environments
• User experience testing plays a much larger role than in traditional TV, where you don't have to worry about whether the viewer can figure out how to "use" the show

Monetization Strategies

Monetization for VR/AR content is still evolving:

• Subscription models for VR platforms (Meta Quest) mirror streaming service approaches
• In-app purchases and virtual goods sales are common in interactive experiences
• Location-based installations charge premium prices for high-end experiences
• Advertising in VR/AR can be more interactive and integrated into the virtual environment (a billboard in a virtual city, for example)
• Branded content offers new sponsorship opportunities, where a brand's product can exist naturally within a virtual world