What is pass-through, and why do all VR headsets have it?

VR headsets released from late 2022 and now in 2023 have high-resolution full-color pass-through boasting impressive mixed reality features. What is pass-through technology and why, all of a sudden, do VR headsets have it?

In this explainer article, we will cover:

• What is VR video pass-through?
• Is video pass-through the same as mixed reality
• Why do VR headsets have pass-through?
• Which VR headsets have mixed reality?
• The future of pass-through is Neural Passthrough

What is VR video pass-through?

In a nutshell, video pass-through on VR headsets is a video feed sent to the displays inside the virtual reality headset. This video feed can be grey scale seen in Meta/Oculus Quest 2 or full-color as on Pico 4.

Before the surge in mixed reality from late 2022 onwards, video pass-through was for the VR user to draw the VR guardian or the tracking zone where the virtual world can be activated safely.

The mixed reality from video pass-through can be enabled with a single or dual-camera setup. When done through two cameras, it is called stereoscopic pass-through, delivering one video feed per eye. In this way, it mirrors how humans see, improving the mixed reality experience with better depth, distance perception, and less jittery 3D objects. In short, our brain composes how we see the world through the overlap our two eyes see, this overlap is called parallax, and MR headsets with a dual-camera setup use the same method.

Pass-through allows for mixed reality

At its core, whether virtual, augmented or mixed reality, the technology is on a path to become smaller, lighter and used as an everyday wearable. With this in mind, VR is set to bridge the gap between VR and AR through high-resolution and color video pass-through. This is called mixed reality and delivers a seamless transition from fully immersive VR to a blended reality in MR.

In fact, when hand tracking and 3D object occlusion are utilized, the digital layer in the real world seems like it is part of the environment, delivering an unparalleled mixed reality experience. An early VR headset to portray the power of mixed reality is Lynx R1. So the VR headset user can then see their environment in high fidelity blending it with digital objects, opening numerous use cases.

Why do VR headsets have pass-through?

VR headsets feature pass-through – outside of enabling a tracking zone to be drawn – because mixed reality enables more possibilities for VR headsets.

Here are a few examples.

Entertainment

In entertainment, the VR wearer can replace their physical wall with a screen to watch a movie. They can even watch it together with their family also wearing a VR headset with mixed reality pass-through.

Healthcare

In healthcare, MR pass-through can create a real-time hologram of the patient’s organs before an operation, increasing the effectiveness of the coming surgery. In fact, low-latency pass-through even enables decentralized healthcare, where a surgeon in Boston can operate on a patient in Paris.

Architecture

An architect can design and draw a virtual schematic in immersive virtual reality to then shrink it down, seamlessly entering mixed reality, for engineers and interior designers to collaborate and iterate on the design.

Remote work

In a home office, the worker can recreate a professional meeting room or work on multiple virtual screens simultaneously with their physical desk setup.

In short, while video pass-through is its own technology, it has become synonymous with mixed reality.

Which VR headsets have mixed reality?

The future of pass-through is Neural Passthrough

In line with VR and AR merging, we got a peek into what Meta is tinkering with inside Reality Labs.

Here, besides the remarkable optics innovations of the “Sandwich Principle”, Meta is also developing a reverse video pass-through technology called Neural Passthrough. This solution enables the VR headset wearer’s eyes to be visible to outsiders using AI eye and face tracking to portray the eyes realistically.

However, for mixed reality pass-through to truly become seamless, VR headsets must include a depth sensor to spatially map the user’s environment. As current mixed reality headsets like Meta Quest Pro don’t feature a depth sensor, interaction with 3D objects through the video pass-through is jittery, but without doubt seamless mixed reality is about to become a reality.