The #1 XR Blog in the World
- English
- Sign In
Hand tracking in virtual reality allows users to interact with virtual objects and environments using their hands. This technology, while simple in real life, is incredibly sophisticated for virtual reality. When achieved, the virtual reality experience gets more immersive and the VR more compelling. What in actuality is hand tracking and how does it work?
In this article you will discover:
Hand tracking is when the headset cameras, LiDAR array, or external sensor stations track your hand’s position, depth, pace, and orientation. This tracking data is then analyzed and translated into a virtual, real-time representation of your hands and their movements inside the virtual world. This representation is then sent to whatever application or video game you are playing, enabling you to naturally interact with the world using your hands.
There are two ways of tracking your hands in virtual reality: from the headset itself or external sensors.
When the tracking originates from a camera array, it is called optical tracking.
Optical hand tracking uses the difference in pixels to determine your hands’ movements. As a result, it requires more processing for software to translate into your virtual hands.
Likewise, optical tracking is bound by the field of view of the cameras (i.e., the active tracking area), their sensitivity to light, and their capabilities for capturing motion (frame rate). Usually, optical tracking is cheaper to install, while a camera array also can enable video passthrough for the wearer.
Alternatively, LiDAR tracking (light detection and arranging) pulses lasers to determine the distance, position, and movement of the environment relative to your hands. This tracking method is more precise, as lasers are faster than a camera’s assessment of pixels, while the bounce-back of the laser into the LiDAR sensor immediately generates a 3D representation of what is tracked.
Alternatively, the tracking can also be done by infrared LEDs, which work the same way as LiDAR. Infrared LEDs, called IR, combined with optical, camera tracking produces high-fidelity tracking. A solution Ultraleap is doing.
For example, Varjo’s XR-3 headset uses LiDAR depth sensing in combination with optical tracking to achieve precision hand tracking. While an IR accessory is included in the hand tracking setup for the Pico Neo 3 Pro and Pro Eye accessory kit.
Because LiDAR uses bouncing lasers – similar to how echolocation works – the quality of tracking is not lost based on distance. Oppositely, because cameras use the changes of pixels to determine movement and position, the quality of the pixels diminishes based on the distance between the camera and your hands.
Now that the types of tracking are known it is important to understand the different types of hand tracking systems used in virtual reality
Tracking originating from the headset is inside-out tracking and is found in standalone VR headsets. For this to function, either a set of cameras or LiDAR sensors on the headset keep track of your hands in relative position to everything, crunching this into raw positional data.
Inside-out tracking works similar to a lighthouse in that a 3D representation of the world must be generated from only one direction. This lessens tracking quality and precision in comparison to outside-in tracking but frees up the wearer to not be constrained within a designated ‘tracking zone’.
When tracking is conducted from multiple external sensors (usually LiDAR or IR technology) forming a ‘tracking zone’, it is called outside-in tracking.
Outside-in tracking can generate a much more precise representation of a person’s position and movements (hands included) from external sensors and the headset itself. A 3D map of your hands within the tracking zone is much easier to compute as your hands are tracked from multiple angles.
Hand tracking lowers the barriers to virtual reality adoption because natural interaction is possible. It improves communication within the virtual world because we talk with hand gestures and it creates an incredible sense of immersion.No matter the hand tracking solution used, it is a critical part of virtual reality. Soon gloves and controllers will step aside for natural hand interactions, and we are here for it.
