Magic Leap 2 controllers reveal internal inside-out tracking

Twitter strikes again. This time it’s Peter Diamond, founder of XPRIZE, revealing previously unknown controllers for the upcoming augmented reality headset Magic Leap 2, showing signs of built-in cameras used for inside-out tracking.

Magic Leap 2 leaked controllers show novel inside-out tracking feature

Previous to this reveal, the upcoming AR headset from Magic Leap hinted at controllers featuring built-in cameras inside-out tracking for higher precision tracking. However, at that time, statements were vague and we didn’t know that it was from built-in cameras in each controller.

Such controller-tracking is unique, as all current AR and VR offerings are either tracked by the headset (called inside-out, but referred to the headset; not the controllers) or require external sensors (called outside-in tracking).
Essentially, implementing more sensory input through sensor systems increases the precision and fidelity of the tracking itself, delivering better immersion. Alternatively, headsets adopting inside-out tracking of controllers makes them stutter or break if moving out of the headset’s tracking range.
So before, you had to buy outside-in virtual reality systems with cumbersome setups such as HTC Vive and PlayStation VR.

How does the tracking work and what does it require?

The upcoming controllers linked with Magic Leap 2 use its built-in cameras and a software system called SLAM. Simultaneous location and mapping, or SLAM, compares the changes in speed, rotation, and acceleration to visible objects around you relative to the cameras/controllers. This live data is then calculated and sent to the headset visor, resulting in the tracking of the controllers. Quite impressive.

There is one problem, it requires high-performing processing power to run, which means that SLAM is difficult to implement in standalone headsets. This creates a further problem with augmented reality headsets which pride themselves on being slimmer and lighter. Potentially this issue can be addressed with an in-house system-on-a-micro-chip (SoC), similar to Apple Silicon and Google Tensor, as this lends itself to higher battery efficiency and processing power per nanometer optimized to the local operating system.

Furthermore, it is questionable how SLAM controller-tracking will work in low-lit rooms as the built-in cameras potentially struggle to calibrate items around the wearer, not to mention motion blur.

In any case, Meta’s upcoming Project Cambria headset is seemingly also adopting this innovative inside-out controller-tracking, so it is a development to keep one’s eyes on.

Conclusion

With the Magic Leap 2 controllers revealed to have built-in cameras for inside-out tracking, it will be interesting to see how well the headset handles augmented reality experiences with this controller technology. As doubts still linger about whether or not the controllers will work well in low-lit rooms, how they will manage motion blur and how Magic Leap will address the processing power needed to realize this innovative tracking feature.

Albeit Meta’s upcoming Project Cambria headset is said to adopt similar controller tracking, it could potentially set Magic Leap 2 apart from other AR headsets and position them back as a leader in augmented reality.