The #1 XR Blog in the World
- English
- Sign In
High-fidelity hand-tracking for VR, and namely AR, is not an added feature – it is pivotal. But it is especially the case with enterprise use cases. VR is still a new technology. So natural interaction – or Skeuomorphism – is needed to lower the barrier of entry and increase the adoption of VR to enable the many use cases of enterprise VR/AR.
It is no secret that the medical and health care industry sees great potential in VR. But with fine-tuned hand tracking, world-class surgeons, released from physical constraints across the world, can operate a set of nimble robotic hands to operate on people. Effectively spiking the level of medical sophistication decentralizing health care.
Research & Developments departments everywhere are self-explanatory for their keen eyes on VR/AR. But specifically, hand-tracking would be greatly adopted and pivotal in “Design Reviews“. Here engineers can get an intimate feel for the design and iterate to improve upon the project. While C-suite execs get a deeper view into their future product line-up and, by extension, make more sound business decisions.
One industry to highlight is the ergonomic sector. In designing ergonomic products like sports clothes, it is crucial not just to use hand-tracking for the development, but also the ability to hand-track product testing.
Bomb-disarming and other disaster-related tasks perk their ears when hand-tracking news comes out. The awe-inspiring robotics company Boston Dynamics has mentioned that their robots are for situations that pose too dangerous for humans to operate. Scenarios like these can be environmental disasters, structural collapses, and bomb disarming. To maneuver hands without physically being in a place will be highly beneficial and open many use-cases.
Next it can be seen as more of an add-on–an extra feature to highlight to close a deal–but would result in greater enterprise VR adoption if implemented: body language and hand-movements in meetings and presentations. Human expression and hand gestures while we speak are paramount in natural communication to present or discuss particular topics and with decentralized meetings, hand-tracking will normalize the use and broaden the adoption rate.
Lastly and the most important is hand-tracking in AR. Augmented reality is for daily use, and in such scenarios, people are not going to wear clunky controllers to operate AR. Therefore, when interacting with the real world, hand-tracking is crucial in the success of AR.
Skeuomorphism is a natural user interface. It is a design and User Interface term for mirroring the real world around us and leveraging our natural human interaction with physical objects. Think of when you interact with a Kindle Reader™, here you swipe from right to left as you would in turning a page of a book. But skeuomorphic design can also be in the look of objects. For example, using a recycle bin logo where discarded files are sent – even deleting files using the “drag and drop” method is considered skeuomorphic.
Usually, the benchmark for a perfected skeuomorphic user interface is that if a user can operate what is in front of them without prior knowledge about what they are interacting with, nor with any onboarding, then that design is a successful Skeuomorphic user interface.
This user-friendly design aims to lower the barriers of entry into novel innovations and systems. And hand-tracking for VR/AR is considered a skeuomorphic practice. It means that VR in enterprise settings must be effortless and intuitive, while being hidden and excel that of alternatives. The best designs are unnoticed.
In a sense, it is about bridging our digital world with our prehistoric evolutionary abilities. We have used our hands since we became bipedal animals, and our tools have advanced ever since. So to make VR natural enough to become effortless for enterprise use, sophisticated hand-tracking is not only needed: it is pivotal.
Likewise, it is helpful to highlight another term of “Mental Models”. Everyone wants to be disruptive and innovative. But we interact with products based on pre-existing expectations of how something should interact. For example, we would be annoyed if a website only scrolled from right to left, rather than top to bottom.
As design expert, Scott Benson writes, “users will transfer expectations they have built around one familiar product to another that appears similar.” A common joke is that a toddler brought up with swiping screens from smartphones and tablets will try to swipe on the TV. This is their Mental Model at work.
The problem is that VR/AR is still so novel that most people, outside of developers and experts, do not have Mental Models to interact with VR/AR. So grounding it in hand-tracking paved its way from touch-control and natural interactions are paramount for its success.
Currently, the mainstream headsets of the Quests have native hand tracking but are still in their infancy and are very crude. Facebook even deemed it a casual feature rather than an enterprise-essential one. Besides that, you have to venture outside the big VR manufacturers and into more niche VR/AR offerings, such as ManoMotion (Pimax) and Ultraleap Gemini (Varjo headsets), while more upcoming headsets are coming with it.
One to highlight in this regard is the Ultraleap Gemini hand-tracking technology from Varjo, implemented in the Varjo XR-3 MR HMDs. Through these premium quality mixed reality headsets design workflows are optimized and cost-cutting.
Ultraleap Gemini is the fifth iteration of Varjo’s hand-tracking software, and is the industry standard when it comes to camera-based, discriminative hand tracking.
There are two forms of hand-tracking: genuine tracking of hands, and gesture recognition. Today hand-tracking usually is gesture recognition enabled from two different systems: the Discriminative approach, and the Generative approach. The Discriminative method takes still frames from the tracking of the hands to correspond it with a database of still images of hands and determines what the hand gesture is. This database can usually be expanded upon and fine-tuned through machine learning and iterative algorithms. The generative approach is similar, other than a virtual 3D hand is created based on your hand movements and corresponded then to a base 3D hand model in the system. The former is more advanced, and the latter is faster.
Likewise, the tracking of one’s hands can be done in two distinct ways: device-based hand tracking versus camera-based hand tracking. The latter is when a camera on the HMD tracks your hand’s movements, whereas a device-based tracking requires some form of wearable, like a glove or finger nubs.
The camera-tracking method removes barriers of entry entirely but is harder to track the hands. Whilst the device-based one requires the user to constantly wear gloves or finger-nubs with them, so everyday use cannot be achieved. But these devices enable a much higher-fidelity tracking.
In either case, hand-tracking is more significant than most make it out to be–especially in enterprising use-cases.
