Oculus Claims To Have Mastered Hand-Tracking Accuracy For Better VR Immersion

There are many components to a fully realized virtual reality experience. The visual aspect is obviously the most important piece of the puzzle, but it's far from the only one. Accurate hand tracking is key as well. While hand tracking is pretty good in its current form on systems like the HTC Vive and Oculus Rift, there is room for improvement, and Oculus believes it's come up with a breakthrough of sorts.

At Facebook's F8 conference this week, Maria Fernandez Guajardo, head of product management on Core Tech at Oculus, talked about a computer-vision based hand tracking system that uses a self-optimizing machine learning algorithm. According to Guajardo, this method produces hand tracking that is "far more accurate than any method before for tracking a single hand, two hands, and hand-object interactions." Here's a look at the system in action:


While the video demonstration isn't mind blowing at first glance, it's the details that tell the story. For example, according to Oculus, the 'Tracking Success Rate' (or accuracy) is far better using the company's machine learning method. In single-hand operations, Oculus claims to have achieved a 100 percent success rate, versus 90.49 percent for other methods.


Both two-handed and hand-to-object tracking are not far behind at 99.29 percent and 98.26 percent, respectively, but what's really impressive is how both of those compare to other solutions. There is a sizable gap in accuracy when moving beyond simple single-handed tracking scenarios.

How this works is that Oculus uses a marker-based tracking system to record hand movements and interactions in high detail. That information is then condensed into 2D, which is fed into a convolutional neural network. With that information on hand (pun intended), the neural network effectively learns what a hand should look like in different scenarios based on a set of markers across a large set of hand imagery. Applied to a VR headset, the system is better equipped to track hand movements based on what the camera is able to capture.

It's not clear if this will be applied to the Rift at some point, or further down the line an another headset. With HTC having released its Vive Pro recently, we imagine Facebook will not be far behind in launching the next version of its Rift, though nothing has been announced yet.

Thumbnail/Top Image Source: Facebook via Road to VR