What you need to know
-Apple is experimenting with ideas around how to best control augmented and virtual reality experiences. - Device patent points to wearable finger controllers. - Smartly, the design leaves fingertips free to use touchscreens.
Apple has long been rumored to be making a virtual reality / augmented reality device, and though its summer software showcase failed to reveal the goods last week, a newly uncovered patent shows Apple's continued exploration of the mixed reality space.
The Apple VR/AR headset may not have been revealed at WWDC 2022, but the patent (number 11,360,558, filed with the United States Patent Office) reveals a set of devices worn like thimbles around a user's fingers, designed to control virtual and mixed reality experiences, and interact with objects within them.
The patent describes arrays of one dimensional and two dimensional sensor elements, taking advantage of capacitive touch sensors, cameras and sensors tracking optical and ultrasonic data. Force and inertial movement can also be tracked. Essentially, you've got something that will let a wearer control elements in 3D space – the VR or AR equivalent of a pointer or mouse, similar to VR controllers we see with Oculus / Meta Quest devices at present, but reduced to their essential elements so as to fit on a user's finger, letting gestures, movements, and inputs be tracked through the air.
The patent specifically references viewing this content through a 'head-mounted device', with haptic feedback included to help a wearer understand a sense of presence in the digital world and through its interactions. It also describes how a connected AR system could better recognise real-world objects through their interaction and proximity to the finger-worn devices, and as such overlay differing information onto physical objects through a headset or similar display based on their rotation and angle. Interestingly, the patent leaves the wearer's fingertip exposed – presumably to keep them free for interacting with a touchscreen device like its best iPhones, or a MacBook's keyboard, anchoring a wearer to their real-world devices.
Pointing a finger into the future
Apple is on the cusp of entering a busy AR/VR space, with lots of competition from the likes of Meta, Sony PlayStation, Snap and other companies betting bigger on the coming importance of the metaverse. Innovation will be key to standing out, so it's good to see Apple experimenting with fresh ideas.
It's interesting to see Apple considering finger based devices too, given that Meta, for instance, is working hard to make hands-free hand and finger tracking the eventual norm, using a headset's onboard camera and space sensors to identify digits and hands and have them tracked accordingly.
But it's a sometimes hit-and-miss affair, and no system has yet managed to offer hands-free control with the same level of accuracy that a held device, be that wand or controller, can offer. Apple's patent here seems to be a concessionary step between both solutions – small enough to be unobtrusive to an immersive mixed reality experience, yet with enough sensors and tech to augment what a person's fingers can achieve.
As ever, a patent is no guarantee of a device ever making it out of the idea phase – it's merely to protect Apple's intellectual property. But it's an encouraging sign as to what Apple may well be cooking up for its augmented and virtual futures.
