Apple WWDC 2023: Apple details visionOS, the software that powers the Vision Pro headset

Apple's Vision Pro mixed reality headset will run on visionOS, company executives announced following the bombshell reveal of its long-rumored wearable at WWDC 2023. The operating system, internally codenamed “Oak,” has reportedly been in development since 2017. It's existence further leaked via source code references last February. Behold, the dawning of the spacial computing era.

While visionOS is based on the existing MacOS and iPadOS platforms, the unique nature of spacial computing requires lower latency than one would need for a mobile or desktop OS. The new headset will an immersive mixed-reality 3D interface that, “frees apps from the boundaries of a display,” according the the company. This means that instead of being displayed in a separate virtual environment, the app windows will appear to float in the physical space in front of the user can be moved and scaled just as they would on a desktop — except now, it can be your actual physical desktop, not just your laptop's home screen. 

The virtual screens Vision Pro displays can appear as gigantic as 100-foot wide wraparound billboards or they can fit into the space of your living room. The same applies to FaceTime calls made with the new headset, which leverages spacial audio to display a life-sized tile image of the speaker positioned where they would be in the room. If the person is speaking from your right, their tile will be on that side of the FaceTime display. 

What you'll see when you look over there is not the actual person you're talking to, but rather a Persona, “a digital representation of themselves created using Apple’s most advanced machine learning techniques.” It's an avatar of what the person looks like without three pounds of screen strapped (however comfortably) to their face. These digital avatars will reflect their user's hand and facial movements in real time.

The system responds to the room's natural light and shadow to provide the user with a better sense of scale and distance. The system's new and novel Eyesight feature will adjust the opacity of the user's surrounding view to increase immersion but automatically clear the visor when another person approaches, allowing each person to look the other in the eye, without having to take off the headset.

Users won't need cumbersome controllers or motion-sensing wands to use the Vision Pro, as the headset utilizes nearly dozen cameras and sensors peppered around the device to monitor their gaze and hand motions turning their eyes, voices and fingers into virtual pointers and clickers. You'll be able to browse through menus by looking at them, selecting items by tapping their fingers and inputting text with the spoken word.

Vision Pro's applications will skew hard towards gaming, media consumption, and communication and will offer Apple apps like Messages, FaceTime and Apple Arcade — the latter of which will offer more than 100 playable MR gaming titles at launch. Apple is already working with a number of media companies to bring their products and content into the new Vision Pro ecosystem. This includes Disney which, as part of its 100th anniversary celebration, announced Monday that it will bring immersive features to Disney+ content, “by combining extraordinary creativity with groundbreaking technology,” Disney CEO Bob Iger said. “Disney+ will be available ‘day one,' [of the headset's availability].” It appears that ESPN content won't be far behind, based on the few glimpses we saw during the demo.

Apple's announcement comes just days after rival Meta unveiled its own mixed reality headset, the Quest 3. The Vision Pro is slated to go on sale next year and retail for $3,499.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission. All prices are correct at the time of publishing.

Source