(Bloomberg) — Apple Inc.’s long-awaited mixed-reality headset is an ambitious attempt to create a 3D version of the iPhone’s operating system, with eye and hand tracking systems that could differentiate the technology from competing products.
The roughly $3,000 device, due later this year under the likely name Reality Pro, will take a new approach to virtual meetings and immersive video, aiming to shake up a VR industry currently dominated by Meta Platforms Inc. It’s a high-stakes move for Apple, which is expanding into its first major new product category since releasing a smartwatch in 2015, and the company needs to impress consumers.
Apple is pushing into an uncertain market with a premium product. The company’s 1,000-plus-person Technology Development Group has spent more than seven years on the project, and Apple expects it to become a new source of revenue — especially with sales growth set to stall this year.
But virtual reality has proven to be a challenge for tech’s biggest titans. Although some projections show the industry will top $100 billion by the end of the decade, headsets are still seen as niche items — and Meta has lost billions on its efforts.
Apple’s goal is to bring something new to the table. The eye and hand tracking capabilities will be a big selling point for the device, according to people familiar with the product, which is expected to cost about twice the price of competing devices. Its core features will include advanced FaceTime-based video conferencing and meeting rooms.
The headset will also be able to display immersive video content, act as an external display for a connected Mac, and replicate many features of iPhones and iPads.
Here’s how it will work: The headset will have multiple external cameras that can analyze a user’s hands, as well as sensors inside the gadget’s casing to read eyes. It allows the wearer to control the device by looking at an item on the screen – whether it’s a button, app icon or list item – to select it.
Users will then pinch their thumb and index finger together to activate the task – without having to hold anything. The approach differs from other headsets, which typically rely on a hand controller.
Like Meta’s latest headset, Apple’s device will use both virtual and augmented reality. With VR, users see images and content in the glasses. AR, on the other hand, overlays digital content on top of real-world views.
The headset will have two ultra-high-resolution screens – developed with Sony Group Corp. – to handle VR and a collection of external cameras to enable an AR “walkthrough” mode. This means that users will see the real world through the cameras placed on the headset. Apple will offer users with prescription glasses custom lenses that sit in the housing itself.
The device will have a so-called Digital Crown – like the Apple Watch – that allows users to switch between VR and AR. In VR, the wearer is fully immersed. When AR is enabled, the content is faded back and surrounded by the user’s real world environment. Apple expects this to be a highlight for the product, according to the people, who asked not to be identified because the project is still under wraps.
Given that the headset is still months away from its release, some features may still be discontinued or changed, the people added. An Apple spokeswoman declined to comment.
The headset’s FaceTime software will realistically reproduce a user’s face and entire body in virtual reality. These avatars will allow two people – each with an Apple headset – to communicate and feel like they are in the same room. The technology differs from virtual meeting rooms on Meta’s headset, creating a more cartoonish avatar of the user.
Due to the massive processing power required for the feature, the headset will only support realistic avatars during one-on-one video chats. It will still allow FaceTime sessions with multiple people, but additional users will appear as an icon or Memoji – Apple’s custom emoji.
Apple plans to unveil the device as early as this spring, though the schedule could still change, according to the people. That would let the company discuss the product at its annual software developer conference in June and then release it later this year.
As with some of Apple’s previous big ventures, the company plans to start slowly. It aims to start early production of the device as soon as February in China and is considering launching the product in the US just to start. The price tag is also expected to limit the product’s appeal, and Apple is already working on a cheaper version — due for release in late 2024 or early 2025 — that could be closer to $1,500. That’s what Meta charges for its mixed-reality headset.
Apple expects to sell around 1 million units of its new headset in the first year. That compares with more than 200 million units for the iPhone, the Cupertino, Calif.-based company’s biggest money maker. In a rare move, it also doesn’t plan to make a profit on the original version — even at the high price — indicating that the company has a long-term view of the platform.
Immersive video viewing will be a core feature of the new device. Apple has discussed developing VR content for the platform with half a dozen media partners, including Walt Disney Co. and Dolby Laboratories Inc. And the tech giant is working to update its own Apple TV+ content to work with the headset. As part of the venture, Apple bought streaming company NextVR in 2020, with the aim of creating sports content in VR.
Apple plans for the headset to have a dedicated video-watching feature that could make viewers feel like they’re watching a movie on a giant screen in a different environment, like a desert or outer space. But while the headset’s video will be immersive, its speakers will be less powerful. So users will need to wear AirPods earbuds to get full spatial sound – a surround sound effect.
The device will also have productivity features, including the ability to act as an external monitor for a Mac. With that feature, users will be able to see their Mac’s screen in virtual reality but still control the computer with their trackpad or mouse and physical keyboard.
The headset’s operating system, internally called xrOS, will have many of the same functions as an iPhone and iPad but in a 3D environment. That includes the Safari browser, Photos, Email, Messages, and the Calendar app. And it will also have apps for the company’s services, such as the App Store for installing third-party software, Apple TV+, music and podcasts. The company is also working on health tracking features.
The experience should feel familiar to Apple users. When they put on the headset, the main interface will be almost identical to that of the iPhone and iPad, with a home screen with a grid of icons that can be rearranged. Users will be able to pin widgets, such as weather, calendar appointments, email and stock market performance, among their app icons.
When users need to enter text, they can use the Siri voice assistant or rely on an iPhone, Mac or iPad keyboard. Unlike an Apple Watch, however, an iPhone is not required for operation. The company is developing technology that allows users to write in the air with their hands, but such a feature is unlikely to be ready for the initial launch.
Gaming is expected to be a popular offering from third-party developers, and Apple has created its own underlying engine to power VR games. In 2017, the company released ARKit and other tools to help developers prepare augmented reality experiences on the iPhone. This helped create the conditions for programmers to build apps, games and services for the headset.
The Apple device will include a variant of the M2 chip found in the company’s latest Macs, as well as a dedicated processor for graphics and mixed reality experiences. The second chip will be called the Reality Processor, according to trademark applications filed by the tech giant.
But making the processors powerful enough raised another concern: the device overheating while on a user’s face. To solve that problem, Apple made the decision to offload the battery from inside the headset to an external pack. It rests in the user’s pocket and is connected via a cable. Another tweak is the inclusion of a cooling fan like on high-end Macs.
The headset can last around two hours per battery pack, in line with competing products. The battery is big, though: about the size of two iPhone 14 Pro Maxes stacked on top of each other, or about 6 inches tall and more than half an inch thick. Nevertheless, some internal prototypes for software development have a built-in battery and charge via USB-C.
The Meta, on the other hand, places its batteries on the back of its headset in a way that helps balance the device on a person’s head. Apple’s approach may ultimately be less convenient for users, especially if they’re watching an entire movie while wearing the headset. Some testers have complained that the product can be cumbersome, according to the people.
The relatively short battery life – about 20 hours less than Apple’s latest MacBook Pro – can create its own problems. If users want to watch multiple movies or play games for hours at a time, they may need to purchase multiple batteries and replace them frequently.
Apple has acknowledged these challenges internally and has tried to set realistic expectations for the product. One benefit of the device, the company believes, is that it can spur customers to visit Apple stores — not necessarily to buy the product, but to try it out. They can then buy another device, such as an iPad or AirPods.
To showcase the new headset, Apple is creating a “store within a store” concept – an area within its stores dedicated to demonstrating the product. The company did something similar when it launched the Apple Watch, which is now central to a $41 billion division.
The first headset will be made of aluminum, glass and cushions – reminiscent of Apple’s $550 AirPods Max headphones. The product will have a curved display on the front that can outwardly show a wearer’s eyes, with speakers on the sides and a headband that will help fit the device around a user’s head.
It will differ from the mostly plastic design of competing products, which usually attach the device to the carrier with several straps.
Eye and hand tracking may end up being the most memorable part of the headset. As with its previous big bets, Apple likes to include a cutting-edge interface that sets its products apart from the competition. With the iPod, it was the click wheel. With the iPhone and iPad, it was the multitouch method. And with the Apple Watch, it was the Digital Crown.
Now Apple hopes the headset’s sci-fi-like interface will make its latest product a winner.
©2023 Bloomberg LP
#Apples #upcoming #Mixed #Reality #headset #work #BNN #Bloomberg
