Apple's Vision Pro is an "over-engineered development kit", and the former person in charge of Meta Oculus sent a 10,000-word summary after returning it!

Vision Pro is an over-engineered "development kit";

There's genius and audacity in its hardware, but it's a depressing story in the software;

The mistakes we made at Oculus, Apple did right;

.....

Recently, Hugo Barra, the former head of Meta Oculus, posted a 10,000-word long article on his blog, sharing his personal in-depth analysis of Apple's first spatial computing device, the Vision Pro, and comparing it with Meta's old VR headset Oculus, to see how strong Apple's hardware and software are. The analysis of this article is well worth reading.

Original link:https://hugo.blog/2024/03/11/vision-pro/

Reproduction without permission is prohibited!

Author | Hugo Barra | Meniscus Editor | Xia Meng

出品 | CSDN（ID：CSDNnews）

The following is the translation:

My friends and colleagues have been asking me what I think of Apple's Vision Pro products. I would like to discuss the following topics through this article:

• Why do I think the Vision Pro is an over-engineered "development kit"?
• The genius and bold decisions behind some of Apple's hardware.
• Gaze and pinch is a powerful UI superpower, a major industry epiphany.
• Why is the Vision Pro software/content story so boring, so unimaginative?
• Why don't most people watch TV or movies with Vision Pro?
• Apple's investment in immersive video will revolutionize the live broadcast of sports events.
• Why I returned my Vision Pro with my top 10 wish list.
• Apple's VR debut is a great thing for us.

The Apple Vision Pro is the North Star of the VR industry, whether we admit it or not

I'm a VR enthusiast who interned at Disney Adventures VR in the '90s, was an early supporter of the Oculus Rift DK1 on Kickstarter in 2013, and led the Oculus VR/AR team at Meta from 2017-2020 (and had the pleasure of working with VR legends John Carmack, Brendan Iribe, and Jason Rubin). I always preemptively test every VR product or experience I have access to.

During my time leading Oculus, I often half-joked with our team (and received a lot of criticism for it!) that the biggest piece of good news for us was Apple's entry into the VR industry and its direct competitor to Oculus. I have always believed that strong competition can propel the team to become the best in the industry. In particular, I've been hovering around the battle center of the iOS/Android ecosystem for almost 10 years, so I know better that by constantly raising the bar for user experience, features, performance, developer APIs, and even improving the products released by the other side, both sides can keep improving. (It's definitely a two-way street: between iOS and Android, it's not necessarily who copied more.) ）

But with Oculus' VR, we never really felt like there was a North Star in the world that could truly capture the human mind, and without such a guiding light, it would be impossible to transform VR from a niche gaming technology to the incredible spatial computing paradigm that we have always thought might represent (and I still believe in that very much). If Apple cares about VR, they can really help us.

The launch of Vision Pro is the fulfillment of a dream I've had for years – to create a huge wave of awareness and curiosity, to elevate the spatial computing ecosystem, and ultimately to lead to mass-market consumer demand and more developers interested in VR. Today, our industry needs to create enough user value and prove that this is really the future of computing.

The instant magic of the Vision Pro boils down to two things: (1) VR has reached new heights like never before, and (2) new UI superpowers that use gaze and pinch

Whether you've used other VR headsets or not, using the Vision Pro is an immediate, magical, and intuitive experience that's because Apple has relentlessly focused on delivering two specific features, both of which are in our human nature:

1) Feel connected and blended with the real world: Thanks to the high-fidelity through-through ("mixed reality") experience, latency is very low, distortion correction is excellent (much better than the Quest 3), and resolution is so high that you can even see your phone or computer screen through the pass-through camera (without taking off your headset).

While there's still a lot of room for improvement in the Vision Pro's hardware (more on that later), this connection to the real world (what VR folks call "presence") is unattainable by any other VR headset, and so far only possible with AR headsets such as HoloLens and Magic Leap, which have physically transparent displays, but have significant limitations in many other ways. Apple's implementation of the optical ID is a layer that floats on top of real-time perspective, and this design is brilliant and enhances this presence.

In my opinion, the Vision Pro's high-fidelity penetrating experience is similar to the iPhone's original Retina display, setting a new experiential standard and gold standard for mobile display fidelity. While there's still a lot to be done on the Vision Pro's penetrating experience, there's no doubt that Apple is setting a new standard for all future headsets, regardless of manufacturer: VR penetration must be so good that it's close to reality.

2) New UI superpowers for gazing and pinching: Thanks to a very precise eye tracking system embedded in the lenses (with 2 dedicated cameras per eye), plus a wide-angle hand tracking system, you can "see" your fingers pinched even when your hands are down or leaning on your thighs. It's so easy for the user to operate, so it really feels like they have a new "laser vision" superpower.

Hardware that tracks eye movements and hands in VR devices has been around for more than a decade, and Apple's unique ability to bring everything together in a magical way makes this UI superpower the most important outcome of the entire Vision Pro product.

In my opinion, the Vision Pro's "gaze + pinch" input mode is the equivalent of the iPhone's capacitive multi-touch gestures. Apple introduced multi-touch in its first iPhone 17 years ago, and it immediately became a new standard that changed the way computing is done. And now that "gaze + pinch" is so breakthrough that it has become the de facto standard for VR interaction, future VR headsets will have to adopt it sooner or later. This breaks the shackles of developers and will spark an eye-opener for us to interact with based on gaze.

hardware

Vision Pro is a well-designed "development kit" that, while too bulky to fit into the market, is enough to spark curiosity around the world

The story of Oculus VR began with the launch of the Oculus Rift DK1 in 2013. The headset was originally launched by the Oculus startup team (a few years before it was acquired by Facebook) with the clear goal of piquing developer interest before it goes commercial. Considering that the VR market didn't exist at the time, releasing a dev kit was the right and necessary strategy to get startups to start building content libraries and build momentum among enthusiasts before releasing consumer products. The team released the DK2 about a year later (2014) and eventually launched the first Oculus Rift consumer headset in 2015.

Oculus 联合创始人 Palmer Luckey 佩戴 2013 年发布的初代 Oculus Rift DK1

I joined Facebook in 2017 after Oculus was acquired by Facebook and am responsible for leading the Oculus team. I soon found myself in the middle of one of the Dev Suite Wars. The Oculus team's DK1 and DK2 are so powerful that it's not uncommon to hear some people arguing at product conferences that we want us to push VR headsets, which are still in the prototype stage, to end users as "dev kits." Since Oculus is no longer a startup and has the resources to test extensively without launching a prototype as a product, there is no longer a need for the Oculus development kit. This stance is not usually welcomed by some of the original Oculus team members.

It's 2024. Following the release of the Vision Pro, the VR hardware enthusiast community (including the original Oculus team members who still keep in touch with me) quickly concluded that Apple's first VR product was indeed conservatively designed and over-engineered. First of all, the Vision Pro is equipped with more sensors than is necessary for the experience that Apple intends to offer. This is a typical phenomenon in the first generation of products that have been developed over many years. This makes the Vision Pro start to feel like a development kit.

Sensor set: 6 tracking cameras, 2 pass-through cameras, 2 depth sensors

(plus 4 unshown eye-tracking cameras)

Here's a quick comparison with existing VR headsets:

This over-stuffing was not surprising and was typical of the first version of the product, and its creators wanted to make sure that the product would stand up to the testing of early adopters, who undoubtedly subjected the product to the most rigorous tests. It's also Apple's way of wanting to see how far developers can push the product's capabilities, and there's no doubt that Apple is relying on this community to build the vast majority of the software/content magic for this new type of computer, as it has done with every other device category before it.

However, Apple's decision to over-stack the Vision Pro inevitably resulted in a headset that weighed more than 600 grams, which weighed more than most other VR headsets on the market to date, making it uncomfortable for most people to wear it for less than 30~45 minutes. Most of the discomfort comes from pressure on the user's face and the back of the head.

I think that due to its heavy weight, the Vision Pro inevitably becomes a high-quality "development kit" whose main purpose is to appeal to everyone's curiosity through its magic (especially through the voices of passionate tech influencers) and to have developers as its primary audience. In other words, Vision Pro is a development kit that prepares the world to embrace a more mainstream Apple VR headset – and it may take 1~2 generations before it fits the market.

With all this in mind, I'm sure that Apple's plan was right, that they prioritized the launch of the first generation at the expense of user comfort. While many people believe that Apple could have avoided the comfort problem by redistributing weight or using lighter materials, these attempts have come at the expense of aesthetics and design.

Against this background, two particularly important decisions made at the time of the launch of Apple's Vision Pro can be understood:

• The main goal of designing an in-store Vision Pro demo experience that would wow people was to experience the magic of VR through Apple's devices as much as possible, and most people didn't even consider buying a $4,000 headset. Actually selling the Vision Pro headset wasn't the main goal of this demo.
• Introducing a signature braided strap that looks great in photos, even if for most people, the strap isn't comfortable on the head. It is easy to conclude that this decision was the right one, since almost all media reports (including especially third-party reviews on YouTube) use a braided belt, despite the fact that it is not as comfortable as the double-loop strap "hidden in the box".

In 2024, the existence of the Vision Pro is entirely due to Apple's successful launch of its first ultra-high-resolution display

From the very beginning, one of the biggest questions within the Oculus VR team about product positioning, especially when trying to convince reviewers, was always inseparable from the poor display. Every headset launched by Oculus, including the latest Quest 3, has issues with too low resolution or noticeable pixelation, some "very bad" and some "a little bad". It's as if we're living in a VR version of the world of VGA computer monitors.

I think that in order for Apple to make a huge splash when it enters the VR market, they need to launch a product that goes far beyond anything that has existed before, after all, this product category has been around in the consumer world for almost 10 years. It's clear that the way to achieve this is to attack the weakness of all existing headsets and reinvent the VR headset, which is exactly what Apple's Vision Pro does.

The Vision Pro is the first VR headset to offer sufficiently good resolution and visual clarity with little to no noticeable "screen effect" or pixel artifacts. This level of realism and fidelity can only be achieved with ultra-high-resolution displays, and there is no doubt that the first time this level of display quality has been achieved is the standard released internally by the Vision Pro.

Apple's relentless efforts and uncompromising hardware madness have largely made it possible for such an ultra-high-resolution display to exist in VR headsets, and it's clear that this product is unlikely to be released before 2024 due to a simple limiting factor: the maturity of micro OLED displays and the presence of an efficient chipset (i.e., M2) capable of performing massive amounts of computation needed to power such displays.

Micro OLED displays are unlike any other consumer display technology that has come before because they are manufactured on a silicon substrate (similar to how semiconductor chips are made). To illustrate the power of the micro OLED display, let's make a comparison: the Vision Pro panel has 7.4 times more pixel density than the latest iPhones and nearly 3 times more pixels than the Quest 3.

Based on the discussion of Apple's Vision Pro micro OLED display that has been heard in many of the hardware supply chain space, many companies (mostly smartphone OEMs) are racing to try to build a product that can provide a Vision Pro-like experience. Apple has signed an exclusive one-year agreement with Sony Semiconductor Solutions Group and its second supplier, SeeYA Technology. There are also rumors that Apple intends to abandon Sony as a display supplier and use BOE instead (according to its website, the panel on the same level as the Vision Pro is in the "sample" stage).

I think the recently announced Meta/LG partnership can create a supply chain advantage for Meta that will allow them to quickly launch the Quest Pro 2 to compete with the Vision Pro, while LG will invest some money to lower the market price of the headset.

Apple deliberately made the Vision Pro's display slightly blurry to hide pixelation imperfections and make the graphics look smoother

The reason why the word "retina" is not used in the marketing materials for the Apple Vision Pro is very clear. To put it simply, the Vision Pro's display doesn't fit the "retina test," which is that the resolution is high enough that the human eye can no longer recognize individual pixels. For VR headsets, the Vision Pro's display is nowhere near retina quality, but our eyes can't see a single pixel when viewing. What's going on?

In the first few days of using the Vision Pro, there was something that caught my eye, but I had a hard time catching (my eyes were hard to see). Everything I saw through the VR device was a little softer than I expected, and I initially attributed this to the Vision Pro's lack of a "screen window effect," which is one of the fatal flaws of any VR headset to date.

Eventually, Karl Guttag (https://kguttag.com/about-karl-guttag/) performed a sophisticated image analysis of the Vision Pro display and came to an intriguing and disturbing conclusion: Apple deliberately adjusted the focus of the Vision Pro display so that the pixels were slightly blurred to hide the screen window effect "in sight."

Here's an image from Karl's blog that illustrates this well. He side-by-side the displays of the Vision Pro and Quest 3 so that we can see the individual pixels, and it's clear that Apple deliberately added a blur effect to the Vision Pro display:

Vision Pro (AVP) 和 Quest 3 (MQ3) 显示器之间的极端特写比较(来源:KGOnTech)

Karl concludes that although the Quest 3's display resolution is much lower than that of the Vision Pro (1,218 PPI for the Quest 3 and 3,386 PPI for the Vision Pro), the Quest 3 displays sharper graphics when displaying high-contrast graphics. In other words, the Quest 3 maximizes the resolution of the display at the expense of a "rough look", while Apple abandons part of the Vision Pro's display resolution in order to achieve a "soft look". My opinion may be slightly different from Karl's.

In my opinion: Apple's deliberate blurring of the Vision Pro's optics is a smart move, as it hides the screen window effect (which in practice means you can't see pixelated artifacts) for smoother graphics across the range. It's also a reflection of Apple's "taste" in tweaking the Vision Pro display to have a distinctive, softer, more refined aesthetic than the Quest 3 (or any other VR headset). This is truly a novel approach to designing VR hardware.

With this design decision, there's no doubt that Apple has sacrificed the Vision Pro display's high pixel resolution to some extent in favor of smoothness of the overall graphics. loss of text clarity in exchange for higher quality images, videos, and 3D animations. This is a huge advantage of micro OLED displays with ultra-high resolution, and Apple has enough pixels to work with to afford the trade-off. Oculus' top VR engineers are definitely against this, and I don't think we'll ever be able to launch a "fuzzy headset."

Unfortunately, the Vision Pro display suffers from severe motion blur and image quality issues that prevent penetration mode from being used for long periods of time

Although Apple's decision to blur individual pixels on the Vision Pro display is very smart, it's a pity that this VR device also has a completely different type of blur, which is extremely detrimental to the overall experience.

From the first time I put on the Vision Pro, I noticed a lot of motion blur in penetration mode, even in good ambient lighting conditions, and there was still some noticeable blur even when watching immersive content. Although my initial instinct was to expect this motion blur to occur on all VR devices, only more noticeable on the Vision Pro, a side-by-side comparison with the Quest 3 quickly turned out to be much more problematic. This is particularly surprising because both the penetrating camera and the display operate at a frequency of 90 Hz.

Since the issue wasn't pointed out in the original Vision Pro review, I even called Apple Support to find out if it was a known issue or maybe even a hardware flaw. But later more in-depth reviews began to point out the same problem.

Motion blur in penetration mode ended up being one of the many reasons I decided to retire the Vision Pro, as it was uncomfortable, caused unnecessary eye strain, and would prevent anyone from using the headset for long periods of time in penetration mode.

There are a few other noticeable issues that also affect penetration mode, including low dynamic range, incorrect white balance in most indoor use, and signs of edge distortion and chromatic aberration. Some of these issues may be fixed with a software update, but I estimate that most of them will not be fixed because they are caused by hardware limitations.

The Vision Pro is equipped with more computing power than most people think: the combination of M2 + R1 brings it to the level of the MacBook Pro

All stand-alone VR headsets are essentially 2-in-1 systems: a regular "computing" computer and a spatial computer bundled together.

• A regular computer is responsible for running applications and performing general calculations: everything on a smartphone, tablet, or laptop, including running the operating system, executing applications under CPU/GPU load, and performing computational work in the background.
• The spatial computer is responsible for the environment: it tracks the entire environment, tracks your hands and eyes, and ensures that everything (the surrounding environment, the OS system UI, and the app) is rendered in the correct physical space position and updated at a rate of 90~120 times per second as your head and body move.

These two "computers" must work seamlessly together, and in VR, any delay of more than 20 milliseconds is noticed, and tends to quickly cause the user to perceive unresponsiveness or instability, which can cause discomfort, eye strain, and even dizziness for many people.

Vision Pro Dual-Chip Design:

"The unique dual-chip design has helped Apple Vision Pro achieve a spatial experience. The powerful M2 chip simultaneously runs visionOS, executes advanced computer vision algorithms, and delivers stunning graphics with extreme efficiency. The new R1 chip is dedicated to processing input from cameras, sensors, and microphones, streaming images to the display in less than 12 milliseconds, enabling a virtually lag-free, real-time view of the world. ”

——摘自苹果 Vision Pro 官网

The Vision Pro comes with the same M2 chip as the 2022 iPad Pro (and the 2022 MacBook Air), as well as the new R1 chip that processes a lot of data from more than 20 tracking cameras and depth sensors. Note that the Vision Pro's performance in benchmarks is very similar to that of the iPad Pro, with both pushing the CPU and GPU to the limit in both single-core and multi-core scenarios (see chart below).

This result is even more impressive than expected, and it shows that the R1 chip takes on most of the very important spatial computing workload (basically the vast majority of the spatial computing workload), giving the M2 chip a lot of computing space, allowing it to provide a level of performance comparable to running in the iPad Pro.

So far, according to all reports, the R1 chip is packaged in a package size comparable to the M2 chip (albeit with a dedicated architecture), pushing the Vision Pro to far outperform all current iPads or MacBook Airs, and closer to the silicon performance of the MacBook Pro. This is definitely a great achievement for the Apple Silicon team.

This also begs the question: what happens if you transfer the Vision Pro's calculations to another Apple device?

As a result of Apple's decision to use an external battery pack, future Vision series headsets will be lighter as they can offload computing tasks to the iPhone, iPad, or MacBook.

One of the most controversial aspects of the Vision Pro is that, unlike all other commercial standalone VR headsets, it comes with an external battery pack. Many people have harshly criticized Apple's decision because of the inconvenience caused by an external battery hanging on the outside of the device.

I agree with Palmer Luckey that this is a necessary short-term decision for Apple given the amount of hardware piling up inside the Vision Pro, but more importantly it's a very conscious long-term decision, which I'll explain in more detail later.

As I mentioned earlier, Vision is a well-designed supercomputer with a range of very power-hungry components:

• 2 laptop-level processors (the R1 chip is almost the same size as the M2, which is the processor used on MacBooks)
• 2 micro OLED displays with very high brightness with high pixel density
• 1 secondary EyeSight display
• 12 cameras and other sensors
• 2 fans
• 2 speakers

As Snazzy Labs' Quin says in this review, the Vision Pro can consume up to 40 watts, which is even more than a MacBook laptop. This also means that it has a power source that can generate a lot of heat. So, in addition to being able to toss the battery weight from the helmet, using an external battery pack also keeps a huge heat source safely away from the user's head.

In my opinion, the long-term strategic reason behind the external battery pack is to give Vision Pro users the understanding that there will always be an external box connected to the headset. In the future Vision series of headsets, Apple will be able to calmly remove a large number of electronic devices from the headset and reduce the weight of the product by about half to about 300g after several generations. This also opens up an extremely interesting path for Apple to use the iPhone, iPad, or MacBook as a connected computer to drive the headset in the coming years, which will greatly simplify the headset.

Interestingly, there's already a connected VR headset on the market today that demonstrates this ideal end state – the Bigscreen Beyond, the world's smallest PC VR headset (which needs to be connected to a computer) and weighs even less than most ski goggles, at just 127 grams. Bigscreen was able to develop this product, which is a bit of a "cheat" in many ways, because this device has removed all sensors (no external camera or eye tracking), but we can experience where the future is going and Apple's goals through this device.

Software

Apple's Vision Pro's software story is a bold rebellion against VR, which was left in ruins due to a lack of exciting AR apps at launch

Apple's Vision Pro's slogan is, "Welcome to the era of spatial computing," and as everyone in the VR industry would expect, Apple is all-in on AR (Augmented Reality) to make that happen. Apple completely ignores what people know about VR over the past decade.

At the heart of Vision Pro's marketing is "to keep users connected to their surroundings and to others." It's easy to see between the lines that Apple is on an anti-VR stance, arguing that Meta's VR implementation promotes human isolation, which is a negative example, while Vision Pro is the opposite of Meta.

In my opinion, Apple's anti-VR stance is a risky move, as they have pushed the Vision Pro into an empty corner by negating much of the immersive content that has been popular in the VR medium so far. It reminds me of Apple's broad stance on privacy, taking a position that is the exact opposite of Meta/Google, and as a result, it finds itself in a tricky position, severely limited in choice, and limited in innovation in the Gen AI era.

There are no 100% immersive games in the Vision Pro app store, and 90% of the apps in the Oculus Quest are immersive VR games. Rather than leveraging the existing community of developers for high-quality immersive VR content, Apple is focusing all of its efforts on AR use cases that leverage the strengths of Apple's ecosystem, namely iOS apps and MacOS productivity.

Apple's lineup of 3D AR apps and games has been disappointing in both quality and quantity, with most of them consisting of simple casual games, some of which were originally 2D games that were hastily converted to 3D. In fact, ARKit has been on iPad and iPhone for years, albeit with limited success, and it should have made it easy for Apple to organize developers to build a plethora of exciting, world-renowned AR apps for Vision Pro. However, on the contrary, what we are seeing is a lack of initial interest from developers in what should be the most decisive and enlightening category.

Ironically, Meta made almost the same mistake in 2022 when it launched the Quest Pro. Despite their announcement highlighting "full-color mixed reality," there are few AR applications on the device.

I think Apple's strategy to attract developers is: "Just get the product right, and they'll come naturally," which is probably the first device category where that strategy didn't work. We'll still have to wait years, maybe even more than a decade, for tens of millions of Vision Pro users to be willing to pay for spatial AR apps. Apple will need to learn from Oculus' approach and actively incentivize developers to build apps for the Vision Pro in terms of funding.

The Vision Pro is positioned as a "big screen" for productivity and watching movies, which is tedious and unimaginative, but Apple has no qualms about targeting it

With a weak and limited lineup of AR apps launched, and not even a decent 3D app or game, Apple had to focus the Vision Pro's overall positioning on how to connect to the existing ecosystem of 2D apps.

In line with Apple's usual product marketing style, the Vision Pro's slogan is clearly listed on the product's official homepage, and every marketing asset echoes it. The way Apple arranges its product messaging is just as important as the messaging itself. Vision Pro has 60% focus on 2D productivity and 40% on the big screen watching media/movies.

Incidentally, FaceTime's profile picture and spatial photos and videos have also become the core pillars of the Vision Pro product message, but these are clearly only ancillary use cases to support marketing. Although the impact is not significant at the moment, it is likely to play an even more important role in the future.

I think the launch of the Vision Pro is a missed opportunity, and while Apple says "welcome to the era of spatial computing," the software and services stack is really only focused on 2D use cases. Although the in-store demo shows an exciting future, the experience that Apple offers when it launches its products can only be described as tedious and unimaginative.

Leaving aside my criticism of Apple's focus when the Vision Pro was launched, let's dive into my thoughts and perspectives on software and experiences that support productivity and media use cases.

Vision Pro aspires to be the "future of work" and continues the goal that Meta Quest Pro has left behind, but ......

One of the biggest themes Oculus encountered early on was the decisive role VR played in the "future of work," from running 2D apps in large virtual displays to having native 3D apps that made it easier to work and collaborate with others on projects.

When Meta released the Quest Pro in 2022, a lot of the marketing hype actually revolved around the Workrooms app. The app allows you to use your Mac within VR, and many of the details you need to consider for working hours in VR have been taken care of, including support for up to 3 virtual displays and the ability to see the physical keyboard in perspective mode, or use a fully 3D rendered keyboard instead.

The Quest Pro is designed to be more comfortable than other VR headsets so that people can wear them for long periods of time. While an attempt with good intentions, the product had a major flaw that did not meet the criteria for a "minimum viable product" and the $1000 list price was simply not worth it. First of all, the display resolution is 22PPD, which is too low and far enough to "work in VR" because the text is not very legible. This drawback (coupled with the very poor see-through quality) was so severe that the product was pointless to launch and I returned it less than 24 hours after first use.

Vision Pro 能否成功解决 Quest Pro(和 Quest 3)失败的问题?

I spent more than 100 hours before and after to actually test the Vision Pro in real-life scenarios and deploy my own real workflows as much as possible, a third of which was spent writing this article. Now, let me share my conclusions.

First of all, before diving into the value proposition of the Vision Pro as a work/productivity computer, I needed to frame my "work" as clearly as possible. In "work mode" (whether doing actual professional work or managing my life), I need to use three different workstations and switch back and forth (except for my smartphone, which I exclude):

• Office Workstation: Mac Pro with 2x Apple XDR 6K displays. It's the highest level of productivity configuration because I have everything I need in just one window and get multitasking with zero barriers. For complex tasks or projects, this is basically my gold standard, with the highest speed and quality.
• Laptop: 16-inch MacBook Pro. It's a mid-to-high productivity configuration, with a retina-quality display large enough to handle more complex tasks, and good enough multitasking capabilities, but I feel that productivity is significantly lower than the first one. A backpack is required when you go out.
• Tablet: 11-inch iPad Pro with keyboard. Medium to low productivity profile, suitable for work that only requires one app, with extremely limited multitasking power (e.g., email, writing without research, some life planning), slightly stronger than mobile phones. A big advantage is that I can more easily carry this "mini computer" without any backpack.

The summary of the three workstations is as follows:

Then I asked myself: Can you imagine using the Vision Pro as a productivity device and abandoning (or paralleling) any of the existing workstations?

The specific questions are as follows:

• Can Vision Pro completely replace my tablet workstation so I can go out with my Vision Pro?
• Can Vision Pro elevate my laptop workstation and make me feel like I have one or two "virtual XDR displays"?
• Can Vision Pro outperform all my workstations for certain productivity tasks?⇒ That's what interests me the most!

Productivity Topic #1: Vision Pro as an Alternative to iPad Pro

Status: Not ready yet (but with a lot of potential!)

I think that the Vision Pro can be a "spatial iPad Pro" with better multitasking capabilities than the iPad and the ability to concentrate on work from anywhere, but in reality, there is still a lot of friction in terms of use, and there are still too many important applications missing, so the probability of achieving this goal today (and in the next 1~2 years) is not high.

Apple's design of the Vision Pro naturally fits well into existing products from the Apple ecosystem (albeit at a relatively high price) and can be used as an alternative to the iPad Pro. The headset has the same computing power as the iPad Pro (the same M2 chip), and it's convenient to support running iPad apps natively. In fact, the Vision Pro was supposed to be better than the iPad Pro because you can run multiple iPad apps side-by-side in full-screen mode, which overcomes one of the iPad's biggest productivity limitations: poor multitasking capabilities.

In reality, however, this argument is untenable in the current situation (at least not yet) for the following reasons:

• Many iPad apps don't work properly (or don't work at all) on Vision Pro, although Apple says all developers can develop apps for Vision Pro. There is a lot of friction and instability in navigating productivity apps because these apps are designed to have a multi-touch UI (e.g., some iPad gestures that don't exist in Vision Pro, and some touch targets that are too small). Many apps require some effort from the developer to work.
• Most productivity apps are absent from the app store (probably due to the reasons mentioned above), which will leave a big gap in most people's workflows. For example, in terms of personal workflows, the most important missing apps include Chrome, Gmail, GDocs/Sheets/Slides, Asana, etc.
• There are still some errors in text input, adding friction in the workflow. Cursor positioning, text selection, and editing are very error-prone. Dictation does not show results in real time.
• For most iPad-like productivity workflows, you'll have to carry a keyboard and trackpad (no mouse support), which can be inconvenient (compared to carrying an iPad or even a laptop with a keyboard). Without these devices, it's almost impossible to edit documents, tables, or presentations.
• There is no reliable long-lasting workspace, which adds more friction. You'll have to reopen the app, and you'll have to resize and reposition the window almost every time. The features we all want, which should soon be available if Apple wanted to, include: (1) persistent workspaces, (2) fixed-location workspaces, and (3) mission-controlled spatial computing.

That being said, these limitations can all be addressed by Apple, and the Vision Pro's potential as an alternative to the iPad Pro does exist. Even though the PPD of the iPad Pro is almost twice that of the Vision Pro, the text of the iPad app on the Vision Pro is clearly visible, enough for you to run 3~4 apps and multiple environment widgets side by side at the same time.

In addition, I believe there is a lot of room for Apple to configure and manage workspaces through a lot of innovation and magic that lets users combine 2D panels and virtual 3D objects. As long as Apple truly empowers developers with these innovations.

I'm really looking forward to the Space iPad Pro, but only if you get all the iPad apps on the Vision Pro and that Apple solves all the issues that cause friction in the workflow. The reason for this is simple, because the Vision Pro boosts focus and really "blocks out reality", allowing us to focus on our work wherever we are, without having to carry around a laptop and still have a certain level of multitasking.

Productivity Topic #2: Vision Pro as a MacBook Virtual External Display

Status: Almost ready (some bugs need to be fixed!)

I think that with just a few software bugs fixed, the Vision Pro could be a nice virtual external display, similar to a 27-inch Apple Studio display, that will allow you to use all your existing MacOS apps on a huge screen and enjoy immersive work (but don't expect the Apple XDR 6K experience!).

One of the best software-only experiences with Vision Pro is that it seamlessly connects to your MacBook while wearing your device and looking at your computer. It's just a simple revamp to the AirPlay UI to create a sense of seamlessness that VR has always lacked.

Before delving into this argument, let me first state that the Vision Pro will never be able to replace my office workstation. My workstation is equipped with dual Apple XDR 6K displays, each 32 inches, with a total of 40 million pixels (218 PPI pixel density and 100 PPD angular >resolution), and I don't need to wear anything on my head, which is not comparable to a headset now or in the future.

The more interesting question is whether the Vision Pro can replace the 27-inch Apple Studio display (or equivalent). To put it simply, for now, the Vision Pro can indeed work in this direction, but Apple needs to address some important limitations to make it a relatively frictionless use case:

• The lack of dual (or triple) monitor support is very disappointing, although there are a lot of legitimate reasons (mainly involving a lot of local Wi-Fi bandwidth). Even though the Vision Pro has a relatively narrow field of view, it still feels like it's through binoculars, and if my MacBook Air can get two or three virtual displays, the situation gets even more interesting.
• The keyboard and trackpad behave inconsistently, making it very difficult to switch back and forth between the iPad/Vision app and the Mac virtual display. I often find myself looking for cursors, virtual keyboards popping up on the screen when I don't need them, and I can't use my beloved Logitech MX mouse.
• There's no such thing as a reliable, long-lasting workspace, which I also mentioned earlier when I discussed the iPad Pro. This should be an easy problem to solve.
• In MacOS, eye tracking doesn't work, not only does it lead to inconsistent input patterns, but it also seems to be a missed opportunity to provide magic features that MacOS has never had before. It's not an easy issue to fix, but I don't think it's a huge technological leap for Apple if the MacOS team is willing to fix it.
• The MacOS app is "stuck" inside the virtual monitor and is not allowed to move around the entire space. It was also a missed opportunity to provide a truly spatial/immersive experience with the Vision Pro, even though the problem that Apple needed to solve was much more complex and required very careful engineering from the MacOS and visionOS teams.

Many of the issues mentioned above are simple software issues that Apple is perfectly capable of solving, and solving them can lead to a huge experience improvement. I suspect they just ran into internal political/collaborative challenges and couldn't get the MacOS team to devote the necessary resources to resolving bugs and feature requests that the visionOS team was experiencing.

The bottom line, in my opinion, is that we could see in the near future that carrying a MacBook Air and a Vision Pro can get a pretty good workstation that offers a lot of advantages, not only to be productive, but to wear the device in a café, on an airplane, or even on the couch at home for hours on end. (Of course, this view does not take into account the cost-effectiveness factor at all).

This is the most powerful advantage that the Vision Pro release brings to Apple, as they have full control and exclusive access to Apple's existing ecosystem. While this is a very tedious and unimaginative use case, it can be very powerful enough for Apple to launch a large number of headsets.

The experience of watching a movie in Vision Pro is truly great, but most people give up after the initial novelty wears off

Watching TV/movies in virtual reality seemed like a very appealing idea, and we (the Oculus team at Meta/Facebook) built a whole product around that idea – the Oculus Go. Launched in 2018, the Oculus Go is the most failed product I've ever been involved in, for the simple reason that despite our strong partnerships with Netflix and YouTube, customer retention is very low. The vast majority of Oculus Go users give up on it altogether within a few weeks of purchase.

The lesson we've learned is that watching traditional (rectangular) TV or movies in virtual reality is really appealing at first, but for most people, the novelty wears off after a few weeks. Here's why:

• Compared to watching TV or movies on a TV, tablet, or laptop, headsets can cause physical discomfort, mainly due to the pressure on the head and face, as well as the inability to sit or lie down comfortably while wearing the device.
• Watching video with a headset can be a lot of resistance, such as more steps (especially finding and putting on the device) and a more cumbersome UI to navigate than on other devices.
• Watching videos in virtual reality means not being able to interact with people and being isolated, which can be a deciding factor for many people.

Back in the day, we came to the conclusion that media consumption in virtual reality is not a core "everyday use" pillar, but can only be used as a side purpose to add some value to other core pillars such as productivity or gaming.

The Vision Pro does bring more advantages than previous virtual reality headsets, with a better display than ever before, and can sometimes create an amazing experience while watching a movie. For example, watching a 3D animated movie from Disney or Pixar is absolutely stunning. But fundamental challenges to the product market remain.

In my opinion, virtual reality is simply not the medium that people would choose to use to view 2D media on a regular basis. Coupled with the comfort and friction issues of the Vision Pro, most people who are excited about watching media in a headset end up going back to a major video device like a TV, tablet, or laptop.

Watching 3D movies on the Vision Pro is a fun and entertaining experience, but the videos are "boxed" and don't feel like real life. Apple has opened the door to a new class of entertainment with the launch of the new Apple immersive video format with Vision Pro.

Apple's immersive video opens up a whole new world of possibilities for media in virtual reality, but its surrealism can present unexpected challenges.

When Facebook/Meta launched the Oculus Go in 2018, we had an important assumption that immersive 180-degree video would capture a lot of consumer interest, which would trigger a ripple effect in the world of entertainment. We've partnered with a small group of media companies that have been specializing in VR video for a long time, so we're in the fray.

The initial excitement quickly passed. On the Oculus Go, the quality of the VR180 video is okay, but the graphics are flat and the colors are muted, which is far from amazing, mainly due to the low resolution. The videos don't create the feeling of being real, nor do they feel transported to another reality. And most of the content is one-offs, there are no real series, no reason to keep people coming back for a while (except for sporting events, which initially failed for other reasons, which I'll get to later).

In less than a year, the Oculus team switched to VR gaming and almost completely stopped investing in immersive video.

In 2020, Apple acquired NextVR, a small but well-known company we worked with, and they were on the verge of bankruptcy (Meta/Oculus abandoned their acquisition). NextVR spent more than a decade building and perfecting the VR 180 camera technology and broadcast quality video production pipeline. NextVR's YouTube channel is still in place today, and there is also a demonstration of the technology (please use your mouse/finger for panoramic viewing while watching videos from their YouTube channel).

The acquisition of NextVR led to the creation of Apple's immersive video format, which captures 3D video at 8K resolution at 90 frames per second in 180 degrees, a format that is a behemoth with eight times the number of pixels of a typical 4K video. We can think of Apple's immersive video format as a new IMAX-3D, but what's really great about it is that it can be projected into a virtual 180-degree sphere (both horizontal and vertical) that occupies your entire field of view.

Vision Pro is the first VR device that allows users to feel like 180-degree 3D video is being played back in 4K quality. At the time of its launch, there are four Apple TV shorts shot in Apple's immersive video format on the Vision Pro. One of my favorite films is Adventure, which is a jaw-dropping film that you can even win a film award. This level of immersion in the Norwegian fjords is amazing, and it's probably my favorite experience in Vision Pro. I've never felt like I've been transported to another place in any other experience.

One of my favorite Apple immersive videos is Alicia Keys: Rehearsal Room, which was a really fun concert that made me feel like I was really in the VR world with another person. It's fun for almost everyone to see Alicia Keys up close in VR, but you might not feel the joy and inspiration of a concert or sporting game on other devices.

I think that because of its surrealism, the ultra-high-fidelity Apple immersive video format will encounter an unexpected major "valley" challenge, where it may feel cool for some people to see someone up close and at such high fidelity, but some may feel uncomfortable or uncomfortable. While this is less likely to occur in concerts or sports games, it is more likely to occur in theatrical narratives and other types of more realistic films.

Originally, Oculus experimented to try to really understand the boundaries that can't be crossed in VR content to avoid people feeling uncomfortable or even unsafe. One finding of these experiments is that over-realism and fidelity may be one of the reasons for crossing the line. In other words, surrealism can quickly take people into the valley of the spooked, one of the two places we always want to avoid in VR (and one that is motion sickness).

This creative challenge takes time and a lot of experimentation, and Apple is the only company we can trust that has enough sensitivity and artistry, and the ability to hire the best talent to conduct this experiment. This may mean that we can expect to watch beautiful experiential films that explore themes such as beauty, wildlife, travel, and music through Apple's immersive format, but deep narratives where the characters are very close to the camera are less likely to be seen.

Luckily for Apple, there's one category of situations where surrealism is unlikely to be a problem, especially for die-hard fans, and that's live sports.

Live sports will be Apple's secret weapon in selling a lot of Vision Pro headsets to die-hard fans, but it will be a long and expensive journey.

In the original 30-second TV commercial for the Oculus Go, Adam Levine and Jonah Hill wore Oculus headsets and sat together on the sidelines of the NBA via VR chatting and watching a game (in fact, they were sitting in their own living rooms).

The TV ad was a huge success, attracting a lot of people to buy the Oculus Go (after all, the device costs only $199) and showed how powerful the potential could be for hardcore sports fans. But as mentioned earlier, for some reason, we failed to make it a reality in the way it was supposed to.

Ultimately, the Oculus team was completely unaware of the opportunity to redefine the sports viewer experience through VR, mainly because we didn't have the patience to develop this market. At first, we failed to build the necessary industry support with sports leagues and broadcast rights holders, so we stopped trying, and the VR sports space was on the verge of death. Today, there have been a few small attempts on the Quest, such as Xtadium and Meta Horizons, but they didn't go through due to the quality of the experience and the limited live content. So far, no one has really made an effort to create this market.

Now, Apple has the opportunity to change that for a number of reasons:

• The Apple immersive experience on the Vision Pro is a transformative experience, both in terms of video quality and real presence. Watching the game in high-resolution VR is a better experience than regular 4K live TV, as die-hard fans can experience the action more realistically.
• Apple's acquisition of NextVR, which already has expertise in VR broadcasting, has been working hard to build a robust 8K live video pipeline, a huge technical challenge that requires significant investment and specialized talent.
• Apple already has an MLS license, is active in the world of live sports, and there are several other rumors that Apple may buy more live streams to further enhance Apple TV (e.g. Premier League, F1 racing).

Apple may be the first to explore the use of Apple immersive experiences and live streaming of Major League Soccer (MLS) via Vision Pro. According to official sources, they could broadcast such events live in late 2024 or early 2025:

The good news is that soon all Apple Vision Pro users will be able to experience the best footage of the 2023 MLS Cup Playoffs captured by Apple's immersive video to feel the thrill of every game. Viewers will view 8K 3D footage from a 180-degree perspective and enter each game with spatial audio.

—Apple press release, February 2024

In addition to MLS (Apple has a long-term agreement to broadcast any event), Apple needs to spend a lot of time and money to reach the necessary agreements with the major sports leagues (NBA, NFL, MLB, Premier League, etc.) to enable this immersive live streaming experience. That being said, it's likely only a matter of time, as the opportunity to rethink viewers' perceptions of sports is huge, and even for a trillion-dollar company like Apple, it will have a big impact.

I think, considering this objectively, ticket prices to watch sports in person have been steadily rising, and today even average or poor seats cost a few hundred dollars, while premium tickets tend to easily sell for a few thousand dollars (the cheapest ticket to the 2024 NFL Super Bowl costs around $2,000). This is a very strong business opportunity for commercial use cases that deliver high-quality, immersive "fieldside" experiences with Vision Pro.

There are two key areas that Apple needs to delve into in order to successfully realize this opportunity. Both of these aspects require a great deal of design, engineering, and experimentation:

• Live sports are very social, which means that Apple will have to invest heavily in providing a shared viewing experience that makes viewers feel like they're in the same room or the same virtual space, as natural as watching a game casually on the couch with family or in a bar with friends.
• The high standard of experience means that Apple will have to really customize every aspect of the experience to look and feel better than the big screen 4K TV, including camera angles, special replays, bird's-eye visualization, analytics overlays, match statistics, and more.

It's a huge space for innovation that will take several generations of Vision Pro to make it happen. I'm optimistic about that, and as part of a team that has tried to seize this opportunity, I truly believe this is an area that needs Apple to change the rules of the game.

PGA App on Vision Pro:

A bird's-eye view of the 3D model of the stadium and the ability to track footage recorded from previous matches

summary

Why I returned the Vision Pro, and the wish list that I wish Apple would fix and improve the product

As a "product person", I generally try to force myself to make real choices like a real consumer. I believe that I should always put myself in the user's shoes, not only at my own product, but also at what others have built. I'll admit that the Vision Pro is the ultimate tech toy, but since I'm not an active developer, I didn't think the $4,049.78 (512GB + tax) price was worth it, so I returned it within the 14-day period and got a full refund.

In the midst of Apple's virtual reality product go-to-market, Vision Pro still has a long way to go before it can truly retain customers. Completely excluding immersive VR games from the Vision Pro app store was a risky decision for Apple, and some of the disappointments, such as the lack of high-quality AR apps at launch, left them with few options to provide user value to non-developers in the short term.

The only low-hanging fruit solution is to increase productivity, which is unimaginative and tedious, but it should be one of Apple's priorities for the next few visionOS releases. If Apple is able to fix all of the friction issues mentioned above, I wouldn't rule out the possibility of trying the Vision Pro again in the future.

In the two weeks I've been using Vision Pro, I've put together a long list of bug fixes and feature requests. Here, I'd like to share the top ten:

1. Fill in the missing gap between Apple and developers by bringing the main apps of the iPad to Vision Pro to reduce friction for productivity use cases, including fixing issues with text input and editing, seamlessly supporting 2 (or even 3) MacOS remote displays, adding persistent workspace windows, building a "Space Mission Control Center" and enforcing a minimum recommended focal length.
2. Encourage people to build great AR games, do everything possible to set high quality standards and reward creativity, add support for shared games and characters, and really drive multiplayer support so that people can participate in the game together.
3. Improve penetration mode to the upper limit allowed by the hardware sensor stack, ideally reduce motion blur, improve white balance, and enable a more seamless experience (when viewing immersive content).
4. Creating persistent workspace spaces allows me to configure different rooms, such as at home or in the office, and let Vision Pro remember the specific configuration of the room.
5. Elevate 3D widgets and objects to first-class citizens in visionOS and enable people to decorate their homes and offices on an ongoing basis.
6. Allowing people to bring their iPhone into VR and simply look at the device (just like the MacOS Virtual Display feature) and see a floating panel that can be placed anywhere in the space can reduce the fear of missing out in VR.
7. Add a guest mode so anyone can try Apple's in-store demo and let Vision Pro users help spread the word, as first-time VR users will find it magical.
8. Add people to watch videos to make people feel together, because VR has a downside: it can feel lonely and isolated, so making VR more socially friendly must be a priority, even though few people will use social features at first. There aren't enough users of the Vision Pro at the moment, so this practical use case isn't practical, but it's important for Apple to set the right tone.
9. Launch a large number of beautiful environments, preferably at a steady frequency, borrowing from the Apple TV screensaver, as well as beautiful interiors (not just landscapes).
10. Implement OpenXR support, establish a partnership with SteamVR, or simply open visionOS to allow VR developers and enthusiasts to build their own compatibility and find a way for people to experience immersive VR games.