Tan Ping, head of the XR Lab of Alibaba Damo Academy: The three major challenges facing the XR Internet - interaction, content upgrading, and computing

From June 10th to 11th, the 22nd Annual Conference of the 2022 Yabuli China Entrepreneur Forum was held in Yabuli, Heilongjiang.

"What is the metacosm?" What is the relationship between the virtual world and the real world in the future and the individual? At the Youth Forum, around topics such as "Metaverse: Science Fiction or Future", Tan Ping, head of the XR Laboratory of Alibaba Damo Academy, elaborated his views.

The following is the transcript of Tan Ping's speech (with deletions):

If the metacosm is the next generation of the Internet, then AR/VR glasses are the next generation of browsers. The browser is a very important product in the Internet era and is the entrance for users to access the Internet. Similar AR/VR glasses are terminal devices for users to get an immersive experience, is the entrance to the immersive Internet, only through AR/VR glasses can we perfectly integrate virtual and real, giving users a new experience.

What is the development trend of AR/VR glasses and application software?

Technically, I divide the existing VR glasses into three generations, the first generation of products is HTCvive, this product technology uses a split type, combined with the positioning technology from the outside to the inside, this generation of products has two relatively large pain points, first of all, because of insufficient computing power, it is necessary to connect the PC through the cable to obtain the support of computing power. At the same time, the user can only move in a relatively small range, and once this range is exceeded, the device will not work properly.

A typical example of the second generation is OculusQuest, which uses an all-in-one solution to solve the problem of computing power, eliminating the need to connect to the PC with cables. At the same time, because of the breakthrough of the positioning algorithm, the user's range of activities is no longer limited, and it is precisely because of the solution of these two pain points that this generation of products began to succeed in the market, but it still retained two pain points. First of all, after wearing ER glasses, the user cannot see the external environment, the action is very inconvenient, there is no sense of security, resulting in the user's willingness to use is reduced. At the same time, the equipment is still relatively bulky, and the comfort of wearing is not high.

The third generation of VR glasses is a series of new products to be launched, according to public information, Facebook will launch a new VR glasses based on Pancoke optical solutions combined with video perspective capabilities in the second half of this year. Through this video perspective technology, users can see the external environment at any time after wearing VR glasses, which can solve the problem of insecurity during use, and because of the new optical solution, the weight of the device can be greatly reduced, and the wearing comfort is greatly improved, so this generation of glasses will accelerate the development of XR Internet.

Compared with VR glasses, the technology of AR glasses is still very immature, and different manufacturers often use completely different technical solutions. For example, Microsoft takes the route from heavy to light, Microsoft's Hololens is very powerful, but the device is very bulky and not very comfortable to wear. Mobile phone manufacturers often take a contrary technical route, from light to redone, for example, we see Xiaomi, oppo, vivo have released very lightweight AR glasses products, but the function is very simple, often mainly as a mobile phone accessories to provide basic text information prompts.

There is also a route, often overlooked, in fact, is MR glasses, that is, on the VR glasses through the video perspective ability to achieve the function of AR glasses, specifically in the VR glasses to install a front-mounted high-definition camera to capture the external picture, while the real-time display on the internal screen, through the video perspective technology to achieve the effect of AR glasses, this video perspective will be the next generation of VR glasses this year to launch the new products quickly popularized, which represents a new technical route.

For the development trend of application software, we judge that the application will first occur in the field of high demand intensity. The survey found that 60% of the content on current VR glasses is games. However, if XR glasses want to become the next generation of computing platforms, they must expand the circle. Core opportunities, such as social scenarios. At the same time, we believe that there will be a good opportunity for live broadcast scenes, because the intensity of demand is very high.

The challenges of the XR Internet: interaction, content scaling, computing

In the process of the development of XR Internet, we have to face technical challenges. The number one technical challenge is the challenge of interaction. A simple and easy-to-use interaction is the key to the rapid rollout of new computing platforms. Glasses need to accurately capture every subtle movement of the user, we also need to design a complete set of interaction logic, so that users can gesture, eye movements and even expressions and immersive three-dimensional content for natural interaction, while glasses also need to be able to perceive the external environment, so that according to the different environment, the user's interaction to make different responses. To solve these interactive problems requires very powerful artificial intelligence algorithms, such as visual SLAM, map creation, object recognition, gesture tracking, eye tracking, expression tracking and many other AI algorithms, all algorithms must be on the lightweight mobile terminal in real time operation, which puts forward high requirements for the power consumption and efficiency of the algorithm, many problems are not completely solved today, and solving these problems is a prerequisite for the development of the XR Internet.

The second challenge is content upsizing. The content of AR/VR glasses should be three-dimensional immersive, however, Internet applications require that the vast majority of content is created by ordinary users, that is, the so-called UGC content, ordinary users can easily edit text, send microblogs, or take photos to the circle of friends, or take videos to Vibrato, but it is difficult for ordinary customers to create high-quality three-dimensional content, even professional users, creating three-dimensional content is also a time-consuming and expensive thing. We need highly immersive content creation tools that allow ordinary users to create high-quality three-dimensional scenes, objects, or characters, and only by solving this problem can the application of the XR Internet flourish.

There is also a technical challenge that will be overlooked – computing. The three-dimensionalization of content will lead to an explosive increase in the computing power demand of the rendering engine, analogous to a large three-dimensional game, more than 90% of the computing power consumed in a game is related to rendering, and only less than 10% of the computing power is in the logic of the game. Today, the vast majority of the computing power consumed by the Internet is the process of business and the logic of the business, if the content of the Internet today is fully three-dimensional, the total amount of computation is likely to increase by more than 10 times, how to cope with such a huge growth in computing power, will be a huge challenge.

To solve this problem, a natural idea is that we can put the rendering on the cloud, the rendering engine renders the picture on the cloud, and pushes the high-speed stream to the AR/VR glasses through the network, which can reduce the need for computing power and battery of the glasses, making the glasses more comfortable and lighter. However, this also means that we have to refactor the entire rendering engine. The rendering engine is designed for game applications on a single PC, where the computing power of a PC is fixed and limited, while the computing power of the cloud can be dynamically deployed and is almost infinite. How to redesign the rendering engine according to the characteristics of the cloud, or even how to fully mobilize the computing power on the cloud and the light for the rendering and calculation of the optical cloud collaboration is a new technical challenge.

The XR Internet is still in a very early stage today, and the future is bright, but the road must be tortuous. One quote that goes well to describe today's XR Internet is that "people tend to overestimate what changes can happen in a year or two and underestimate what will happen in 10 years." Today's XR Internet still has many factors that are difficult to accurately predict, but the best way to predict the future is to create the future, hoping that through our joint efforts, we can create a wonderful new generation of the Internet.

For more reports on this article, we have published on the [Hexun Finance APP], search for "Hexun Finance" in the app store, download and participate in the index guessing activity to win the JD card and the 10,000 yuan cash prize