Text/VR Gyro Small Drill Wind
Eye tracking has become standard in the new generation of XR devices.
As early as 2020, shortly after the launch of Quest 2, many people secretly wondered what new features the next generation of XR headsets would have, and what killer skills other manufacturers needed if they wanted to challenge Quest 2.
After a detailed interpretation of the dynamics of the manufacturer, it can be found that the eye tracking that is highly respected by XR manufacturers and what users expect is eye tracking. Eye tracking technology is imaginative and has the practical possibility of landing, and can almost be voted the most anticipated new feature.
Eye tracking is standard with the XR headset
Ushered in the inflection point of development from the B-end to the C-end
By the beginning of 2022, the product information of Apple's MR headset, Sony PSVR 2, and Meta's next-generation headset Cambria, which is the most concerned by the market, will become the next XR hardware weather vane.
Compared with the previous generation, PSVR 2 and Cambria have iterated on performance specifications, while adding eye tracking as important new features.
Although the Apple MR headset is mysterious, from the beginning of 2021 when it was exposed by Bloomberg to be launched in 2022, many sources unanimously revealed that the eye tracking function will be adopted. In addition, in recent years, a large number of eye tracking patents for head-mounted displays have been exposed, which further confirms that eye tracking will play an important role in Apple's MR products.
Although the above three major manufacturers have a strong vane nature, the market has high expectations for their consumer products, so they are extra cautious when launching new products to the market, and they will be later than the market to test the water products in the iteration of new functions.
Big manufacturers such as HTC, HP, and Pico have taken the lead and launched products equipped with eye tracking functions. In the past three years, we have seen a lot of eye tracking technology in the XR direction on the headset. Like what:
At CES 2019, Xiaopai Technology launched the Pimax 8K headset with eye-tracking function;
The VR-1 headset launched by Varjo in 2019 features a customized eye tracking solution;
In 2019, HTC Vive launched the Vive Pro Eye with a new eye tracking function;
At CES 2020, Pico released a Neo 2 Eye VR all-in-one with eye tracking;
In 2021, HP launched the Reverb G2 eye tracking headset;
In 2021, Pico released the Neo 3 Pro Eye VR all-in-one machine equipped with eye tracking;
In 2022, Sony announced the PSVR 2 equipped with eye tracking;
Due to the high threshold of eye tracking technology, it is not easy to be professional. Although from the perspective of patent disclosure, major manufacturers have studied the direction of eye tracking, the eye tracking function equipped with the hardware products launched by them is still based on cooperation with eye tracking technology providers.
The products mentioned above are no exception, and most of its eye tracking technology comes from the Swedish eye tracking manufacturer Tobii. Tobii eye tracking technology can also be seen in NVIDIA VRSS technology and Qualcomm's XR reference design.
NVIDIA VRSS 2 integrates Tobii eye tracking technology (Source: NVIDIA)
Qualcomm and Tobii collaborate to integrate eye tracking technology for XR kit (Source: Network)
In addition, these eye tracking products, which are one step ahead, have a common feature - mainly for the B-side.
First B-end and then C-end is the default development path for many new technology applications. Generally, after the application matures in the B-side, the next step will be pushed to the consumer side. Eye tracking in the consumer product PSVR 2 and the next generation of Meta headset is the key signal of its popularity on the consumer side. Eye tracking will usher in an inflection point in 2022 for large-scale application from the B-end to the C-end.
What stage is the eye tracking technology that will soon be applied in consumer XR headsets? What are the problems waiting to be solved? Gyro Jun asked to learn about it from Tobii's technicians.
The state and challenges of eye tracking technology
The basic principles of eye tracking technology are not complicated, and there is more than one way to implement it. At present, the most common is the pupillary corneal reflex method used by technology providers represented by Tobii.
Eye tracking under this scheme is mainly done by eye tracking cameras, light sources and algorithms. The light source emits infrared light to form a flicker point in the cornea reflection, and the eye tracking camera captures the high-resolution image of the eye, and then through the algorithm analysis, locates the location of the flicker point and the pupil in real time, and finally estimates the user's line of sight direction and landing point with the help of the model.
A brief diagram of how eye tracking works in XR equipment (Source: Tobii)
Adaptation challenges for integration solutions
Although the principle of eye tracking technology is not complicated, the current XR equipment form configuration is different, and the optical design used is different, so it puts forward no small challenge to the eye tracking integration platform and program, and requires the solution provider to have strong R & D capabilities and integration experience.
Backed by two decades of experience in R&D and integration of major XR equipment, Tobii has evolved itself over the past few years to keep pace with changing market demands. Today, whether the XR headset is based on a Fresnel lenshot mirror or a folding optical path pancake optical design, is an all-in-one pc or needs to be connected to a PC for virtual reality or augmented reality, it can be supported by tobii XR solutions.
To help XR OEMs understand the value of Tobii eye tracking in their devices, Tobii also offers an eye tracking XR module as a quick proof-of-concept tool.
For XR OEMs, the self-evolving capabilities of eye tracking technology are not only a guarantee of current product quality and performance, but also the key for manufacturers to provide developers with intergenerational product consistency and reduce repetitive development.
High-precision crowd coverage challenges
When eye tracking technology moves toward consumer-grade devices, the quality determinants of its solutions are no longer just precision and accuracy, but whether such metrics can be reproduced in a large consumer base.
On this subject, Tobii mentions the concept of population coverage: "Population coverage is not a common indicator in eye tracking schemes outside of Tobii, but we think it is extremely important. Consumer XR headsets need to take into account the different situations that can occur among millions of users, such as droopy eyelids, cosmetics covering important eye features, wearing glasses and contact lenses, having weak or dominant eyes, and varying eye features depending on the race. ”
"Getting eye tracking to work on all consumers means that its solution needs to achieve high accuracy and accuracy in 95 or even 99 percent of the population." Otherwise, the XR app can't be sure that all users are satisfied. Tobii is proud to lead this important indicator, the result of more than two decades of effort and experience in scientific applications and product commercialization. ”
The smaller the error in eye tracking accuracy under the same population coverage, the better (Source: Tobii)
Cost challenges for consumer-grade devices
In hardware products, the cost and price/performance of any technology components and new features are important considerations for XR manufacturers. Most of the products that we have contacted or heard about released that include eye tracking are positioned at the high end or B-side, which is due to cost factors, and manufacturers see the broad prospects of application innovation of this technology and adopt integration, which shows a high degree of recognition of cost performance.
Tobii believes that there are many visible trends that are further improving the cost performance of eye tracking technology. One of the trends is that better technical processes are reducing hardware and engineering costs. For example, in VR devices based on folding optical path design, Tobii's solution can use only one eye motion camera without affecting quality, achieving a better balance between quality and cost. In addition, mass production using standard components as well as consumer-grade equipment can also greatly optimize costs.
In addition to the above-mentioned visible costs, XR manufacturers have also begun to pay attention to the invisible cost of eye tracking technology integration, such costs include that if they cannot obtain high accuracy and accuracy under 95% or even 99% of the population coverage, XR devices will lose their product promises to consumer groups, losing hardware and brand value.
As a global leader in eye tracking technology, Tobii's years of research and global product integration experience provide the largest data collection and subset for its XR integration platform, whether XR devices are aimed at global users or local markets (such as Asia or China), can obtain high accuracy and precision in high crowd coverage.
In addition, Tobii believes that "the integration of the certification, production, testing and other aspects required in the eye tracking technology process requires a lot of experience to avoid additional costs." Therefore, the more experienced the technology solution provider integrates and the more products are listed, the more unnecessary costs can be controlled. ”
As a cutting-edge technology, eye tracking has developed over the years, what is the value or imagination that can attract the unanimous optimism of manufacturers in the XR field?
Core values of eye tracking:
Optimize resources, improve experiences, and gain insights
Eye tracking has gained widespread adoption in XR enterprise device and app development in recent years, and will drive more disruptive features and experiences on consumer devices in the future. Overall, its core values are in optimizing system resources, providing a comfortable and natural interactive experience, and revealing insights into user behavior.
1. Optimize system resources
Fixation rendering technology based on eye tracking is an efficient means and core function to optimize the resources of XR equipment system, and it is also one of the main purposes of eye tracking in the products listed above.
The pursuit of better visual effects is the trend of the times, so the future XR equipment resolution and field of view will continue to improve, the current resolution of VR equipment has reached 8K or even 12K, the maximum field of view of the product has reached 210 degrees, followed by the pressure on the rendering, 8K screen with the mobile terminal processing platform can not be loaded at all, but the use of fixation point real-time rendering of the local can be achieved.
Fixation rendering can be simplified to mean that only the center area of the user's gaze in the XR field of view is rendered with high precision, while the other areas of the field of view are rendered at low resolution, reducing the GPU shading load while maintaining a high frame rate, so as to better configure limited computing power resources and give users a great experience. This "focused" visual effect also corresponds to the natural visual behavior characteristics of the human eye.
Fixation rendering is divided into two types: fixed and dynamic. It is understood that with fixed fixation rendering, the XR device will preset the center part of the display as a high-quality rendering area. Therefore, the user's gaze point needs to always be in the position of the area, and the head should be rotated to divert the line of sight. Back in 2018, Oculus Go began to adopt this approach.
Oculus Go (Source: Network)
Dynamic fixation rendering captures fixation points as the eye shifts for more precise real-time rendering. Tobii has done a series of benchmarks in different devices and environments. Test data show that dynamic fixation rendering reduces gpu shading load by twice as much as fixed fixation rendering, maintains high frame rates and optimizes system resources more effectively. Whether it's an all-in-one pc or a PC-connected headset, dynamic fixation rendering can make it even better.
In the hardware environment of the Pico Neo 3 Pro Eye, the benchmark showed that Tobii's dynamic fixation technology, Tobii Spotlight, increased frame rates by 78% and reduced all-in-one power consumption by 10%. This is undoubtedly an excellent solution for all-in-ones to enhance the performance of your hardware.
One of the Tobii Dynamic Fixation Rendering Benchmarks (Credit: Tobii)
Tobii's dynamic fixation rendering is also used in NVIDIA's VRSS solution. The new vrss 2 provides drivers with up-to-date eye tracking information with minimal latency to control the oversampled area of the rendered frame, improving image quality based on where the user is looking. VRSS is a zero-coding solution, which means that developers don't need to add any code to implement their functionality.
2. Comfortable and natural interactive experience
Social is one of the most important applications in VR, where users communicate freely through their avatars and expect a real immersive experience.
In order to create such an experience, XR has put a lot of effort into Avatar, such as Meta has developed a good enough motion capture from body language to details such as face and lips to increase the expressiveness of the avatar.
Meta launches new Avatar system (Source: Network)
In addition to this, one of the important hallmarks of a user's social presence is vivid and authentic "eye contact", which will also be an important use for eye tracking in consumer-grade XR applications. Effective communication and a stronger sense of authenticity are achieved through natural eye contact and eye movements, reducing the "Uncanny Valley" effect.
Ready Player Me avatar supported by Tobii Eye Tracking (Credit: Tobii)
A natural and comfortable user experience is an inevitable requirement for wearing a headset device for a long time. Tobii's gaze menu feature simplifies user interaction with content, allowing users to make intuitive, effortless choices based solely on gaze. When the user needs to indicate an object of interest and issue a command, it can also be done using gaze without the need for laborious finger pointing with the hand.
In AR applications, virtual objects need to be better mapped into the user's physical world. Eye tracking can provide the correct depth of view information, help place objects in the right place in the augmented world, reduce the parallax difference between the real and virtual worlds, and enhance a comfortable and natural experience.
3. User behavior analysis
In 2021, the HP Reverb G2 Omnicept Edition VR headset with Tobii eye tracking is another important example of the value of eye tracking – combined with sensors such as face tracking and heart rate sensing, to analyze user behavior and reveal insights through a software platform.
HP Reverb G2 Omnicept Edition (Source: HP)
This headset is an upgraded version of the original Reverb G2, and the key to the upgrade is that it includes a series of sensors such as Tobii eye tracking, as well as the HP Omnicept platform that can process the sensing information. This combination of hardware and software enables monitoring, analysis and feedback on VR user activities such as brain power.
Manufacturers and developers can use eye tracking information to analyze user behavior, so as to optimize and customize products, game production, etc.
epilogue
As Quest 2 breaks through tens of millions of sales to reach the consumer threshold, the VR headset market will also usher in further breakthroughs. As the standard of the next generation of XR headsets, eye tracking will also bring innovative applications and experiences to more users.