Zuckerberg pointed out in the interview that the focus of the next generation of Quest is on facial and eye-tracking technology, and Project Cambria will also be equipped with eye-tracking technology. Still, eye tracking is a major R&D challenge that Meta needs to solve, and the technology has challenges in terms of ubiquity, latency, accuracy, and so on.
Judging by the latest paper published by Meta Reality Labs, it is exploring an eye-tracking scheme based on event cameras. So what is an event camera? Its English name is Event Camera, which is a camera technology that has emerged in recent years.
The event camera is different from the traditional camera data backhaul format, it adopts an asynchronous communication method, that is, the clocks between devices are not synchronized, the event camera is not the data captured by the fact of the backhaul, but when the pixel changes (events) such as brightness are captured, the data will be returned.
In simple terms, an event camera works similarly to the human eye, with its main function being to capture changes or movements in objects.
Meta notes that the event camera can operate at low power and the response delay can be reduced to microseconds. These features are expected to solve the limitations of existing eye-tracking technologies in terms of real-time performance, power consumption, and even eye-tracking technologies suitable for mobile devices such as AR/VR headsets.
About the Meta Eye Tracking Solution
Currently, the common design of eye-tracking systems is to use flashing lights to illuminate the cornea to capture data, and this scheme requires only sparse data to operate. This means that the sampling rate of the eye-tracking system does not need to be very high, so the researchers expect that even the use of an event camera will be sufficient, although the event camera will only take pictures when capturing motion, but as long as the key data is captured, the eye changes can be calculated.
The biggest difference between this scheme and traditional eye-tracking technology is that it does not have a fixed sampling rate, nor does it sample all pixels, but mainly returns the change data of pixel brightness (with time information), and only samples when the change occurs. Although less data is collected, the accuracy and efficiency are good enough.
Another advantage of event cameras is their high dynamic range (≈120dB), virtually no motion blur, and lower power than traditional cameras, with latencies as low as sub-milliseconds.
Meta said: This is the first completely event-based corneal flicker/eye-tracking protocol. The principle is that the light source shines on the cornea of the eye, and the reflected light generated enters the image sensor and is displayed as visible data. This data can be used to locate the movement of the cornea and thus predict fixation.
Encrypted differential lighting
Researchers point out that the eye is usually composed of two surfaces: 1) a mirror-like cornea that produces specular reflections on light sources; and 2) Lambert surfaces such as skin, iris, and sclera, which diffuse light.
In the process of tracking the eyeball, in order to avoid errors caused by the light source shining on other parts of the skin, iris, sclera and so on, each light source used in the scheme has a matching compensation light system to keep the total brightness unchanged. Without affecting the specular reflection of the cornea, the events generated by the diffuse reflection of the scene are suppressed.
To this end, Meta designed a lighting scheme called "Coded Differential Lighting", which features enhanced specular events while suppressing other events, and only blinking when the event is triggered, sampling rates in the kHz range on standard hardware.
The scheme adopts dual LED light source, which can improve the efficiency and accuracy of corneal specular reflection detection. In order to illuminate the correspondence between the light source and the corneal reflection/flicker, it uses binary mode, allowing the light source pulse to run in two cycles.
Typically, eye-tracking schemes use a high-speed flickering light source. However, scenes illuminated by flashing light sources often saturate the event phase machine, causing brightness changes throughout the scene. Therefore, the researchers let the two LED lights switch between flashing, rather than flashing a single light, in order to keep the overall illumination roughly constant.
They also found that the optimal distance between the LED lights was at points where the eyeball's reflective points just touched and did not overlap. When the LED lights are close together, the two flashes cancel out where they intersect, resulting in a net brightness change of zero and reducing the effective frequency of flicker.
What issues were resolved
Meta pointed out that eye tracking is a key feature of AR/VR headsets, which can enrich the interaction between users, and can also achieve dynamic fixation rendering effects to improve the visual perception of AR/VR. In addition, it can also be used for experimental, medical, training and other scenarios to analyze by using the user's eye movement data. The zoom display system and the focal surface display system also realize the zoom function of eye tracking.
In AR/VR headsets, eye-tracking systems need to be low-power in order to extend battery life and reduce heat generation. At the same time, it is also necessary to support high sampling rates, such as in user authentication scenarios, the sampling rate needs to be as high as 1kHz.
At present, many video eye tracking systems on the market are based on the principle of pupilular corneal reflection (PCCR), which usually shines infrared light on the eyeball and captures the position/angle of the light reflected back by the mirror surface of the cornea to estimate the center of the cornea and the position of the pupil (dark oval). The direction of fixation is then calculated by the distance between the center of the pupil and the center of the cornea.
In contrast, Meta's scheme is more power-efficient, consuming about 5mW per LED, and subpixel-level accuracy can also be achieved at low power. If you replace the wide-angle LED with the position of the eyeball, the performance will be further improved. The event camera consumes about 10mW, and the entire sensor consumes about 35mW. At the same time, the sampling rate of the event camera can reach 1kHz, which can capture subtle changes in the eyeball. Reference: scontent