Recently, researchers at Meta Reality Labs created a prototype VR headset, equipped with a custom accelerator chip specifically for AI processing, making it possible to render the company's realistic Codec Avatars on standalone headsets.
Long before the company changed its name, Meta had been working on a virtual digital human project called Codec Avatar, which aimed to make the near-photolike avatar in VR a reality. The system uses a combination of sensors on the device (such as eye tracking and mouth tracking) and AI processing to present the avatar to the user in real time in a realistic way.
With breakthrough 3D capture technology and artificial intelligence systems, Codec Avatar can help people quickly and easily create realistic avatars in the future, making social connections in virtual reality as natural and common as the real world. The company believes that realistic virtual representations will change everything.
It is understood that the early versions of the company's Codec Avatars research were supported by powerful graphics cards such as the NVIDIA Titan X GPU. So, the team has been working on how to implement Codec Avatars in low-power VR all-in-ones. A paper published last month at the 2022 IEEE CICC conference proves this. In the paper, Meta revealed that it created a custom chip built on a 7nm process specifically to be used as an accelerator for Codec Avatars.
According to the researchers, the chip is still far from being on the market. This chip is specially customized, and the footprint is only 1.6mm. The team specifically considered an important part of the Codec Avatars processing pipeline when designing: analyzing incoming eye tracking images and generating the data needed for the Codec Avatars model. In turn, they reconstructed part of the Codec Avatars AI model to take advantage of the chip's specific architecture.
Reality Labs researchers said: "By reconstructing the eye tracking extraction model based on convolutional neural networks and customizing it for hardware, the entire model fits into the chip to reduce the system-level energy and latency costs of off-chip memory access." By efficiently accelerating convolution at the circuit level, the prototype chip demonstrated achieves performance and low power consumption of 30 frames per second in a small form factor. ”
By accelerating the dense portion of the Codec Avatars workload, the chip not only speeds up the process, but also reduces the power and heat required. Thanks to the chip's custom design, it was able to do this more efficiently than a general-purpose CPU, which then informed the redesign of Codec Avatars' eye tracking component.
But the headset's general-purpose CPU will run at full power, and while the custom chip can handle part of the Codec Avatars encoding process, the actual visual effect of managing the decoding process and presenting a virtual digital person is the Snapdragon XR2.
Impressively, Meta's Codec Avatars can run on standalone headsets, even if a dedicated chip is required. But one thing we don't know is the processing effect of Codec Avatars visual rendering. The user's underlying scan is very detailed, but may be too complex to render completely on Quest 2. It's unclear how much of the "realistic" part of Codec Avatars is preserved in this case.
It's worth mentioning that the study represents a practical application of the new computing architecture, which Reality Lab's chief scientist Michael Abrash recently described as the only way to make XR's sci-fi vision a reality. In an earlier speech, he said that in order to implement the true XR devices described in science fiction works, we need to move from highly centralized processing to a distributed processing architecture.
It is conceivable that a range of XR-specific features could benefit from specially designed chips. Spatial audio, for example, is ideal in XR for added immersion, but realistic sound simulations are computationally expensive (not to mention power-hungry). In addition, the hardware and algorithms that will be uniformly designed can be of great help in terms of speed and power.