Recently, NOLO, an important member of GSXR, released a demonstration clip of its gesture recognition technology on his self-media account. It can be seen from the video content that the staff wears NOLO's upcoming device NOLO SONIC 2, and smoothly taps, presses, pinches, drags, slides and other actions with bare hands. NOLO officially said that this technology will not only be installed on NOLO SONIC 2 equipment, but also will be developed and cooperated with the outside world. With a more natural and intuitive way of operation, NOLO's gesture recognition technology will complement the new paradigm of interactive input technology solutions under the GSXR standard.
In March 2021, China Mobile led by China Mobile and jointly initiated by industry partners launched the China XR General Standard (GSXR) officially released and rapidly promoted in the industry under the leadership of the CCSA Extended Reality Industry and Standards Promotion Committee, which aims to build China's XR cross-platform interoperability system, so that industry participants can carry out hardware, platform, and content development based on the standard, which can ensure that the developed software applications can run on hardware that has been adapted to the standard. The formal implementation of this standard has greatly reduced the difficulty of XR software and hardware adaptation and accelerated the development of the industry.
The formal implementation of the GSXR standard not only provides a common interface specification for the XR industry, but also reduces the access cost of developers. At the same time, the software functions will be integrated into the form of SDK to serve the XR industry.
Tencent Sogou Input Method has become the first 'Chinese input' solution in the GSXR standard through this path, and it is also one of the important human-computer interaction solutions in the standard. Tencent Sogou Input Method provides exclusive, intelligent and efficient voice input and keyboard input modes for XR systems, and can also match multi-form keyboards according to the type of system input box. Among them, the industry's first voice input allows users to free their hands and type with their mouths; Sogou innovates through the combination of software and hardware to optimize the controller input experience and improve input efficiency; Combined with the ability of gesture algorithm, bare hand input can also be performed, allowing users to experience more immersive and intelligent input methods. At present, Sogou VR input method has served the hardware products of China's XR industry, and has been applied as the first baseline product in China Mobile x NOLO co-branded VR terminal.
The gesture tracking technology released by NOLO is also based on the GSXR standard protocol, and developers can use the gesture recognition, bone nodes, pinching gestures, hand tracking and other functions in the XR feature module for XR application development through the GSXR's general API interface. At the same time, in order to facilitate developers to quickly start developing XR applications, GSXR provides detailed documentation and forum discussion functions for Unreal SDK and Unity SDK. After docking, users can achieve more natural and immersive operations such as pointing, pressing, selecting, dragging and dropping with bare hands in the XR device.
Referring to the research and development process of this technology, the person in charge of NOLO's relevant technology said: "NOLO's technical team relies on artificial intelligence deep learning technology to train data fusion with the help of real hand images and 3D generated hand images. trained a specialized deep neural network with millions of hand images in which wrists and fingers have 3D spatial poses; The resulting gesture data is transmitted to the 3D graphics engine for rendering via the GSXR protocol, and then displayed in VR glasses, finally achieving gesture synchronization between the real world and the VR world. At the same time, in order to ensure the real-time recognition, NOLO deeply optimizes the network structure to ensure that the single-frame recognition speed on the Snapdragon XR2 chip reaches less than 5ms; Deep learning algorithms dynamically extract gesture information from multi-frame data to improve motion coherence, so as to achieve smooth and silky recognition effect. ”
Regarding whether the technology is open to the public, the relevant person in charge of NOLO said: "This gesture tracking technology is not only used in NOLO SONIC 2 to continuously enrich the user's experience. As a member of the GSXR working group, we are open to opening this technical cooperation to members of the CCSA Extended Reality Industry and Standards Promotion Committee to help the rapid development of China's XR industry. ”
The GSXR standard has been implemented for two years, and the CCSA Extended Reality Industry and Standards Promotion Committee will further promote the implementation and implementation of the extended reality standard, develop test solutions suitable for industry development, carry out product testing and certification, explore new application models, serve application developers, and build a healthy and sustainable industrial ecology. Next, China Mobile (terminal company and Migu culture) will rely on the unified standard of GSXR, gather content in all fields, provide one-stop service of "standard-content (application)-platform", create a "1+N+N" service system of "one platform, N content resources, and N application services", open XR general capabilities, and provide end-to-end solutions. From the aspects of standards, content, terminals, etc., build an all-domain XR matrix, and empower the application of XR technology in education, medical care, industry, sports, performing arts and other fields and scenarios. And with an open attitude, we will work with industry partners to build a healthy XR ecosystem and help the rapid development of China's XR industry.