Reporting by XinZhiyuan
EDIT: Good sleepy peach
【New Zhiyuan Guide】At the Beijing Winter Olympics, the GOOSE Factory AI Sign Language Digital Man was officially put on duty. Their sign language commentary has given more than 27.8 million special groups Chinese exclusive romance.
At the Beijing Winter Olympics, the track was cold, but the hearts of these more than 27.8 million people were warm.
Why?
On the morning of the 8th, Gu Ailing won the championship, and for a time the whole network brushed the screen.
3D sign language digital person Xiao Cong explained the exciting moment after the Chinese women's snow sports won the first gold medal in Tencent Sports in fluent sign language.
After winning the gold medal in the short track speed skating mixed team 2000m relay competition, "ice rookie" Fan Kexin has been unable to cry during the interview.
In this tearful 1 and a half minutes, it is also Xiao Cong who allows the hearing impaired to share the joy of winning like everyone else.
During the Winter Olympics, the same 24-hour on-the-job sign language anchors in addition to Xiao Cong, there are 3D sign language digital people listening.
"The Chinese team is the first to cross the finish line!"
In the short track speed skating mixed team relay final on February 5, Ting Yu expressed the excitement of the Chinese team winning the championship to the hearing impaired through sign language broadcasts.
These two AI sign language anchors are Tencent's 3D sign language digital people, and they use vivid and accurate sign language to bring wonderful content of the Winter Olympic Ice and Snow Events to people in the silent world.
They do all this for nothing else, only for these people.
Who are they?
According to statistics, more than 466 million people in the world have hearing impairments, and more than 27.8 million people in the mainland meet the standard of hearing disabilities, accounting for more than 30% of the disabled population.
Unable to hear the world like an auditor, the hearing impaired uses a combination of hand movements, facial expressions, and even lip-syncer changes to create a language system that, unlike any other language, relies entirely on visual communication: sign language.
However, even with ways to communicate with each other, there is still an invisible wall between them and the able-bodied.
From TV news to long-form popular science, and now the short videos that are popular around the world, media and video platforms have always been the basic path for the public to understand the world and integrate into society.
However, existing media platforms not only lack large-scale sign language teachers, but also smaller sign language playback windows also limit the clear presentation of non-hand-controlled information such as expressions and body movements. In addition, TELEVISION programs are mostly arranged in normal word order when providing sign language commentary, and rarely take into account the special expression structure of sign language.
As a result, the vast majority of people with hearing impairments can only understand less than 60% of what is in sign language news.
You may ask, can it be solved by subtitles?
The answer is, yes but hard.
One of the most important factors is that there is a big difference between the expression of sign language and the written word.
For those young people with higher levels of education, reading subtitles may not be a problem. However, it is still quite difficult for hearing-impaired people who use sign language as a "native language" to rely on subtitles alone, but it is much easier to understand the video content after combining sign language.
In addition, some expressions in sign language can convey emotional meanings such as degree, likes and dislikes, and may be lacking in expressing them simply through subtitles.
Therefore, in order for the hearing impaired to properly get the content in the news broadcast, the following three challenges must be solved:
1. Sign language is a completely different expression order from Chinese
For example, in Mandarin, "cat chases mouse", the expression of sign language is "cat, mouse, chase". The sign language expression of "Beijing is often stuck in traffic" is "Beijing, traffic jam, often".
2. Sign language is not only about hand movements, but also about expressions, mouth shapes, and so on
For example, the gestures of "Am I doing well" and "Am I doing it right" are the same, and the distinction between the two needs to be judged according to the shape of the mouth. In addition, if you want to express the tone of "question", you need to cooperate with the frowning expression, and the tone of "exclamation" corresponds to an eyebrow-raising action
3. There are no imaginary words and quantifiers in sign language, and they need to be appropriately deleted when converting
For example, "I buy two pencils, one book" is expressed as "I buy pencils, two, books, one." In the sign language of "heavy snow", there is neither "big" nor "flying", but on the basis of "snow", the body swings to reflect the degree of the adverb.
AI was also pulled to do the "special training" of the Winter Olympics
So, how can people with hearing impairments better watch and understand the Winter Olympics?
As the world's first 3D sign language digital person, Xiao Cong shouldered this heavy responsibility.
Before officially taking up his post, Xiao Cong underwent rigorous training, completely in accordance with the National General Sign Language Dictionary, and could successfully complete the word order conversion and translation process from Chinese to sign language.
In addition, with the help of a team of consultants from the China Association for the Deaf, sign language teachers and hearing impaired people, Xiao Cong was not only able to better understand the content that needed to be broadcast, but also completed the synchronization of sign language movements and facial expressions, improving the expressiveness of sign language.
After the preliminary preparation work is completed, it is time for the "special training" for the Winter Olympics.
For sporting events, when it comes to the white-hot stage, the speed of the commentator's speech may become very fast. In addition, in post-race interviews, there may be ambient noise and reverb in the audio. For the special scenario of the Winter Olympics, there is a lack of relevant data. These will have a great impact on Xiao Cong's translation.
In response to such problems, the team targeted the capture of a large number of sports event commentary data, and also developed a data enhancement scheme and a multilingual fusion training algorithm, respectively, from the data and model training two levels of optimization of the final effect.
After special training on sign language vocabulary in sports events, Satoshi got a pass for sign language commentary for sports events at the Winter Olympics.
After evaluation, the intelligibility of Xiaocong's sign language expression can reach more than 90%, and the delay time between sign language and oral broadcast speed has also been reduced from 20%.
Xiaocong was able to achieve such outstanding results by relying on the first complete Text to Pose, Video to Pose sign language translation system in China.
As a mature PaaS system, it can quickly convert and output from text, video to sign language video.
In addition, with the blessing of the PaaS system, it is not difficult to add sign language commentary to live programs in the form of video streams.
As the R&D team behind it, Tencent PCG AI Interactive Department has a deep accumulation of AI capabilities, and has made breakthroughs in multiple tracks such as voice, digital people, computer vision, and natural language processing.
Goose Factory Next
In a Winter Olympic ice and snow event, Xiao Cong and Ling Yu brought convenience and warmth to more than 27.8 million people, and also practiced Tencent's long-standing concept of "science and technology for good".
In the next ice and snow event, Xiao Cong and Ting Yu will also provide sign language interpreters, so that more people can witness the moment when the Chinese team won the gold!
After several years of accumulation, the AI sign language digital people who have gone from the laboratory to the front of the stage have made technology have become warm.
In the future, Tencent's 3D sign language digital human application scenarios will also be infinitely expanded. In addition to providing fluent sign language commentary in traditional news reporting, real-time live broadcasts and other news release scenarios.
It will also carry out the exploration of offline scenic spots, airport hospitals and other cultural and life service places to solve the problems encountered by the hearing impaired in daily life.
At the same time, more and more sign language figures will continue to emerge, providing diversified and personalized services for the hearing impaired, filling the gap in information transmission, and gradually implementing the construction goal of information accessibility in the whole society.