Tianjin, Feb. 18 (Xinhua) -- For the hearing impaired, fierce events can often only be seen, but not heard. At this year's Beijing Winter Olympics, many hearing-impaired people found that there was a beautiful-looking Winter Olympic News AI anchor in the event news, and "her" accurate sign language gestures allowed the hearing impaired to feel the charm of ice and snow sports at the same time as the hearing people (the abbreviation of the hearing-able).
"Use technology to cross the sound barrier and let the hearing impaired 'hear' the Winter Olympics." Yuan Tiantian, deputy dean of the School of Deaf Artificial Labor of Tianjin University of Technology and head of the sign language database support team of the AI anchor of the Winter Olympic News, said.
Poster Maker: Fang Jinyang
Build a "bridge of communication" in 6 years
Yuan Tiantian has the tenacity to persevere to the end.
She led the team to build a "bridge" for hearing impaired people to communicate with hearing impaired people in 6 years. And at the Beijing Winter Olympics, the "first step" of landing applications was taken.
"We have developed a system called the 'Chinese Sign Language Real-time Translation System in Complex Scenarios', which can translate the input sign language movements and expressions into words or speech after being recognized by computer vision technology. To this end, we have collected more than 200,000 corpora. Yuan Tiantian said.
Talking about the original intention of doing this, Yuan Tiantian said that it was to better communicate with the hearing-impaired students.
Students at the School of Deaf And Artificial Medicine of Tianjin University of Technology collect data for sign language recognition systems through mobile apps. Photo by Xinhua News Agency reporter Sun Fanyue
"I came to the college 16 years ago as a teacher, and in the process of contacting the hearing impaired students, I found that what they most desire and fear most is to communicate with hearing people. Because of my professional background in networking, I was determined to change this situation through technology. Yuan Tiantian said.
From one person groping, to only one hearing impaired student joining, to setting up a barrier-free intelligent technology research and development team. Today, this "special" technical team composed of hearing-impaired people and hearing-impaired people has more than 50 core members, many of whom are "post-00s" hearing-impaired students.
With the development and growth of the team, the technical difficulties have also been broken. "At present, we have achieved a system that recognizes the language of the system at a rate of up to 95% in well-lit and stable rooms, and can achieve 'synchronous translation' in some scenes." Yuan Tiantian proudly added, "As long as you type the sign language in front of the camera, the text translation will appear on the display with coherent meaning." It can now be used in education, legal consultation, catering, transportation and other scenarios. ”
Let the hearing impaired "hear" the Winter Olympics
In November 2021, Yuan Tiantian received an urgent task - to teach the AI anchor of the Winter Olympic News to "speak" sign language.
"The process of artificial intelligence deep learning is like teaching children to learn to speak, and every word and every sign language action needs to be repeated 'learning' to achieve accuracy." Yuan Tiantian and her companions began to overcome the problems one by one.
Min Di, a teacher at the Deaf Artificial College who participated in it for the first time, has a deep understanding of this process. "One word is 'difficult.'" He said.
The first thing in front of us is the "unpopular" Winter Olympic vocabulary, as well as the language of professional journalism, how can it be converted into sign language that most people with hearing impairments can understand?
The team members first formed a certain sign language play method of the new words sorted out, and then let the sign language experts determine whether the hearing impaired could understand them one by one. When encountering words that are difficult to express in sign language, there is a lot of discussion. "Each word goes through at least two of these back-and-forths, so we've added more than 1,000 sign language uses for more than 1,000 professional words." Min Di said.
With more than 200,000 corpora as the basis and new professional vocabulary added, I thought the work was close to completion. But it soon became apparent that sign word order was not the same as the ordinary person's speaking word order, and even if these words were "instilled" to AI anchors, they could not type coherent sentences that the hearing impaired could understand.
"Through years of accumulated technical experience, we have enabled AI anchors to gradually master fluent literal translations from ordinary people's language to sign language, and constantly adjust the speed of AI anchor sign language so that people with hearing impairments can understand more clearly." Min Di said.
Use technology to crack more "52 Hz" passwords
Wang Xiaoyang, 22, is a junior at the College of Deaf ArtificialIty. As a hearing-impaired student, he participated in the whole process of "learning" sign language by the AI anchor of the Winter Olympic News. Since the opening day of the Winter Olympics, he has turned on his phone on time every day to "stare" at every sign language gesture of the AI anchor.
"I'm afraid she didn't study well, I'm afraid she'll make a mistake!" Wang Xiaoyang explained in sign language. But to his relief, more and more hearing-impaired friends are sending messages telling him that the AI anchor's sign language is "clear and clear." "Sign language is beautiful and we are proud of ourselves." Wei Xiangyu, a classmate on the side, added in sign language.
Students at the School of Deaf Artificials of Tianjin University of Technology debugging sign language recognition systems. Photo by Xinhua News Agency reporter Sun Fanyue
Technology transcends the barriers of sound and allows more hearing-impaired people to move to the wider world.
"Information technology has helped us a lot." Wang Xiaoyang said in sign language that he always felt uneasy when communicating with Jianting people before, and with these technical means, he had more courage to integrate into society. "I want to be a teacher in the future because my hometown lacks deaf teachers. Or I'll go be a programmer and use technology to make more deaf people 'hear'. ”
For Yuan Tiantian, there is still a lot of work waiting for her to lead the team to complete. "We will continue to iteratively upgrade our system until it is able to allow hearing impaired people to communicate with us smoothly in a variety of scenarios."
Yuan Tiantian told the story of a "52 Hertz whale" - in the ocean, there is a whale with a different frequency of calls, and it is lonely because it cannot communicate with its peers. The whale is very much like the hearing impaired group, "We are going to crack the '52 hertz' code, and use technology to connect lonely whales with other whales, so that they can get more opportunities and happiness." 」 ”