Editor: Rayyan David
【New Zhiyuan Introduction】Light moving mouth, no sound, can also be voice recognition? The smart necklace created by the Chinese team of Cornell University not only does it, but also has universal Chinese and English, and the recognition accuracy rate is more than 90%, and the mystery lies in the infrared camera installed inside the necklace.
Existing speech recognition technology allows users to talk directly to Siri, ask about tomorrow's weather, or ask Alexa to play a song or something.
But these technologies still require people to speak their voices. But what if there is a person who can't speak? Or is it inappropriate to speak out on certain occasions?
Cheng Zhang, an assistant professor of information science at Cornell University's Bowles School of Computer and Information Sciences, and Ruidong Zhang, a phD student, gave the answer: Speechin.
It is a device that recognizes silent commands by recognizing skin deformations around the neck and face. It utilizes a neck-mounted infrared camera to capture these pictures.
"SpeeChin: A Smart Necklace for Silent Speech" was published on December 31, 2021 in Proceedings of the Association of Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies Recognition" article details this technique.
In October, Ruidong Zhang sent the paper to the UbiComp 2022 conference.
"This technology involves two issues. One is, why do you have to use a necklace? The other is, why identify lip synchles?"
For the first question, we think of a necklace as something that people get used to wearing. It may be a little more comfortable than wearing it on your ears.
For the second question, many people may think that their mobile phone already has a voice recognition function and no longer need to wear a necklace. But sometimes you may not be able to speak and speak, or some people may not be able to make any sound."
"And we're introducing this technology that learns the patterns of various parts of a person's body when they speak, so that they can know what he's talking about even if he doesn't make a sound."
"We've introduced a whole new form factor, hardware." Ruidong Zhang said. He built a prototype of the new device at his home in China in 2020. At the time he was working remotely in China on his first year of phD program.
The device looks a lot like the Neckface technology that Cheng Zhang demonstrated last year.
Neckface was done by Cheng Zhang and members of his Scifi Labs team. Neckface uses infrared cameras to continuously capture the trajectory of changes throughout the face, collecting images of the skin on the chin and neck.
Like Neckface, Speechin uses an infrared camera to capture changes in the wearer's skin, which is mounted on a 3D printed necklace. In order to improve the stability of wearing, the developers installed a small wing on each side and a coin at the bottom.
Compared to setting up a camera in front of a face, it is more convenient to hang an infrared camera around the neck and protect privacy. Because if there is a camera in front of you, you will inevitably capture the scene behind you, which may invade privacy.
For the first experiment, they recruited 20 participants. Half Speaks English, half Chinese Mandarin. Team members determine the baseline position of the chin primarily through testing. Then train the device with different pictures and learn to recognize some simple instructions.
Ruidong Zhang asked participants to speak 54 commands in English, including digital instructions, interactive instructions, voice assistant instructions, punctuation instructions, and navigation instructions. Then say 44 simple words or phrases in Chinese, the content is basically the same.
Speechin's average accuracy in recognizing English and Chinese is 90.5% and 91.6%. In order to test the limit to the maximum, the experimenters found 10 participants to conduct another experiment.
The experimenters asked the 10 men to silently utter the contents of 72 carefully crafted syllables made up of phonemes, including 18 consonants and 4 vowels.
Finally, the group members found 6 more people and said 10 Chinese phrases and English phrases as they walked. The success rate decreased slightly this time, because there are some variables when walking, such as walking more head movements than when not moving.
Ruidong Zhang built a laboratory at home with a welding station and recruited people to participate in experiments in his hometown.
The place where I live is a small town and people who speak English are not easy to find. So we ended up going to Hangzhou and looking for English speakers at Zhejiang University to participate in the experiment. It was an unforgettable experience for me.
Resources:
https://dl.acm.org/doi/abs/10.1145/3494987
https://www.inceptivemind.com/speechin-smart-necklace-recognizes-wearers-silent-speech-commands/23335/