Imagine if you were sitting in a conference room or library with a pin-dropping pin and blurting out "Siri, see tomorrow's weather," which might not be appropriate.
What happens when a person can't speak or has to be quiet, but needs to give instructions to a smart device?
To that end, Cheng Zhang, an assistant professor of information science at Cornell University, and Ruidong Zhang, a doctoral student, designed a necklace called "Speechin."
Image courtesy of Cornell University
Based on chin movements, it captures "deformed images of the neck and facial skin" to recognize silent language, and can currently recognize simple phrases in English and Chinese.
GIF produced by: YouTube@Cornell University
SpeeChin's infrared camera is mounted on a 3D printed "necklace box" that hangs on a silver chain and the camera is pointed at the wearer's chin. To improve stability, the developers designed "wings" on both sides and placed a coin at the bottom.
In addition to these, SpeeChin is also equipped with a microprocessor, battery and Bluetooth module.
Using machine learning-based algorithms, the device can determine which commands the wearer is silently saying based on the movement of the wearer's chin, and then relay those commands to the paired smartphone, like Siri in another way of communicating.
To avoid privacy concerns, SpeeChin will only lean down to the chin and will not point directly at the user's face.
In the original trial, there were 20 participants (10 english-speaking, 10 Mandarin-speaking), and the researchers measured the baseline position of their chins and then trained SpeeChin to recognize simple commands using differential images.
The 10 English participants silently uttered 54 commands, including numbers, interactive commands, voice assistant commands, punctuation commands, and navigation commands, as did 44 commands from the other 10 Mandarin participants.
It turns out that SpeeChin's average accuracy in recognizing English and Mandarin commands is 90.5% and 91.6%, respectively. The researchers say the device has the potential to learn a person's silent voice patterns.
The researchers also asked 6 participants to speak 10 Mandarin and 10 English phrases silently while walking. The study had a lower success rate, largely because participants' heads moved in unpredictable ways.
Why do silent speech recognition in the form of a necklace? One researcher elaborated on the purpose of their study:
We think necklaces are a form of habit, rather than ear-mounted devices, which may be uncomfortable; as for silent voice, people may think ," I already have a speech recognition device on my phone. But you need to speak up for people and occasions that can't speak up.
It is worth mentioning that SpeeChin is similar in appearance to NeckFace. NeckFace, a device introduced last year by Cheng Zhang and his SciFi Lab team, continuously tracks facial expressions by using an infrared camera that captures images of the chin and face under the neck and generates a 3D reconstruction of the full expression.
Animated image from: YouTube@CornellScifiLab
NeckFace can be particularly useful in the area of mental health because it tracks people's emotions throughout the day. While people don't always express emotions on their faces, the amount of change in facial expressions over time may indicate mood swings.
NeckFace can also be used in other scenarios, such as virtual meetings when the front camera cannot be selected, facial expression detection in virtual reality scenes, and so on.
If SpeeChin develops further, its use cases will also increase, including environments that must be muted, noisy environments that cannot be recognized, and people who lack language skills.