Zhi DongXi (public number: zhidxcom)
Author | Cheng Qian
Edit | Panken
Is it reliable to let the robot do nucleic acid sampling?
Over the past two years, several R&D teams have developed systems that allow healthcare workers to remotely manipulate robotic arms to complete nucleic acid sampling. However, compared to "direct start", these operations have a difficulty - it is difficult to grasp the weight of the start.
▲The new intelligent throat swab sampling robot system jointly initiated by the team of Zhong Nanshan, an academician of the Chinese Academy of Engineering, and the Shenyang Institute of Automation of the Chinese Academy of Sciences
After all, it is not the hands of the medical staff themselves, and the medical staff can only rely on their rich experience to imagine how much force should be controlled to control the machine to poke the nose, throat and eyes of the person being examined.
But if the robot can be allowed to feedback the tactile synchronization to the medical staff, is the problem solved?
On January 14, a research result cooperated by scholars from City University of Hong Kong, Dalian University of Technology, Tsinghua University and China University of Electronic Science and Technology was listed in Science Advances, the top international academic journal. The robot VR system project in the paper solves the above problems.
The research team has developed a flexible electronic skin patch that can be applied to robotic VR as a wireless human-computer interaction system. As long as this "magic sticker" is attached, humans can share their sense of touch with machines.
The patch can be either small to insensible or large as a blouse that responds to the sense of touch by the force and frequency of vibrations. When the user puts on the VR device, he can remotely control the robot to complete the delicate movements that are like being on the spot.
▲Demonstration of the application of electronic skin in the detection of nucleic acid in intelligent robot VR
In the past, most VR immersive games on the market often required users to install bulky and huge haptic perception hardware devices such as exoskeleton control to increase the haptic experience. The new electronic skin is expected to make it easier for users to have a real tactile feeling in the virtual world.
How to achieve robot VR haptic feedback? What difficulties have been solved by the research? What are the new applications in the future?
Recently, Zhidong exclusively interviewed Yu Xinge, the corresponding author of this Science paper and a doctoral supervisor of biomedical engineering at the City University of Hong Kong, to understand the behind-the-scenes story of this research and have a deeper understanding of its technical principles and application prospects.
First, the medical staff remote control, robot to help you test nucleic acid
The start of the robot VR project also begins with the new crown pneumonia epidemic that swept the world in early 2020.
At that time, the number of confirmed cases of covid-19 was rising, medical resources for medical staff were in short supply, nucleic acid sampling efficiency was low, and the risk of infection among medical staff was very high. This made Yusinger think about whether it was possible to develop a remote non-contact controlled sampling robot.
There are many benefits of nucleic acid sampling robots, including reducing medical costs and alleviating the risk of infection, Zhong Nanshan's team, Tsinghua University's computer department professor Sun Fuchun's team, and researchers from the University of Southern Denmark have developed intelligent nucleic acid sampling robot systems.
▲The fully automatic nasopharyngeal swab sampling robot exhibited at the 2021 service trade fair (picture source is Global Times)
However, these robots, which are controlled by remote control handles, have high requirements for the accuracy of medical staff's operation.
Yu Xinger's team observed that during the operation, the doctor needs to judge the surgical progress through the screen and imagine how much feedback there is, which often requires clinically experienced doctors to control.
So, is it possible to share the sense of touch between the doctor and the robot, so that the robot can synchronize the haptic feedback in the field to the remote medical staff?
The research project of applying electronic skin as a wireless human-machine system to robot VR was launched.
The researchers have proposed an idea that as long as the intelligent electronic skin is "attached" and the doctor remotely controls the robot, it can be perceived through the vision, hearing, and touch, which is similar to the zero-distance experience of the surgical process, and to a certain extent, it will reduce the standard of the surgical robot certification doctor.
▲ Demonstration of the application of electronic skin in virtual reality of intelligent robots
However, to turn the idea into reality, there are still many difficulties to overcome.
In the progress of the robot VR project, the biggest problem encountered by Yu Xinger's team is the two-way feedback link between the robot and the human body. This is also the longest time-consuming part of the project's progress.
There are two core difficulties: First, after the electronic skin captures the human body action, how to accurately transmit the instructions of this action to the robot? The second is how to accurately feedback the tactile sense obtained by the robot to humans?
In addition, the robotic VR research process involves not only biomedical engineering, which Yu Singh's team is good at, but also materials, circuits, mechanical optimization, wired feedback pathways, software development, and interface optimization.
Therefore, they worked together with many outstanding talents from Dalian University of Technology, Tsinghua University and China University of Electronic Science and Technology to finally make progress in such a huge application engineering project.
Second, the skin can also change into a jacket, vibration mapping tactile feedback
Going back to the study itself, the overall dimensions of the control panel of the new electronic skin are 57mm x 39mm x 0.8mm.
The researchers employed a multi-layer stacked layout in the electronic skin, in which a layer of skin tone elastic silicon (polydimethylsiloxane, PDMS) can be used as a soft adhesive for the skin, and a series of chips and sensors interconnected by polyimide (PI), including resistors, capacitors, Bluetooth modules, microcontroller units (MCUs), bridges, and soft sensors and actuators developed by the researchers.
▲The composition structure of CL-HMI electronic skin
The design in the electronic skin is formed into a filamentous snake-like structure according to the rules of perfect mechanical design, thus making the entire system stretchable. Another layer of skin tone elastic silicon (PDMS) is used to encapsulate all functional components.
▲Serpentine structure design in electronic skin
"The final form of an electronic skin depends on what we want to achieve, it can be a small patch that is almost insensible, or it can be a pair of gloves, a top or a jumpsuit," Yusinger said. ”
Extending to robotic VR, the paper reads, the researchers wore four electronic skin-integrated patches to control a 13-degree-of-freedom humanoid robot capable of performing most of the human's movements.
Equipped with the corresponding sensors, the robot can simultaneously experience pressure from the forearm, upper arm, thigh, thigh side, and abdomen.
During the experiment, the number of bending sensors and haptic actuators can be freely selected according to the specific application, and the VR device is worn at the same time to synchronize the visual information obtained by the robot electronic eye.
In this way, the robot VR system can be used as an operating platform for remotely controlling the robot in wireless operation mode.
▲ Glove-shaped electronic skin manipulation remote control car simulation experiment
Regarding the implementation of haptic feedback, YuSinger said: "We replace touch by different vibration intensities and frequencies. "The vibration can range from a low-frequency knock-on vibration to a state similar to the vibration of a mobile phone.
In the course of the study, researchers will establish mapping standards for users, and users can gradually adapt to vibration feedback after training and familiarization, but so far, the mapping standards will vary depending on body parts, users and other factors. Yusinger told Zhidong that they are also exploring technologies to achieve more realistic mappings.
Third, through the Internet transmission, less than 50 milliseconds can easily respond
In addition to tactile perception, the wireless transmission function of the robot VR system is equally magical.
Seven bending sensors and five actuators are installed on this electronic skin. These sensors and actuators can be connected directly to the MCU's analog-to-digital converter (ADC) and general purpose input/output (GPI/O) interfaces in the electronic skin for multichannel sensing and actuation.
Electronic skin includes sensors that read information, wireless transmitters for sending information, and small vibrating magnets that facilitate haptic feedback.
The sensors are made up of wires placed in a zigzag manner that are pulled straighter when the patch is bent, and when bent and straightened, data about body movements can be transmitted to the robot.
The researchers say the sensors can convert human movement into electrical signals, which are further processed by the MCU and transmitted wirelessly to the target robot. At the same time, the pressure sensor on the robot detects contact with the external environment, and controls the vibration intensity of the haptic actuator through the Bluetooth module to provide the user with haptic feedback.
To further illustrate the performance of the electronic skin's real-time wireless operation, the researchers measured the cyclic response speed of the electronic skin in the range of 1-5 meters, including signal sensing, data transmission, signal reception, brake response and other links.
The results of the study show that the factors affecting the wireless transmission distance of electronic skin are: the thickness of the elastic silicon encapsulation layer of the top skin tone, and the complex biological tissue in the human body may absorb electromagnetic radiation.
In terms of improving the haptic feedback of robot VR, Yu Xinge said: "We can now solve the above problems by adjusting the position and structure of electronic skin, and will also cooperate with more professional communication experts to integrate related technologies and continue to explore the potential of robot VR in the field of wireless communication transmission." ”
▲Wireless transmission distance experimental data of electronic skin
In addition to the Bluetooth operating mode, the electronic skin also supports two wireless communication modes: Wi-Fi and Internet.
For Wi-Fi mode, the wireless transmission range can be extended to hundreds of meters as long as the operating area is covered by the same Wi-Fi network. In Internet mode of operation, the wireless transmission distance can be extended to anywhere covered by the Internet, and its response time ranges from 30.2 to 47.8 milliseconds.
Fourth, create an offline version of the meta-universe, in addition to listening, seeing can also touch
In addition to the medical field, haptic feedback can also be applied to social, entertainment, prosthetic sensing and other aspects to further enhance the user experience.
If the VR device, haptic feedback and the existing online video software are combined, from hearing and vision to touch, across the screen, users can feel the real sense of shaking hands and hugging with distant relatives and friends.
In last year's explosion of the "metacosm" concept, people need to build their own avatars in the virtual world. To make this avatar feel empathy with humans, it is inseparable from the evolution of haptic technology.
As early as 2018, Yu Xinger has long focused on haptic feedback research, exploring the combination of VR, robotics and electronic skin. Previously, the progress of skin VR developed by its team was published in the international academic top journal Nature at the end of 2019. The skin VR system is equipped with near-field communication technology (NFC), and people thousands of miles away can wirelessly control the tactile drive of skin VR by touching and sliding the screen.
In his view: "Robot VR can be understood as an offline version of the metaverse. ”
In the real world, many tasks with high risk factors and extreme environments require robot assistance, which requires a closed-loop human-computer interaction system that integrates haptic feedback and sensing.
Yu Xinge said that in the virtual world to control the robot, the user only needs to send instructions to the software, and in the real world, there are more instructions and hardware interaction links, and the difficulty of controlling the robot is higher.
Robotic VR systems are mainly aimed at human-computer interaction. Yu Xinger also added that from the perspective of accurate perception of robots, their research team has already achieved relevant results, and now the size of the vibration unit in the electronic skin has been successfully reduced to nearly 1/4 of the previous one, and the unit integration degree per unit area has also been greatly improved.
In terms of entertainment applications, Yusinger gives us an example of robot fighting in the "Iron Fist" movie replacing violent, bloody fighting. In the future, the use of robot VR and skin VR to control robot duels is also a potential application scenario in this field.
▲ Synchronized action clips of people and robots in "Iron Fist"
Robotic VR can also be used for prosthetic sensing in the field of health.
Special populations who have lost their upper or lower limbs often cannot grasp the magnitude of the force by observing themselves alone after installing prosthetic limbs, and installing electronic skin on the end of their amputated limbs is expected to improve this problem.
Yu Xinge explained that while controlling the movement of the prosthesis, the electronic skin can also feedback the hardness and weight of the object to the electronic skin at the end of the amputation, so that the user can feel the real force and hardness feedback, thereby controlling the size of the force.
According to Yu Singh, they are also preparing startups to explore the landing of robotic VR, and are currently more inclined to industrial partners such as medical applications.
Conclusion: A new way of human-computer interaction or a new outlet for robot applications
The emergence of closed-loop human-machine interface provides an updated interaction mode between the human body and the robot, which further increases the application scenario of remote control robots, in addition, two-way feedback also provides the possibility for robots to perform more complex tasks. Examples of remotely controlled high-degree-of-freedom robots in long-range operations, prosthetic control, and medical applications represent a wide range of potential applications in different fields.
Developing remote control technology that combines precise haptic sensing and feedback is important, and YuSinger says they are still exploring the diversity of electronic skin morphologies, the accuracy of haptic feedback, the efficiency of wireless transmission...
In the future, the metacosm may be an important area for the application of robotic VR systems. But YuSinger believes that the research of this project is far more than this, the skin as the largest tactile sense of the human body, there is more potential for development, but the basic exploration time in this field is still very short, basic research is still indispensable at this stage.
For the perception of hearing and vision, we have become commonplace in daily life, if tactile perception can be further popularized, will it make the virtual world more and more real? We are also looking forward to the future of not only hearing, seeing, but also truly feeling the virtual space.