In human tactile exploration and dexterous object manipulation, it is essential to process dynamic tactile signals quickly and efficiently. Traditional electronic skins generate frames of tactile signals when interacting with objects, but they are often unable to effectively encode temporal information and quickly extract features.
On May 9, 2024, Zhi-Bin Zhang and Ayça Özçelikkale et al. from Uppsala University in Sweden published a paper titled: Spike timing–based coding in neuromimetic tactile system enables dynamic object classification in the world-renowned journal Science. The first author of the paper is Assistant Professor Libo Chen at Uppsala University, Sweden. This paper reports a novel neuromorphic tactile system that utilizes a time-coding mechanism for sparse pulses in biology that can encode highly dynamic tactile information with millisecond-level (1.2 ms) temporal resolution, enabling rapid object classification. The design of this system allows it to process tactile signals with similar efficiency and precision as the biological nervous system, providing an important basis for the development of neuroprosthetics and neurorobotics.
At the same time, Silvestro Micera of the Institute of Biorobotics and the Division of Artificial Intelligence and Robotics at Santa Ana High School published a news article in Science titled: Toward more naturalistic tactile sensors, saying that the highly dynamic biomimetic tactile system developed by Zhi-Bin Zhang et al. represents an important part of future robotics and prosthetic systems.
The hardware consists of 64 tactile sensors, distributed across the electronic skin, that translate mechanical stimuli into receptor potentials across a network of spike neurons implemented across software. The sensors include artificial receptors that mimic the function of specific human mechanoreceptors, and a circuit is designed to simulate the function of tactile sensory neurons by using a "leakage-integral-discharge" model. The tactile spiking neural network (t-SNN) then uses these signals to enable dynamic and fast feature extraction, including curvature and hardness, from the surface touched.
This highly dynamic information has a very fast temporal resolution, which enables efficient classification of different objects at the time of grabbing. Notably, the system is robust to uncertainty (including the use of e-skin by different humans) and damage (e.g., the loss of some tactile sensors in e-skin).
Design and coding principles for neuromorphic tactile systems
This neuromorphic tactile system mimics the human tactile system using artificial perceptrons and a network of spike neurons. When the system interacts with an object, the artificial perceptron converts the mechanical stimulus into a potential signal and encodes it into a spike train to represent tactile information. These spike trains are then processed by a network of spike neurons that are used to identify objects and extract features. This design enables the processing of tactile information in an efficient and dynamic manner, contributing to the development of neuroprosthetics and robotics.
Figure 1. A neuromorphic tactile system with a pulse-timing coding strategy.
Encode dynamic haptic information about touch into spike timing
Artificial sensing neurons are able to efficiently encode tactile information during contact events and consume very little energy when driven by events. They also touched surfaces of different shapes and hardnesses and found that the contact of these surfaces resulted in different patterns in the spike timing. On top of that, the pattern of the first spike is a good representation of the dynamic information during touch, such as curvature and hardness.
Figure 2. Use your fingertips to encode tactile information on the touch surface.
Leverage spike timing to achieve dynamic feature extraction by touch
They also tested the accuracy of the system when touching different surfaces and found that the system was able to extract tactile features quickly and accurately, and was able to classify surfaces effectively. In addition, they have developed a biomimetic sensory-motor system that has been tested on the motor nerves of mice, and the results have shown that the system has a good perception of surface hardness and is expected to play a role in the development of neuroprosthetic devices.
Figure 3. Use your fingertips for dynamic feature extraction and object classification by touch.
Experiments were conducted on 22 types of everyday objects of different shapes, sizes and hardness, and it was found that the system was able to quickly and accurately identify objects at the initial stage of grasping. The classification accuracy of the system for objects increases rapidly over time, which is closely related to the count of the first spike. Even when the experiments introduce differences in behavior between different subjects and large variations in data samples, the performance of the system remains robust and reliable.
Figure 4. Subject's hand grip was used for dynamic feature extraction and object classification.
In conclusion, the research of Zhi-Bin Zhang et al. lays the foundation for the development of biomimetic tactile systems that can more realistically simulate human touch. The idea is to use artificial receptors to mimic various mechanoreceptors in natural skin, thereby enhancing information processing power and overall efficiency. In practical use, these receptors can be integrated into aesthetically pleasing gloves or embedded directly into prostheses to ensure good haptic feedback without compromising appearance and function.
Original link:
https://doi.org/10.1126/science.adf3708
https://doi.org/10.1126/science.adp2623
Source: BioMed Technologies
Disclaimer: It only represents the author's personal point of view, the author's level is limited, if there is anything unscientific, please leave a message below to correct!