How are sequence memories stored in the brain? Chinese scientists document the workings of macaque brains and offer new insights.
The human brain processes sequence information all the time, whether it is language communication, action implementation or episodic memory, which essentially involves the representation of timing information.
Sequences, on the other hand, take time to execute, and the brain needs to remember the entire sequence before applying timing information. For example, we need to remember a series of directions given by the guide when asking for directions, and remember a series of movements when learning a new dance. In these cases, not only do individual pieces of content need to be remembered, but the order between them cannot be confused.
Paper cover
At 2 a.m. on February 11, 2022, the journal Science published a research paper titled "The Geometric Structure of Sequence Working Memory representation in the prefrontal lobe of macaque monkeys" in long form. The research was completed by the Center for Excellence in Brain Science and Intelligent Technology (Institute of Neuroscience) of the Chinese Academy of Sciences, the Wang Liping Research Group of the Key Laboratory of Primate Neurobiology of the Chinese Academy of Sciences, the Min Bin Associate Researcher of the Shanghai Brain Science and Brain-like Research Center, and the Tang Shiming Research Group of the School of Life Sciences of Peking University.
In the study, the scientists trained macaques to memorize sequences consisting of multiple spatial locations, recording neuronal activity in the prefrontal cortex of the macaques' brains.
The researchers found that neurons characterized each spatial location in the sequence in the form of swarm encodings, and found similar ring geometries in these representations. The study overturns the key assumptions of classical sequence working memory models, providing new insights into the conundrum of how neural networks perform symbolic representations.
The "three screens" in the monkey brain
On the morning of February 11, researcher Wang Liping introduced that human beings generally have two modes of thinking, which are divided into intuition, emotion, and thoughtful logic, and the latter relies on working memory. This process consists of two parts, namely information retention and information manipulation. And that's where the scientific question of their research group begins — how are sequence memories stored in the brain?
Cognitive psychologists have been thinking about the representation of sequence information as early as the beginning of the 19th century, sequence information coding is also considered to be the premise of the syntactic structure of human language, and the exploration of sequence translation in the field of machine learning has given birth to the Transformer model that is now in full play. However, little is known about the neural coding mechanisms of the brain with memory of time-series information.
Macaques are evolutionarily the closest model animals to humans, and to explore the problem of time series memory encoding, researchers trained macaques to memorize spatial sequences consisting of multiple location points.
Macaque spatial sequence memory task
During the mission, three different dots flash in turn on the screen in front of the macaque, and the macaque needs to report the order in which these taps were presented before a few seconds later. During the memory retention period a few seconds before reporting, information about spatial sequences is temporarily stored in the brain in the form of working memory.
To record the activity of the brain neuronal population as the macaque monkeys performed their tasks, the researchers imaged the lateral prefrontal cortex, the home of working memory.
How does the brain characterize multiple pieces of information in a sequence at the same time during memory? The researchers hypothesized that macaques also had a "screen" in their brains on which they could write down the dots that had appeared. But if three points are displayed on this screen at the same time during the memory retention period, how should the order of each point be reflected? Will there be three different screens in the macaque's brain at the same time? This way, each screen only needs to record the information of one point, and there is no interference between the screens.
Characterization of sequential memory in the high-dimensional vector space of the nerve
In this regard, the researchers analyzed the high-dimensional data obtained by calcium imaging and found that the two-dimensional subspace corresponding to the information of each sequence can be found in the high-dimensional vector space, that is, the corresponding "screen" can be found. Within each subspace, the spatial positions of the different points correspond to the ring-like structure of the real visual stimulus. Moreover, the subspaces corresponding to different sequences are close to orthogonal, indicating that the brain does use three different screens to represent sequence information.
To further explore whether the brain always uses the same "screens" to memorize different types of spatial sequences, the researchers decoded and analyzed the data, that is, using machine learning methods to train linear classifiers to distinguish spatial information in different orders. For example, training the decoder with the neuronal population activity of the macaque monkey when it responds correctly can achieve better decoding results in some of the correct sequences. These results suggest that the "screen" used for encoding order is stable and generic.
The researchers also found that similar ring-like structures are shared between subspaces of different orders, except that the radius size of the ring decreases as the order increases. One possible explanation is that information in the lower order is assigned fewer attention resources, resulting in smaller corresponding rings and less differentiation. This structure also corresponds to the behavioral manifestations of sequential memory, for example, if we remember more content in our daily lives, the more information we have later, the more likely it is to be wrong.
This finding can also be summarized as the nature of spatial information coding geometries at the population level that are subject to timing modulation. Interestingly, this property does not fully apply at the individual neuronal level, and the enhanced modulation of single neuron activity is a key assumption of the classical sequence working memory model, suggesting that the coding of sequence memory should pay more attention to the nature of population neurons.
Photo of the research team working
Researcher: A key assumption that overturns the classical sequence working memory model
Wang Liping uses the performance of the symphony orchestra to analogize the group encoding of neurons in memory, and each player is a neuron. The research team excavated how order subspaces are implemented on individual neurons, such as "whether there are players participating in multiple symphonies" or "whether players participate in the same way". The findings suggest that a significant number of performers participate in different symphonies in very different ways, which overturns the key assumption of the classical sequence working memory model that different neurons were previously thought to always participate in the work of information storage in the same way.
This study is the first to illustrate the principles of calculation and coding of sequenced working memory at the population neuron level, and also provides new ideas for how neural networks perform symbolic representations.
In the 1980s, researchers in the field of artificial intelligence proposed the concept of tensor product to achieve the representation of the symbolic structure of neural networks, but how it naturally emerges at the level of neural networks has not been well solved. The neural representation of sequence working memory corresponds to embedding the symbolic representation from the subspace of the corresponding order into the high-dimensional vector space, and supports the linear reading of the symbolic structure information by the downstream neural network.
Xie Yang, a postdoctoral fellow in the Research Group of the Center for Excellence in Brain Science and Intelligent Technology of the Chinese Academy of Sciences, and Hu Peiju, a research assistant, were the co-first authors of the paper, and Li Junru, research assistant, Chen Jingwen, a doctoral student, and Song Weibin, a graduate student of Peking University, made contributions at different stages of the project. Professor Stanislas Dehaene of the French National Institute of Health and Medicine, Professor Wang Xiaojing of New York University and Yang Tianming, researcher of the Center for Brain Intelligence Excellence of the Chinese Academy of Sciences, participated in the study. The Chinese Academy of Sciences, the National Science and Technology Commission, the Ministry of Science and Technology and the Shanghai Municipal Government funded the work.
According to the evaluation of academician Guo Aike, a well-known neuroscience and biophysicist, the innovation of this paper lies in the experimental paradigm of coexistence of two clues of time and space information based on the sequence learning of macaque monkeys, and found that the state space of high-dimensional neurons can be decomposed into the sum of multiple two-dimensional subspaces, thus revealing the simple geometric structure of the working memory of sequence information in the prefrontal cortex of macaque monkeys. This discovery reveals that sequence information encoding utilizes the principle of dimensionality reduction, thereby reducing the complexity of neural computation. In addition, Academician Guo Aike pointed out that this paper will have an impact on brain-inspired artificial intelligence research.