Reporting by XinZhiyuan
EDIT: Sleepy
As we all know, social animals have a hierarchy. However, a recent Nature article found that this kind of consciousness is inherently engraved in the brain! Tonight at 19:00, XinZhiyuan invited the author Professor Lu Cewu, and interested readers can leave a message at the end of the article or in the live broadcast room to ask questions. (Online Tencent conference number: 498-116-229, or scan the poster QR code, or add WeChat ID: aiera2015_2)
In the field of AI for Science, DeepMind wants to say second, I'm afraid no one dares to call the first.
The front foot solved the protein structure problem that had plagued the academic community for 50 years, and climbed Nature several times in a row; the back foot used deep reinforcement learning to perfectly control the nuclear fusion reactor and then go to Nature.
Recently, teams from China have also made breakthrough contributions in this cutting-edge direction!
On March 16, a work on the mechanism of behavioral understanding was published on Nature, and the neural circuits that form the behavioral mechanism of "social hierarchical identity" in the brains of mouse populations were successfully discovered and analyzed.
The paper uses machine learning to understand behavior to reveal how the mammalian brain encodes social hierarchies and uses that information to shape its own behavior.
The author is from the team of Professor Lu Cewu of the Shanghai Jiao Tong University. Another co-communicator of the paper is Professor Kay M. Tye of the Salk Institute.
Thesis link: https://www.nature.com/articles/s41586-022-04507-5
Based on computer vision, the team analyzed the association of social and competitive brain nerve signals in large mice and found that the "social rank" behavior generated by the mouse population was actually controlled by neural circuits in the brain.
That is to say, mammals are born to judge the status of other individuals and themselves in social groups, and make behavioral decisions accordingly. For example, low-grade mice will let high-grade mice eat preferentially, and low-grade mice will show obedience behavior and so on.
As soon as the article was published, the melon-eating masses were confused.
I never expected that the cognition that I had always believed in would be "subverted".
The social hierarchy of the rat swarm is actually engraved in the brain?!
For ease of understanding, we can solve this study in two parts.
When a mammal (a behavior agent) performs a certain behavior, does its brain produce a corresponding stable mapping of cranial nerve patterns?
If stable mappings exist, can machine learning be used to discover and resolve neural circuits of unknown behaviors (such as social identity-related behaviors)?
So to answer this series of questions about the nature of behavioral understanding, the team wore a radio physiological recording device for each mouse to record sequences of brain nerve signals from the lateral prefrontal cortex (mPFC) in specific brain regions during social activities, and tracked and localized each mouse through multiple cameras at the same time.
Based on the research results of gesture estimation (such as AlphaPose) and behavior classification research developed by Professor Lu Cewu's team, behavioral semantic labels are extracted, so that behavioral understanding can be scaled and quantitatively correlated with brain nerve signals. The system integrates the most advanced technologies for computer vision behavioral understanding, such as the accuracy of algorithms on mouse attitude estimation points to a higher level than the human eye.
Then, using a large amount of data collected automatically, a regression model from "neural activity signals in the mPFC brain region of mice" to "behavioral labels" was trained by the hidden Markov model. The team found that the trained model still had a stable mapping relationship on the test set. It can also be determined that there is a stable mapping relationship between the type of behavioral vision and the cranial nerve signal pattern in the brain of its actor.
Therefore, after having such a visual behavior detection-brain neural signal correlation model, it is possible to explore those new behavioral neural circuits.
Figure 1. Visual Behavior Detection-Cranial Nerve Signal Correlation Model (Left: Mouse Visual Machine Behavior Detection; Right: System Framework and Model Learning)
For the "mammalian social hierarchy" originally mentioned, it involves very complex behavioral concepts, such as low-level mice will give high-level mice preferential food, low-level mice will show obedience behavior, and so on.
So, how do these mammals judge the status of other individuals and their own social groups? What is the neural control mechanism behind it? For the academic community, this has always been an unbreakable difficulty.
Figure 2. Analysis of the neural mechanism of social-level behavior based on computer vision and machine learning
However, with the blessing of the "Visual Behavior Detection-Brain Nerve Signal Association Model", Lu Tsewu's team and collaborators successfully recorded the brain activity of mice when performing "social hierarchical" behavior.
On top of this, the team further discovered the mechanism by which this behavior is formed—the medial prefrontal cortex-lateral hypothalamus (mPFC-LH) circuit that controls social hierarchical behavior. Moreover, this conclusion has been confirmed in rigorous biological experiments.
Arguably, this study brings a whole new paradigm of research based on machine vision learning to discover neural circuits of unknown behavioral functions.
Machine behavior understanding – three big questions
The above work is part of the basic research of behavioral understanding and is also an important issue of artificial intelligence.
After detecting a real-world entity, the machine wants to further understand what she/he/she is doing, and follows up as an executive entity (human or robot) to understand what he himself is doing.
However, in order for AI to truly understand behavior, it has to answer the following three questions:
Neurocognitive Perspective: What is the intrinsic relationship between machine cognitive semantics and neurocognition?
Machine Cognition Perspective: How to make machines understand behavior?
Embodied Cognitive Perspective: How to Transfer Behavioral Understanding Knowledge to the Intelligent Ontology (Robotic System)?
The Nature paper just now is the first "neurocognitive perspective" problem.
Figure 3. Lucewu's team works primarily around behavioral understanding
Of course, for the latter two problems, Lu Cewu's team also has many years of accumulation.
Eyes: I will! Brain: No, you won't!
Humans feel relaxed about seeing an act at a glance, but it's very challenging for machines. Behavioral understanding is a more abstract and elaborate concept in our brains than common object recognition.
For example, when you close your eyes and imagine a concept of behavior, there are thousands of possible patterns, unlike objects (e.g., apples, tables) that have a single pattern.
Such a huge space of possibilities makes it difficult to "understand behavior" with deep learning brute force as before. Experiments have also shown that the accuracy of behavior recognition is still very low.
In the face of this challenge, Lu Cewu's team conducted research from multiple dimensions such as behavioral knowledge reasoning, generalizability of behavioral objects, and posture estimation, the basic tool supporting behavioral understanding, and the main results include three parts:
Human Activity Knowledge Engine
Different from the general direct deep learning "black box" model, Lu Zewu's team built HAKE (http://hake-mvig.cn/home/), a knowledge-guided and data-driven behavioral inference engine.
First, HAKE divides the behavioral understanding task into two phases:
Mapping visual patterns to the primitive space of the local state of the human body, and expressing diverse modes of behavior in primitives of finite and near-complete atoms;
Primitives are programmed according to logical rules to infer behavioral semantics.
Figure 4. HAKE System Framework
Second, HAKE provides a large knowledge base of behavioral primitives to support efficient primitive decomposition and complete behavioral understanding with the help of combinatorial generalization and reducable neural sign reasoning:
Rules can be learned: HAKE can automatically mine and verify logical rules based on a small amount of prior knowledge of human behavior-primitives, that is, summarize the rules of primitive combinations and perform deductive verification on actual data to discover valid and generalizable rules and discover unknown behavior rules.
Figure 5. Learn the rules of unseen behavior
Human performance upper bound: On the 87-class complex behavior instance-level behavior detection test set (10,000 images), the performance of the HAKE system with complete primitive detection is even close to the performance of human behavior perception, verifying its great potential.
Behavioral Comprehension "Turing Test": HAKE's "erasure method" is very similar to that of humans, which confirms that it is similar to humans in the understanding of the "interpretability" of behavior.
This special "Turing test" allowed HAKE and human subjects to erase some of the key pixels in the image, respectively, so that people could not distinguish what the picture was trying to say.
The human volunteers responsible for verifying the results need to make judgments about the processed images. If the answer is wrong, it means that the AI/person performing the "erase operation" can better understand the behavior in the diagram.
The results showed that for the images that had been erased by HAKE, the accuracy rate of humans was only about 59.55%, which was less than 10% higher than the random guess of 50%.
Figure 6. HAEK's "erasure method" is very similar to that of humans
Second, the behavior object can be generalized brain-inspired computing model
For a specific behavior (such as "washing"), the human brain can abstract the generalized concept of behavioral dynamics, apply it to different visual objects (such as clothes, tea sets, shoes), and use this to make behavior recognition.
Studies in the field of neuroscience have found that for continuous visual signal input, in the process of human memory formation, space-time dynamic information and object information reach the hippocampus through two relatively independent information pathways to form a complete memory, which brings the possibility that behavioral objects can be generalized.
Simply put, when you have seen "dog jumping", if a completely different animal, such as a cat, does the same action, then you can still understand that what you see is "cat jumping".
Figure 7. The generalization that comes with a decoupled approach to the concept of behavioral objects and the concept of dynamic behavior
Based on brain science inspiration, Lu Tsewu's team proposed a semi-coupled structural model (SCS) suitable for high-dimensional information by mimicking the mechanism of human cognitive behavior objects and dynamic concepts working independently in various brain regions.
SCS can independently explore the concept of behavioral visuals and dynamic concepts of behavior, and store the two concepts on relatively independent two parts of neurons, and design the information independent error back-propagation mechanism under the framework of the deep coupling model to constrain the two types of neurons to only pay attention to their own concepts, thus initially realizing the generalization of behavioral understanding on the actor object.
The proposed semi-coupled structural model work was published in Nature and Machine Intelligence and won the Outstanding Youth Paper Award at the 2020 World Congress on Artificial Intelligence.
Figure 8. Visualizing neurons representing "visual objects" and "dynamic concepts of behavior" (left: video sequence; center: object neurons; right: dynamic neurons)
Third, human posture estimation
Human posture estimation is an important basis for behavioral understanding, and it is also a problem of how to obtain accurate perception under structural constraints.
To this end, the team proposed algorithms such as graph competition matching, global optimization of attitude flow, and inverse motion optimization of neural-analytical hybridization, which systematically solved the problems of large crowd interference, unstable posture tracking, and serious common sense errors in the three-dimensional human body in the human movement structure, and published more than 20 computer vision top papers such as CVPR and ICCV.
Figure 9. Structural sensing work (left: dense crowd posture estimation; medium: posture tracking; right: three-dimensional human body shape estimation)
The relevant research results have accumulated to form an open source system, AlphaPose (https://github.com/MVIG-SJTU/AlphaPose), which is widely used in the field of sensors, robotics, medicine, and urban construction.
Figure 10. AlphaPose
After posing to estimate AlphaPose, the team further developed an open source video behavior understanding open source framework AlphaAction (https://github.com/MVIG-SJTU/AlphAction).
Figure 11. AlphAction
Brain: I will too! Hand: No, you still won't!
Okay, now that the machine can understand these behaviors, doesn't that mean that my AI can come in handy?
Don't worry, it still can't work!
Fast forward 59 years, when two scientists did a very famous experiment in 1963.
The researchers first connected a pair of kittens who had never seen light to a carousel. One of them, though trapped, can stand on the ground and walk around on its own, while the other is fixed to a crane. When the cat that can move around begins to move, the other one will follow suit.
After letting the two cats go through a wave of "learning", the researchers found that although the surrounding environment changes in the eyes of the two cats are the same, in the end, only the kitten who can only walk develops normal visual perception.
The reason is that the cat in the crane only learned that when something approaches, it will look "bigger", but it does not know that this actually means that the physics is "closer" to itself.
Even in later tests, when objects were about to stick to the face, the cat would not even blink its eyes. That is to say, the pattern change in the field of vision has no spatial significance to it.
So, in order for AI to be able to obtain normal visual perception with depth, it is necessary to give it a "body" to interact with the real world on a physical level.
To generalize this conclusion, it is not difficult to see that only when the agent (robot) can learn human behavior and complete the common task accordingly, can it be proved that the machine understands the nature of behavior.
图13. 「Movement-produced stimulation in the development of visually guided behavior」
Therefore, the exploration of understanding the nature of human behavior from a first-person perspective also needs to expand from simply considering "what she/he is doing" to jointly considering "what I am doing", which is also the research idea of "Embodied AI".
Explore the transfer of this understanding ability and learned behavioral knowledge to the embodied intelligent ontology (humanoid robot), so that the robot initially has "human behavior ability", and finally drive the robot to complete some of the real-world tasks, laying the foundation for the universal service robot.
The solution of the above scientific problems will:
Improve behavioral semantic detection performance and improve the scope of semantic understanding;
Improve the ability of agents (especially humanoid robots) to understand the real world, and at the same time test the machine's understanding of the nature of the concept of behavior according to the real-world feedback during the completion of the task, laying an important foundation for the realization of universal intelligent robots.
In recent years, Lu Cewu's team has cooperated with Feixi Technology to build a universal object grabbing framework GraspNet (https://graspnet.net/anygrasp.html) in the field of embodied intelligence, realizing the capture of various types of invisible objects such as rigid bodies, deformable objects, and transparent objects in any scene.
For the first time, GraspNet surpassed the PPH (picks per hour) indicator at human levels, three times the previous best-performing DexNet algorithm, and the paper was cited 70 times in a year.
5.7MB
Figure 14. GraspNet
About the author
Lu Cewu is a professor and doctoral supervisor at Shanghai Jiao Tong University, whose research interests include computer vision and robotics learning.
Personal homepage: https://mvig.sjtu.edu.cn/
He was a high-level overseas youth introduction talent in 2016, was rated as one of the 35 Chinese science and technology elites under the age of 35 (MIT TR35) by MIT Science and Technology Review in 2018, and was sought to be an outstanding young scholar in 2019, and has published more than 100 papers in high-level journals and conferences such as Nature, Natural Machine Intelligence, and TPAMI as a corresponding author or first author.
In addition, he has served as a reviewer for Science and chaired the top of artificial intelligence and machine meetings such as CVPR, NeurIPS, ICCV, ECCV, IROS, etc.
References: https://www.nature.com/articles/s41586-022-04507-5