Over the past few decades, AI and machine learning have advanced rapidly, with significant advances in areas such as visual recognition, language understanding, and natural language processing.
However, while these systems are increasingly approaching or even surpassing humans in their performance on specific tasks, they still have significant limitations in their ability to understand complex scenarios, make effective reasoning, and have long-term memory.
Especially when dealing with visual scenes, existing models often struggle to separate and identify individual object entities from the scene, let alone track the changes and interactions of these objects over time.
In addition, many existing models lack an intuitive understanding of how objects exist and interact in the physical world, which limits their reasoning and prediction capabilities.
Human cognitive processes rely heavily on an intuitive understanding of objects and their physical properties, which allows us to easily handle complex dynamic scenarios and make effective reasoning and predictions in our daily lives.
Therefore, we can draw inspiration from human living habits to explore more reasonable model architectures that are more in line with human behavior, and make up for the shortcomings of existing AI systems in complex scene understanding, object segmentation and tracking, and intuition and prediction based on physical intuition.
Based on this, Professor Sun Hao's team at Renmin University has carried out a study that aims to solve the following key questions:
First, it solves the problem of object segmentation and tracking in complex scenes.
Existing models tend to perform poorly at recognizing and tracking multiple objects in a scene, especially when there is occlusion or interaction between objects.
By developing new inference modules and memory mechanisms, they hope to improve the model's ability to perceive objects in these scenarios.
The second is to achieve reasoning and predictions that are more in line with human behavior.
Many models lack the ability to make effective inferences and predictions based on physical intuition. In this study, we try to simulate the human reasoning and prediction process by introducing slot-based spatio-temporal transformers and memory buffers to improve the intuitive physical understanding of the model.
Third, it explores object-centered cognitive processes.
By mimicking human object perception and intuitive physical abilities, this research aims to gain a deeper understanding of how humans learn the laws of the physical world through observation and interaction.
This not only helps to explain human cognitive processes, but also has important implications for the development of smarter AI systems that can mimic these processes.
Figure 丨Model architecture (source: arXiv)
Once the research objectives were defined, the team began to design specific areas for improvement and a preliminary model architecture.
Based on the model architecture of the preliminary design, they carried out model construction and preliminary testing. This needs to be done on a simple or publicly available dataset in order to quickly validate the feasibility of the direction of improvement.
Next, they conducted in-depth experiments on a wider dataset with the goal of fully verifying the validity of the research hypothesis and precisely determining the optimal model structure.
最终，相关论文以《面向视频的推理增强型以对象为中心的学习》（Reasoning-Enhanced Object-Centric Learning for Videos）为题发在 arXiv。
Figure丨Related papers (source: arXiv)
Li Jian is the first author, and Sun Hao is the corresponding author.
Photo丨Li Jian (Source: Li Jian)
It is expected that the results will enable the following applications:
First, it can be used for autonomous driving.
In the field of autonomous driving, this achievement can accurately identify and track objects on the road (such as other vehicles, pedestrians, and obstacles).
At the same time, this technology can improve the ability of autonomous driving systems to understand the surrounding environment, especially in complex traffic situations, and better predict the behavior and possible changes of other objects.
Second, it can be used for visual surveillance.
In a security monitoring system, this achievement can accurately segment and track each object in the video, which is very useful for tasks such as event detection, behavior analysis, and anomaly recognition. In other words, this technology can make the surveillance system more intelligent and effectively improve public safety.
Third, it can be used in robotics.
In the field of robotics, this achievement can improve the understanding of complex environments and the ability to manipulate objects, allowing robots to better understand their surroundings and plan and interact effectively, especially when performing tasks such as searching, grasping and handling.
Fourth, it can be used for interactive entertainment and games.
In game design and interactive entertainment products, this achievement can provide virtual environments and objects with real physical behavior, thereby greatly improving the user experience.
Fifth, it can be used for image analysis.
In the field of medical impact and chemical image processing, the accurate identification and tracking of specific structures in images (e.g., tumors, organs, etc.) is important for disease diagnosis and treatment planning, and this achievement can play a role in improving the accuracy and efficiency of medical image analysis.
Figure丨Experimental results (source: arXiv)
In addition, based on the basic principles of human intuitive physics, the team built a hidden space time series prediction model from an object-centric perspective to further understand and predict the dynamic changes in the physical world.
At the same time, they combined the advanced large model and diffusion generation model to build a multimodal basic model of video generation for physical scenes, which is more in line with physical laws.
In this study, they will also embed the general prior physics knowledge into the effective mechanism of the current model, which improves the consistency of the prediction of hidden space feature sequences.
This strategy not only enhances the coherence of video frame prediction, but also ensures that the generated video satisfies the basic physical laws, thereby improving the video realism.
Furthermore, the research group constructed a set of hidden space sequence prediction models and methods based on symbolic learning and inference. The model can combine the spatiotemporal slot attention mechanism to achieve more robust video generation and prediction for complex physical scenes.
Through this series of innovative methods, it also provides strong technical support for the generation of realistic videos in complex physical scenes.
Resources:
1.https://arxiv.org/pdf/2403.15245.pdf
Typesetting: Liu Yakun