Recently, a team from the University of California, San Diego proposed a student agent system based on generative model, EduAgent. The large language model is used to simulate the fine-grained physical behavior, mental state and learning process of students in an all-round way.
Experiments show that EduAgent can not only simulate and predict the learning behavior of real students, but also generate reasonable learning behavior of virtual students without real data.
A related paper titled "EduAgent: Generative Student Agents in Learning" was recently published on the preprint website arXiv [1].
Songlin Xu, a Ph.D. student at the University of California, San Diego, is the first author and corresponding author, and the study was completed under the co-supervision of Assistant Professor Qin Lianhui and Professor Zhang Xinyu.
Figure丨Related papers (source: arXiv)
Challenge: Simulate the fine-grained behavior of students with large language models
Although there have been many studies using deep learning to simulate student behavior before large models, deep learning is limited by a large amount of training data, and it is difficult to directly simulate students with different personalities.
With the explosion of large language models, more and more studies have proven that large language models have strong learning and simulation capabilities even if new training data is not provided to adjust the model.
Pre-training based on large language models has taught it strong contextual learning capabilities and a wide knowledge base. Therefore, it is more feasible to use a large language model to simulate student behavior, just give it contextual information.
Although large language models have been well proven to have a strong ability to mimic human behavior, it is more challenging to use it to model the fine-grained behavior of students.
Building on previous research, the team provided more fine-grained dataset labels, including real-time eye-tracking information collected by students during online classes, various psychological states, and final learning outcomes.
However, there are too many student behaviors that need to be simulated (including physical behavior, mental state, and ability to understand knowledge points), and the context is too complex, making it difficult for large language models to grasp the key points from the large amount of information.
Xu Songlin said: "We propose a cognitive priori-based method to guide the model to think about and reason about the potential connections and interactions between different behaviors, so as to achieve better student behavior simulation. ”
图丨EduAgent 框架(来源:arXiv)
In addition, in many scenarios, it is difficult to obtain real student data. Therefore, the researchers also explored whether virtual student agents simulated by large language models can still produce plausible behaviors without any real student data.
By simulating students with different personalities, they found that the cross-relationship between the physical behavior, mental state, and ability to grasp knowledge points generated by the virtual student agent is consistent with the relationship between the three as proven by many existing cognitive science studies.
"This proves that it is possible to generate reasonable, fully virtual student behavior using large language models without relying on real data." He said.
It is reported that in terms of digital twin systems for educational scenarios, researchers have developed agent simulation systems that are more fine-grained and closer to the real learning state of students.
Such a system can comprehensively simulate a variety of real learning behaviors and states of students.
For example, it can not only simulate students answering questions, but also simulate students' eye movements during the lesson, and even whether the students' mental state is confused about a certain knowledge point, and whether they are very focused or distracted in class.
At the same time, the intelligent teaching system based on digital twins has been further improved than the existing intelligent teaching systems, and is no longer limited to simply providing suggestions to teachers or providing answers to students, but is deeply integrated into the teaching process.
Provide timely and personalized educational support throughout the student learning process through fine-grained learning behaviors (such as focus) and each student's unique knowledge background and comprehension ability.
In addition to the field of education, the idea of "using large language models to simulate human physical behaviors (such as eye movements)" proposed in this paper is expected to be extended to other scenarios that include user behavior.
For example, the interaction between a real person and a virtual digital human. Through the generation of virtual digital human eyes and real people's eye movements for eye contact and interaction, so that users can gain a sense of identity and emotional communication.
Develop student agents based on generative models
According to reports, the research roughly goes through five stages, including: finding important questions, literature research, proposing new models, designing experimental evaluation models, and thinking about the potential application scenarios and limitations of the model.
Specifically:
First, look for important questions.
In fact, researchers have identified this problem in previous projects: how can we more realistically model more granular and comprehensive learning behaviors of students with less data?
Existing deep learning models require a large number of datasets for training, so it is urgent to study new models to meet good results without requiring a large amount of additional training data.
Second, literature research to understand the solutions, effects and limitations of the latest existing papers on this issue.
Xu Songlin said: "Through the survey, we found that although there have been studies using large models to simulate student learning, the effect is not even as good as that of deep learning models. ”
One of the main reasons for this is that existing studies only use the accuracy of students' answers to questions for modeling, ignoring contextual information. For example, the content of specific knowledge points and the behavior of students in the process of learning this knowledge point.
Thirdly, a new model is proposed to address the limitations of existing research.
In view of the limitations of the existing research mentioned above, the research group proposes to combine students' physical behavior, psychological state and knowledge comprehension ability to model students' behavior more comprehensively, and create a real student agent EduAgent.
"The agent can have its own eye movements, mental state, and comprehension just like a real student, rather than simply making predictions about the student's accuracy in test scores." Xu Songlin said.
Figure丨Comparison between the dataset and the existing student learning behavior dataset (source: arXiv)
Fourth, an experimental evaluation model was designed.
The researchers designed two experiments to evaluate the model. The first experiment was used to model a specific student to predict future learning behaviors and states through a small amount of historical learning behavior data from real students.
The second experiment is to generate specific virtual learning behaviors for virtual students with different personalities without relying on any real experimental data. It can be applied to the generation of virtual student data to train specific instructional strategy models.
Xu explains: "Among them, the third and fourth stages are often iterative of each other, because it is impossible to achieve a very ideal effect in a single model design. ”
Fifth, think about the potential application scenarios and limitations of the model, as well as the unsolved problems.
For example, conduct a more in-depth study of the generated virtual student behavior to guarantee its plausibility. In addition, there are issues such as possible bias against large language models.
Want to enhance personalized instruction to students
Currently, Songlin Xu is a third-year Ph.D. student at the University of California, San Diego, with a research focus on AI-based Human-AI Integration. In other words, AI can be used to enhance people's cognitive abilities and its application in education, health, and other scenarios.
Based on this research, the research group will conduct more extended research in the future. These include: exploring a more powerful model based on the dataset of this study, applying it to the team's experimental platform to develop an intelligent teaching system, and solving the problem of personalized training of students through targeted teaching.
Picture丨Xu Songlin (Source: Xu Songlin)
According to reports, they have developed a teaching website to provide targeted teaching and guidance for students with different personalities and learning backgrounds by integrating the latest models and algorithms.
For example, through the students' historical learning behaviors, a personalized database is automatically established for each student to describe each student's unique learning status and ability to grasp knowledge points.
A targeted student simulator is built for each unique student to train an intelligent teaching model (AI teacher), so that AI teachers can repeatedly explore different teaching strategies and select the best teaching strategy based on the historical learning records of different students.
"Hopefully, our system can provide truly personalized instruction for all students, overcome the barriers to individualized and diverse instruction, and ensure that education is inclusive." Xu Songlin said.
It is reported that they are considering recruiting volunteers (including teachers and students) to experience this personalized teaching system remotely for free, and partners interested in the project are welcome to contact Songlin Xu [email protected].
Resources:
1.https://arxiv.org/pdf/2404.07963.pdf
Operation/Typesetting: He Chenlong