The birth of large language models has effectively promoted the development of artificial intelligence. But as models get bigger and more training data increases, people don't know much about them.
For example, GPT-4, a typical representative of large language models, still gives wrong answers to questions that seem simple to humans (in the two cases shown below).
Picture丨Screenshot of the case (source: Zhu Zeyuan)
So, is this a problem with GPT-4 itself, or is it not enough training data, or is it too math weak?
For scientists who pursue rigor, it is necessary to think about the causes of this series of problems and try to discover the universal laws that exist behind them.
Six months ago, Zhu Zeyuan and collaborator Professor Li Yuanzhi of MBZUAI, Meta's AI Basic Research Laboratory (FAIR Labs), discovered some unexpected complications in the process of studying "how to learn knowledge" in large language models.
For example, there is some knowledge that the model can remember but cannot say, and some knowledge, the model can speak but cannot deduce.
Some sequential knowledge, such as the idiom "carry on the past and forge ahead into the future", always appears in order, so no matter how big the large language model is and how long it has been trained, it can only remember the positive order, but not the reverse order. This phenomenon, which involves the "sequential nature of knowledge", has been called the "reversal curse" by the academic community.
(来源:arXiv [3])
In order to overcome this problem, FAIR Labs has recently proposed an alternative training scheme called "reversal training", which basically means that all data should be trained twice in the forward and "reverse" directions at the same time, and then the problem of reversal curse can be effectively solved by finding the most reliable "reverse" training method.
近日，相关论文以《逆转训练攻克逆转诅咒》（Reverse Training to Nurse the Reversal Curse）为题在预印本平台 arXiv 上发表[1]。
Authors include FAIR Labs research engineer Olga Golovneva, research scientist Zeyuan Allen-Zhu, research scientist Jason Weston, and research scientist Sainbayar Sukhbaatar.
Figure丨Related papers (source: arXiv)
A reversal training scheme was proposed to overcome the reversal curse problem of large language models
In fact, when exploring the reasons behind large models giving wrong answers to simple questions, Zhu Zeyuan believes that excessive pursuit of the performance of large language models on benchmark datasets may also lead to the drift of humans and general artificial intelligence.
For example, AlphaGeometry[2], recently published in Nature, is an AI system developed by DeepMind that is capable of solving 25 of the 30 plane geometry problems of the International Mathematical Olympiad.
But its main algorithm is a brute force search without AI involvement, and the steps of the search are selected from hundreds of human-selected lemmas.
Is it possible that DeepMind has hand-picked hundreds of lemma tailored for the 30 International Mathematical Olympiad questions?
We dispute this (on behalf of the team only, not Meta's official position). But from a scientific point of view, we should try to avoid human intervention in case 'there is as much intelligence as there is human being'. Zhu Zeyuan said.
Picture丨Zhu Zeyuan (source: Zhu Zeyuan)
Based on similar concerns, Zhu Zeyuan proposed a new concept of "language model physics".
This concept advocates simplifying the complex under the inspiration of physics, splitting "intelligence" into multiple dimensions, including grammar, knowledge, reasoning, problem solving, etc., and creating new synthetic data for each dimension, and building an ideal large language model training and testing environment to explore the universal laws of the model. Similar to studying Newton's laws in a vacuum, or studying gas equations in an ideal environment.
It is important to note that researchers should not limit themselves to individual models such as GPT-4, but should summarize the generalizability of any model under an ideal dataset.
"For the field of artificial intelligence, by removing the false and preserving the true in an ideal environment, we can eliminate factors such as data cheating and manual selection, truly find out the universal laws of large language models, and propose solutions to enhance performance. Zhu Zeyuan said.
It is understood that the first part of the "Language Model Physics" project focuses on grammar research, the second part focuses on reasoning research, and the third part focuses on knowledge research.
"However, due to the sheer number of discoveries, the third part alone, 'Knowledge Research', has been split into at least three papers, Part 3.1, 3.2, and 3.3, each with several or even a dozen conclusions, all of which have been published in arXiv. Zhu Zeyuan said.
Figure | Schematic diagram of the third part of "The Physics of Language Models" (Source: Author Twitter)
For the phenomenon of "sequentiality of knowledge" published in Part 3.2, Zhu Zeyuan and Li Yuanzhi first observed it in an ideal environment, and then verified its existence in pre-trained models such as GPT-4 and LLaMA-2 that are available on the market.
So what are the benefits of doing research in an "ideal environment" rather than a realistic model?
For example, in this case, we can fix the order of knowledge in an ideal environment without worrying about the contamination of the test data.
If we always say "so-and-so, born on XXXX, X/XX", to ensure that the knowledge in the dataset is the person's name before the birthday, and then extract half of the information about the person in the dataset, the reverse knowledge extraction ability of the training model is trained, such as "what is the name of the person born on X/XX/XXXX".
We will find that no matter how big the model is and how long it is trained, it can only complete reverse knowledge extraction for this half of the people (100% correct because this half is in the training set), but cannot generalize it to the remaining half (0% correctness).
Figure | In an ideal environment, the accuracy of all reverse knowledge extraction would be almost 0 (source: arxiv[3])
In other words, in an ideal environment, not only can the test set and the training set be completely separated, but the amount of data can be increased infinitely, and even the model can be opened to observe "why" knowledge cannot be reversely extracted, and the necessary conditions for extracting knowledge can be obtained.
What's more, research in an ideal environment can be generalized to real-world models, including GPT-4, where the "reversal curse" can also be observed.
For example, in addition to the idiom reversal mentioned above, you can also ask the large language model for the previous sentence of "no reason in the west of Yangguan", or give the birth date/work unit/city of the celebrity on the encyclopedia, to ask the large language model who the name is.
"A large number of tests have told us that real-world models are not very good at answering such inverse knowledge questions. Zhu Zeyuan said.
However, it is important to point out that it is difficult to determine on a real-world model whether the cause of these incorrect answers is whether the model has not been trained long enough or whether there is not enough data.
Even if the real-world model answers correctly, will it see the original problem in its training data (i.e., data pollution). To sum up, it is difficult to get convincing and scientific conclusions from direct research on realistic models.
"That's why we're doing Language Model Physics, which is to explore a whole new way of thinking about AI models. Zhu Zeyuan said.
Finding the problem is one thing, and if you want to solve the "reversal of the curse", it is a new extension of the problem. To this end, Zhu Zeyuan and the "Inference Memory" research group of FAIR Labs have teamed up to provide a real-life solution based on the findings in the ideal environment—random word splitting inversion training.
The main thing is to randomly split every 1-25 consecutive tokens (corresponding to about 1-15 English words) into a group, and reverse the entire article on the premise of keeping the order of each group unchanged.
The language model is trained using both the forward text and the inverted text. If the same data will be inverted for multiple times, the words can be split by a different random method each time, which invisibly increases the diversity of the data and enhances the access efficiency of the large model to knowledge.
On the other hand, randomly splitting and flipping words also simulates human speed reading. In other words, when we read a text quickly, the eyes are also randomly disassembling and even reading in disorder. This includes turning books and reading repeatedly while learning important knowledge.
The researchers referred to this method as "reverse training" and tested real data on the LLaMA-2 model.
Figure | Tested on a real LLaMA-2 model, reversal training can overcome the reversal curse (source: arxiv[1])
At the same time, they also made an important discovery: if both forward and reverse training is performed, it will not affect the forward training results, and will not reduce the score of the traditional benchmark dataset.
Regarding the impact of the "Language Model Physics" series on the application field, Zhu Zeyuan believes that it will be very comprehensive. As a spin-off of the series, Reverse Training Conquers the Curse of Reversal is likely to be used in all application scenarios for all companies while helping to solve one of the many problems of large language models.
"Of course, there is a process for all theoretical research to be put into practice. I welcome all researchers to refer to the theoretical guidance given in our paper to find gains in practical applications. Zhu Zeyuan said.
In addition, it is worth mentioning that in July 2024, Zhu Zeyuan will be invited to hold a series of lectures on "Language Model Physics" at ICML 2024.
He is committed to challenging every dimension of artificial intelligence, hoping to explore the universal physical laws of large language models
It is understood that Zhu Zeyuan studied in the Department of Physics of Tsinghua University as a bachelor's degree, graduated from the Department of Computer Science of the Massachusetts Institute of Technology in the United States, and was a disciple of Professor Silvio Micali, a Turing Award winner.
He was a two-time gold medalist in the International Olympiad in Informatics, a gold medal in the global finals of the International Collegiate Programming Competition, and a world runner-up in the Google Code Jam.
Prior to joining FAIR Labs in 2022, Zhu worked at Microsoft Research headquarters.
"When I joined FAIR Labs, I was given 100% research freedom to independently initiate projects and select what I consider to be the most important AI topics for long-term research. The "Physics of Language Models" project is a long-term project that I am responsible for. Zhu Zeyuan said.
As mentioned above, "Reversal Training to Overcome the Reversal Curse" is a spin-off of the project.
However, when he first participated in the project, Zhu Zeyuan was not very "active". This is mainly due to the fact that he has always been cautious about participating in scientific research projects, given his limited energy.
"When I was contacted by Sukhbaatar, the project leader, I told him from a theoretical point of view that he had proven that the data inverse training was effective in an ideal environment. So, I think the reverse training method is too simple, and it just needs to do more large-scale experiments. He said.
But Sukhbaatar asked rhetorically, "Then why did you publish LoRA in the first place?"
This question prompted Zhu Zeyuan to think and reflect for a long time, and finally made the decision to change his mind.
Among them, LoRA is a simple and effective fine-tuning tool that Zhu Zeyuan participated in developing when he was working at Microsoft Research. At the time, he thought the tool was too simple, but now the latter has become the most commonly used fine-tuning algorithm in the industry, and almost everyone in the industry knows about it.
After the project "Reversal Training Conquers the Reversal Curse" began, Zhu Zeyuan and his collaborators found that different reversal training strategies had different effects than they had initially expected. In this regard, they also made a detailed comparison in the paper.
"In general, if an algorithm is simple and useful, and does not require complex mathematical formulas, isn't that what we humans want most?" Zhu Zeyuan said.
In addition, on the basis of the current research, he told us that the "Physics of Language Models" project has also developed a follow-up plan, including two papers from the second part of the project, "Language Model Inference Research", which can be published in 2 months, and will study and improve the reasoning ability of AI models on elementary school math problems in an ideal environment.
Zhu Zeyuan said: "We have a very ambitious goal, that is, to eliminate the false and retain the true in an ideal environment, challenge every dimension of artificial intelligence, and summarize the universal physical laws of large language models." ”
At the same time, he also believes that the "Language Model Physics" project, which focuses on studying large language models in ideal environments, is different from most scientific research.
"To me, it's like a new discipline and a new way of studying a problem, and it's very exciting. Therefore, I almost stopped all the scientific research directions I was working on and threw myself into them. He said.
Even though he was criticized and questioned during the research process, including whether the measured data was too idealistic, perhaps too limited, and different from reality, he remained unconcerned.
He always adhered to the words of Giordano Bruno, an Italian scientist who adhered to heliocentrism, who once said, "Truth does not change because most people believe or do not believe".
Resources:
1. O.,Golovneva, Z., Allen-Zhu, J., Weston. et al. Reverse Training to Nurse the Reversal Curse. arXiv:2403.13799v1（2024）.https://doi.org/10.48550/arXiv.2403.13799
2. Trinh, T.H., Wu, Y., Le, Q.V. et al. Solving olympiad geometry without human demonstrations. Nature 625, 476–482 (2024). https://doi.org/10.1038/s41586-023-06747-5
3. Z. Allen-Zhu, Y. Li. Physics of Language Models: Part 3.2, Knowledge Manipulation.arXiv:2309.14402（2023）. https://arxiv.org/abs/2309.144027
Typesetting: Liu Yakun