MOSS can perform a series of tasks such as dialogue generation, programming, and fact answering, opening up all technical paths for generative language models to understand human intent and have dialogue capabilities. The passage of this path provides important experience for domestic academia and industry, and will help the further exploration and application of large language models.
Qiu Xipeng said: "Although MOSS still has a lot of room for improvement, its emergence proves that domestic scientific research teams have the ability to overcome important technical challenges on the way to developing ChatGPT-like products. "The MOSS R&D project is strongly supported by the Shanghai Artificial Intelligence Laboratory. Later, this work will be shared with the industry community through open source.
"End-to-end" walk through the big language model
Since ChatGPT is not open source, and the details of its technical solution have not been disclosed, researchers have a lot of speculation about this. Some experts believe that ChatGPT is a complex combinatorial system that cannot be implemented by a single generative language model; Some experts believe that the technological gap between home and abroad in this direction is widening.
As an expert in natural language processing, Qiu Xipeng has been working on general models with multi-task learning ability in recent years. In order to explore the technical route of ChatGPT and gain first-hand experience in large language model research, he led the team to accelerate MOSS development. Today, MOSS has become the first ChatGPT-like language model publicly unveiled in China, and "end-to-end" has gone through the whole process of development of large language models.
The basic steps of MOSS development are the same as ChatGPT, including two stages: base training of natural language models and training of dialogue ability to understand human intentions. In the dialogue ability training stage, OpenAI collected at least hundreds of thousands of human instructions, asked professional annotators from all walks of life to write command responses, and then entered them into the model base to help ChatGPT gradually understand various instructions. The Fudan team adopted different technical routes to significantly improve the learning efficiency and R&D efficiency by allowing MOSS to interact with humans and other dialogue models, and efficiently completed the dialogue ability training in a short time.
Qiu Xipeng said frankly: "The gap between MOSS and ChatGPT is mainly in the pre-training stage of natural language model base. The number of parameters of MOSS is an order of magnitude smaller than that of ChatGPT, and there is still a lot of room for improvement in task completion and knowledge reserve. According to reports, this artificial intelligence assistant has entered the internal testing stage, which will obtain data with the user's permission, and will also collect user feedback, hoping to greatly enhance the dialogue ability of MOSS.
Measured dialogue, programming and other capabilities
How well does MOSS answer questions? The reporter asked it a few questions with Chinese.
"Some people think that AI will replace a lot of people's jobs, what do you think?" MOSS replied: "I don't think AI will completely replace humans, but rather help people do their jobs to some extent." For example, machines can improve efficiency in data analysis, decision-making, etc., and can save a lot of time and effort. In addition, AI can also be used to assist humans with more complex tasks, such as diagnosing diseases and developing new products. In conclusion, AI can make human work more efficient, but it will not completely replace humans. ”
The reporter asked: "What suggestions do you have for how Shanghai can better develop the artificial intelligence industry?" MOSS replied: "I think Shanghai can better develop the AI industry by: 1. Strengthen research and development in the field of artificial intelligence, support academic research, and promote scientific development; 2. Strengthen the integration of industry resources to promote the development of the artificial intelligence industry; 3. Strengthen the research and development of artificial intelligence-related products and improve the competitiveness of products; 4. Strengthen investment in the artificial intelligence industry and promote investor participation; 5. Strengthen the laws and regulations of the artificial intelligence industry to ensure the healthy development of the artificial intelligence industry." ”
It can be seen that MOSS's answers are fluent, logical and correct. However, the research team also frankly pointed out that the current version of MOSS is not stable enough, and some answers have factual errors or logic that are not smooth. "MOSS's English response level is higher than Chinese because its model base learns more than 300 billion English words, and Chinese only about 30 billion words."
During the demonstration, the research team entered multiple instructions in English, demonstrating the ability of MOSS multi-round interaction, table generation, code generation and interpretation.
"Tell me 5 sci-fi movies." "Generate a table showing these 5 films and directors." "Add a list to show when these movies were released." In this multi-round interaction that included table generation, MOSS successfully completed the task.
Like ChatGPT, MOSS has code generation and interpretation capabilities. The demotor asked it to generate a piece of Python code to implement quicksorting, and MOSS quickly completed the task.
MOSS also has ethical judgment and legal knowledge. For example, if you ask it to "make a plan to destroy humanity" and ask it "how to rob a bank", it will give a value-based answer.
Open source lowers the threshold for R&D for small and medium-sized enterprises
They also plan to combine Fudan's research results in artificial intelligence and related interdisciplinary disciplines to give MOSS more capabilities, such as drawing, speech, composition and teaching, and strengthen its ability to assist scientists in efficient scientific research.
At present, the biggest shortcoming of MOSS is that the Chinese level is not high enough, mainly because there are many Chinese web pages on the Internet, such as advertisements, and it is very difficult to clean. To this end, the Natural Language Processing Laboratory of Fudan University is stepping up the cleaning of Chinese corpus, and the cleaned high-quality Chinese corpus is used for the next stage of model training. The research team believes that this will effectively improve the model's ability to Chinese dialogue.
In the future, the research team will share the results of this work through open source to promote the analysis and development of pre-trained language models in academia and industry. The development threshold of pre-trained large language models is very high, requiring a lot of computing power, training corpus and manual labeling. In the mainland industry, only large institutions have the strength to develop large models. After MOSS is open sourced, it can effectively reduce the threshold for the development and application of pre-trained language models, allowing small and medium-sized enterprises to develop various vertical products such as intelligent customer service, smart home, and artificial intelligence lawyers on the basis of it.
"MOSS is relatively computationally intensive, and small and medium-sized enterprises can use it." Qiu Xipeng said, "I look forward to the continued cooperation between the Fudan team and the Shanghai Artificial Intelligence Laboratory to promote AI inclusiveness through MOSS and subsequent research and exploration, and empower the domestic artificial intelligence industry as soon as possible." ”