What does the huge paradigm shift brought by ChatGPT mean |

Finish|Kong Yuexin

Editor|Ma Jiying

Head photo photography|Deng Pan

ChatGPT "fainted".

When he first talked to ChatGPT last year, Zhou Bowen deliberately gave it a difficult question: "My friend is 10 months older than his leader, they are now married for 3 years, how many months is he older than his leader when he is married for 50 years?" ”

ChatGPT was indeed stumped by this simple reasoning like a brain teaser, "It analyzed and calculated with me one by one, to the effect that it is now 10 months older, married for 3 years, 47 years away from their 50 years of marriage, 12 months in a year, 47×12 equals, plus 10 months, he will be more than 400 months older than his leader at that time." ”

Zhou Bowen, Huiyan Chair Professor of Tsinghua University, Tenured Professor of the Department of Electronic Engineering of Tsinghua University, Director of the Collaborative Interactive Intelligence Research Center, and founder of Zhiyuan Technology, is clearly aware of the advantages and shortcomings of current technology due to his long-term research on natural language, large models and human-computer dialogue related technologies. Zhou Bowen said that at present, ChatGPT can answer questions related to writing and summarizing with high emotional intelligence, but once it involves the combination of knowledge and reasoning, its shortcomings are revealed.

Despite this, Zhou Bowen is still very optimistic about the prospects of ChatGPT, "Our attitude towards ChatGPT is 'not surprised by the arrival, the impact is not underestimated, and there are ways to do it in the future'. ”

At present, ChatGPT is widely used, which requires a lot of data and computing power to support large models. In Zhou Bowen's view, although the domestic large factories with abundant funds and full of ability have come down, but limited by their own business development, and will also experience a long research and development time, coupled with various concerns of large manufacturers in the company's development decisions, such as whether the cognition and expectations of the management and technical leaders are consistent, but also consider the stock price, return on investment, etc., who (the big model) can run out Now there is a question mark.

Zhou Bowen also proposed another feasible path, "I think that combining the application of end-to-end training in the vertical field and gradually developing into a large model model model is a path worth paying attention to and exploring, especially for startups." Because the current market is large enough, startups only need to do a good job in a vertical field first, help their own large models bring a closed loop of data and scenarios, and because the customer value is clear and has clear payment model support, the path of more and more powerful models will be opened. ”

This end-to-end path has also been gradually verified in Zhou Bowen's entrepreneurial process. At the end of 2021, Zhou Bowen founded ProductGPT and developed a multi-round dialogue model ProductGPT, aiming to deeply interact and collaborate with professionals to help enterprises efficiently create explosive innovative products. On March 1, Zhiyuan Technology announced that it has completed a hundreds of millions of yuan angel round of financing led by Qiming Venture Capital and co-invested by Matrix Partners.

Recently, Zhou Bowen delivered a keynote speech "When we chat about ChatGPT, what do we chat about?" in an internal exchange with "China Entrepreneur". It mainly introduces the latest progress in the field of collaborative interactive intelligence and multimodal learning, and the prospect of ChatGPT application in the future industry.

Highlights of the presentation include:

1.The emergence of ChatGPT is inevitable, and OpenAI was the first to make it by chance.

2. ChatGPT is still far from the level of general artificial intelligence, and its reasoning, knowledge and other capabilities have obvious shortcomings.

3.ChatGPT realizes multiple rounds of collaborative co-creation between humans and AI for the first time, which brings a huge paradigm shift. Once the paradigm is established, a new flywheel of AI driving new knowledge discovery will be formed.

4. The cost of developing a general large model is very high and it is a constantly improving mobile target, starting from the vertical large model plus scene end-to-end, and slowly iterating a larger business model, which may be a more suitable approach for startups.

The next breakthrough in 5.AI will shift from a purely virtual presence to helping people gain more effective insights, form new knowledge, and complete tasks in the physical, biological, and information worlds, creating higher-value scenarios.

The following is the content of the speech, which has been deleted.

ChatGPT's weaknesses and strengths

For those of us in the industry, we foresaw the emergence of AI capabilities early on. There is a long chain of AI development, from machine learning to neural networks to deep learning, to Transformer models, to GPT-3, InstructGPT, all the way to ChatGPT, the whole path is very clear, so the emergence of ChatGPT is not a completely unforeseen thing.

Even without OpenAI, AI capability technology would have emerged in the next two years, so I think the emergence of ChatGPT is inevitable, and OpenAI is a company that made it by some chance.

Why does OpenAI make ChatGPT?

I think that although this work itself is OpenAI's own involvement, it is definitely inseparable from the help of the entire academic community.

Like the Transformer model we said, it is based on the further deepening of academic theories such as multi-head self-attention, and after success, more research is obtained to analyze the rationality of its mechanism. These academic community studies have verified that large models can indeed store rich knowledge after being trained on a large amount of data, and their performance is not randomly generated overfitting. These advances have given companies the courage to spend a lot of money on training models, and they need evidence to be on the right track, and these studies are a very good piece of evidence.

In addition, there must be competition, if the GPT model does not compete with the BERT (Google's pre-trained model) model for a long time, it will not progress so fast.

ChatGPT also tested two theories, one is that the more data, the stronger the ability, that is, the theory of linear amplification "Scaling Laws"; The other is "capability emergence", that is, after the model is large to a certain extent, it will suddenly begin to integrate and penetrate.

But in my opinion, ChatGPT is still nowhere near the level of general artificial intelligence. ChatGPT demonstrates effectiveness through massive amounts of unsupervised learning. But for artificial intelligence to be more versatile, it also needs to have the ability to combine the three capabilities of knowledge, calculation, and reasoning.

ChatGPT is currently doing better calculations, and there are also some capabilities in reasoning, but the degree of complex reasoning is relatively low. For example, people can perform multi-hop reasoning and derive C directly from A. But ChatGPT's reasoning ability quickly drops to 20% accuracy after more than two hops, so I say this ability is not enough. At the level of knowledge, the current knowledge of ChatGPT is incomplete.

The second shortcoming of ChatGPT is that bigger models don't necessarily mean better. When the scale of the training model reaches a certain limit, the performance of AI on certain tasks will decrease. Because the real imagination and reasoning ability of AI are currently flawed, once the more data is fed and the larger the model, its thinking and creativity will be solidified.

ChatGPT can do where it is now, in addition to the emergence of the capabilities of the underlying large model, it has a very major contribution, that is, to strengthen the ability of AI through human collaboration and interaction.

For example, if we ask GPT-3 (the basic model of ChatGPT), explain the moon landing to a 6-year-old. Finding possible answers is not difficult for GPT-3 at all, it can find at least 4 answers: a is to explain this problem from gravity; b can start from history, such as the Soviet satellite went to the sky, which led to a sense of crisis in the United States, so the moon landing project was launched; c can start from astronomy, the moon is an ancient satellite of the earth; D can start from the good wishes of human beings, there is Chang'e and jade rabbits on the moon, and we humans always want to go there.

What's so hard about this problem? It was GPT-3 who didn't know which answer was better for a 6-year-old.

OpenAI has come up with a very good way to have people sort the answers. When facing a 6-year-old child, the order of people is d>c>a=b, that is, starting from d, 6-year-old children are more acceptable. When people give feedback to GPT-3, it will learn the feedback, learn how to order the answers, and the model after the order will change from the original GPT-3 to InstructGPT (the initial version of ChatGPT). GPT-3 has about 175 billion training parameters, while InstructGPT is only 1% of its size, or 1.3 billion parameters. But after learning the model, you ask it again, write a story about a frog for a 6-year-old, and he will say once upon a time at the beginning, similar to the beginning of a children's song.

It can be seen that human feedback is very important to ChatGPT, allowing it to learn and sort with human values and understanding, which is why human users feel that ChatGPT has high emotional intelligence. But this is what many people worry about, ChatGPT can learn output with values that train users, and scoring through sorting will change the priority of its answers, which is why many regulators are paying attention to this issue.

In addition, ChatGPT's multimodal collaborative interaction is also very important. At the same time, the co-evolution of AI and the environment is also important. ChatGPT, a human-machine collaboration system, is difficult to use in very complex real-time terminals. How to effectively use ChatGPT-like capabilities under different edge computing resources and communication bandwidth? This is a very important topic in the industrial Internet and its production practice.

We are also now studying how to make the cloud ChatGPT capabilities can be deployed in the production, design, and practice of the factory for effective use, through the interaction and collaboration between the terminal and professional users, the human professional knowledge learning is put into the edge end model, and at the same time, the cloud ChatGPT can continue to iterate according to the progress of a large number of edge ends. According to the InstructGPT model, it has become the original cloud model directly interacting with people, and now the cloud large model interacts with the edge small model, and these small models interact with people, which is the collaboration between AI and the environment that our team researched. This is also a major topic in academia that is worth studying in the next 10 years.

A huge paradigm shift

The emergence of ChatGPT is a milestone event, over the past two decades, the well-known AI applications such as DeepBlue, IBM Watson and AlphaGo, often by competing with humans to create hot spots, with effects that surpass humans to gain widespread attention, and ChatGPT for the first time proposed human-AI collaborative co-creation, which brought about a huge paradigm shift. That is, the new round of innovation in artificial intelligence will definitely revolve around the collaborative co-creation of man and machine. This pathway will lead to a larger number of applications, which in turn will lead to an evolution of the productivity landscape. The impact on AI technology companies is profound.

In addition, at a higher level, ChatGPT has a profound impact on related industries.

Let me give you a scientific research example, the cover article of Nature magazine on January 5, mainly discusses the problem of more and more papers in human science in the past few decades, but fewer and fewer breakthrough results, not only in China, but in the whole world.

I think one of the important reasons is that with the development of science and technology, each discipline has been very perfect, and information cocoons have begun to form in the discipline. In a very small subfield of the discipline, there are more and more papers, and in this field, researchers have to read a large number of papers to cover a very small field, so the information overload in the information cocoon, the barriers between the cocoon rooms are too high, and eventually a researcher has to spend a lot of time to master this knowledge, then he has less time to create, and there are fewer opportunities for cross-innovation.

Imagine if artificial intelligence systems can master all the basic content in a large number of fields, and then talk to people, communicate, help people do all kinds of verification, argumentation and calculations, and inspire people, so that eventually people can have more time to do the most creative breakthrough work.

This is a new paradigm that we are discussing. Once this paradigm is established, it is extremely significant and will form a new flywheel of artificial intelligence and knowledge discovery. That is, the stronger artificial intelligence does, the stronger its knowledge, understanding, reasoning and other combination capabilities, the more it can help human beings discover more and better breakthrough scientific advances and new knowledge, including the discovery of new drugs, cancer treatment, human brain research, etc.

And with the acquisition of this new knowledge, the more we can help the world create better artificial intelligence, and new artificial intelligence breakthroughs can bring more knowledge, forming a positive flywheel of artificial intelligence - knowledge.

AI accelerates industrial innovation and application

I believe that artificial intelligence will help human beings to do better high-quality development.

Because AI can help enterprises innovate, this is also something I pay great attention to in the field of industry-university-research integration. We hope to use the data analysis, understanding reasoning, computing power and design ability of artificial intelligence to help people better understand the market, gain insight into consumers, and then design innovative products. That is, generating ideas, generating products, completing design drawings, generating sales plans, and other work that originally required many professionals to complete, can let AI help speed up the completion, and finally let people make judgments, which is a new innovation brought about by generative artificial intelligence and human-machine collaboration.

If you think about it this way, there are many industry opportunities worth paying attention to. Therefore, many investors ask me, how do I see the future industry opportunities of artificial intelligence and large-model ChatGPT?

My answer is, look at a VC in Silicon Valley called A16Z. This very well-known high-tech venture capital firm divides the entire generative artificial intelligence AIGC technology into the following categories: the lowest level is computing hardware, such as GPUs; The upper layer is the cloud computing platform, such as AWS, as well as the domestic Alibaba Cloud and Tencent Cloud; Further up is the closed-source artificial intelligence general large model, such as GPT-3, ChatGPT and other companies such as OpenAI, and some open source models. Based on these models, there are companies doing various applications; A particularly noteworthy category is the integration of the underlying self-owned large model capabilities and applications, which we call end-to-end companies.

A16Z's view is that at present, ChatGPT and other very new cutting-edge technologies, the specific sustainable business model is not clear. Unlike the underlying computing hardware and cloud, which will certainly profit, including companies such as Microsoft; The application layer does APP, because there is no moat, it is difficult to ensure profits. The only thing that is clear is that end-to-end companies, such as Midjourney in the US, have a very clear prospect, and this company now receives $100 million a year in sustainable subscription fees. According to the A16Z classification of business models, it is an end-to-end application.

Why is this model more competitive in the long run? From a technical point of view, it is because it forms a very important closed loop of infrastructure, large models, application scenarios and end users. When the company has specific functions for end users to use, it will generate a lot of usage data, and the data feedback can help improve the application, but also help improve the basic model capabilities, and finally the model will continue to be optimized and iterative.

In addition, the big model is also the focus of future industrial development, but the business model of the big model is worth exploring. Because the cost barrier for large models is very high, both large companies and small businesses have their own burdens. So I think it might be more appropriate to start end-to-end and slowly iterate on a larger business model.

This raises the question of whether the closed-source model and a vertical application can be packaged together to achieve an end-to-end business model.

ChatGPT has some existing weaknesses, so the answer to this question is a clear no. At present, ChatGPT is like a typical "panacea", it knows a little about everything, and can integrate some information very logically, but many answers have problems that cannot guarantee the accuracy and quantification of information; Second, for the correlation of information, especially the data itself, it is difficult to establish the logic behind it; Third, it doesn't provide unique insights, it's basically a more advanced gramophone.

Once a vertical professional user uses ChatGPT, he will find that the answer provided by ChatGPT is either what professional users know, or it may be serious and cannot guarantee that it answers correctly.

Therefore, Zhiyuan Technology has made a vertical model, called ProductGPT, to help enterprises do product innovation. It helps vertical business workers to innovate and give a very comprehensive analysis when answering. Because it has very detailed data support, in addition, it can carry out in-depth analysis according to brand, category, and characteristics, which really helps professionals.

ProductGPT, as a collaborative interactive artificial intelligence in the vertical field, according to the tests we have collected so far, it can increase innovation opportunities by 10 times, speed the time-to-market by nearly 10 times, and greatly reduce the cost of innovation, helping enterprises bring revenue, business growth and profits.

Implications of ChatGPT for industry, policy, regulation, etc

ChatGPT's multiple rounds of generative conversations may raise some new AI governance issues. I summarized three points:

1. How to label the output of ChatGPT from the perspective of governance?

Label AI-generated content with potentially contradictory and conflicting information, wrong knowledge, outdated knowledge, lies, prejudices, racial discrimination, etc., to help users treat it with caution. For example, we need to discuss whether and how to add a digital watermark to AI-generated content so that people know that it is AI-generated and be more cautious when they see it?

2. Research on intellectual property issues related to ChatGPT and related legal issues.

ChatGPT will inevitably cause many copyright problems in practical application scenarios, and its training process uses a large number of Internet corpus, and this imitation of existing text data will also constitute infringement. Does the intellectual property rights of ChatGPT-generated content belong to the user, or to the owner of the original content, or to OpenAI? At the same time, because of who should bear the responsibility for the adverse consequences caused by the use of ChatGPT, the relevant legal issues also need to be studied.

3.The definition of the scope of use of ChatGPT, which has already had an impact on many industries.

Example 1: Middle school students will use ChatGPT to complete their homework, but due to the uncontrollable nature of the content they generate, it will bring legal, security, ethical risks, and even lead students to commit crimes.

Example 2: In academic research, if the thinking task is given to an automated chatbot, it also violates academic ethics.

At the beginning of this year, I published a paper in the ACM Computing Surveys, a journal of the American Computer Association, "Trustworthy Artificial Intelligence: From Principles to Practice", which proposed that trustworthy artificial intelligence is not an isolated problem, and that the explainability and anti-attack of artificial intelligence must be considered together, including the problems encountered by ChatGPT, which are worth studying academically. Issues related to generalization, explainability, transparency, reproducibility, value alignment, and responsibility all need to be addressed. So in general, there is still a lot of work to be done in artificial intelligence, especially in the context of ChatGPT, academic research, regulation and practice are trinity, to iterate on each other.