Generative AI Report: Talent has an impact on the future of AI beyond computing power

Whether it is a model or an application, it is inseparable from hardware manufacturers or cloud service providers, computing power is currently the most scarce resource, is the most significant part of the cost structure of large models, GPU is the key computing power hardware for training models and accelerated inference, GPU performance actually determines the pace of this emerging industry. But in the long run, the impact of talent on the future of artificial intelligence exceeds computing power.

Giants are busy developing large models and have not yet considered deep cutting into specific application scenarios, which is the blue ocean of start-ups and also has reefs on the road to development. The current generative AI market is in the early stages of technology-led, and there are opportunities for platform companies with a market capitalization of hundreds of billions of dollars. Within 3 years, the core driving force of disruptive AI applications will come from the innovation of the underlying model, and the role of the model will be greater than the role of product design.

Zhou Zhifeng, partner of Qiming Venture Capital.

On July 7, at the 2023 World Artificial Intelligence Conference Qiming Venture Capital Forum "Generative AI and Big Model: Change and Innovation", Qiming Venture Capital jointly released the "Generative AI" report. In the AI 2.0 era, the large model obtained after large-scale data pre-training can be directly used by various downstream tasks, whether it is a model or an application, it is inseparable from hardware vendors or cloud service providers, computing power is currently the most scarce resource, GPU (graphics processing unit) is the key computing power hardware for training models and accelerating inference. But in the long run, the impact of talent on the future of artificial intelligence exceeds computing power.

The report believes that the current generative AI market is in the early stages of technology-led, and there are opportunities for platform companies with a market value of hundreds of billions of dollars. Within 3 years, the core driving force of disruptive AI applications will come from the innovation of the underlying model, and the role of the model will be greater than the role of product design.

Zhou Zhifeng, partner of Qiming Venture Capital, said that the progress of human science and technology is constantly accelerating, just like a wave on the ocean, the frequency is getting higher and higher, and the waves are increasing. We are in the early stage of generative AI development, the future development speed will be very fast, any great technology trend appears, will be mixed with bubbles, I hope everyone can ignore the industrial cycle, ignore noise and bubbles, and do things steadily, in order to promote the development of AI.

Computing power is the most scarce resource, and theoretically the cost of training large models decreases over time

The development of artificial intelligence has gone through more than 70 years. The report shows that the advancement of four generations of underlying technology has driven the development of four waves of artificial intelligence. The first wave of small-scale expert knowledge took 40 years to complete; The second wave of shallow machine learning took 20 years to complete; The third wave of deep learning took 8-10 years to complete and achieved certain achievements. The latest wave of AI started with the Transformer-based pre-trained model in 2017 and broke through the technological singularity after the release of the GPT-3 model in 2020.

The report proposes the AI 1.0 era and the AI 2.0 era. In the AI 1.0 era, it is necessary to develop specific models for specific tasks and use relevant data, and task and model coupling. In the AI 2.0 era, large models obtained through large-scale data pre-training bring good results and generalization capabilities, and can be directly used by various downstream tasks.

Companies in the AI 2.0 era will be divided into three layers: one is the infrastructure layer, which is mainly a tool chain manufacturer that solves the training/inference/deployment of large models and an intelligent computing center that provides GPU resources, and the intelligent computing center is a new generation of AI chips or the next generation of general-purpose GPUs. The second is the model layer, which is mainly to develop large models and provide AI model services or API (application programming interface) services, including GPU resources required for training and inference. In addition to such large models of the base, companies that offer vertical models for specific industries or scenarios are also included. The third is the application layer, that is, application companies that focus on solving a specific field, including application companies that develop large models by themselves and application companies that use third-party large models.

New applications require new infrastructure. The infrastructure of AI 2.0 is an intelligent computing center centered on providing intelligent computing power. Whether it is a model or an application, it is inseparable from hardware manufacturers or cloud service providers, computing power is currently the most scarce resource, but also in the easiest to profit the most important resources, is the most significant part of the cost structure of large models, GPU is the key computing power hardware for training models and accelerated inference, GPU performance actually determines the pace of this emerging industry. The report shows that training a large model similar to GPT-3, that is, 175 billion parameter scale and 300 billion tokens, requires 3.15*10^23FLOP computing power requirements. If only 1 NVIDIA V100 chip is used, under the theoretical computing power of 28TFLOP with FP16 accuracy, it takes 357 years of training; To shorten the training time, it is necessary to increase the hardware investment, but the efficiency of computing power will decrease. If only one NVIDIA A100 chip with a theoretical computing power of 312TFLOP at FP16 accuracy is used to retrain GPT-3, it will take 32 years. The report mentions that theoretically, with the improvement of hardware performance and software optimization, the cost of large model training will decrease over time.

Generative AI closely combines research and innovation, and there are more problems to be studied on the way to AGI

With the advancement of computing power and models, more startups are pouring in, facing competition and possible giant crushing. But competition fosters innovation, and unlike startups that emerged rapidly in the direction of productivity tools in 2022, a larger percentage of new companies in 2023 focused on innovation in the underlying technology. Large-model startups have also begun to diverge, and while general-purpose large-model startups are in the ascendant, vertical large-model companies for specific directions such as healthcare, e-commerce, scientific research, industry, autonomous driving and robotics have begun to emerge.

The report argues that it is still the early days of AI 2.0, and the infrastructure and core technologies are not particularly mature; The giants are busy developing large models and have not yet considered deep cutting into specific application scenarios. This is a blue ocean for start-ups, and there are also reefs in the path of development. The current generative AI market is in the early stages of technology-led, and there are opportunities for platform companies with a market capitalization of hundreds of billions of dollars. Within 3 years, the core driving force of disruptive AI applications will come from the innovation of the underlying model, and the two cannot be decoupled, and the role of the model will be greater than the role of product design.

In cutting-edge research, 2022 and 2023 are two years of breakthroughs in generative AI technology, and the report finds that a prominent feature of the field of generative AI is the close integration of research and innovation processes, many of which are implemented within enterprises and quickly launch products. From GPT-4's technical report to Microsoft's research papers, it has shown that large models have word processing ability and mathematical reasoning ability close to humans, but on the way to general artificial intelligence (AGI), there are more problems that need to be studied and solved, such as confidence calibration, long-term memory, continuous learning, personalization, planning and conceptual leapfrogging, transparency, cognitive fallacies and irrationality. The most important research direction in the past six months is to crack and understand the mysterious and exciting intelligent "emergence" of large models, which need to go beyond the ability to predict the next word, and also need a richer and more complex "slow thinking" deep mechanism to supervise the mechanism of "fast thinking" to predict the next word.

The report shows that the best cutting-edge research must be to study and solve problems encountered in technology-scale applications, such as studying how to reduce hallucinations, tune large models to output real content more accurately, and train stronger reasoning ability; How to train models more intensively, lower the threshold, and launch new products that can be used by all walks of life and consumers; how to interact with the real physical world like a human being; how to be an assistant to the complex work of humans, designing and helping to carry out scientific experiments; how to influence employment and thus policy responses; How to make AI safe and trustworthy.

In the long run, the impact of talent on the future of artificial intelligence exceeds computing power. Chinese researchers have published more papers than the United States in terms of numbers, but at the top of the pyramid, whether it is research or entrepreneurship, the United States still has a clear advantage. Globally, the focus of AI research and innovation is shifting from universities to enterprises, and the top three institutions with the most top scholars in the United States are Google, Microsoft and Meta, which together attract 30% of the top scholars in the United States, and China is still dominated by universities.