Editor's note:
Computing power construction is set as one of the key tasks in the "14th Five-Year Plan" period, and under the active layout of a group of cloud vendors, computing power centers have been completed and landed. Under the background of the rapid expansion of computing power demand in the whole society, how to revitalize computing power resources has become the key, so the computing power network has become a strategic requirement for the development of the country, society and industry.
The full text is about 6168 words, and the recommended reading time is 16 minutes.
In the intelligent era, computing power will become the key productive force for social development. The growth of computing power is not only positively correlated with GDP growth, but is also changing the model of scientific innovation, so that experiments that previously required a lot of time, manpower, and material resources can be completed by computers.
On this basis, just as electricity has crossed from various power generation categories such as hydropower and thermal power to the power network with unified services, the computing power network is an inevitable trend in the development of computing power infrastructure.
From hash power to computing power network
Computing power is becoming an important way to change the economic growth model, and the scale of computing power is also becoming an important indicator to measure the economic growth of a country. According to the 2021-2022 Global Computing Power Index Assessment Report jointly compiled by Tsinghua University and the Internet Data Center (IDC) (see Figure 1), there is a strong positive correlation between computing power and economic growth. Every 1 point increase in the hashrate index can bring about 1.8‰ of GDP growth and 3.5‰ of digital economic growth. Up to now, the highest computing power index is the country of the United States, followed by China, followed by Japan, Germany, the United Kingdom, France and so on.
Computing power not only has a strong positive correlation with the economy, but also brings a huge impact in the field of scientific innovation. Three-dimensional structure prediction of protein molecules has been a problem that has plagued the field of structural biology for more than 50 years. In the past few years, due to the rapid development of intelligent computing power, protein structures that originally needed to be predicted by biologists' repeated experiments can now be accurately predicted by computers and the Alphafold2 algorithm. Therefore, intelligent computing power based on artificial intelligence is also changing the model of scientific innovation.
The current situation of China's computing power development
China has become one of the main forces in the development of computing power in the world, especially in terms of the increase in computing power, China is the leader. At present, the top two countries in terms of global computing power are the United States and China, then Japan, Germany, the United Kingdom, France, Canada, South Korea, Australia and other countries, and India, Italy, and Brazil are just getting started. But even in terms of the growth rate of power development, the mainland ranks first in the world, reaching 13.5%, and the United States is 5.0%. Like economic development, the mainland is continuing to rush forward with very high computing power development growth. For the future development of the mainland network, the computing power advantage will no longer be limited to the volume of computing power, but higher computing efficiency, wider application of emerging technologies, and more sound infrastructure support.
From grid computing to computing networks
After the concept of grid computing was proposed in 2003, many scientists in the field of supercomputing around the world have participated, deeply promoting the development of grid computing. Data centers can now be interconnected on x86 CPUs' synchronous processors at a lower cost of bandwidth and computing power. In the past 20 years, the cost performance of bandwidth has increased by 100~1000 times, and the cost performance of computing power has also increased by 15 times. So when discussing computing power and computing power networks today, the whole background and environment have changed drastically compared to grid computing 20 years ago.
In today's society, the development of the power network is more mature. Electricity networks send electricity to where customers need it. Electricity is composed of a variety of different sources of electricity, including thermal power, hydropower, solar power, wind power and other electricity sent from a variety of different power plants. Users don't care where the electricity comes from, and the electricity they get may come from different power plants in a certain proportion, such as thermal power generation accounting for 60% and hydropower accounting for 40%. The proportion of new green power sources, such as solar and wind power, is gradually expanding, but it does not affect the end-user experience. This is where power networks come in. Our country's power network, including UHV engineering, "source-grid-load-storage" network-wide coordination, "West-to-East Power Transmission" DC UHV, smart grid, etc., have all developed very well.
Taking the computing power network and the power grid as an analogy, the computing power in the computing power network basically includes three categories: supercomputing, intelligent computing power and cloud computing power. When using it, we hope to be able to design a computing power network, encapsulate these different computing power, and deliver it to the place where users need it. This computing power network can consider a variety of factors and is beneficial to society, economy, science, etc.
Possible technical approaches to computing power networks
Developing a computing power network requires four technical steps (see Figure 2): the first step is to submit tasks, data, and run the application in a single data center; The second step is that for homogeneous data, the application can be submitted in the same environment but in different data centers, the computing power scheduler uniformly schedules resources, and the application still runs in a single data center; The third step is that the application can run across multiple homogeneous data centers, or it can be submitted uniformly across heterogeneous data centers, but ultimately run in a single data center. The fourth step is similar to the power grid, completely oriented to users of computing power, users only need to submit application requirements, and then the computing power network performs resource scheduling from the network, and integrates different heterogeneous computing power through the compatibility layer of the network for unified use. At this point, an application can run simultaneously across multiple heterogeneous data centers to get results.
The first step to the second step is from a single computing power center to a homogeneous multi-computing power center, which is a problem that should be solved at present, including the uneven utilization rate of the isomorphic multi-computing power center, and the main applications include offline data processing, model training, scientific computing, etc. The main technical challenges include: application packaging technology, computing power network scheduler, rapid data migration, billing, permissions, and data security assurance. Therefore, the application needs to be containerized first, just like containers, in the past, ships, trucks, and trains were loaded with their own storage structures, and intermediate links required a variety of buckets, cranes and even manual transportation of goods from one container to another, and with containers, the whole process became simple and efficient. Similarly to applications, at this stage, the computing power network should also do a good job of containerized packaging of applications, which is conducive to compatibility with homogeneous hardware system packaging and completing the centralized transmission of computing power resources and data.
The computing power network scheduling platform includes resource management, job scheduling, scheduling strategy, etc., so that the resources and application data between different computing centers can be well adapted through the computing power network scheduling resource management. The scheduling strategy includes a variety of strategies, one strategy is to use the concept of global raw data to establish a global unified data catalog, quickly find the associated data, so that the storage of data and resources can be clear at a glance. The other two strategies, one is to follow the number, and the other is to follow the number. "Number follows calculation", that is, where the computing power is, the data is there, which is generally used in scenarios where the amount of data is relatively small.
The second to third step span consists of two stages (see Figure 3), the first stage is the isomorphic multi-computing center to the isomorphic cross-computing center. The advantage of this is that the software at the top level of the application is the same, although it may have different versions, but in most cases it can reduce the difficulty of adaptation, and then it can be submitted uniformly through homogeneous data centers, and slowly achieve submission and operation across multiple homogeneous data centers. This stage mainly solves the problem of insufficient computing power in a single center and data not leaving the domain. The main applications include: large model training, federated learning, federated querying, etc. The main technical challenges are the network interconnection and transmission technology of computing power center, the communication optimization technology for parallel applications for cross-computing power center network, privacy computing and federated learning technology. The important things in the network interconnection and transmission technology of computing power center are network interconnection, perception network orchestration, and native data transmission and expression of computing network. In the stage of parallel application communication optimization technology for cross-computing power center networks, some optimization tasks need to be solved, such as considering the communication mode between different data centers, the division of intermediate tasks, the allocation of computing power and bandwidth, and so on.
Another stage of step 2 to 3 is the crossing of homogeneous multi-computing power centers to heterogeneous multi-computing power centers. The CPU, computing resources, etc. used on the homogeneous multi-computing power center are the same, so the packaging and scheduling on it are easier. On the heterogeneous multi-computing power center, the adaptation difficulty of packaging and scheduling is greatly increased. Because it uses different computing power resources, it is necessary to propose better adaptation methods for heterogeneous data and resources.
For the entire programming environment, resource management organization, load balancing is a big challenge, which requires unified programming and optimization technology on heterogeneous platforms, and existing compatibility layers, such as MPI (scientific computing), PyTorch and Spark (big data), in which the portability of AI to various artificial intelligence accelerators is still a pain point. This involves the optimization of the underlying operators of computing power, for example, for some heterogeneous computing power, including the computing resources provided by NVIDIA's GPUs, it is necessary to propose a unified programming heterogeneous architecture to support their operator optimization. In addition, the challenge of heterogeneous scheduling at the workflow level is relatively greater, and it is necessary to take advantage of the advantages of different computing centers to solve problems collaboratively. Predecessors have some effective attempts in distributed computing that can be used to solve such problems.
The third to fourth steps are the leap to heterogeneous cross-computing power centers. From the use of homogeneous computing power centers to completely heterogeneous cross-computing power centers, it takes a long time to complete after the scheduling of the computing power network and the problem of the compatibility layer of the computing power network are solved. Its main applications include large model training, federated learning, and workflows combined with other types of business tasks.
Pengcheng Cloud Brain II practice in Pengcheng Lab
Pengcheng Lab has designed and completed the E-level intelligent computing platform "Pengcheng Cloud Brain II.", which can not only meet the local computing needs of Guangdong Province and Shenzhen City, but also provide some support for the national strategy; It can not only be used for theoretical research support, but also for core technology management and development, and can also meet the needs of some intelligent applications.
Dedicated architecture for AI
Pengcheng Cloud Brain II is an architecture specifically for AI (see Figure 4), which has 10 billion billion semi-floating-point operation capabilities, is equipped with 64PB of storage, and the delay between any node is only 2 microseconds, which is a fully node cross-interconnected machine.
The construction of Pengcheng Cloud Brain requires artificial intelligence dedicated acceleration hardware with super processing capabilities, open software architecture and a complete open source ecosystem. Pengcheng Cloud Brain II has reached the E-level computing power, which is an AI infrastructure that has reached the world's leading level in computing density, computing power scale and training speed, and can provide the best support and services for the development of Chinese intelligence.
Pengcheng Cloud Brain II performance evaluation
Since its launch in October 2020, Pengcheng Cloud Brain II has participated in the lO500 ranking five times, all of which are No. 1, and its IO performance is outstanding. In terms of AI computing power, in 2020, 2021 and 2022, for three consecutive years in the field of supercomputing in China, the AIPerf500 list jointly launched by CCF and ACM SIGHPC experts ranked No. 1. In addition, Professor Jin Hai's team of Huazhong University of Science and Technology optimized their program on Pengcheng Cloud Brain II., and obtained the first and second best results on the two tracks of this year's GRAPH500, respectively.
Since the launch of Pengcheng Cloud Brain II, it has supported the training of many large models, including Pengcheng Pangu, Pengcheng Shennong, Pengcheng Dashen, Pengcheng Bian Que, Pengcheng Tongyan, Pengcheng Changxi, etc., and has also provided some top domestic institutions for large model training.
Pengcheng Cloud Brain II has been allocated for about 492.78 days after conversion since its launch, with an allocation rate of 96.53% and an actual utilization rate of 77.17%, which is a fairly high utilization rate for a supercomputer. 25.76% of Pengcheng Cloud Brain II is used by Pengcheng Lab, 60.92% is provided to co-construction cooperation units, 7.53% is provided to public welfare institutions, 2.45% is provided to universities, and 3.34% is provided to some other related institutions. Pengcheng Cloud Brain II, as a scientific device, embodies a good openness. This machine is not only used by scientific research institutions, but also by local governments and small and medium-sized enterprises, such as Guangzhou Laboratory, Shenzhen Bay Laboratory, Shenzhen Municipal Health Commission, Traffic Police Bureau, Yuntian Lifei, etc. In view of the large demand for Pengcheng Cloud Brain II, Pengcheng Laboratory is considering designing Pengcheng Cloud Brain III, which has a computing power of about 16 times that of Pengcheng Cloud Brain II, which can further meet the super computing power needs of scientific computing.
Challenges and prospects of China's computing power network
Development vision and goals
The vision of China Computing Power Network (China Computing NET, C2NET) construction is: to build a national computing power network like a power grid, operate a computing power network like an Internet, and allow users to use computing power as easily as electricity. In order to achieve the above vision, the main goals of China's computing power network construction include: building an independent innovative computing power network technology system, building a national computing power network infrastructure covering the interconnection and efficient coordination of large heterogeneous computing power centers such as national supercomputing centers, intelligent computing centers, and data centers, and promoting the transformation of computing power supply mode.
Several challenges faced by China's computing power network
First, cyber challenges. Computing power centers are scattered throughout the country, through "ultra-high voltage" long-distance high-speed communication, so that all computing power network nodes can achieve the sharing of heterogeneous computing power resources, and it is urgent to break through ultra-wideband, ultra-low-latency network connections, such as bandwidth of more than 100 Tbit/s, latency of no more than 1 millisecond every 200 kilometers; Multi-core optical fiber, coherent optical communication, wavelength division multiplexing, etc. are possible breakthrough technologies.
Second, the challenge of computing power diversity. Centralized large-scale cloud computing power nodes (CPU clusters), intelligent computing center nodes (GPU clusters), supercomputer nodes (hybrid clusters), and decentralized edge nodes (embedded devices) urgently need to realize direct interconnection of heterogeneous nodes.
Third, the heterogeneous challenge of chip and instruction systems. The underlying chip is heterogeneous, and the CPUs are provided by Intel x86, AMD, ARM, Moore Thread, Zhixin and other different manufacturers; AI chips are heterogeneous, and sources include Huawei NPU, NVIDIA GPU, Cambrian MLU, Haiguang DCU, Pingtou Gehanguang NPU and other manufacturers. Different chip manufacturers have low willingness to research and develop the unified adaptation and scheduling of heterogeneous computing power centers, and prefer autonomous management. How to obtain the support of the underlying chip manufacturers to achieve unified adaptation and scheduling of computing power centers is a major challenge.
Fourth, operators are challenged to fight separately. Network operators focus on user experience and efficiency, it is difficult for different enterprises to communicate with each other, and how to improve the willingness of enterprises is also a major challenge due to large investment but lagging output during the computing power network construction period.
Exploration of Peng City (C2NET-0.1)
With the deployment and support of the National Development and Reform Commission, Pengcheng Lab launched the China Intelligent Computing Network Construction Pre-research Project in 2019, and developed a core software stack and distributed scheduling platform compatible with a variety of heterogeneous AI chips, with a construction cost of 350 million yuan. Acceptance completed in June 2022. The total computing power of the set is more than 2.3E of semi-floating-point accuracy.
In December 2021, the AI Industry Technology Innovation Strategic Alliance (AITISA) officially established the Intelligent Computing Center and the Intelligent Computing Network Task Force to coordinate and promote the standardization and development of the Intelligent Computing Center and the Intelligent Computing Network. Pengcheng Laboratory and a number of units proposed a series of standard plans for the "Artificial Intelligence Computing Power Network" in the intelligent computing task force (see Figure 5), and a total of 5 technical proposals and 7 demand proposals have been submitted so far. Different heterogeneous intelligent computing power is divided into different levels for standardization, which is convenient for packaging, data definition, and unified resource allocation.
The heterogeneous interconnection technology of computing power networks is also developing rapidly. Pengcheng Lab is considering the use of high-speed, ultra-wideband, low-latency private networks to achieve the connection between machine nodes. For example, 10 TB all-optical network interconnection was carried out in Pengcheng Cloud Brain and Guangzhou Supercomputer. Pengcheng Cloud Brain is interconnected with Jinan Supercomputing SD-WAN, and Pengcheng Cloud Brain is interconnected with MPLS of the University of Science and Technology of China. At the same time, research on 360-kilometer long-distance WRDMA transmission technology based on all-optical network is carried out to provide support for the realization of the "big switch" of the national computing power center interconnection.
Ultra-wideband low-latency communication between nodes is currently the main technology, and experiments with a bandwidth of more than 100 Tbit/s and a transmission distance of more than 2000 km can be achieved in the laboratory. In the future, 1.0 and 2.0 of China's computing power network are expected to realize 100 Tbit/s direct communication between all cluster nodes of "East Data and West Computing". The so-called direct connection is that the optical fiber is directly connected from one point to another, and there may be some amplification in the middle, but there is no router exchange, so the delay in the middle is controllable, not only the bandwidth is guaranteed, and the delay is very low.
China Computing Power Network Phase I Plan
The first phase of the China Computing Power Network (C2NET-1.0) is from July 2022 to December 2025, the project is supported by the Ministry of Science and Technology of the People's Republic of China, and is now being promoted and implemented, and its overall construction goal includes three contents (see Figure 6). First, computing power aggregation builds high-speed network interconnection of different nodes, develops a cloud platform, realizes unified operation and maintenance management and elastic distribution of computing power, and provides a super computing power network that can distribute and learn across nodes for large models. Second, resource aggregation gathers the most complete public data resources to achieve safe, open, shared, and trusted flow of public data, models and other resources between different nodes. Third, build the strongest ecological aggregation platform from ecological convergence, realize unified and open model capabilities between different nodes, share application innovation achievements among different nodes, and operate an open source community based on intelligent computing networks.
summary
Just as the leap from power to power network, computing power network is an inevitable trend in the development of computing power infrastructure, and applications can be efficiently executed in heterogeneous cross-computing power centers through the flexible allocation of computing power. Under the guidance of this goal, the construction of computing power networks will lead to breakthroughs in a number of core key technologies, including standardized packaging of heterogeneous computing power resources and ultra-wideband low-latency communication of computing power nodes, making the mainland the first to enter the uninhabited area in computing power technology. In the future, Pengcheng Laboratory will follow the requirements of the National Development and Reform Commission and the Ministry of Science and Technology, and the strategic scientific and technological forces of all parties in the United Nations to fully promote the research and development and construction of China's computing power network.
Source: CNCC2022 Invited Report
Author: Gao Wen