Since 2012, the amount of computation on datasets used for large-scale AI training has been growing at a rate of 10 times per year. In the case of ChatGPT, for example, its November 2022 version used 175 billion parameters, while the March version used 1.5 trillion parameters. There are three main reasons behind the increasing number of training parameters: 1) increasingly complex AI models, 2) a large amount of online generated data that can be used for training, and 3) the increasing accuracy and robustness expectations of AI applications. When we translate these needs into memory requirements, it means higher bandwidth, and higher capacity. In fact, there may not be too many challenges in the evolution of computing power at present, and the industry is facing more of a storage wall problem, that is, the development of memory and bandwidth is too slow. This is the main reason why NVIDIA can make the H200 inference speed twice that of its predecessor H100 without adjusting the GPU architecture. It is reported that thanks to HBM3e, the H200 graphics card has 141GB of memory and 4.8TB/s bandwidth.
图 | 内存接口和互联IP，图源：Rambus
According to data released by TrendForce, the mainstream demand of the AI industry has shifted from HBM2E to HBM3 in 2023, with the proportion of HBM3 demand increasing to 39%, and the market demand is expected to account for 60% in 2024. This was confirmed by Joe Salvador, vice president of IP product management and marketing at Rambus. When it comes to memory and interfaces, we have to mention a company that engraves the memory gene in its name - Rambus (Ram and bus stand for storage and secure and fast transmission, respectively). Founded in the nineties of the last century, the company is headquartered in San Jose, Silicon Valley, and its main businesses include: basic professional licensing, semiconductor IP licensing, and chip business. Among them, semiconductor IP is mainly divided into interface IP and security IP. Rambus' technologies and products target market segments such as the data center and edge computing markets, as well as the automotive Internet of Things. According to the financial report released by Rambus, the company's business revenue from chips and IP reached a new high in 2022, of which the business revenue from products achieved a year-on-year growth of 58%, and the operating cash flow reached $230 million.
Figure | Rambus data center solutions, source: Rambus
The reason behind Rambus' outstanding performance in 2022 is that data centers are its main focus market, with more than 75% of chip and IP business revenue coming from data centers. In recent years, driven by the needs of big data, artificial intelligence, Internet of Things and other industries, the data center track has developed rapidly and shown a continuous growth trend. Against this backdrop, Rambus recently introduced the HBM3 memory controller IP, which delivers up to 9.6 Gbps of performance to support the continued evolution of the HBM3 standard. With a 50% increase in data rates compared to HBM3 Gen1's 6.4 Gbps data rate, the Rambus HBM3 memory controller delivers a total memory throughput of more than 1.2 TB/s for training recommender systems, generative AI, and other demanding data center workloads.
Figure | Rambus HBM3 controller module diagram
It is reported that the Rambus HBM3 memory controller will be available for licensing from now on. As for whether the Rambus HBM3 memory controller can meet the needs of the new HBM3e on the market, Joe Salvador said: Indeed, several mainstream memory manufacturers in the current market claim to have HBM3e memory, but from the perspective of specifications, the current HBM3e is not a formal industry standard, it is an expansion on the basis of HBM3, comparing HBM3E and HBM3, it can be found that in fact, the thickness of the stack has not changed, the supported DRAM capacity has not changed, and the total bandwidth of knowledge has changed. ”
图 | HBM内存的演变，图源：Rambus
It is worth mentioning that Rambus recently sold its very well-developed PHY business to its former customer Cadence, and in response to this strategic change, Joe Salvador said: "This move will help Rambus to better cooperate with other PHY partners, because it no longer constitutes a direct competition with them, but a good upstream and downstream partnership." In addition, for the Chinese market, Su Lei, general manager of Rambus Greater China, said: "The development of AIGC represented by ChatGPT this year has completely brought AI chips to fire. We see that there are also some emerging companies in the Chinese market that are starting to focus on AI training chips, and we are currently working closely with these cloud vendors and AI chip companies, and this year is mainly for HBM3 projects. ”