Core stuff (public number: aichip001)
Author | ZeR0
Edit | Desert Shadow
Core East & West reported on November 15 that at the 2021 Tencent Digital Ecosystem Conference held on November 4, Intel and Tencent announced a cooperation to build a high-performance, high-capacity storage product and a diversified database.
After the meeting, Luo Yun, deputy general manager of Tencent Cloud Database, and Intel technology experts were interviewed by Xin Dong and other media to share the technological innovation cooperation between the two sides in the database field, the core advantages of Optane persistent memory, and the development trend of database products.
In October last year, Tencent and Intel reached a vision for cloud block storage (CBS).
All data in the cloud disk is stored in the SSD, and all data passes from the user terminal through RDMA or TCP/IP network storage to the storage server on the other side. Since the latency of SSD disks is very unstable under high pressure, Tencent Cloud and Intel hope to refactor and optimize a fast SSD product, so they take into account Intel Optane persistent memory.
Optane persistent memory can achieve a latency close to DRAM, and at the same time has the characteristics of persistence, large capacity, high life, and byte accessibility. "In past architectures, we could add a layer of persistent memory, write data to persistent memory and return it, which can greatly reduce end-to-end latency." Intel technical experts say.
Compared with traditional DRAM memory, Intel Optane persistent memory has two major advantages: <b>1) its higher storage density and lower unit storage cost can help users expand cloud storage capabilities more economically; 2) The durability features of Intel Optane persistent memory in App Direct mode make it effective as a data persistence storage carrier for CBS products. </b>
For performance-intensive user business scenarios, CBS products based on the new storage engine design and Intel Optane persistent memory optimization can achieve better bandwidth, lower latency, and higher input/Output Per Second (IOPS), which not only shortens the data read and write latency, effectively improves the service life of the system, but also greatly enhances the storage space use efficiency after the implementation.
In order to cope with changes in storage requirements from multiple dimensions such as data capacity, storage performance, and data security, Tencent Cloud is also exploring the construction of a cloud database with capacity, performance, availability, and security.
Luo Yun, deputy general manager of Tencent Cloud Database, said that in the past decade, various database product trends in various segments have evolved in another direction, and they want to have more single databases with stronger capabilities to cover more scenarios. In order to better integrate Redis+MySQL databases and alleviate the difficulty of customers solving various complex problems such as data double write, double read, and cache consistency, Tencent Cloud has created a new high-performance distributed KV database product Tendis.
After more than two years of communication, Intel and Tencent Cloud explored how to use Optane persistent memory to achieve better architecture and performance in the direction of persistence, and gradually matured Tendis by relying on the advantages of Intel Optane Persistent Memory 200 Series in terms of performance, capacity and persistence in the form of "software + hardware" combination innovation, integrating innovative technologies such as automatic separation of hot and cold data.
When building the new generation of Tendis 2.0, Tencent Cloud did not simply put the Redis engine on Tendis, but designed the DB engine completely from scratch.
According to Luo Yun, the design idea of the entire engine is to solve several problems: <b>1) how to use Optane to complete data persistence</b> and achieve stable latency; <b>2) solve the problem that data should be accessed efficiently</b>.
The design of the entire engine and Optane are very closely integrated, which can be understood as building Tencent Cloud's own DB engine around the characteristics of Optane.
At the access layer, a new set of random memory byte addressing features that exactly matches Optane is designed, so Tencent Cloud has built an indexing engine that can be understood as a Hash-enhanced core database based on the traditional Hash.
In the storage layer, it is necessary to combine the two layers of data of Optane on the hot data storage effect and the SSD cold storage to complete the request to the user, there is no block, and at the same time, it is necessary to do a good job of eliminating the hot data mechanism, so the engine should do a good job of distinguishing between hot and cold data.
Luo Yun mentioned that if higher production efficiency can be achieved, it will also save more resources for the whole society in a certain sense. By adopting the new generation of Optane technology, Tencent Cloud database products compress the two-tier architecture of the traditional development model into one layer, and developers can significantly reduce the cost of cloud and the number of servers purchased after using this new engine, and improve production efficiency.
At present, Tendis can provide three different product forms <b>of storage version, persistent memory version and hybrid storage version</b> to meet different needs. At the same time, the product can also cooperate with the reliable and stable platform provided by Tencent YunxingHai Lingling Water System XC221 self-developed server to improve the efficiency of application development and maintenance, and can also achieve cost reduction and efficiency improvement through the separation of hot and cold data and large capacity memory, and bring higher reliability to form a strong support for the core business.
Tencent Cloud also continues to focus on distributed database products that have advantages in terms of storage capacity, scalability and cost performance.
Its "star-star" enterprise distributed database product, TDSQL, integrates the 3rd generation intel Xeon Scalable processor, which greatly improves TDSQL's parallel processing capabilities through its new architecture and traditional advantages of high frequency, multi-core, and multi-threading.
TDSQL also applies Intel Optane persistent memory in all directions to help its customers expand storage capacity more economically, and Intel Optane persistent memory in App Direct mode can effectively serve as a data persistence storage carrier for Redo logs.
According to Intel technical experts, Intel offers differentiated solutions for cloud databases and distributed databases. Open source cloud databases can take advantage of the large capacity of Optane persistent memory to solve cache conflicts, or take advantage of Optane's persistence features to turn in-memory database products into persistent in-memory database products. Distributed databases are mostly closed sources, in terms of transactional or financial distributed databases, strong heterogeneity and strong synchronization are very necessary, and Intel Optane persistent memory will have a large application in this regard.
In addition, Tencent and Intel jointly launched the first TDSQL Developer Competition to encourage developers to implement technology and business innovation based on TDSQL through development and application practice.
Luo Yun revealed that the competition will give a lot of questions from shallow to deep, and developers can answer questions in it, verify their understanding of the database, slowly embrace the technology of the database, and more perceive the TDSQL database products of Optane and Tencent Cloud.
Talking about the experience of how to balance high performance, low cost, and data reliability, Luo Yun shared that if the database is defined as a triangle, these three characteristics are the three vertices of the triangle, pulling any edge or a vertex, the triangle will change, and their relationship is very close.
In theory, it is difficult to balance high performance and data reliability perfectly. Network partitioning results in either availability or data persistence, and access performance and availability may be sacrificed if you want to guarantee greater reliability of your data. The total cost is equal to the cost of a single GIgabyte× scale, you should choose a reasonable medium to store reasonable data, the best solution is to provide hierarchical storage, so that customers can choose between cost and performance.
Therefore, Luo Yun suggested that database vendors should provide more mechanisms than policies, that is, to give users different data reliability levels to choose, so that users can find a balance suitable for their business between different reliability and performance.
"We also introduced Optane's logic because it's cheaper than memory and much faster than disk." He believes that this gives users a good example of making choices.