Abstract:With the advent of the industrial Internet era, the discipline of database, which has a history of nearly fifty years, has bloomed with new vitality, and the NoSQL database that plays an important role has also shown more development trends. At the 11th China Database Technology Conference (DTCC2020), Jie Zhu, senior product expert of Alibaba Cloud, introduced the latest trends in the NoSQL database industry, as well as the innovative practice of Alibaba Cloud NoSQL technology and the open practice of Alibaba Cloud NoSQL ecosystem.
The content of this article is based on the recording of the speech and the PPT.
<h1>Guest Profile:</h1>
Jie Zhu, a senior product expert at Alibaba Cloud, is currently responsible for Alibaba Cloud's RDS and NoSQL database products.
The content of this sharing mainly includes the following three aspects:
New trends in the NoSQL database industry
Alibaba Cloud NoSQL Technology Innovation Practice
Alibaba Cloud NoSQL ecosystem open practice
<h1>New trends in the NoSQL database industry</h1>
Database history
First, let's briefly review the development of databases. In fact, databases are an industry with a very long history. Database technology has been enlightened since the 1950s, but the general saying is that it is calculated from the birth of the relational model in the 1970s, so the birth of the database has a history of nearly 50 years, and the database has been evolving and developing in terms of technology, business and application scenarios. For example, in the 1990s, for the scenes of personal office, personal entertainment and enterprise informatization, very famous databases such as SQL Server and PG were born based on the X86 server. The NoSQL database to be shared today was born around 2000 with the development and promotion of the entire Internet. The development of Internet business has given birth to the very famous relational database MySQL, and there have also been NoSQL databases such as MongoDB and Redis, which solve a series of performance, cost, scalability and other issues encountered by traditional relational databases. Finally, around 2015, with the development of the cloud computing era, there have also been some new changes in the database field, and the development of AWS in the international arena is very strong, in China, Alibaba Cloud provides cloud-native database products for the entire system of TP/AP/NoSQL/tools, typical such as PolarDB and ADB.
The database is on the cusp of investment
In computer technology, the industry with more than ten years of history is already very, very old, the database industry has been close to fifty years of history, and can even be described as "dinosaur", but it is very interesting that database technology is still very vital. The above figure briefly lists several recent large investment and financing events in the field of domestic and foreign databases, and it can be found that the database industry is not weakening day by day, but has increasingly become an investment outlet. Domestic Dameng, Oceanbase, Renmin Gold Warehouse, Even Number, PingCAP have a good development, while internationally, Snowflake is listed and has a market value of more than $100 billion, which represents the database business to the point of rebirth.
The new track of cloud database has achieved a new breakthrough in China's software industry
For databases, cloud computing is an important new track for its development. Gartner made a very optimistic prediction that 75% of databases are expected to be in the cloud by 2022, and this prediction is gradually becoming a reality. In 2020, Alibaba Cloud Database officially entered the Leader Quadrant in Gartner's Magic Quadrant, which is not only a breakthrough for Alibaba Cloud, but also the first time that the entire Chinese software industry has entered the Leader Quadrant in nearly four decades of history, which is unprecedented. As we all know, our country's software industry is relatively backward compared to Europe and the United States, because it started late, and Alibaba Cloud database can go to such a position, its core is still relying on the new track of cloud computing, which also represents a new breakthrough in China's software industry, which is a milestone event.
Rapid hardware changes drive data technology development
On the other hand, the rapid change of hardware is also driving the development of database technology. In fact, hardware and software are complementary, the rapid development of hardware to promote the software to continuously change its own technical architecture, and software through continuous update and iteration to fully use hardware resources, so as to better and faster support business development. For example, NoSQL databases have done a lot of things to change their organizational model and data organization structure, all in order to solve problems like disk and CPU usage, and organize hardware resources more efficiently. In the past, it was CPU and disk, and in recent years, there are several very important trends in memory, first of all, the cost of memory has dropped very quickly; secondly, the capacity of single machine has increased on a large scale, and now the basic single-machine server has reached hundreds of GIGabytes or even 1TB; and finally the rapid development of non-volatile memory technology. It should be said that these trends are not a news, but this year a very big change has taken place, that is, these technologies have begun to really be applied and large-scale commercialization, I looked at the Alibaba Cloud persistent storage in-memory database in China is the first cloud service vendor to achieve large-scale commercial use.
To sum up simply, the rapid change of hardware has driven the development of the entire database and even the entire software industry, and in the past two years, the industry has paid great attention to changes in memory, not just CPU and disk, so it can be said that "RAM IS THE NEW DISK" is no longer a trend, but has become a reality.
Evolution of noSQL-related database concepts
NoSQL databases are due to the birth of Internet business. As early as 2006, Google introduced BigTable, so HBase, Cassandra, MongoDB, and Redis were followed by different forms of underlying data organization to solve different problems. Around 2010, Google introduced new products represented by Spanner, and newSQL databases such as F1, SequoiaDB, and TiDB appeared, even if SQL was used to solve application problems, but also retained the scalability of NoSQL. And back to today, you will find that the debate over these concepts is no longer a hot topic, like AWS this year's aws re:invent conference to talk about Pure-build Database, that is, dedicated database, which is not a bad thing, that is to say, today's driving data development is the scene, we are entering a more innovative, more connotation, more depth of the deep water area. Today, Alibaba Cloud Database has done more of a series of related technologies such as software and hardware integration technology, industrial Internet of Things technology, and AI technology, and integrates and applies them in the database field, especially in the field of NoSQL databases.
NoSQL is the most active section of the database
Although there have been many mentions of "going O" recently, noSQL is actually the most active sector in the entire database field. According to Gartner's report, the global database market in 2019 was about $55.2 billion, an increase of only 15.2% year-on-year, while NoSQL increased by 51.7% year-on-year, so NoSQL is the real market growth engine for the entire database field. In the NoSQL segment, businesses are thriving and technology is constantly innovating and changing. In summary, for technicians in the Field of NoSQL, there will be a very good historical opportunity, NoSQL is in a good era of investment outlet, hardware changes, and new business ideas superimposed. This is also a very good opportunity for entrepreneurs involved, and if you are in the NoSQL related sector, there will be enough room to show off.
<h1>Alibaba Cloud NoSQL Technology Innovation Practice</h1>
Next, we will introduce the most active section in the database field of NoSQL, and what technological innovation practices Alibaba Cloud has done in the past year.
Innovation breaks through the depth of the scene
In the past year, Alibaba Cloud's innovation and breakthrough in the Field of NoSQL has four main aspects. First of all, in terms of the combination of software and hardware, Alibaba Cloud released a large number of software and hardware in the Tair cloud native memory database released this year; secondly, in terms of the graph + AI one-stop intelligent platform, Alibaba Cloud has made a lot of attempts in the graph database GDB, and has done a lot of things in AI; third, in terms of cloud native multimode, facing the future industrial Internet era, Alibaba Cloud released an innovative multimode database; finally, Alibaba Cloud saw that 75% of the databases in the future will be on the cloud. In addition to the original startups, there are many large companies that need to go to the cloud, in response to their needs, Alibaba Cloud has realized the innovative MyBase new form database, for medium and large customers, to solve the core requirements of their database on the cloud autonomously controllable, flexible and exclusive. All in all, Alibaba Cloud hopes to break through these in-depth scenarios through innovation and bring more use value to customers.
The cloud-native era of in-memory databases
The figure above shows the approximate architecture of Alibaba Cloud's cloud-native in-memory database Tair. Tair is a self-developed cloud-native in-memory database that has been honed by Alibaba for more than a decade, which mainly solves the customer's data real-time online, system acceleration and high-frequency high-throughput processing. The Tair database has several characteristics, first, it is fully compatible with the community's Redis database; second, it has the ability of high-performance persistent storage through expanded memory, including supporting SSD versions, but also supporting non-volatile memory versions; because the Tair database is located on the cloud, it is very convenient to connect with data on the cloud, and at the same time, it can seamlessly match data management tools on the cloud to help customers support large businesses such as games, e-commerce, and government affairs.
Combine hardware and software to expand new scenarios
Alibaba Cloud is no longer volatile storage in fact has a very long layout, in the past year, Alibaba Cloud has done a very important thing is to commercialize this internal technology, although many enterprises are also studying this aspect of technology, but to achieve real commercialization, Alibaba Cloud is the first in China. The underlying layer of Alibaba Cloud's Tair in-memory database is based on the DPCA bare metal instances and Optane persistent memory, which enable Tair to have both low memory latency and SSD storage capacity and cost balance. Alibaba Cloud Tair not only combines software and hardware to achieve technological breakthroughs, but more importantly optimizes and processes the previously low utilization scenarios. Through the combination of software and hardware, some new scenarios have been successfully expanded, typical of which include scenarios where computing intermediate data has high performance requirements; scenarios in which big data result sets provide query services to the outside world in games, live broadcasts, and data analysis; and high-performance real-time online data access. Through the new media and some key capabilities of the original database on the cloud, such as elastic scaling, the cost is reduced by more than 60% while achieving similar memory performance.
AI algorithms empower graph computation to achieve graph analysis and reasoning capabilities
In recent years, the development of AI technology has also led to very big changes in the industry. AI and database in many aspects have been combined, this focus on the expansion of the application of the graph database, in fact, the graph database in the industry has a lot, such as the more famous Neo4j, and the so-called original graph application refers to a new data organization structure to solve the original relationship model can not solve the problem, such as the correlation relationship fast query, etc., the focus of research and development is usually to improve query performance and import performance, etc., which is also the development trend of common NoSQL databases, Examples include MongoDB, Redis and HBase. This year, Alibaba Cloud has made new explorations on this basis, hoping to provide customers with training and application of AI algorithm models by increasing AI capabilities, and helping customers to conveniently build knowledge graphs. In addition to improving performance and solving query problems, Alibaba Cloud can also directly provide knowledge and algorithm models and decision models, and this year Alibaba Cloud will only support simple things such as original image query to logical analysis and reasoning capabilities, truly helping customers provide intelligent solutions, so as to better solve overall business problems.
Best practice graph database, 100+ successful application scenarios
The ability of the graph database has been widely used and verified within Alibaba, such as Tmall, AutoNavi and Hema, etc. have fully applied the capabilities of the graph database and realized commercialization, in addition to our graph database also widely serves external customers, such as external customers in the Internet business, games, finance and other industries, and has accumulated more than 100 successful scenarios, it should be said that the application scenarios and scope of the graph database are becoming more and more extensive.
China's independent intellectual property rights map database was shortlisted for the first time
Alibaba Cloud's graph database GDB, which was just commercialized in February this year, is a very young product, through the combination of AI, it has become a very good product that can truly solve customer problems end-to-end, and has quickly achieved a breakthrough. In Q4, Alibaba Cloud Image Database was first shortlisted for Forrester Wave as a map database with independent intellectual property rights in China, which is a very remarkable achievement as a product that has been commercial for less than a year.
Multimode database concept
Next, we will share a new cloud-native multimode database for the future of industrial Internet and IoT. Focusing on Alibaba Cloud's understanding of Alibaba Cloud's native multimode database and our ability to innovate, alibaba cloud.
First of all, cloud native capabilities use cloud capabilities to achieve auto scaling and on-demand use, and have an open ecosystem, while multi-mode native capabilities refer to the ability to have multiple vertical engines and realize cross-mode fusion and unified access on this basis. Through the combination of the above two capabilities to form a cloud-native multi-mode database, Alibaba Cloud released Lindorm this year, a new cloud-native multi-mode database, we hope to create a scenario-driven database based on customer needs, which can simply and efficiently help customers process business, and provide an open and integrated ecosystem as a whole.
Lindorm's multimodal capabilities
The concept of multimode has been around for a long time, and you can see a lot of databases that support multimodality on DB-Engines, but there are very big differences between them, and most of the industry's databases may only stay in the first or second phase of multimode, may be able to support multiple data types, or have multiple vertical engines, but can not do the ability of cross-mode fusion. Alibaba Cloud Lindorm has reached the third stage of multimodality, realized cross-mode fusion, and is working towards the fourth stage, that is, the stage of unified access and the connection of model engines. Although many of the databases in the industry are called multimode, there are essentially cross-generational differences.
Cloud-native multimode database Lindorm - multimodal massive storage capability, the preferred database for the Internet of Things
IDC predicts that in 2025, 70 ZB of the 163 ZB of data produced worldwide comes from Internet of Things devices, and the "4 Vs" of the database mentioned before refers to Internet data, and now "4 V" refers to the data of Internet of Things devices, which is summarized as the number is getting larger and larger, more and more types, more and more types, and more and more valuable.
In the past, the value of Internet data was very high, so the storage cost could not be considered, but now in the face of data generated by IoT devices, if the original storage method is still adopted, it will cause higher costs, so it is necessary to reduce the cost of cloud storage and make enterprises affordable. Lindorm re-storage cost this piece of precipitation a full range of capabilities, from the storage cost, computing resource elasticity, compression algorithm, specific through the cloud disk + OSS hybrid storage, with local buffering, to ensure performance; with the help of storage and computing separation, in accordance with flexible scalability and Serverless resource management mode, reduce the user's use threshold, to achieve cloud-native elasticity; through adaptive compression algorithm and intelligent cold, hot and warm heat identification, optimize storage costs and take into account access efficiency, Enable intelligent compression.
In addition, after the data is stored, it is necessary to have rich processing power to exert the value of the data, which is also the key ability to process Lindorm multimode data, Lindorm has a comprehensive layout in the time series engine, multiple data models, and data analysis. At the same time, to solve the complexity of IoT data, by supporting a variety of data types, with wide table, timing, search and file model processing capabilities, and 100% compatible with open source access protocols, especially in the time series scenario, the self-developed time series data engine realizes time series processing such as high throughput, high compression, and multi-dimensional aggregation of the Internet of Things; in addition, it can also be deeply integrated with Alibaba Cloud internal products, such as DLA deep integration, providing powerful data analysis capabilities. Lindorm hopes to be able to make Lindorm the database of choice in the future era of the Internet of Things by providing cost-effective storage capabilities, as well as the ability to target multimodal data storage in the future. We focus on building and precipitating key capabilities for scenarios such as the Internet of Things, big data storage, interactive real-time storage, and data lake storage centers.
After fully moving to the cloud, you also need to be exclusively controllable– creating a private cloud experience on the public cloud
Gartner predicts that in the future, 75% of databases will go to the cloud, so in addition to startups and small and medium-sized enterprises, the most important trend in the next step is that more large and medium-sized enterprises will go to the cloud. There is a huge difference between the cloud migration of large customers and the cloud of small and medium-sized customers, that is, the requirements of business complexity, data complexity, business flexibility and openness are completely different. Therefore, one of the things that Alibaba Cloud has done this year is to be able to combine the advantages of hosted and self-built databases to help customers build myBase, a dedicated cluster for cloud databases, so as to fully release the technology dividend, allow customers to use flexibly, and support multiple databases.
MyBase has a number of excellent features:
First, exclusive exclusivity, can solve the full hosting service all customers mixed deployment together caused by a variety of conflicts, scrambling, isolation of the problem, a single large customer with exclusive server cluster, which is very different from the traditional model, to help customers achieve exclusive physical resources, but also to meet the customer's desire to take into account the allocation of small instances.
Second, the ability to open up, from the underlying physical cluster to the upper layer of the database permissions are all open; completely different from the traditional fully managed closed system, the open system to meet the more autonomous and controllable demands of medium and large customers.
Third, the various settings and parameters of server clusters provided by cloud computing vendors in the past were often not adjustable, or a small number of them could be adjusted. In the MyBase exclusive cluster, give customers the ability to fully adjust, in the exclusive mode, from the underlying physical resources to the upper layer of software and algorithms are exclusive, all open to customers, can be combined with their own business to adjust and adapt, through the combination of database capabilities and customer business, and then achieve the overall resource minimization and best performance.
In addition, MyBase clusters are the same kernel as RDS, so they do not degrade the quality of service, that is, create a private cloud service experience on the public cloud. This is also a very important trend in the future, now everyone needs to go to the cloud, then the need for resource pools based on the public cloud to reduce costs, but at the same time there are security and autonomous controllable demands, so in the face of new problems, there is a need for new solutions to solve problems, you need to be able to create a private cloud on the public cloud like experience, Alibaba Cloud hopes to give large and medium-sized customers a new experience.
Open OS permissions
Among them, Alibaba Cloud MyBase has done a lot of key work, such as permissions, such as open OS permissions is not so simple, if you do not ensure that the database is highly available, what is the value of other things? Therefore, behind the first thing to ensure the high availability of the database and the deep openness is years of practice and key technologies.
Open this small step, but for customers is very valuable, through the open, in the original operation and maintenance work mode at the same time, but also give full play to the value of DBA, so that DBA can continue to solve database problems in a timely manner. In addition, the original business system has accumulated a lot of operation and maintenance management systems, and how to play its original role in the form of the cloud is also a very big challenge. With this open capability, the various tools of the business can be well integrated. In addition, in different systems, flexibility is also available, and independent capabilities such as resource resiliency policies can be set at the instance level. Therefore, through the open ability, in one fell swoop, it solves the demands of large and medium-sized customers for autonomy, controllability and openness, which is a relatively large innovation in the database of the entire industry, which is a model that has never been used in the cloud database before, and the world is the first.
Unleash Alibaba Cloud database technology dividends – agile, open, and low TCO
In addition to openness, Alibaba Cloud technically exports various technologies that it uses internally. Typical over-provisioning capabilities, database mixed capabilities, small instance management capabilities in a single cluster, and flexible online migration of instances are open to everyone, allowing customers to enjoy Alibaba Cloud's mature management capabilities and services. The ability provided by the customer's business and service provider is very effective to fully enjoy the technology dividend, achieving agile, open and low TCO.
This year, Alibaba Cloud Group database cloud also fully adopted the form of MyBase, Ali Group's various e-commerce, payment and other businesses should be the largest database users in China, MyBase this flexible and exclusive, autonomous and controllable is the best practice of Alibaba Group database cloud, successfully withstood the severe test of the Double Eleven scenario.
Dedicated resource pool for autonomous driving
In addition, we have observed that in addition to being flexible and exclusive, autonomous and controllable, there will be more companies that hope to achieve automatic driving of the database, just like Tesla. In most scenarios, simple and repetitive work can release the burden of database management through the database's autonomous driving capabilities. Really in complex scenarios, as well as scenarios that need to be closely aligned with the business, when the autonomous driving technology really can't handle it, it can allow humans to flexibly intervene in the project.
At present, in the DBA common repetitive work, such as hosts, instances and disks and other automatic scaling capacity, automatic migration and other issues, Alibaba Cloud MyBase provides an exclusive resource pool for automatic driving, what is done is like Tesla, in the general road scenario to solve the problem of automatic driving, in special scenarios can be flexibly adjusted by manually.
<h1>Alibaba Cloud NoSQL ecosystem open practice</h1>
In the future, 75% of databases will be on the cloud, why cloud databases can replace traditional databases, in addition to continuously providing new capabilities to new scenarios and bringing new value through innovation, a very core point is the ecological capabilities that the cloud can provide. Alibaba Cloud has a very open attitude in practicing the cloud ecosystem, and we have always hoped to work together to serve our customers by working with the best vendors in the industry.
Alibaba Cloud exclusively supports the new version of MongoDB
In October last year, Alibaba Cloud and MongoDB officially reached an in-depth cooperation, alibaba cloud is currently the only cloud vendor that can provide a new version of MongoDB. A brief review of the version development of MongoDB, which supports transactions in version 4.0 and penetrates into the traditional RDS market; in version 4.2, distributed transactions are supported and transformed to NewSQL; in version 4.4, it is fully enhanced by user-driven, and some of the features are deeply cooperated and jointly developed by Alibaba and MongoDB officials, while the latest version of MongoDB can only be exclusively provided on Alibaba Cloud.
Internet of Things, industrial network networking full scenario coverage
In addition to working with database vendors like MongoDB, Alibaba Cloud also provides one-stop solutions in specific industries. In this part of the industrial Internet, in the face of the new Data generated, Alibaba Cloud has created various scenario solutions with manufacturers at home and abroad, such as OSIsoft+ Intel in the industrial production line IT and OT fusion monitoring, Honeywell of intelligent buildings, etc., to provide full-scenario solutions, these solutions have successfully formed capabilities, and landed in very large industries such as electricity, steel, etc., so that data can be truly stored and visible. We hope that more and more excellent manufacturers will cooperate with us for a win-win situation.
Innovation and openness of Alibaba Cloud NoSQL database
The innovation and openness of Alibaba Cloud NoSQL database in the past year can be roughly summarized into the above five aspects, namely, the combination of software and hardware for non-volatile storage, the AI technology empowerment map database, the multi-mode scenario for the industrial Internet, the new form for the independent control of medium and large customers, and the open and win-win ecology for new data.
The core concept of Alibaba Cloud NoSQL is innovation and openness, we continue to innovate product capabilities, to expand the depth of the scene, innovate product forms, solve customer key problems, and we hope that more partners and us to build an ecosystem, win-win cooperation, to jointly create solutions to enable customers.
This article is the original content of Alibaba Cloud and may not be reproduced without permission.