The birth of cloud native is to solve the problems of traditional applications in architecture, fault handling, system iteration, etc., while open source has contributed to the backbone of enterprises to build cloud-native architecture. In the process of devoting himself to open source and participating in cloud native every day, the authors of this article have different thinking and practices on the open source industry and cloud native stream system solutions.
Author | Li Penghui is responsible for editing | Tang Xiaoyin
Produced by | New programmers
As business and environments change, the trend toward cloud native is becoming more and more pronounced. Now is the era of enterprise transformation from cloud computing to cloud native, the concept of cloud life has been widely recognized by enterprises after several years of polishing and practice, and cloud application management has become a must for enterprise digital transformation. It can be said that today's developers are either using products and tools derived from cloud-native technology architectures, or developers of these products and tools.
This article is from New Programmers: Cloud Native and Comprehensive Digital Practices
Cloud native became the foundation strategy
So, what is cloud native? Everyone has a different interpretation. I think, first of all, cloud-native applications are developed to run on the cloud, and are solutions for enterprises to continuously deliver business quickly, reliably, and at scale. Several keywords of cloud native, such as containerization, continuous delivery, DevOps, microservices, etc., are all interpreting its features and capabilities as solutions, and Kubernetes has laid the foundation for cloud native with its pioneering declarative API and regulator model.
Second, cloud native is a strategy. Cloud native was born to solve the problems of traditional applications in terms of architecture, fault handling, and system iteration. From traditional applications to the cloud, it's not so much a technology upgrade as a strategic transformation. Enterprises on the cloud are facing the full integration of application development, system architecture, enterprise organizational structure, and even commercial products, whether to join the cloud native tide once is a strategic decision that will affect the long-term development of enterprises from all aspects.
Cloud native with an open source background
Most of the architecture-related open source projects born in recent years use cloud-native architecture design, and open source contributes to the backbone of enterprises to build cloud-native architecture.
Open source technology and ecology are trustworthy, and the cloud can bring good scalability to users and reduce resource waste. The relationship between cloud native and open source can also be glimpsed from the CNCF-led open source foundation's continuous promotion of cloud native development. Many open source projects are themselves built for cloud-native architecture, which is a basic software feature that users will prioritize when they go to the cloud.
Take, for example, the Apache Software Foundation, a neutral open source software incubation and governance platform. In the long-term open source governance of the Apache Software Foundation, the Apache Way has been taken as a guideline, in which "community is greater than code" is widely circulated, that is, projects without communities are difficult to last. An open source project with a high degree of activity in the community and code, after being polished by developers around the world in a variety of scenarios, can be continuously improved, frequently upgraded and iterated, and a rich ecosystem is born to meet different user needs. The superposition of the two factors of the cloud native tide and the current open source environment will make those excellent technologies that accompany the continuous upgrading of the technology environment innovate and stand out, and the technologies that do not adapt to the times will gradually lag behind or even be eliminated. As I've said before, cloud native is a strategic decision, and the most advanced and reliable technology must be preferred for the strategic decisions of the enterprise.
A message flow data system built for the cloud
The importance of open source in a cloud-native environment was discussed earlier, so how does a cloud-native open source project need to be designed, planned, and evolved? How should enterprise digital transformation in the cloud-native era choose messaging and streaming systems? In this article, I'll take the design and planning of Apache Pulsar, an open source cloud-native messaging and streaming data system, to which I devote myself to an example. I hope to be able to provide you with reference ideas and inspire you to find a solution for the message and stream data system.
A Look Back at History: A Dual Track system of messages and flows
Message queues are typically used to build core business application services, and streams are typically used to build real-time data services, including data pipelines and so on. Message queuing has a longer history than streams, which is the messaging middleware familiar to developers, focusing on the communications industry, and common systems such as RabbitMQ and ActiveMQ. Relatively speaking, the streaming system is a new concept, mostly used for moving and processing a large amount of data scenarios, such as log data, click events and other operational data is displayed in the form of streams, common streaming systems such as Apache Kafka and AWS Kinesis.
For previous technical reasons, messages and flows were treated separately into two models. Enterprises need to build a variety of different systems to support these two business scenarios (see Figure 1), resulting in a large number of "dual-track" phenomena in the infrastructure, resulting in data isolation, data islands, data can not form a smooth flow, governance difficulties are greatly improved, architecture complexity and O&M costs are also high.
Figure 1 The "two-track system" caused by enterprises building different systems to support business scenarios
Based on this, we needed a unified real-time data infrastructure that integrated message queuing and stream semantics, and Apache Pulsar was born. Messages are stored once on the Apache Pulsar topic, but can be consumed in different ways through different subscription models (see Figure 2), which solves a number of problems caused by the traditional messaging and streaming "dual-track system".
Figure 2 Apache Pulsar integrates message queuing with stream semantics
A key element of natural cloud nativeization
As mentioned above, the cloud-native era brings developers the ability to rapidly scale capacity, reduce resource waste, and accelerate business advancement. With a natural cloud-native messaging and streaming data infrastructure like Apache Pulsar, developers can better focus on application and microservices development rather than wasting time maintaining complex underlying systems.
Why is Apache Puslar a "natural cloud native"? This has to do with the underlying architecture of the prototype that was originally designed. The cloud-native architecture of storage and computing separation and hierarchical sharding greatly alleviates the expansion and O&M difficulties encountered by users in the messaging system, and can provide users with high-quality services at a lower cost on the cloud platform, which can well meet the needs of the message system and streaming data system in the cloud-native era.
Biology has a conclusion called "structure and function adaptation" From single-celled protists to mammals, their life structure is becoming more and more complex and their functions are becoming more and more advanced. Similarly with the underlying system, "architecture and function" is reflected in the following points on Apache Pulsar:
- The storage compute separation architecture guarantees high scalability and can take full advantage of the elastic benefits of the cloud.
- Cross-region replication can meet the needs of cross-cloud data multi-provisioning.
- Tiered storage to take advantage of cloud-native storage such as AWS S3 to reduce data storage costs.
- Lightweight function compute framework Pulsar Functions, similar to the AWS Lambda platform, introduced FaaS to Pulsar. Function Mesh is a Kubernetes Operator that helps users use Pulsar Functions and connectors natively in Kubernetes, giving full play to the features of Kubernetes resource allocation, auto scaling, and flexible scheduling.
Infrastructure: Storage compute separation, tiered sharding
As mentioned above, Pulsar adopted a cloud-native design from the beginning of its birth, that is, the architecture of storage and computation separation, and the storage layer is based on the Apache Software Foundation open source project BookKeeper. BookKeeper is a highly consistent, distributed Append-only log abstraction, similar to the message system and streaming data scenario, where new messages are constantly being added and applied to the message and streaming data domain.
In the Pulsar architecture, the data service and data storage are two separate layers (see Figure 3), the data service layer consists of stateless broker nodes, and the data storage layer consists of Bookie nodes, and each node of the service layer and storage layer is peered. The broker is only responsible for the service support of the message and does not store data, which provides independent scalability and high availability for the service layer and the storage layer, greatly reducing the service unavailability time. Peer-to-peer storage nodes in BookKeeper can ensure that multiple backups are accessed concurrently, and also ensure that even if only one copy of data is available in the storage, it can also provide external services.
Figure 3 Pulsar architecture
In this layered architecture, both the service layer and the storage layer can be scaled independently, providing flexible elastic scaling, especially in elastic environments (such as clouds and containers) that can automatically scale and dynamically adapt to traffic spikes. At the same time, it significantly reduces the complexity of cluster expansion and upgrade, and improves the availability and manageability of the system. In addition, this design is also very friendly to containers.
Pulsar stores the subject partitions at a smaller sharding granularity (see Figure 4). These shards are evenly distributed and will be distributed across the Bookie nodes in the storage layer. This shard-centric approach to data storage treats topic partitioning as a logical concept, divided into multiple smaller shards, and evenly distributed and stored in the storage layer. Such a design can lead to better performance, more flexible scalability, and higher availability.
Figure 4 Sharded storage model
As can be seen from Figure 5, instead of most message queues or streaming systems (including Apache Kafka), which use a monolithic architecture, message processing and message persistence (if provided) are on the same node within the cluster. Such architectures are designed for deployment in small environments where traditional message queuing or streaming systems face performance, scalability, and flexibility when used at scale. As network bandwidth increases and storage latency decreases significantly, the architectural benefits of storage-compute separation become more apparent.
Figure 5 Traditional monolithic architecture vs storage compute tiering architecture
Read-write distinctions
Following the above, let's take a look at the differences in the writing and reading of messages.
First look at write. On the left side of Figure 6 is the application of the monolithic architecture, the data is written to the leader, and the leader copies the data to other blowers, which is a typical architectural design with storage and computation without separation. On the right side of Figure 6 is the storage compute separation application, where data is written to the broker and broker is written to multiple storage nodes in parallel. If three copies are required, the return of two copies is considered successful when strong consistency and low latency are selected. If the broker has the role of leader, it will be limited by the resources of the machine where the leader is located, because the leader returns, we can confirm that the message was written successfully.
Figure 6 Comparison of monolithic architecture and hierarchical architecture writes
In the hierarchical architecture of peering on the right, any two of the three nodes are successfully written when they are written back. When we performed performance tests on AWS, we found that the latency of the two structures when brushing the disc will also be a few milliseconds apart: there will be delays for topics that fall on leader in stand-alone systems, and less affected by delays in layered architectures.
In real-time data processing, real-time reading occupies 90% of the scene (see Figure 7). In the hierarchical architecture, real-time reading can be carried out directly through Broker's topic tail cache, without touching the storage node, which can greatly improve the efficiency and real-time nature of data reading.
Figure 7 Comparison of real-time data read by a monolithic architecture and a hierarchical architecture
The schema also causes differences when reading historical data. As can be seen from Figure 8, in the monolithic architecture, the leader is found directly when the message is played back and the message is read from disk. In the architecture of storage computation separation, the data needs to be loaded into the broker and then returned to the client to ensure the order of data reading. When reading data does not have strict requirements for sequentiality, Apache Pulsar supports reading data segments from multiple storage nodes in parallel at the same time, even reading data from a topic can also use the resources of multiple storage nodes to improve the throughput of reads, Pulsar SQL also uses this way to read.
Figure 8 Comparison of reading historical data between a monolithic architecture and a hierarchical architecture
IO isolation
BookKeeper does a good job of IO isolation inside data writes and reads. BookKeeper can specify two types of storage devices, the left side of Figure 9 is the Journal disk where the writeheadlog is stored, and the right side is where the data is actually stored. Even when reading historical data, write latency is guaranteed as much as possible.
Figure 9 IO isolation of BookKeeper
If the resources of the cloud platform are utilized, Pulsar's IO isolation allows users to choose different resource types. Since the Journal disk does not need to store a large amount of data, many cloud users will configure according to their own needs to achieve the purpose of low cost and high quality of service, such as the Journal disk uses low storage space, high throughput and low latency resources, and the data disk selects a device that can store a large amount of data.
Scaled capacity
Storage computing separation allows Broker and BookKeeper to scale separately, and the following describes the process of scaling the topology. Assuming that n topics are distributed on different brokers, the new broker joins the ability to transfer the topic ownership within 1s, which can be regarded as the transfer of the stateless topic group. In this way, parts of the topic can be quickly transferred to the new Broker.
For storage nodes, multiple data shards are spread across different BookKeeper nodes, and a new BookKeeper is added when it is expanded, and this behavior does not lead to the replication of historical data. After a period of data writing, each topic will switch to the next data shard, that is, switch to the next data shard. When switching, the Books are reselected to place the data, thus achieving a gradual balance. If a BookKeeper node hangs up, BookKeeper will automatically make up the number of copies, and the topic will not be affected during the process.
Multi-provision data across the cloud
Pulsar supports cross-cloud data multi-backup (see Figure 10), allowing cross-room clusters to be formed for two-way synchronization of data. Many foreign users deploy cross-cloud clusters in different cloud vendors, and when one cluster has problems, they can quickly switch to another cluster. Asynchronous replication creates only minor data synchronization gaps, but provides a higher quality of service, and the status of subscriptions can be synchronized across clusters.
Figure 10 Cross-cloud data provisioning
Enter the era of serverless architecture
Pulsar Functions and Function Mesh brought Pulsar into the era of serverless architecture. Pulsar Functions is a lightweight computing framework designed to provide a platform that can be very simple to deploy and operate. Pulsar Functions is lightweight and simple, and can be used to handle simple ETL jobs (extraction, conversion, loading), real-time aggregation, event routing, etc., which can basically cover more than 90% of stream processing scenarios. Pulsar Functions draws on serverless architecture (Serverless) and function-as-a-service (FaaS) concepts to allow data to be processed "nearby" and value to be mined on the fly (see Figure 11).
Figure 11 A single Pulsar Function message flows
Pulsar Functions is just a single application function, in order to associate multiple functions together, combined to complete the data processing goals, the birth of the Function Mesh (open source). Function Mesh also adopts a serverless architecture, which is also a Kubernetes Operator, with which developers can use Pulsar Functions and various Pulsar connectors natively on Kubernetes, giving full play to the characteristics of Kubernetes resource allocation, auto scaling, flexible scheduling and so on. For example, Function Mesh relies on Kubernetes' scheduling capabilities to ensure that Functions are recoverable from failures, and Functions can be scheduled appropriately at any time.
Function Mesh consists mainly of two components: Kubernetes Operator and Function Runner. The Kubernetes Operator monitors the Function Mesh CRD and creates Kubernetes resources (i.e. StatefulSets) to run Functions, connectors, and mesh in Kubernetes. Function Runner is responsible for invoking Function and connector logic, handling events received from the input stream, and sending the processing results to the output stream. Currently, Function Runner is based on the Pulsar Functions Runner implementation.
When the user creates a Function Mesh CRD (see Figure 12), the Function Mesh controller receives the submitted CRD from the Kubernetes API server, then processes the CRD and generates the corresponding Kubernetes resources. For example, the Function Mesh controller creates a StatefulSet when processing a Function CRD, and each of its Pods starts a Runner to invoke the corresponding Function.
Figure 12 Function Mesh processes the CRD process
The Function Mesh API is based on the existing Kubernetes API implementation, so the Function Mesh resource is compatible with other Kubernetes native resources, and cluster administrators can use the existing Kubernetes tools to manage the Function Mesh resource. Function Mesh uses Kubernetes Custom Resource Definition (CRD), where cluster administrators can customize resources through CRD to develop event streaming applications.
Users can use the kubectl CLI tool to submit CRDs directly to the Kubernetes cluster without using the pulsar-admin CLI tool to send a Function request to the Pulsar cluster. The Function Mesh controller monitors the CRD and creates Kubernetes resources, running custom Functions, Source, Sink, or Mesh. The advantage of this approach is that Kubernetes directly stores and manages Function metadata and runtime state, thus avoiding the problem of metadata inconsistencies with the running state that may exist in Pulsar's existing schemes.
epilogue
In this article, I share my thoughts on the open source industry and the technical practice of cloud native streaming platform solutions in a cloud-native environment. As a fully committed open source person, I am pleased to see that in recent years more and more people have embraced the open source concept and become open source developers and contributors, and the open source industry is booming. Like countless developers, I hope to move forward on the open source road and help more enterprises accelerate the process of cloud native and digitalization."
This article is from New Programmers: Cloud Native and Comprehensive Digital Practices. In New Programmer 003, we focus on the two themes of "developers in the cloud-native era" and "comprehensive digital transformation". Alibaba, ByteDance, NetEase, Kuaishou
The enablers of cloud-native technologies of Internet manufacturers such as Amazon, from technology definition, technology application, and practice case sharing
In terms, the hardcore output of the direct core is used to comprehensively analyze cloud native, helping developers quickly find a technical paradigm suitable for their own development in the cloud native era.
At the same time, we will also be on Microsoft, Intel, Huawei, and Shi
Ned, Siemens and other first companies to start digital transformation launched a report, through more than a dozen technical experts to share a vivid case, a glimpse of finance, new retail, industry
Digital transformation achievements in areas such as the Internet of Things help more developers who are concerned about digital transformation to gain inspiration from the experience of pioneers
。
Subscribe year-round (001-004) to enjoy more membership benefits