Recently, when I was reading "Microservice Architecture Design Patterns", I was very curious at first, because in my impression, design patterns are the 23 design patterns that are often said, and what are the design patterns of microservices? The content of this article is mainly about the application of microservice design patterns in work, and I hope it can inspire you.
Transaction Outbox mode
Transactional Outbox Pattern: Saves messages in the database Outbox table as part of a transaction
policy is a microservice that handles insurance application, taking insurance transaction as an example:
As shown in the figure above, there are 4 steps in the insurance process (pay attention to the operation of Task A), which is the embodiment of the transaction outbox mode, why is it said to be the transaction outbox mode?
- The message task persisted in the Task table is eventually sent to MQ, so it's the outbox
- The operation of persisting a Task record is included in the transaction, so it is called the transaction outbox
The message body saved in the Task record is a task that scans the initialization state by another service service-job, and sends it to MQ, and the task is modified to the completed state after the delivery is successful, which is called the polling release data pattern in the transactional outbox pattern.
One downside to this design pattern is that frequent polling of the database can be costly as the amount of data grows. Two measures are used in our system to optimize:
- Database and table sharding: Exchanges space for time to avoid overhead caused by a large amount of data in a single table
- The task in the completed state is designed to be carried forward by "data carryover": the task is first saved in the Task table, and then archived in the TaskRecord table after the execution is completed
There is also a more efficient, but slightly more complex way to develop: the trailing database log pattern.
Trailing database log mode
Each update of the database corresponds to a database transaction log, which is read through the transaction log miner and sent to the message broker for each message-related record (open source framework: Github: debezium, which can read the MySQL binlog into Kafka), but the disadvantage of this method is that it requires some development effort, because it needs to listen to the database transaction log and call the underlying API of the database.
Believe in the "Postman"
In engineering practice, there is another method that does not adopt the transaction outbox pattern to ensure data consistency: the task in the completed state is persisted in the transaction first, and then the message is directly sent to the message queue, if the message fails to be sent, the exception is captured and the task is modified to the initialized state, and then the service-job service is relied on to compensate: that is, the initialized state task is sent to MQ. Let's take the insurance process as an example, as shown in the code below:
// 1. 持久化保单数据
savePolicy();
// 2. 持久化保单明细数据
savePolicyDetail();
// 3. RPC 调用投保接口
rpcPolicy();
// 4. 持久化完成状态的任务，任务中记录了要发送给MQ的消息体
int num = insertTask(TaskStatus.COMPLETE);
// 5. 如果插入成功了，借助线程池发送消息
if (num > 1) {
threadPoolExecutor.execute(() -> {
try {
mq.send(task.info);
} catch(Exception e) {
// 发送失败，抛出异常，修改任务状态为初始化状态，依赖 service-job 服务进行补偿
updateTask(TaskStatus.INIT);
}
});
}
This design pattern assumes that MQ clusters are always highly available, and I named this design pattern Trust "Postman". In production practice, this design pattern is relatively stable because the MQ O&M team is working to ensure the high availability of MQ clusters.
In the surrender process, which involves the modification of different databases, such as the persistence of the policy, the modification of the policy status, and the push of related settlements, to ensure the data consistency in this process, it is no longer possible to rely on ACID local transactions, but to use the cross-service Saga design pattern to maintain data consistency. Let's introduce the two types, namely collaborative saga and orchestrated saga.
Saga maintains data consistency across multiple services by using asynchronous messages to drive a series of local transactions.
协同式Saga
Let's take a look at the insurance process with the concept of Saga:
In this process, the decision-making and execution sequence of Saga is distributed among each participant of Saga, and communication is carried out through message exchange, and one participant of Saga triggers another Saga to execute after execution to ensure data consistency, which is called collaborative Saga.
The advantages and disadvantages of this design pattern are as follows:
- Advantages: Relatively simple, loose coupling between services
- Disadvantages: It is difficult to understand because it does not have a place to define the execution process of Saga, and the processing logic of Saga is distributed in different services, and the entire process needs to be organized according to the tasks triggered by the code
Why isn't XA used for distributed transactions?
XA implements distributed transactions using a two-phase commit protocol, committing when all participants in the transaction succeed and rolling back when there is a failure. To use this pattern, on the one hand, all transaction participants (databases or message brokers) meet the XA standard, and on the other hand, it is essentially synchronous inter-process communication, which has the drawback that it reduces the availability of a distributed system (assuming that the availability of each service in a distributed system is 99% If the service is called synchronously from one service to another: the service must get a response from another service before it can return its client call, then the availability of the distributed system is the product of the availability of each service, and the availability decreases as the synchronous interaction service increases, and the way to maximize availability should minimize the amount of synchronization operations between systems). Therefore, Internet companies rarely adopt strong consistency design, but adopt eventual consistency design (banks may use strong consistency). In addition, XA implements distributed transactions that rely on a coordinator of the transaction (such as Seata), which is more complex to implement than the above methods.
Choreographed Saga
Another implementation of Saga is orchestration, which requires a transaction coordinator (DTM), and the global transaction initiator defines the orchestration information of the entire global transaction, including the forward and reverse compensation operations of each step, and submits it to the transaction coordinator (DTM), and the coordinator executes the Saga logic asynchronously according to the step.
If the application process uses orchestrated saga, the process of successful application is as follows:
The transaction definition execution steps of orchestrated Saga are very flexible, if we want to do the compensation logic for cancellation settlement in the event of insurance failure, we can define it ourselves, as shown below:
The advantages and disadvantages of this design pattern are as follows:
Advantages: Ability to centralize process control, easy scalability and loose coupling between services, and orchestration saga is appropriate if the dependencies between services are complex and business processes change frequently
Cons: Introducing a coordinator increases development complexity (Extended Learning: DTM Open Source Project Documentation)
Now let's go back to the question we asked at the beginning of the article: what is the difference between the microservices architecture design pattern and what we often call the design pattern?
The design pattern we often say is an object-oriented design pattern, and its solution element is a class, while the microservice design pattern is a design pattern that stands at a higher dimension, that is, the system architecture level, and its object is the service in the system, and the solution is also composed of services that cooperate with each other.
Shoulders of giants
- Microservices Architecture Design Patterns: Chapters 1 - 4