Preface
This article summarizes the problems encountered in the inventory field and the solutions to build inventory pre-occupation capacity. Thanks to [Jin Peng], [Sun Jing], and [Chen Rui] for their content and help in the writing of this article!
1. Overview of inventory pre-occupation business
After a consumer takes a picture of an order for a product, the inventory system reserves inventory for the order, and this act of reserving inventory is called inventory pre-occupation.
In the system, inventory pre-occupation is mainly to deduct inventory data. For example, if a product has 5 available stocks, and the order buys 1 of this product, the inventory system needs to deduct the available stock quantity from 5 to 4
Inventory pre-occupancy is a core logistics process. If there is a problem with the pre-possession ability, the product may not be sold properly or oversold.
2. Challenges and responses to the capacity building of inventory pre-occupation
In business scenarios such as flash sales and live promotions, there are often situations where multiple orders go to pre-occupy one or several product inventories in a short period of time. How to handle inventory deduction operations for hot commodities in high-concurrency scenarios is a major technical challenge faced by the inventory pre-occupation business
2.1 Performance Challenges
When multiple threads concurrently make inventory deductions for product data in the same database, locks will be added to the database to ensure that the data is operated correctly. When the product data is [hot] enough, a large number of lock waits can cause performance problems, as shown in the following figure:
In the past, the peak value of online business for fixed goods was hundreds of times per second, but the conventional database pre-occupation solution can only support 50 times per second after verification by stress test data. Once the high-frequency pre-occupation of hot commodities occurs, TP99 will soar, and if the high-frequency pre-occupation time is long, it will bring stability risks to the system
2.1.1 Solution research
1. Asynchronous current limiting. Make the hotspot less hot
When business allows, slow down the pre-occupation operation, thereby reducing the heat of hot spots and relieving the performance pressure on the inventory system. See the image below:
Advantages: The logic is simple, and the transformation risk is small. Optimize the problem from the perspective of the entire call chain, not just the bottleneck
Disadvantages: The interaction mechanism with the single party needs to support the asynchronous mechanism, which may involve process transformation;
2. Horizontal splitting of commodity inventory improves database processing capacity and reduces the impact of database locks during concurrent requests. Split a hot spot into multiple not-so-hot spots
(1) When the product is put into storage, the quantity is split into N copies and placed in N tables or N rows of a table
(2) When pre-occupying, take the remainder according to the pre-occupied document number and access different data sources for pre-occupancy
If the performance supported by a single record is 50 singles/second, then after splitting into 3 parts, the performance of the support can be increased to 100+ singles/second. See the image below:
Advantages: The upstream is imperceptible, and the transformation can be controlled in the inventory field
Disadvantages: 1. The logic is relatively complex, and the risk of transformation is high
2. Business is damaged. There may be cases where there is inventory, but it is not available in advance. Example: (1) There is only 1 available inventory for all 3 data sources, but the quantity on the order is 2, and the pre-occupation is unsuccessful (2) The first data source is out of stock, and the other data sources are in stock, but the order is routed to the first data source. You can use other sub-database retry, sub-inventory addition, and post-retry to solve this problem, but the logic is complex
3. Use cache write-resistant traffic. Improve hotspot handling capabilities
The time taken to pre-occupy hot commodities is mainly focused on database operations, and the faster Redis cache is used to replace the database to provide pre-occupancy capabilities, as shown in the following figure:
The processing power of the cache is much higher than that of DB, and after stress testing, it can support 1200 hotspots per second
Advantages: the upstream is imperceptible, and the transformation can be controlled in the inventory field;
Disadvantages: Complex processing logic. You need to add cache processing logic and ensure data consistency between cache and db
2.1.2 Comparison and selection of various schemes
| scheme | Whether it can support the business without loss | Realization costs |
| Asynchronous throttling | No, only asynchronous interaction with a single party is supported losslessly | low |
| Horizontal splitting of commodity inventory | No, there may be cases where there is inventory but it is not possible to pre-occupy it | middle |
| Cache write-resistant traffic | be | high |
Combined with the current business situation, there are currently some KA merchants with large volume and high transformation risk, but these KA merchants have asynchronous interaction with the order maker, so this part uses the [asynchronous rate limiting] solution, and other merchants use the [cache anti-write traffic] solution
2.1.3 Performance Optimization Results
Through optimization, the TPS of hot commodities was successfully increased from 50 to 1200, an increase of 24 times. TP99 is reduced to 130 ms, which is 4.3% of the original duration (from 3000 ms to 130 ms).
The orange part is the optimized result:
2.2 Thread synchronization issues
Problem definition: Multiple threads query and manipulate the inventory of the same product, making the inventory data confusing
DB pre-occupied mode
Solution: Use MySQL transactions and row locks to prevent threads from influencing each other and manipulate changes in SQL statements
a. Locate inventory. Use information such as item ID, warehouse ID, inventory status, etc. to locate inventory IDs
b. Operate inventory. Deduct inventory based on inventory ID, set current inventory = current inventory + operation volume. In this step, MySQL adds a mutex lock to the ID to prevent different threads from interfering with each other. Batch updates are used here to improve the performance of multiple products in one order
UPDATE stock
SET stock_num = stock_num + CASE id
WHEN 1 THEN 'value1'
WHEN 2 THEN 'value2'
WHEN 3 THEN 'value3'
END
WHERE id IN (1,2,3)
c. Check the inventory. In order to prevent overselling, query the inventory according to the inventory ID, if the inventory of any item in the order is deducted to less than 0, an exception is thrown, and the database transaction mechanism is used to roll back
Cache (Redis) pre-occupied mode
Solution: Put Redis operations into Lua scripts, and use the single-threaded execution of Redis and the characteristics of Lua scripts that will not be inserted by other operation statements during the execution of Lua scripts to prevent threads from interfering with each other
2.3 Deadlocks
• Causes of deadlocks
1. Deadlock between pre-occupied processes. When multiple orders pre-occupy commodities and contain multiple identical commodities, and multi-threaded concurrent requests, threads hold the locks that each other depends on, and then wait for the other party to release the locks on which they depend. See the image below:
2. Deadlock between multiple processes. When multiple threads make concurrent requests, threads hold locks that each other depends on, and then wait for the other party to release the locks they depend on.
Note: The inventory data that needs to be operated by the current logistics inventory platform can be divided into warehouse inventory, logical inventory, and batch inventory. Among them, logical inventory and batch inventory can be regarded as splitting the inventory of a warehouse in different dimensions.
• How to avoid deadlocks
Lock sorting, keeping the order of locks consistent. In the case of multiple transactions requesting resources, the order of lock requests should be kept consistent, so as to ensure the sequential execution of threads. The pseudocode is as follows:
public Result handleOccupyRequest(List<CalcOccupyRequest> paramList) {
//XX业务逻辑
//Long类型比较器，根据库存id进行排序
Comparator<Long> comparator = new Comparator<Long>() {
@Override
public int compare(Long o1, Long o2) {
return o1.compareTo(o2);
}
};
//对要操作的各类库存进行排序
if(saleableStockIds!=null){
Collections.sort(saleableStockIds, comparator);
}
if (otherStockIds!=null){
Collections.sort(otherStockIds, comparator);
}
//XX业务逻辑
}
2.4 Data Consistency Issues
Problem definition: As two independent data sources, Redis and DB need to maintain inventory data, how to ensure the eventual consistency of the two data sources?
This question can be broken down again
1. How to ensure the eventual consistency of data between Redis and DB from the process processing mechanism?
2. In case of inconsistencies, how to find and solve them?
How to ensure the eventual consistency of data between redis-db from the process processing mechanism?
•Initialize the process scheme. Use locked DB inventory + Redis transactions to ensure data consistency. See the image below:
• Data synchronization process. Use MQ retry + task system to ensure that synchronization can be completed
In the unlikely event of a discrepancy, how to identify and resolve it
•How to determine whether the data is consistent due to the constant changes between DB and Redis databases, especially in the synchronization process?
◦Introduce cache operation data to eliminate the impact of synchronization delay on data consistency comparison as much as possible. When operating the cache, increase the operation volume to the Cache Operation Data, and subtract the operation amount from the Cache Operation Data after the DB synchronization is completed. The data consistency comparison formula is: [Cached Inventory Data] + [Cached Operation Data] = [DB Inventory Data]
◦Use multiple comparisons to minimize the effect of residual delays. Set the threshold for the number of comparisons, and as long as one of the multiple comparisons is judged to be the same, the consistency check is passed
• With millions of inventory changes every day, how can you save performance while ensuring that inconsistencies are detected in a timely manner?
◦For commodities whose inventory is lower than a certain water mark, consistency comparison is carried out after each inventory change to avoid affecting the business due to data inconsistency
◦For products with sufficient inventory, set the comparison time interval, and only compare once within the interval to avoid affecting the system performance
3. Inventory pre-occupation processing process
Cache processing process: