Text / Cui Wei, Vice President of China Foreign Exchange Trade System
Zhan Hanglong, Senior Engineer of China Foreign Exchange Trade Center
The efficient operation and interconnection of financial infrastructure provide a solid guarantee for the high-quality development of digital finance. General Secretary Xi Jinping pointed out at the opening ceremony of the seminar on promoting high-quality financial development that it is necessary to establish and improve an independent, controllable, safe and efficient financial infrastructure system. The realization of the goal of independence, controllability, safety and efficiency is inseparable from the support of scientific and technological innovation. Among them, autonomy and controllability are the premise, safety is the bottom line, and efficiency is the focus. This requires us to speed up the independent research and development of science and technology, master core technologies and key resources, and continuously improve the level of financial market infrastructure.
Adhere to the system concept and optimize the overall efficiency of the trading system
As the organizer and operator of the interbank market, CFETS (CFETS) focuses on its core trading business, adheres to the system concept and problem orientation, promotes independent research on innovative technologies, and continuously optimizes its independent, controllable, safe and efficient core trading system. The construction of the core trading system is a complex system engineering, which has extremely high requirements for high performance, security, reliability and other indicators. At the same time, Xinchuang products are also facing an urgent need for function optimization and performance improvement in the process of adapting to application systems. Therefore, we should not only pay attention to the innovation of application algorithms such as trading engines and digital intelligence models, but also promote the research of underlying technologies such as communication, storage, and operating systems, promote the efficient collaboration of software and hardware, and realize the overall optimization of the whole process and combination. In short, with the goal of building an independent, controllable, safe and efficient core trading system, the CFETS actively uses the system concept to promote the collaborative optimization of the application layer and the technical layer. For the ultra-fast market distribution scenario, the three major innovations of fine coding, streamlined scheduling, and accurate transmission are used to improve high-speed and reliable communication at the microsecond level.
In the transaction market distribution scenario, in order to achieve high-throughput and low-latency message transmission, we start with network communication technology, deeply analyze the technical characteristics of high-speed and reliable UDP communication protocol, systematically study a number of technologies such as offset coding, kernel bypass, and multi-modal transmission, continuously improve the performance of financial message transmission, and build a high-speed and reliable communication base.
New transport protocols are creating new opportunities for communication acceleration
TCP and UDP have their own characteristics in the network transport layer protocols. TCP is a connection-oriented transmission, which is reliable, but it takes a long time to establish a connection, and the performance is significantly degraded when the network environment deteriorates. UDP is a connection-oriented transport that is lightweight, fast, but unreliable. A variety of reliable UDP solutions, such as QUIC, KCP, and MoldUDP64, have been proposed to make up for the shortcomings and improve communication performance while giving full play to the advantages of UDP (see Figure 1).
Figure 1 Comparison of the performance of reliable UDP and TCP in different network environments
QUIC is an encrypted transmission protocol developed by Google. The Internet Engineering Task Force (IETF) has identified it as the underlying protocol for the HTTP/3 standard. QUIC has the advantages of 0-RTT communication, multiplexing without head-to-head blocking, and user-mode flow control (see Figure 2).
Figure 2 Comparison of QUIC-based HTTP3 and HTTP2 protocol stacks
KCP is a fast and reliable communication protocol that increases the flow rate through mechanisms such as non-delayed confirmation and non-concession flow control. KCP can reduce the average latency by 30%~40% and reduce the maximum latency by 3 times on the basis of 10%~20% additional bandwidth cost.
MoldUDP64 is a multicast distribution protocol developed by NASDAQ. To solve the problem of out-of-order arrival and unreliability of UDP data packets, MoldUDP64 provides a packet loss detection and request retransmission mechanism for clients to ensure reliable market and order-preserving distribution.
Three major innovations help the rapid market to make new breakthroughs
Combined with the characteristics of reliable UDP technology, we focus on the rapid distribution scenario of market conditions and rely on the financial technology laboratory of the trading center to promote high-speed and reliable communication technology research. In the financial market, in order to meet the needs of timeliness and fairness, exchanges generally use native UDP to build market multicast networks. Financial institutions generally receive market information through the border router of the exchange through the colocation mode or leased line. In order to solve the problem of unreliable native UDP messages, some exchanges use the "UDP multicast + TCP retransmission" mode, and some exchanges use the "multicast push + unicast retransmission" mode of MoldUDP64.
On the basis of investigating the practices of domestic and foreign peers, we further started from the underlying mechanism of operating system and network communication, explored and developed a combined solution for high-speed and reliable communication, and completed microsecond-level single-hop market communication, with a throughput of 400,000 transactions per second, realizing the independent and controllable key core technologies.
1. Fine coding. In the message initialization process, the offset binary coding technology is studied. According to the common characteristics of the financial packet structure, the in-memory data house number is pre-designed, and the process directly accesses the data according to the house number, and the analysis efficiency is improved by more than 10 times through cache prefetching.
2. Streamline scheduling. In the packet sending/receiving phase, the traditional multi-step copy mode from memory to kernel to NIC is used, which takes a long path and consumes CPU resources. We've worked on kernel bypass technology to provide direct access to memory and network cards, simplifying the data "moving" path.
3. Precise transmission. In the communication transmission stage, we study the message communication mechanism based on the reliable UDP protocol, and optimize the packet packaging and security control based on the size and frequency of messages of different lines, so as to further improve the bandwidth resource utilization.
Three types of scenarios deepen the new value of communication technology
The financial industry has the ultimate pursuit of improving information and communication efficiency. In the future, we will continue to explore the application of innovative communication technologies in more business scenarios, improve the digital infrastructure of the financial market that is independent, controllable, safe and efficient, and continuously improve the efficiency and quality of financial market services.
1. Research on SD-WAN-based wide-area market distribution. In recent years, some major foreign exchanges have cooperated with large cloud service providers to promote the migration of trading business to the cloud. For example, NASDAQ has fully migrated peripheral functions such as market supervision, data analysis, and data distribution to the Amazon cloud platform, and will further pilot the cloud-based deployment of the matching engine in 2021. CME provides real-time and delayed market data based on Google Cloud Platform and plans to migrate all trading infrastructure to the cloud over a 10-year period.
By tracking the development trend of domestic public clouds and industry clouds, software-defined wide area network (SD-WAN) technology is used to build a hybrid cloud-based multicast market network and a regional market forwarding center based on cloud virtual routers. Take advantage of the high bandwidth and high-quality links of the mature cloud platform to further expand the coverage of trading services.
2. Explore RDMA-based high-performance storage and computing clusters. In the context of distributed architecture transformation, the storage-computing separation framework puts forward extremely high requirements for high-throughput interaction between nodes. Scenarios such as big data processing and large model training require high-speed data collaboration between CPUs and GPUs, and the transmission delay bottleneck of traditional network architectures is becoming more and more prominent.
Through the study of RDMA and its subdivision technology RoCE, the link layer optimization is realized by using kernel bypass and memory zero copy, and the message routing and traffic control algorithms are optimized to further reduce the message communication delay within the data center and improve the core transaction performance.
3. Promote a multi-site multi-active architecture based on peer-to-peer multicast networks. In order to improve the business continuity of the core trading system, financial institutions have widely adopted the multi-location and multi-active solution. In the multi-active architecture, different data centers need to synchronize service status in real time to ensure rapid switchover (RTO) and data loss (RPO) when the production master node is abnormal. At present, the industry mainly uses database stream replication technology for state synchronization, and some organizations have begun to explore state synchronization technology based on peer-to-peer multicast networks.
By enabling PIM-SM and IGMP V3 between production and intra-city data centers, and configuring the core switch with the same Anycast RP address to establish a peering relationship, multicast sources can be registered and receivers can join nearby. On this basis, application components use UDP multicast to implement message interaction and data synchronization across data centers.
In the future, CFETS will deepen and expand the application of innovative technologies in the financial market, and continuously improve the service level of digital infrastructure in the financial market.
Resources
[1] ITCH Protocol https://www.onixs.biz/insights/itch-protocol-usage
[2] Zeng Kaohsiung, Hu Shuihai, Zhang Junxue, et al. Journal of Computer Research and Development, 2020,57(1):11.) DOI:CNKI:SUN:JFYZ.0.2020-01-006.
[3] Li Xuebing, Chen Yang, Zhou Mengying, et al. A review of Internet data transmission protocol QUIC[J].Journal of Computer Research and Development,2020,57(9):13.)
[4] Kuang Quan, Li Xiaoyong, Bai Yingcai. Research on hierarchical multicast congestion control algorithm based on RTP/RTCP[J].Computer Applications and Software,2006,23(2):3.)
[5] Yu Xueshan, Zeng Jin, Yang Piaopiao. China Financial Computer,2022(12):65-69.)
(This article was published in the first half of April 2024)