Graph calculations may be a good track in the coming years

Guest | Pan Zhenxuan

Edit | Zhongliang

In the Internet era, graph data increasingly presents the characteristics of massive and dynamic, and the models and methods of static graph computing are difficult to cope with the needs of data processing. Flow graph computing can be based on real-time changing data, streaming to build dynamic graph data relationships, and based on dynamic changes in graph data in real time analysis, calculation and mining, is the mainstream technology branch of graph computing.

InfoQ, as a technology media, has maintained a special focus on technology trends, and this time we interviewed Pan Zhenxuan, head of the Ant Flow Chart Computing team. Flow diagram calculation is an important part of The Ant Large-scale Graph Computing System TuGraph, which can effectively mine the trends and changes in data relationship changes, and undertake important functions such as nearline asynchronous graph calculation. Pan Zhenxuan shared with us the application experience of ant flow diagram calculation and the development trend of graph calculation in the future.

"Around 2017, there were not many people who knew about the concept of graphs at that time, and even if there were, they were only familiar with databases (such as neo4j in the industry) and offline graph computing systems (Google's Pregel, etc.), at that time, in the industrial world, there was no mature flow graph computing system, nor did we need to see typical application cases, we could only feel the stones to cross the river."

When talking with InfoQ about flow graph calculation, Pan Zhenxuan said that at that time, the flow graph calculation project was only an internal exploration project, and no one knew whether it could run through. "As a pathfinder for this project, I only know that it is feasible in principle, and whether it actually works or not needs to be marked with a question mark."

In the initial stages, the flow graph computing team had only two members. "I have to believe in this thing myself to attract more people to join in," Pan zhenxuan said when it comes to the most difficult things. According to his recollection, at that time, on the one hand, they needed to explore how the system should be designed and built, and on the other hand, they needed to find typical application cases to prove the business value of the flowing real-time graph, so that more people could believe in the value of the direction.

Try, explore, and verify

After stepping on 2018, the exploration time has been more than a year, and Pan Zhenxuan has also found a scene that can be calculated using flow graphs. "At that time, our team mainly looked for application scenarios suitable for flow graphs, and also fit the ability of business scenarios to build core engines. In small scenes such as credit risk control inside the ants, some landing attempts have also been made. Pan Zhenxuan said.

It wasn't until Singles Day 2018 that flow graph calculations really proved their worth. In an article by Pan Zhenxuan, it was mentioned that flow graph calculation has been able to dynamically identify abnormal capital risks exceeding the six-degree relationship chain (strong concealment) in the case of the double eleven to promote extreme traffic peaks, and this risk control ability is also very leading in the industry.

At this time, there are two other problems in flow graph computing, that is, how to make users use more and better? First of all, the flow link will be much more complex than the offline graph computing of the overall link, and there will be problems such as weak interactivity relative to the graph database, which is often wanted by the business side, but cannot be really used due to the high threshold. Secondly, at that time, the flow diagram computing team only had 3 students, and since the system was completely independently developed, there were many functions and features on the kernel that needed to be improved.

When talking about how to solve the threshold problem and manpower problem, Pan Zhenxuan frankly said: "The better solution is to make everyone believe in the value of the flow chart calculation itself, on the one hand, through the business scenario brought by the double eleven benchmarking scene, let everyone perceive that the use of flow chart calculation can obtain very good business results." On the other hand, it is better linked with the middle office, and quickly covers a type of scene user by supporting a specific middle office (such as knowledge graph), so as to form a flywheel with scale growth and allow more services to use flow graph calculations. ”

Indeed, only by constantly validating the value of flow graph computations can the challenges that ensue can be met. When these problems are solved, ant flow graph computing has entered the third stage, about 2020, as graphs are more and more widely used in ants, the systematic construction has brought great challenges to the flow graph computing team. So they extended the power of the flow graph from both sides to provide the ability to integrate offline, allowing users to support experiments based on off-line data based on a set of DSLs.

In this way, the flow chart computing system has gradually become one of the core members of the ant graph computing system. According to Pan Zhenxuan, the current graph calculation engine is widely used in scenarios such as ants' security risk control, credit risk control, knowledge graph, data lineage, capital analysis, traffic attribution analysis, and membership relationships.

After entering 2022, the Ant Graph computing team began new explorations and innovations, such as exploring the frontier capabilities of large-scale distributed graph machine learning systems, next-generation graph databases and online graph computing systems, and also opening up mature graph computing technologies to the outside world and applying them to finance, energy, government affairs and other fields. At present, the GeaFlow team is also actively communicating with the outside world, hoping to open up the capabilities of flow graph computing to external scenarios, so that the industry can use the power of graph computing more and better and exert the value of graph computing.

Innovation and upgrading

The upgrades in recent years mainly include a series of work around the systematization of flow diagrams, while also continuing to deepen the core of the system. In terms of systematization, around the flow chart calculation engine, the Ant Graph calculation team has built a complete set of flow chart calculation system from interactive offline graph exploration to long-term offline graph simulation based on historical data, to flow nearline graph calculation and dynamic timing graph calculation. On the system kernel side, they built compute and storage separation to support hyperscale graph state management and storage. At the same time, it also made relevant depth optimization around the hot spots/large points unique to the graph.

Flow chart computing in the application of ant infrastructure technology is more and more extensive, at present, ant's large-scale graph computing system continues to break through, has become one of the core infrastructure of ant risk control, flow chart computing is an important part of it.

Flow chart computing in Ant Group mainly assumes the following two responsibilities. First of all, as the application of graphs in business becomes more and more extensive and the understanding of graphs becomes more and more profound, the online query of simple graph relationships cannot fully meet the requirements of business scenarios, so businesses urgently need online and real-time processing capabilities that support complex graph queries /graph calculations. Since the latency requirements of online scenarios are very high, the current business will build a complex graph query / graph calculation preprocessing based on the streaming graph calculation engine, so as to write the computed data to the KV storage in advance, thereby providing extremely low latency online query capabilities. Based on this capability, ant risk control system currently better combines the low-latency simple graph query capabilities provided by the online graph database, and the real-time graph computing capabilities of the complex graph query/calculation provided by stream graph computing, thereby further enabling the business to better use the graph computing system within ant.

Secondly, with the further evolution of data and intelligence, the flow graph engine is also more integrated with the intelligence of graphs, such as working with ant graph learning teams to build offline training and real-time graph inference capabilities for dynamic graphs. Among them, the flow graph engine provides the computing power of the off-line integrated graph, which can support both simulation back-reference verification based on historical data and real-time graph processing capability of streaming data. At the same time, the current streaming graph calculation engine also supports a set of DSLs, which opens up the user's offline training and streaming on-line stages, which greatly improves the efficiency and use experience of research and development.

In the next few years, the track can be expected

When we talk about the nature and future development of the flow graph calculation engine GeaFlow, Pan Zhenxuan said: "GeaFlow is technically a computational technology that integrates graph computing and stream computing, and from the computational semantics and computational model, it is closer to the semantics and model of graph computing. From the perspective of business use, it is more inclined to flow computing, and it will also use related technologies of flow calculation. Flow graph computation is essentially an incremental computation, but from the perspective of computational semantics and data model, it is a graph-centric perspective. ”

At the same time, Pan Zhenxuan also said, "I personally believe that the field of graph computing will become more and more mature and standardized in the future. In terms of data system, graph computing will become more and more complete in the future like the general big data system. For example, around the real-time graph data, you can build an overall real-time graph number warehouse system, based on such a set of real-time graph number warehouse system, you can build graph data real-time data processing, real-time graph data access. It can not only improve the vividness of graph data, but also give greater play to the value of graph data. At the same time, in terms of computing power, I believe that the ability of graph computing will become stronger and stronger, and more business scenarios will adopt graph data structures to further explore the value of data, and with the enhancement of computing power, there will be more and more graph computing scenarios from offline to real-time, from real-time to online. ”

Just last year, the People's Daily published an article saying: The development of high-performance graph computing in the mainland has a good technical foundation and practical conditions. On the one hand, people's daily lives today are inseparable from high-performance computing. Weather forecasting, new drug research and development, new materials, cars with higher safety factors, high-speed rail and airplanes all need high-performance computing as support. On the other hand, due to the excellent expression ability, visualization effect and solid mathematical foundation of the graph, the graph calculation has also been of high value in terms of national security and financial security. According to a research report, by 2023, graph technology will promote the rapid decision-making scenario of 30% of enterprises worldwide, and the annual growth rate of graph technology application will exceed 100%.

Taking Ant Group as an example, graph computing has been maturely applied to Ant Group's payment and digital finance scenarios, providing stable decision support capabilities for risk control, anti-money laundering, anti-cash-out and social networking applications. Among them, the important risk identification ability supporting Alipay has increased by nearly 100 times, and the efficiency of risk trial and analysis has increased by 90%.

The wave of digitalization is getting bigger and bigger, and the online data of enterprises is also increasing. Decision-making through data will definitely involve data-based analysis and operation, and graphs, as a high-dimensional data structure, can better mine the correlation characteristics of data and provide accurate and efficient decision-making for enterprises. The future development of the graph calculation track is worth looking forward to.

Event recommendations

On June 19th and 20th, archSummit Global Architect Summit will soon land in Shanghai, Pan Zhenxuan lecturer will also come to the scene to communicate with you, in addition, at this summit, we set up a total of fifteen topics, including big data and artificial intelligence, middleware development practice, mobile terminal development practices, microservice architecture design, etc., the detailed topic content can be understood through the following Banner scan code, looking forward to communicating with you on the spot.

Click on one to see fewer bugs