1. Background
In the early stage, we introduced in detail the exploration of the customized data center (R2-AZ2) project of Bilibili [1], mainly focusing on the technical iteration and implementation of smart energy-saving data centers. The efficient operation of a data center does not exist in isolation, it relies on a complex and sophisticated network of interconnections to ensure connectivity between servers, storage, and network equipment within the data center.
The cabling system is a key component of the interconnection of the data center, and the improper management of the data center cabling will cause the delivery cycle of the production environment to be extended, the reserved cable is too long, the cable layout is chaotic, the equipment installation is difficult, the troubleshooting and maintenance time is increased, and even the airflow organization of the cabinet will be affected, resulting in local overheating and affecting the safe operation of electronic information equipment.
In addition, with the rapid development of AI technology and business applications, intelligent computing centers are rapidly rising, and networks are developing in the direction of high bandwidth, low latency, and low power consumption, which also means that the requirements for networks and cabling systems are continuously increasing.
As one of the key infrastructures of large-scale data centers, how to use digital management tools to improve the efficiency of delivery and operation and maintenance management is also a problem we have been thinking about and exploring the direction of practice.
1.1 Introduction to Data Center Cabling
Before designing the integrated cabling of the data center production system, it is necessary to understand the network requirements of the business side, determine the topology of the network cabling, and determine the design scheme of the integrated cabling of the data center according to the designed network topology [2].
Figure 1 Flow chart of network cabling implementation
There are two common types of cabling: structured cabling and unstructured cabling [3].
Structured cabling provides standards-based connectivity for electronic information equipment by using one or more wiring areas and cabling products (such as cabling network cabinets, high-density patch panels, etc.) by adopting standardized predefined connection points and path designs, and when the electronic information equipment moves, increases, or changes, only the cables need to be re-patched in the centralized wiring area, while the permanent link section does not change due to the movement, addition, or change of the equipment. The structured cabling system uses high-density pre-terminated fiber optic cables, which greatly reduces the number of fiber optic cables in the equipment room, but the design and installation of the cabling system requires more time and cost, and the same link may reduce the network transmission quality due to the addition of intermediate connection points.
Unstructured cabling, also known as point-to-point cabling, is a type of cabling system that does not use any predefined standards, connection points, or paths, and makes an end-to-end cable connection. Compared with structured cabling, unstructured cabling systems have lower installation costs and shorter installation time, but higher costs for later operation and change.
At present, Bilibili flexibly selects the cabling mode according to the network requirements of different computer room environments and services, taking into account factors such as cost, delivery cycle, and on-site environment. Unstructured cabling is usually preferred for conventional server cabinet areas, and structured cabling design is considered for network core cabinet areas of important levels to optimize the number of cables and later operation management.
Figure 2 Cabling field diagram of the network core cabinet
1.2 Development of intelligent management platform for wiring at station B
For the network cabling task of a single-building data center, it is generally completed in several batches according to the service delivery rhythm. When dealing with each batch of cabling projects, the whole life cycle information flow and management rely on personnel in different roles to manually complete it through office software tools, and there are the following problems and challenges in the face of the rapid delivery requirements of today's complex medium and large data center cluster production systems:
(1) Document standardization and accuracy problems: In the process of wiring engineering, the lack of standardization and accuracy of document circulation may lead to missing or wrong information. We need to ensure that the file format, data content, etc. meet certain standards, and reduce errors caused by human operations.
(2) Routing planning efficiency: The current manual planning method is inefficient, which is easy to cause error in route length estimation, resulting in errors and additional workload in actual construction. We need to improve the automation and accuracy of route planning to reduce manual intervention and errors.
(3) High complexity of wiring management: due to the large number of personnel, roles and basic data involved, wiring management has become a cumbersome and long-term work. The traditional single-batch cabling project and delivery information have become islands, resulting in incomplete historical cabinet wiring status and information, which is not conducive to improving the delivery efficiency of the production environment of the computer room and the operation and maintenance management and cabling optimization after operation. We need to establish a more efficient information management and data sharing mechanism to achieve smoother information transfer and more optimized cabling strategies.
Based on the above background, combined with our actual wiring engineering management experience, the development project of the network cabling intelligent management platform of the B station data center is proposed, which aims to digitize, visualize and intelligentize the offline network cabling process, so that professional and technical personnel can easily, quickly and accurately complete the cabling planning, management and operation and maintenance work.
2. Wiring intelligent management platform
For the cabling management platform of station B, sort out the complete cabling management process, and the following is a schematic diagram of the process.
Figure 3 Schematic diagram of the cabling management process
This article will focus on the automation module of the red frame wiring, and the accuracy of this part of the data will directly affect the final effect of the entire wiring task.
2.1 Function 1: Basic information maintenance
The objects involved in cabling management include but are not limited to the following information:
- wire rod
- Cabinets
- Sideboard
- passage
- column
- Column head cabinet
- Column tail cabinet
- Private room bridge
- Production of bridges across private rooms
- ……
Figure 4 Cabinet layout & production tray plan
For the above basic data, we extract the basic data strongly related to the wiring project, integrate and abstract several data models, such as cabinet model, channel model, cross-package interconnection model, etc., combined with the cabinet layout & production bridge plan, etc., and the operation and maintenance personnel maintain the above basic data information in advance according to different physical environment conditions, so as to lay the foundation for the calculation of the wiring project.
2.2 Function 2: The wiring is calculated by automation
Method 1: Scenario inductive calculation method
In order to automatically complete the calculation of the layout of the route, Station B adopts a cost-effective way to quickly build a simple and clear schematic diagram of the layout of the private room of the data center, and the operation and maintenance personnel draw the physical layout structure of the private room by adding the form of rows and columns, and associate the basic data to establish the relative position relationship of columns, channels, and cabinets.
Figure 5 The layout of the data center built on the platform
Summarize the daily wiring requirements and integrate 6 abstract high-frequency wiring scenarios, which are:
- Wiring in the same channel in the same private room
- Cross-channel cabling in the same compartment [Specify Route Direction]/[Do Not Specify Route Direction].
- Routing is not specified across private rooms, where the starting cabinet, the end cabinet, and the tray exits of the private room are in the same column
- Routing is not specified across private rooms, where the starting cabinet or end cabinet and the tray exit of the private room are not in the same column
- For the routing of the specified route across the private room, the starting cabinet and the end cabinet are not in the same column as the tray exit of the private room
- Other scenarios can be added flexibly.
For different scenarios, a unified formula for calculating the routing distance is defined, and the routing method with the shortest distance is selected.
Method 2: Shortest path optimization method
Compared with the scenario induction method in method 1, the calculation efficiency is higher, and the layout and wiring scenarios of the computer room are wider, but the difficulty lies in how to transform the actual engineering problems into mathematical problems and solve them.
We first number the strong and weak current bridge, computer room cabinet and distribution cabinet in a segmented and orderly manner in the design stage, and then establish a mathematical relationship model for the cable connection equipment such as the bridge, cabinet and distribution cabinet in the wiring planning stage, clarify the geometric relationship between each other, and input the initial data of the wiring length of each section of the tray, and finally automatically calculate the shortest path of wiring between any equipment through the optimal path planning algorithm, and output the planning path result.
Figure 6 Flowchart of shortest path optimization
2.3 Function 3: Create a cabling task
2.3.1 Task Generation
For single cabling tasks, you can create cabling details in batches according to cabling scenarios (such as vertical cabling in a cabinet, cross-cabinet cabling from a service switch to the network core, and cross-cabinet cabling between network cores) for a single cabling task.
The following core information is specified in the cabling details:
Routing task name: Name the task requirements
The starting point information of wiring: including the starting private room, starting cabinet, and starting U position
The end point information of the wiring: including the end point private room, the end point cabinet, and the end U position
线材类型：六类非屏蔽网线，LC-LC，AOC，MPO……
Cabling cable number: Automatically generated
……
After the basic wiring information is clarified, the system automatically calculates the shortest path and outputs the cabling length, and gives a list of all cabling tasks.
Figure 7 The details of the new wiring on the platform are displayed
2.3.2 Task Execution
After the platform wiring task is generated, the task execution stage is carried out, and the process management documents are digitized and platformized through process approval, and the relevant information of the wiring project is updated in real time until the final wiring acceptance and delivery.
2.4 Function 4: Cabling operation and maintenance management
After the pre-cabling task is completed, the basic information data of the cabling will continue to provide value for subsequent cabling O&M management. Through the visualization tool, you can quickly understand the key information of the wiring in the data center and judge the delivery of network cabling in the business environment. Through data analysis, you can further analyze whether the cable type matches the port of the connected device, the failure rate of different types/brands/batches of cables, and the trend of cabling costs, so as to help the sourcing of faulty lines.
Figure 8 Visualization of the cabling O&M management interface
3. Platform application and future prospects
At present, the 1.0 version of the B station wiring intelligent management platform has been officially launched, and the invention patent "computer room wiring method and system" submitted by us is in the substantive review stage, which has realized the functions of wiring and computer room information platformization, wiring and automatic calculation of length, etc., which not only greatly improves the efficiency and effect of the implementation of wiring engineering, but also reduces the routing length of the platform automatically planning by 10% on average compared with the length manually calculated in the early stage, so as to help network and data center operation and maintenance management personnel do a good job in cabling cost reduction and cabling system management.
In the future, we will continue to integrate the implementation, delivery and network operation and maintenance experience of cabling projects, and strive to open up the information docking and application between network management, IDC operation and maintenance management and cabling management. In this way, we will build a more comprehensive, intelligent and powerful cabling management platform for Bilibili, so as to help realize the rapid delivery, intelligent operation and maintenance and fine management of network cabling.
Reference:
[1] The practice of customized project of station B of a new generation of smart energy-saving data center
[2] Lu Lei. Research on Integrated Cabling Technology for Data Center Network. Telecommunication Letters: Networks and Communications 8 (2019): 5.
[3] Which cabling should be used in the data center
Author: General Engineering: Tao Shuai, Lei Mutou
Source-WeChat public account: Bilibili Technology
Source: https://mp.weixin.qq.com/s/PLYoQNRN-Wh_rivYjJK1hg