Digital twins (DT) can virtualize the entire lifecycle of a system and are ideal for use in autonomous driving testing. This paper proposes a method for using DT for connected autonomous driving testing in a limited environment, that is, in the simulation test environment of automatic driving, the mapping of DT is used to realize the real connected autonomous vehicle test in virtual complex road scenarios. Relevant experiments show that this method can effectively support the test of networked autonomous driving.
A digital twin (DT) is a digital representation of a physical system (physical twin) capable of simulating the entire lifecycle of a running system and mapping synchronously with the physical twin. The concept of DT began in 2002 and was first used in the aerospace sector. Recently, other industrial sectors such as manufacturing, industrial engineering, and robotics have gradually begun to understand and try this technology.
With the development of autonomous driving, the testing and verification of DT functions has become one of the major challenges in the development of autonomous vehicles. Some researchers believe that this challenge can be solved well by using simulation testing, such as software testing (SIL), hardware-in-the-loop testing (HIL), vehicle-in-the-loop testing (VEHIL), and hybrid simulation testing in virtual simulations. It can quickly simulate any scenario, but it cannot verify the real situation. Compared with simulation testing, the traditional automotive industry relies more on field testing. However, real road testing is expensive and time-consuming in extreme cases, and there are some scenarios where testing is not even possible. In 2017, M-City published a study proposing a data-driven approach to evaluating self-driving cars. Unlike pure virtual simulation, it uses real-world driving data to build test scenarios. This is a DT-oriented approach, but this approach was developed from an active security perspective without the introduction of automotive wireless communication (V2X) technology.
V2X technology can not only provide non-line-of-sight perceptual information for road vehicles, but also establish communication links between the vehicle and the cloud data center; therefore, we believe that V2X technology can play an important role in DT-based autonomous driving testing as a link between physical and cyberspace. V2X technology can send scene information to the vehicle under test on the road and provide virtual testing of the road. Although DT is considered a disruptive technology, it is still in the conceptual stage, with only a few studies devoted to its approach to building and implementing in the manufacturing space. Therefore, there is no mature DT-based autonomous driving test scheme. In recent years, the China Academy of Information and Communications Technology has developed a DT-based networked automatic driving test prototype system, using V2X technology to achieve the whole process of sensor data upload and virtual scene information release, and conducted road vehicle testing. The corresponding test results show that the system can support low-latency networked autonomous driving tests.
DT-based connected autonomous driving test framework
In practice, DT may be defined and understood differently in different industries. Self-driving developers see it as an augmented reality solution. In a sense, a DT-oriented test system refers to an information-physical system (CPS) that collects data in the real world (physical space) through a communication network, uses large-scale data processing techniques in cyberspace to analyze the data, and feeds the results back into the physical space to solve real-world problems. Each CPS includes smart machines, storage systems, and production facilities that can autonomously and intelligently exchange information, make decisions, trigger actions, and control each other.
Figure 1 A networked autonomous driving test framework based on a digital twin
The DT-based connected autonomous driving test method consists of two key steps: one is to collect real driving data, and the other is to generate complex scenarios. Road vehicles collect and publish driving information through sensors and V2X, complete data fusion processing, and then upload the corresponding information to the simulation platform. The simulation platform selects the test scenario based on the real-time driving information and feeds the corresponding information back to the road vehicle. The road vehicle control system responds to the scene information and outputs the response and uploads it to the simulation platform. The simulation platform judges the test results and generates a test report. As shown in Figure 1, the test scheme consists of three layers, namely the field test layer, the network transport layer, and the experimental test layer.
(1) Field test layer. The field test layer consists of 3 parts: the vehicle under test, the virtual car and the head-up display (HUD), the real test site, and more.
Vehicle driving information is collected by in-vehicle sensors, real driving environment information is collected by surrounding vehicles and roadside devices such as radar and cameras, and virtual scene information is provided by cloud databases and displayed on the HUD. This assumes that all vehicles are equipped with LTE-V2X and 4G/5G modules.
(2) Network transport layer. The network transport layer includes two communication modes: a direct communication link for LTE-V2X and a 4G/5G cellular communication link. LTE-V2X is used to collect environmental information such as road information, surrounding vehicle driving status, pedestrian status, etc., and 4G/5G is used to establish the connection between physical space and virtual space. Obviously, the performance of the network transport layer can have a fatal impact on the real-time nature of the autonomous driving test, which can be reflected by the response delay of the vehicle controller. Here we define the basic performance parameters as follows.
● Measured vehicle speed: 0 ~ 130 km/h;
● Communication coverage radius: >300 m;
● Vehicle status information update frequency: 10 ~ 20 Hz;
● Data rate (downstream) > 100 Mbit/s, data rate (upstream) > 20 Mbit/s;
● Transmission delay
In addition, Table 1 lists the content of reference messages that need to be sent via V2X for testing connected autonomous driving.
Table 1 Reference message content based on digital twin connected autonomous driving test
(3) Experimental test layer. The experimental test layer includes three parts: channel modeling, performance index calculation, virtualization, and performance acquisition. In the process of scene generation, a 1:1 digital scene model should be constructed by considering the influencing factors such as the road environment (lanes, lane lines, road surfaces, weather and lighting, scene elements), traffic conditions (traffic flow, pedestrian congestion, adaptive cruise control), traffic participants (vehicles, pedestrians, obstacles), and environmental sensors (radar, cameras, GPS/maps, wireless communications). The simulator should support complex road scene modeling, as shown in Figure 2. During the test, the tester selects the test scenario and sends the corresponding scene information to the vehicle under test through the 4G/5G network.
Figure 2 Road Scene Simulation Simulator
Test the scene design
The test scheme consists of three phases: pure virtual test, sensor data test, and real vehicle test, as shown in Figure 3.
Figure 3 The Test Phase
2.1 Pure Virtual Testing
Phase 1 of the DT test scenario is pure virtual testing. This test is similar to traditional virtual simulation, with the main steps as follows:
● Step 1, build a road scene based on test requirements and data;
● Step 2, set the vehicle parameters (driving parameters and sensor parameters);
● Step 3, increase mixed traffic interference;
● Step 4, add the control algorithm;
● Step 5, start the experimental simulation.
2.2 Sensor data testing
Phase 2 of the test is based on real sensor data, as shown in Figure 4. In the process, real-world sensor data is collected and sent to data centers over a cellular network. The remote driving system makes decisions based on sensor data and feeds the decision information back to the test system.
Figure 4 Sensor Data Testing
2.3 Actual vehicle testing
At this stage, the real vehicle is used as the device under test. Vehicle information is sent to the virtual emulator over a 4G/5G network. Two typical simulation scenarios are shown in Figure 5. On the basis of the virtual simulation test, traffic flow and autonomous driving algorithms are added to simulate the real scene and timely return to the real autonomous vehicle. The decision results of the controller will be sent back to the central control simulator, and the test will be completed under the control of the autopilot algorithm.
Figure 5 2 typical real-car test evaluation scenarios
Demonstration and analysis of test results
3.1 Demonstration of test results
In order to verify the effectiveness of the test architecture, we built a DT-based networked autonomous driving test prototype system at the China Academy of Information and Communications Technology Office, and the specifics of the relevant tests are shown in Figure 6.
On-board sensors and roadside units collect information about the environment around the vehicle under test. Each device captures the surrounding information according to its configuration, and then synchronizes the collected data under the same timestamp label with other devices on the network. The information about the physical system is uploaded to the server that is running the DT application.
Figure 6 The test results are demonstrated
Figure 6 c) shows the forward collision warning application validation process. Virtual objects are displayed in the center controller simulator, and the real vehicle behavior is fed back to the center console through the camera.
3.2 Analysis of test results
Through the test of dt-based connected autonomous driving test prototype system, we believe that V2X technology can not only support application functions such as vehicle driving safety and efficiency improvement, but also be used for connected automatic driving testing. In addition, autonomous driving testing can also be considered one of the important application scenarios of V2X technology.
Although this test protocol has proven to be effective, it is only a basic prototype system, and there is a lot of room for improvement.
(1) The automotive industry has not yet reached an agreement on the delay in response to autonomous driving. That said, we are not sure whether the proposed solution can meet the response delay requirements of autonomous driving. With the development of autonomous driving, the corresponding communication performance requirements need to be defined. DT-based connected autonomous driving testing will also be considered an application for 4G/5G communication networks. The testing process should continue to be improved on this basis.
(2) In this scheme, in-vehicle testers can not see the virtual scene generated in cyberspace, the user experience is not good, therefore, the combination of virtual reality (VR)/augmented reality (AR) can be a later research direction.
(3) It is necessary to ensure that the generated virtual scene matches the real traffic scene. So far, the traffic database is not complete. That is, there is currently no guarantee that the test scenario will cover and represent a realistic traffic scenario. With the improvement of the traffic database, the test scene library will also be continuously improved.
(4) As an application of communication scenarios, DT-based networked autonomous driving tests must be implemented based on unified communication protocols, and data sets and data exchange formats need to be defined; therefore, we need to develop corresponding message layer protocols to ensure that products of different manufacturers can be tested in different test demonstration areas.
(5) How to choose a test scene that matches the real scene. One of the key points of autonomous driving testing is the choice of scenarios. Obviously, all virtual test scenarios are derived from real traffic scenarios. In order to identify the weaknesses of autonomous control algorithms, we always choose the worst scenarios to build a virtual scene library, so virtual tests cannot accurately assess the risk or probability of real-world situations. So far, however, we have not been able to provide a trusted mapping between real-world and virtual-scenario tests. In other words, we should establish a DT-based networked autonomous driving performance evaluation system for government standardized tests or inferring the expected safety performance of autonomous vehicles.
As V2X technology evolves, the automotive industry is considering embedding V2X-related application capabilities into products. V2X is not limited to applications such as vehicle safety and traffic efficiency improvement, it is a promising technology in other application areas. In dt-based connected autonomous driving test scenarios, V2X technology plays an important role in connecting virtual and physical spaces.
Reprinted from the forefront of intelligent driving, the views in the text are only for sharing and exchange, do not represent the position of this number, such as copyright and other issues, please inform, we will deal with it in a timely manner.