Many potential applications of AI involve making real-time decisions in physical systems while interacting with humans, and racing is an extreme example of this. The driver must perform complex tactical maneuvers to overtake or block the opponent while driving the vehicle under the limits of its traction. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real cars, while also including complex multi-agent interactions.
Here, Peter R. Wurman, sony's AI division, and others describe how to use modelless, non-strategic deep reinforcement learning (deep RL) to build championship-level racing agents and name it Gran Turismo Sophy (GT Sophy), a highly realistic PlayStation 4 (PS4) game developed with Polyphony Digital Turismo (GT) Sport (https://www.gran-turismo.com/us/) competes with the world's best gamers.
To be successful, a racer must be proficient in four areas:
(1) Racing control, (2) Racing tactics, (3) Racing etiquette and (4) Racing Strategy.
The authors combine state-of-the-art, model-free deep reinforcement learning algorithms with hybrid scenario training to learn an integrated control strategy that combines superior speed with impressive strategies. In addition, the authors construct a reward function that enables GT Sophy to be competitive while adhering to the important but unspoken rules of sportsmanship in racing.
Figure 1. Impact of collision penalties and other key design choices in the project
Next, the author showcases the GT Sophy by competing against top human players on three car and track combinations that pose different racing challenges. The first track, the Dragon Trail Seaside, used a high-performance road bike. On the second track, the Lago Maggiore GP (Maggiore), uses a car equivalent to the International Automobile Federation (FIA) GT3 class. The third and final race was held at the Circuit de la Sarthe (Sarthe), famous for hosting the 24 Hours of Le Mans.
The race features racing with the Red Bull X2019 Competition, which can exceed 300 km h-1. Despite a lack of strategic acumen, gt Sophy has shown significant progress in the first three of the four skill areas mentioned above in the process of winning all 4 human races.
Figure 2. GT Sophy matches against human top players