Compile the | ZeR0
Edit | Desert Shadow
The new Sony Dafa is coming!
Zhidong reported on February 10 that today, Sony AI, Sony AI, announced that its AI program beat the world's top racing game players and landed on the cover of Nature, the top international academic journal.
Thesis Link:
https://www.nature.com/articles/s41586-021-04357-7
This is another milestone achieved by game AI after AI defeated human champions in poker, chess, Go, StarCraft, DOTA and other games.
As the world's first racing AI agent to be able to defeat the strongest human players in a highly realistic racing simulation game, Sony's racing game AI GT Sophy has honed its tactics and skills in just one or two days to surpass the 95% of human players in the racing simulation game GT Racing. After a total of 45,000 hours of training, the AI program is already able to compete with the top GT racing players.
Compared with the board games and some multiplayer strategy games that AI has mastered before, GT Racing is more complex because it highly simulates the real world, every car and every track is modeled, and the visual, audio and dynamic aspects all restore the real-world driving experience as much as possible.
This makes AI must have a strong continuous judgment and rapid response ability, under the condition of high-speed changes, comprehensive consideration of friction, aerodynamics, driving routes, speed, direction and other factors, within a few inches of the opponent, the vehicle with complex nonlinear dynamics of real-time control, and know how to surpass the opponent without violating the law.
"To surpass outstanding human drivers in a head-on race is a milestone achievement in the field of AI." Professor Chris Gerdes, co-director of the Stanford Center for Automotive Research, believes the technology used to develop this AI is expected to play a role in self-driving car software.
GitHub Links:
https://sonyai.github.io/gt_sophy_public/
A stronger agent than playing StarCraft, proficient in control, tactics, and etiquette
Launched in April 2020, the GT Sophy Research Project is an autonomous AI agent trained using a new deep reinforcement learning platform and one of the key challenges Sony AI has been working on since its inception in November 2019.
Sony AI is based in Japan, the United States and Europe, focusing on promoting three AI flagship projects in gaming, imaging, and sensing. Sony AI global head Hiroaki Kitano also said: "By 2050, AI will win the Nobel Prize with its own scientific research results!" ”
And the racing game AI that appeared on the cover of Nature today is a big move that Sony AI has been raising obscure and planning for a long time!
Over the past two years, the Sony AI team, polyphony Digital (PDI), the game development studio behind the GT Racing series, and the cloud gaming team at Sony Interactive Entertainment (SIE), have worked closely together to train the AI using the cloud gaming infrastructure managed by the SIE.
To recreate the real-world racing environment as much as possible, PDI created the surrealist drive simulator GT Sport for the PlayStation 4 and provided API access.
The GT Sport is equipped with some of the latest car dynamics simulations that realistically recreate physical phenomena such as racing, tracks and even air resistance, tire friction, and are guided by automakers to buckle every detail from body curves, body panel clearances to headlight shapes.
Designed in collaboration with the FIA, the simulator is an esports community of over 400,000 people worldwide, and it brings a fair racing environment with clear rules and criteria for judging.
GT Sophy was trained in this ultimate simulation environment, and the distributed training platform DART is also instrumental in this new AI achievement.
Based on this custom platform, Sony AI researchers were able to train GT Sophy on the PlayStation 4 console on the SIE Cloud Gaming Platform.
DART allows researchers to easily specify experiments, run them automatically when cloud resources are available, and collect data that can be viewed in a browser. In addition, the platform manages the PlayStation 4 console, proxy computing resources, and GPUs for training across data centers.
It has access to more than 1,000 PlayStation 4 consoles, each of which is used to collect data on trained GT Sophy or evaluate a trained version. The platform consists of the necessary computing components (GPU, CPU) to interact with a large number of PlayStation 4s and supports long, large-scale training.
DART enables Sony AI's research team to seamlessly run hundreds of experiments simultaneously and explore techniques to take GT Sophy to the next level.
Backed by these infrastructures, GT Sophy surpassed about 95 percent of gt-sport players in just one or two days. After 10 days of 45,000 hours of driving learning, gt Sophy achieved superhuman time trial performances on all three tracks.
To test the power of this racing game AI, the researchers had GT Sophy compete with four of the world's best GT racers in the "2021 Racing Challenge" on July 2 and October 21, 2021, and successfully surpass these top human racers.
Second, how is the top racing game AI refined?
To create super-powered racing game AI, Sony AI researchers and engineers have developed innovative reinforcement learning techniques, including a new training algorithm called Quantitile-Regression Soft Actor-Critic (QR-SAC), an understandable coding of racing rules, and a training program that facilitates the acquisition of subtle racing skills.
Deep reinforcement learning is a key component of most AI milestones in arcade games, chess, Go and other real-time multiplayer strategy games, and is particularly suitable for developing game AI agents, as reinforcement learning agents consider the long-term effects of their behavior and can independently collect their own data during learning, thus avoiding the need for complex, hand-coded behavioral rules.
Dealing with complex areas like GT Racing requires equally complex and subtle algorithms, rewards, and training scenarios.
The AI takes information from multiple GT Racing games and learns how to win by maximizing the rewards for running laps and minimizing the penalty for collisions. For example, if it overtakes another car, it will receive a certain weight reward, but accidents such as cutting corners, collisions, skids, etc. will be punished.
GT Sophy was trained in a variety of scenarios on GT Racing's three car and track combinations. Some of them only have AI agents on the track, while others add 7 NPC opponents for normal gameplay. Each track position, starting speed, spacing between cars, and opponents' skill levels are random.
Through continuous learning and experience, GT Sophy has mastered the skills of car control, racing tactics and racing etiquette.
(1) Racing control: Racing is essentially trying to drive a car that is on the edge of control or traveling farther away. Estimating braking points, finding the best route, finding grip to maximize speed and control, and so on are very interesting machine learning problems in themselves.
A new algorithm, QR-SAC, explicitly reasons various possible outcomes of GT Sophy's high-speed actions. Explaining the consequences of driving action and the uncertainties involved helps GT Sophy navigate corners at the limits of the car body and consider complex possibilities when racing against different types of opponents.
Let's look at an example of GT Sophy's extreme driving skills, where an agent can drive through a series of corners that stick to the wall without contact.
(2) Racing tactics: Drivers need to be able to make quick decisions in high-speed changing racing situations to overtake their opponents on the line, taking into account the opponent's reaction to overtaking attempts. While AI agents can collect their own data, training specific skills such as slipstream passing requires the opponent to be in a specific position.
To address this, GT Sophy's learning includes mixed-scenario training using manual race situations that may be critical on each track, as well as specialized opponents who help agents learn these skills. These skill-building scenarios helped GT Sophy acquire professional racing skills, including handling crowded starts, slingshot wake overtaking, and even defensive maneuvers.
▲ GT Sophy successfully overtook the human racer with sharp turns
(3) Racing etiquette: Drivers need to follow specific rules to limit the extent to which the car can slide off the track and who should be held responsible in the event of a collision. At the same time, drivers need to drive aggressively to win, and finding the right balance is a challenge.
To help GT Sophy learn exercise etiquette, Sony AI researchers found ways to encode written and non-cost race rules into complex reward functions. The research team also found that it was necessary to balance the number of opponents to ensure that GT Sophy had a competitive training match, while not becoming too aggressive or timid toward human competition.
For example, the GT Sophy overtook human riders without blocking the driving circuit, leaving them plenty of room to maneuver and show fairness and sportsmanship.
These features set GT Sophy apart from the early AI agents that had previously beaten human champions in some of the classic games.
Chess, Go, etc. are completely informational games, and AI does not need to master the physics of the real world, but only focuses on game strategy. Even AlphaStar, which plays StarCraft, and OpenAI Five, dota, aren't trying to master real-world physics.
Now, GT racing cars are trying to simulate the real world, so tactics, strategy and etiquette are crucial, and more difficultly, AI needs to have these skills when the car accelerates at the physical limit.
Third, it can also be applied to robots, drones and automatic driving
Like other AI that beat human champions, gt Sophy's value can be limited to playing games.
During the development of GT Sophy, researchers regularly interacted with top drivers to test the latest version.
"Sophie's racing route is something that a human driver would never have imagined." Kazunori Yamauchi, creator of GT Racing and a real-life racer, said the technology will be part of its future version of the game and is expected to help novices and professional drivers improve their skills. "I think a lot of textbooks on driving skills will be rewritten."
THE GT Sophy has also given new inspiration to top human riders. Igor Fraga, 2018 winner of the FIA Gran Turismo Championship, praised: "GT Sophy shows us new possibilities that we never imagined before. ”
Japan's top Takuma Miyazono, who won an unprecedented triple crown in esports racing, has been playing virtual racing since he was 4 years old, but he has never met a racer like GT Sophy. "Sophy is very fast and the laps are better than the best drivers expect." He believes that seeing Sophy, some actions are possible.
EMILY Jones, the FIA Gran Turismo Championship 2020 World Finalist, was also inspired by GT Sophy, whose lap time on the Dragon Trail was 107.964 seconds and ai's lap time was 106.417 seconds.
▲Emily Jones
"In some corners, I drove the car very big and then reversed, and the AI drove the car very close, so I learned a lot about the line. Also know what to prioritize. Taking entering corner 1 as an example, I braked later than the AI, but the AI would have a better exit than I had and beat me in the next corner. It wasn't until I saw AI that I realized this and thought, "Okay, I should do this." Emily Jones said.
What's more, this research breakthrough will spark a debate about the best computing methods used in driverless cars.
Sony AI global head Hiroaki Kitano said that the purpose of GT Sophy is not only to surpass human players, but to provide players with a stimulating opponent, accelerate and enhance players' technology and creativity. The AI algorithms developed for GT Sophy may also be applicable to other types of machines such as drones and robots.
"In addition to contributing to the gaming community, we believe this breakthrough also brings new opportunities in areas such as autonomous racing, autonomous driving, high-speed robotics and control." Hiromi Kitano said.
▲Hiroaki Kitano, Sony AI CEO
Avinash Balachandra, senior manager of driving research at the Human Center at Toyota Research Institute, said: "The use of machine learning and automatic controls in racing is exciting. The institute is testing self-driving cars capable of operating at extreme speeds. Toyota is working on "human amplification techniques that could one day improve active safety systems using technologies that experts have learned from motorsport," he said.
Bruno Castro da Silva, a professor of reinforcement learning at the University of Massachusetts Amherst, described GT Sophy as "an impressive achievement" and an important step toward training AI for self-driving cars.
But he thinks going from GT Racing to the real world will be a challenge, because reinforcement learning algorithms like GT Sophy have a hard time considering the long-term implications of decisions, and it's also hard to guarantee the safety or reliability of these algorithms.
"If we want such AI systems to be deployed in real life, security is paramount." Da Silva said, "The lack of security is one of the main reasons why machine learning-based robots have not yet been widely used in factories and warehouses." ”
IV. Conclusion: The double victory of AI and gamers
Commenting on the progress of this research, Kenichiro Yoshida, Chairman, President and CEO of Sony Group, said: "Sony's mission is to "fill the world with emotions through the power of creativity and technology", and GT Sophy is the perfect embodiment of this concept. ”
Overall, racing game AI not only shows the technological advances of how AI learns strategies to work in complex situations, but also how AI can provide players with new gaming experiences.
Sony AI and PDI will explore how to integrate gt Sophy into future versions of the GT Racing series. Kazunori Yamauchi, president of Polyphony Digital, believes that this AI concept will advance the future of gaming and the automobile.
Source: Sony AI, Nature, Ars Technica, Wired