Edit: Good Sleepy Peach Yuan Xie
The dignity of human gamers has collapsed into a corner, and this time it is the game manufacturer's father himself who humiliates him: Sony developed its own AI players and defeated the world's first-class e-sports players in GT racing games.
High-powered AI defeats humans in a variety of games, and these messages have appeared frequently in recent years.
Whether it is chess and quiz competitions in the early years, or Go, StarCraft, and Turret 2 in recent years.
In front of the AI players, the human professional players are literally beaten to the point of crying.
Today, AI racers once again beat human players and landed on the cover of Nature!
Sony's own development of the AI "GT Sophy" has made electronic racing games join the above list: whether it is The top e-sports players in Japan or other countries, they are inferior to the AI trained by Sony with 1,000 PS4 consoles in GT racing.
GT Sophy, or "Sophie," is a joint venture between Sony's in-house AI division, PDI Studios, and interactive entertainment subsidiary, and took more than five years.
The AI division provides deep learning algorithms and is responsible for training models; games from PDI Studios serve as an environment for AI training and verification; and interactive entertainment companies' cloud computing architecture connects thousands of PS4 hosts as large-scale training architectures and platforms.
As a neural network driver, gt Sophy exhibits exceptional speed, handling and driving tactics while adhering to the rules of the car.
Stanford Professor Christian Gerdes wrote in an article on Nature: "In this face-to-face competition, AI drivers are so adept at outpacing human players represent a milestone achievement in artificial intelligence."
Dragon Trail stages "Fast and Furious"
What's so powerful about that???
In other words, the purpose of racing is simple, that is, to surpass the competitors in less time, and you win.
The hardcore collisions in Fast & Furious are blood-boiling, and the ultimate showdown on the real field requires real tactics.
Sony PDI Studios offers a venue in Tokyo to compete with GT Sophy and its variants from leading GT Racing games worldwide.
Excerpts from the competition
In gt-Sophy's first match with human players, the Dragon Trail was featured in a spectacular duel.
The entire runway is 5209 meters long, divided into three sections of S1, S2 and S3, as well as 17 curves.
The starting position of the race is arranged in the order of AI and real people, with a total of 8 drivers.
However, at the beginning of the race, the GT Sophy AI variant named Violette completed the surpassing of human drivers, ranking second.
AI teammate Bordeaux has always maintained the number one position.
At the T2/3 turn, AI player Verte swooped in front of human player Tomoe Yamanaka.
In the middle of the race, the First and Second Sophy AI chose the optimal route without obstruction.
The race progressed to zone S3, where racers had to go around the T17 bend to quickly reach the finish line.
The results of the race showed that gt sophy occupied two seats among the top three racers, and only human racer Ryota Kokumi rushed into the top three.
Ryota Kokubu has always been a calm and composed racing driver, winning the 2019 FIA Sports Car Championship In Tokyo National Cup in the fifth round and the 2020 FIA GT Championship National Cup in third place.
The emergence of this result, in fact, can be seen from the qualifiers.
The AI player on the right side of the screen is not only more stable when cornering, but also chooses a route that is significantly better than that of a human player.
In another race on the in-game Lake Maggiore circuit, four AI models defeated four human opponents.
The most prominent of the AI legions in the competition, the GT Logue, was in the first place in all three scoring laps of the competition, more than 5 seconds ahead of the fastest human player. However, on the Lake Maggiore circuit, there are also AI drivers who make mistakes, understeer when cornering, and hit the wall.
In its July 2021 experiment, GT Sophy was able to surpass human runners in pure racing on virtual empty tracks with no other cars. Train until October 2021 to beat a group of human opponents in the real game racing session.
Takuma Miyaen, a star player at the 2020 GT Racing World Cup, said: "We can't think of an AI driving style." But he also acknowledged that gt sophy's tactical decisions were useful.
Takuma Miyaen, who was defeated by the AI, was disobedient
It can be seen that racing is not just a matter of speed and reaction time, driving a car is extremely challenging for a person's extreme strategy, so getting the machine to master is not an easy task.
As Nature points out, achieving this will require overcoming extremely complex physical challenges, as the "limited friction" between the tyres and the ground needs to be carefully used to race the track.
For example, braking with friction reduces the force that goes around the corner.
More specifically, each grounded tire can produce a friction force, or load, proportional to the vertical force. When the car accelerates, the load shifts to the rear tire, and the friction of the front tire decreases.
This can lead to understeering, in which case the steering wheel cannot generate more cornering force and will maintain a de facto handbrake state when rushing out of corners.
Conversely, when the car brakes, the load is transferred to the front of the car. This can lead to oversteering, meaning the rear tire loses traction and the car spins furiously.
Add to that the complex track terrain, as well as the complexity of adjusting load transfers with suspended vehicles, as well as the challenges of racing become apparent.
In order to win the race, the driver must choose to keep the car as close as possible within the range of friction limits.
If you brake too early when cornering, your car will slow down and waste time. Brake too late and there won't be enough cornering to maintain the racing line you want. In addition, braking too hard may cause the car body to rotate.
Although the driving limits of a car are very complex, physics has explained these limits very well. Therefore, it is also natural that they can be computed or learned.
In a face-to-face match, gt Sophy, instead of taking advantage of lap time, easily overtook the competition in the end.
For example, at the final sprint node of the first race, 2 human racers try to block the path of 2 AI racers.
But GT Sophy managed to find two different routes, finishing the way past the human players and rushing to the finish line.
After training through a neural network model, GT Sophy learned to take different routes through corners in different situations.
So, how exactly did this AI Superman racer come to be?
How do AI racers get made?
Unlike other games with more fixed rules and gameplay, gt racing games have very open player tactics options. And gt racing games are characterized by better simulations of the laws of real-world physics.
Therefore, a GT racing game with both virtual and real world difficulties is not easy for AI to play well.
First, we need an ultra-realistic simulator as a training environment.
Gran Turismo® Sport (GT Sport) is a driving simulator for the PlayStation 4 designed and produced by Polyphony Digital in collaboration with FIA (International Automobile Federation).
GT Sport has clear rules and judgment standards, and it is possible to ensure a fair playing environment without "cheetahs". （doge）
In addition, GT Sport reproduces the real-world racing environment as realistically as possible, including its cars, tracks, and even physical phenomena such as air resistance and tire friction.
Under the direction of the car manufacturer, the details of the car are accurately reproduced, from the curve of the body, to the width of the gap between the body panels and the shape of the turn signals and headlights.
After the game environment is available, it is necessary to configure the training environment.
DART is a network architecture tailored by Sony AI for this purpose, allowing researchers to use Interactive Entertainment's cloud computing game platform to connect 1,000 PS4 game consoles to train GT Sophy for a long time.
Thousands of PS4 host arrays, the luxurious dream of the game house
In such an architecture, all computing resources between various offsite data centers are effectively consolidated. Researchers can easily determine experimental parameters, set experiments to be automated when cloud resources are available, and collect data that can be observed in the browser.
Researchers have smoothly executed hundreds of simulation experiments on this platform, raising the technical tactics of AI to unprecedented heights.
Finally, there is the training of AI driver GT Sophy.
Like its AI predecessors who defeated humans in games, GT Sophy also used deep reinforcement learning to train in order to avoid artificially encoding game behavior into a complex and huge set of behavioral rules at the beginning.
When an agent, or GT Sophy, takes action in the training environment, the algorithm rewards or punishes them based on the results it results. After receiving a reward (or punishment), GT Sophy updates its perception of the world to determine its next move.
Sony AI researchers and engineers have developed innovative reinforcement learning techniques, including a new training algorithm, QR-SAC, to give AI a rational consequence analysis within the rules and physical limits of various high-speed driving decisions. And code with intelligent body-understandable racing rules, as well as a training program that promotes subtle racing skills.
In reinforcement learning, AI racers need to consider the long-term consequences of their actions and be able to collect their own data independently during the learning process, avoiding the need for complex hand-coded behavioral rules.
Of course, dealing with a complex field like Gran Turismo still requires equally complex and subtle algorithms, rewards, and training scenarios.
Later in the training, the researchers added different numbers of opponents to train GT Sophy when playing against human riders
Judging from the results, the algorithm developed by Sony can be described as the best in the world.
After just a few hours of training, the GT Sophy was already on the track, and the speed was "in a day or two" faster than the 95% of the drivers in its training data set.
Of course, 95% is far from enough.
After about 45,000 hours of training, gt Sophy finally beat the human drivers at three circuits ( the Croatian Seaside Dragon Trail, the Italian Maggiore Lake Grand Prix and the French Sartes Circuit).
However, in the game summary with humans, AI has many innate advantages, such as perfect memory and fast reaction time.
GT Sophy, in particular, has an accurate map of the track with coordinates of the track boundaries and "precise information about the load of each tire, the angle of slide of each tire, and the state of other vehicles."
However, two other elements can still be limited: the frequency of action and the reaction time.
The GT Sophy's input signal is limited to 10 Hz, and theoretically the maximum input signal for humans is 60 Hz, which sometimes causes human riders to show "smoother motion" at high speeds.
In terms of reaction time, GT Sophy is able to react to events in the competition environment within 23-30 milliseconds, which is much faster than the maximum reaction time estimated by professional athletes of 200-250 milliseconds. To compensate for this deficiency, the researchers increased artificial latency, training GT Sophy with response times of 100 milliseconds, 200 milliseconds, and 250 milliseconds, respectively.
Even so, GT Sophy was "superhuman in all three tests."
The head of Sony's AI business unit admitted that it is quite difficult to let AI learn to civilize the game without being humble, and it is not too grumpy or too cautious to make tactical decisions in the face of opponents, and the amount of tasks is still very large.
First of all, the AI must learn to drive on the basis of understanding the virtual racing position, virtual aerodynamic model, track graphics, and basic driving action in the game environment; then there are various GT racing game tactics, such as sliding flow obstacles, plugging overtaking, and various blocking card slots; and finally, let the AI learn the necessary track etiquette rules, such as avoiding malicious foul collisions, respecting the opponent's lane safety, and so on.
Racing controls
The QR-SAC algorithm can unambiguously deduce various possible outcomes of GT Sophy's high-speed actions. Accounting for the consequences of driving behavior and the uncertainties involved helpED GT Sophie to corner under the physical limits and consider complex possibilities when racing against different kinds of opponents.
Racing tactics
By adding training in mixed scenarios, using artificially crafted race situations that may be critical on each track, and specifically playing against opponents to help agents learn these skills.
These skills-building programs helped GT Sophy acquire professional racing skills, including handling crowded starts, countering slipstream obstacles with catapult overtaking, and defensive maneuvers.
Racing etiquette
To help GT Sophy learn track etiquette, Sony AI researchers have found ways to encode the written and non-written rules of the car into complex reward functions.
The team also found it necessary to balance the number of opponents to ensure that GT Sophy showed a degree of competitiveness in training matches and did not become too aggressive or timid when playing against people.
Resources:
https://www.nature.com/articles/d41586-022-00304-2
https://www.nature.com/articles/s41586-021-04357-7
https://www.cnet.com/tech/gaming/sony-ai-defeats-human-racers-in-gran-turismo-playstation-game/
https://www.engadget.com/sony-ai-gran-turismo-sophy-racing-gaming-183604864.html
https://www.gran-turismo.com/us/gran-turismo-sophy/technology/
https://www.gran-turismo.com/us/gran-turismo-sophy/race-together/