The following article is from Tencent Game School, written by Tencent Game School
Introduction: At this year's Game Developers Conference (GDC 2024), experts from Tencent Games continued to share cutting-edge information, focusing on AI, rendering, cross-terminal game development, animation and other game technology applications and game production, which attracted the attention of the industry. This article is the dry content of the graphic version shared by "MotorNerve: A Character Animation System Using Machine Learning".
Speakers:
李松南 腾讯互娱天美F1工作室Machine Learning Team副总监
Liao Yuchen is a senior engine developer at Tencent Interactive Entertainment TiMi F1 Studio
Hello everyone, I'm Liao Yuchen from Tencent Games. My colleague Songnan Li and I will share with you how we apply machine learning in character animation. Let's get started!
The theme of this sharing, MotorNerve, is the name of a character animation system that uses machine learning. We expect to use machine learning to improve animation, productivity, and performance
We have a very talented team and we are very grateful for the dedication of our team members. Here I would like to introduce TiMi J1F1 Studio, which is the studio to which Songnan and I belong. MotorNerve is a technology that will also be used in studio products such as QQSpeed and NFS:Mobile.
一、Motion Matching
Okay, let's dive into our talk, which will be divided into two parts: one is a walking animation based on the improved Learned Motion Matching, and the other is an interactive animation based on our own animation intermittent complementation technology.
Here's an outline of the first part: First, I'll introduce some animation optimization methods and tools based on Motion Matching. I'll then discuss Learned Motion Matching and Base Motion Matching (or LMM and BMM for short) and explain why we use machine learning in Motion Matching. Finally, I'll share some of the work we've done for performance optimization.
As we all know, Motion Matching is a well-established technology that is commonly used in Locomotion. Here are two Locomotion demos that we have completed, and you can see that the movements are very coherent and smooth. To achieve this effect, we implemented a number of methods and tools to polish the animation. I'll talk a little bit about how to implement them.
We all know that Motion Matching switches back and forth between different animation clips during the run, affecting the completeness and continuity of the animation. Therefore, we have added some strategies to the algorithm of Motion Matching, such as allowing you to increase the cost of animation jumps, or blocking nearby frames when jumping, etc., to avoid unreasonable animation jumps and make the whole animation smoother.
In addition, when the player manipulates the movement of the character, in most cases, there will be a certain error between the animation and the logic trajectory. For example, in this video, the yellow one is the target trajectory operated by the player, and the red arrow is the trajectory of the animation matched by MM, which is still relatively biased. We have implemented a set of dynamic trajectory correction functions, which can automatically correct the trajectory of the animation so that it is close to the target trajectory of the player's operation, and the green arrow is the result.
In addition, we provide powerful editing features and debugging tools to help us quickly complete the animation polishing work. On the right is our debugging tool, which records each frame and shows the details of the Motion Matching match, helping us quickly analyze ways to make the animation perform better.
Now that we've seen how we polish our animations, let's talk about why we use machine learning. Using traditional Motion Matching (we call it Base Motion Matching) allows for rapid development iterations, but when it comes to running games, there's a problem that can't be ignored: high-quality animations require a huge amount of animation, resulting in a high memory footprint, which is often unacceptable.
Machine learning-based Learned Motion Matching can solve this problem perfectly, but the disadvantage of LMM is that it is difficult to develop. Every time you simply change some parameters and want to see the results you have to wait for the model to train, it usually takes hours or more, so what should we do? One possible way is to use BMM in the development phase and LMM in the running phase.
So it's important to know that the LMM and BMM have to behave the same, and we've done a lot of work for that. You can see this video, with the LMM on the left and the BMM on the right, and the difference is almost invisible. In addition, we also measured some objective data and did subjective double-blind experiments, which showed that our LMM performance can be aligned with BMM.
In addition, the previous LMM solution could not implement curves and Notify functions, which made it difficult to actually apply them in the game. We've implemented this feature to automatically handle animation curves and notify.
Okay, now let's talk about performance. The Learned Motion Matching algorithm, while satisfactory in terms of memory, is less than ideal in terms of CPU time, which also makes it difficult to apply in game projects. We have improved the inference architecture of the LMM, using framerate-independent steppers and nonlinear strategies, resulting in a significant performance improvement and enabling the same model to be applied at different frame rates. As you can see in the video on the right, the models are all trained at 30FPS. The original LMM model doesn't work at 60FPS, while the MotorNerve does.
In addition, we have implemented batch inference optimization, which can save up to 50% of CPU overhead when multiple people are on the same screen. Here's our demo that achieved 160 characters at 60FPS with a single thread. Pretty cool, right?
That's it for the first part, where we discussed how machine learning can be used to create an amazing Locomotion animation system that can achieve smooth, high-quality animations with great performance. Next, my colleague Li Songnan will introduce you to the second part of the content of interactive animation.
二、Motion In-Betweening
Now, let's dive into the second part, which is the complementary technique that can be used to generate transition animations. If you have some experience with animation and machine learning, then this will be easy to understand.
Here's the outline: First, I'll cover what animation complement is and why we need it in game development. Then, I'll cover the classification of animation complement methods, which category our method falls into, how it works and why it's better than others. Finally, I will introduce its application in our interaction and displacement systems.
In the image on the left, the orange pose is a keyframe that can be manually set by the animator. The blue pose is generated by our animation intercompensatory algorithm. To put it simply, the inter-animation complementation algorithm is a method of generating intermediate animations based on some keyframes. These keyframes are relatively sparse, so you won't get good quality with linear interpolation.
Nowadays, animation complementation algorithms are mostly based on machine learning. In order to generate high-quality intermediate animations, it needs to learn from large amounts of animation data. The video on the right shows the results of our methodology, a collaborative work between Zhejiang University and Tencent Games, published at SIGGRAPH 2022. The blue pose is generated by our method. You can see that the movements are of high quality and the transitions are smooth.
Good. So why do we use animation complements in our MotorNerve project? In video games, we can use at least two scenarios for animation complements. The first is the transition of the interactive animation, and the second is the transition of the bit moving painting.
In the first part of this talk, my colleague introduced the bit movement painting. Walking is an important part of a character's activities. But in addition to walking, the character may also need to interact, i.e., interact with objects or other characters in the environment. In order to achieve this, we usually need a transition that changes the character's state from displacement to interaction. We can use animation complements to make this transition, as shown in the image on the left.
Second, walking itself has different states, such as walking, running, sprinting, moving in different directions, and so on. We can use animation complements to generate transitions for these different walking states, as shown in the image on the right. I'll go into more detail later when I talk about the app.
First, let me introduce the classification of animation intermittent complementation methods. From now on, I will use MIB as an abbreviation for animation complement. In general, there are offline MIB methods that can generate animations offline in the DCC tool, as shown in the image on the right. The inputs are the current frame f1 and the target frame fn. The offline method generates all the intermediate frames at once, as shown in the image, which usually takes a long time and therefore cannot be used in the game in real time. On the other hand, we also have online methods that generate intermediate frames frame by frame. If the model is efficient enough, it can be used in the game in real-time. Our approach falls into this category. It has a relatively low complexity, so it can be used both online in games and offline in DCC tools.
Here's an overview of our approach. In simple terms, it takes the current frame, the target frame, and the difference between these two frames as input, and generates features for the next frame in the latent space. The decoder is then used to decode the feature to generate a pose for the next frame. The output frame is then used as the input frame, and the process is repeated until it ends.
We use a variational autoencoder to encode the motion of the lower body into the latent space, and then sample it in the latent space of this high-quality motion. This is a key factor in achieving higher quality in our method than others, especially in terms of reducing foot slippage. We can reduce foot slippage by 50% compared to other methods. Due to time constraints, I don't think I can clearly explain this complex diagram, especially for viewers who are not familiar with deep learning. If you are interested in the details, please refer to our papers, which are listed below.
In MotorNerve, we expand the dissertation approach in two ways. We've added trajectory control and frame rate prediction.
For trajectory control, we can not only set the pose of the target frame, but also make the character move along a specific trajectory. In the video on the left, we use only one frame and one track to generate this run sequence.
For frame rate prediction, we have developed a method to predict the number of frames to be generated based on the pose of the current frame and the target frame. Intuitively, if the target frame is farther away from the current frame, we should generate more intermediate frames, otherwise if the target frame is close to the current frame, we should generate fewer intermediate frames.
In addition to distance, other factors such as the direction the character is facing, the speed at which they are moving, etc., can also affect the number of intermediate frames. In the video on the right, you can see that if we use a fixed number of intermediate frames, the movement is unstable, i.e. sometimes too slow, sometimes too fast, sometimes it looks like the character is trying to pose for the photographer. On the other hand, as shown on the right, if we can predict a reasonable number of intermediate frames that fit into the input, the resulting motion is more natural.
Next, let me introduce the application of MIB. The first app is to generate transitions for interactive animations. A transition is the transition between the current walking pose and the upcoming interactive animation.
When we press the button to start interacting, the character is usually in a walking state, possibly in any pose. Normally, we can use motion blending to generate this transition, but the blended motion tends to be linear, so it's not natural. There will be a lot of problems with slipping feet. From a programming standpoint, you may need to manually set up some entry points, which is cumbersome. Instead, we can use MIBs to generate transition actions. As shown in the video, it can be used for various characters to interact with objects, such as sitting on chairs, picking up items, opening doors, opening boxes, and more. There is less foot sliding and the transition is natural. You can barely notice which part is the transition action.
In these videos, we use MIBs in the game in real time. On a PC, it takes about 1 millisecond to generate a frame.
We also implemented our MIB method in the action builder to generate transition actions offline. This video shows the results of the transition from a stationary position to an open door position.
Another application of MIB is for walking transitions, to be precise, to generate walking motions that match the trajectory. In addition to looping animations like stationary loops, walking loops, running loops, etc., there are more animations that are transitions between these looping animations that are used to transition the character to different states, such as from stationary to walking, from walking to running, or changing the direction of movement.
In Motion Matching, it's important to match these bits to the control tracks, otherwise you'll have to slide. During motion capture, the actor tries his best to follow a predetermined chart of dance moves, but only approximately. As a result, post-processing is required to fine-tune these bits, which can take time. As mentioned earlier, our MIB method can generate character animations that match the trajectory. Therefore, we can use MIB to generate those animations that do not match the predetermined track.
As an experiment, we asked the actors to perform freely on a motion capture stage, rather than strictly following a dance movement schedule. We then use these motion capture animations to train our MIB model. Finally, our MIB model was used to generate a bit movement plot that matched the control trajectory. Our DCC tool can generate a large number of bit-motion drawings in a short period of time, most of which have high-quality motion effects. In the video below, we compare the effects of using MIB animation and using blending animation for steering motion. As you can see, the character on the left has detailed movements such as legs stepping on the ground.
Okay, that's it for the second part. Let's summarize: First, MIB is a deep learning-based motion generation method. Second, we can use MIBs to generate motion transitions for interactions and displacements. Finally, due to the efficient temporal performance of our MIB method, we can use MIB directly in the game, and we can also use MIB in the DCC tool to generate offline transition animations. We believe that machine learning-based motion generation methods will play an increasingly important role in game development in the near future.