Introduction: At this year's Game Developers Conference (GDC 2024), experts from Tencent Games continued to share cutting-edge information, focusing on AI, rendering, cross-terminal game development, animation and other game technology applications and game production, which attracted the attention of the industry. This article is the dry content of the graphic version shared by "Delta Action" double-end high-quality integrated production pipeline and technical support".
Guest speaker: Shi Yang, head of the DF project client of Tencent Interactive Entertainment Tianmei Y2 Studio
Good afternoon, and thank you for joining us. My name is Shi Yang, and I'm in charge of the development of the Delta Ops game client. This presentation was supposed to be given by my colleague Liang Shangli and me. But due to some travel issues, he couldn't join us, so I'll be alone for the whole presentation.
Today, I will share "Double-end high-quality integrated production pipeline and technical support", covering the various production challenges and problems encountered in the development of "Delta Action".
First, let's watch a video to understand the basics of the game.
Operation Delta: Black Hawk Down is the sequel to the classic Operation Delta series from the late '90s and early 2000s, and we decided to bring it back into a next-generation operator tactical shooter for PC, console, and mobile.
The scale of our product is actually quite large, and we want to bring the ultimate FPS experience to players on all platforms. Today, given the hardware differences between PC and mobile platforms, we will also focus primarily on cross-terminal development between the two platforms. Let's start by looking at the challenges we face.
The first is that there is a huge gap in performance, and the frame rate is the benchmark for FPS games. Our goal was to cover low-end devices on each platform, and in order to make better use of the platform's capabilities, we decided to run different renderers. Another issue is that we have to develop and publish content at the same time, which means that porting is not a viable option. Moreover, two separate teams responsible for each platform are also unacceptable in terms of production and maintenance efficiency. Therefore, we decided to build a unified end-to-hand development process. Our goal is to deliver high-quality cross-platform games with the same team at the same time.
Let's take a look at the big picture, in the early stages of our decision to unify cross-platform development, we actually had a lot of heated discussions and envisioned a lot of potential problems. Basically, we've focused on two main areas, one is the production process part, which focuses more on the production of assets and content, and the other is cross-platform running support, which mainly involves game development and performance tracking tools.
This is not a complete list, this is just part of the problem, and the reality is much more complicated. Today we can't go into every detail in detail, but we've selected some typical questions that we think are important and challenging, and share some ideas and approaches with you.
Let's start with the asset making part.
Basically, we have different sharing policies set up for different types of assets. But how to build a cross-platform scene is one of the most complex issues, involving the production of individual models, LOD management, shader management, level layout editing, performance overhead, and even the level artist's workload. This is the part of the production process that we are mainly focusing on today.
So, first let's look at what problems arise when sharing assets across platforms?
For shared assets, the first problem we need to solve is how to support two sets of shaders in the same material slot. Second, we make the most of the LOD chain, but as an FPS game, we can't tolerate any collision mismatches. In addition, there are differences in texture resolution and pixel ratio between PCs and mobile devices. This results in changes in texture size and compression settings. When we share resources between different platforms, we need to remove redundant data on non-target platforms to avoid wasting memory and disk storage. Of course, there are many other more detailed issues.
It's tricky to manage different shaders in the same model, especially since we have multiple renderers, we run deferred renderers on PC and high-end mobile devices, and forward rendering on other mobile devices. To do this, we implemented a virtual material system. As the name suggests, it is used to virtualize materials. It provides the ability to decouple assets and shaders, as well as customize resource organization rules, such as texture channel remapping. It also maintains multi-platform and multi-quality level resource allocation, as well as our modular material and feature reuse.
In Editor mode, virtual material templates store basic material information for different platforms and different quality levels. It is then instanced as a Virtual Material Instance and assigned to a Material Slot in a shared model assembly.
At runtime, in the packaged version, the Virtual Material Instance will fall back to a normal Material Instance, and the data from the non-target platform will be stripped away.
Here's a diagram that illustrates how Virtual Materials are used. The data provider collects texture data, exposes texture slots used by virtual material templates, manages resources at different quality levels, and adapts the data. The Effect Layer consists of two functions: the Material Effect Logic and the Blending Method with the Upper Layer. The hybrid controller is a separate logic layer that handles complex hybrid methods. With the Virtual Material System, we basically decouple the Material from the renderer, making the shader very flexible across different platforms and even within the same model component.
In addition to materials and shaders, the second typical problem is collisions. Let's take a look at this short video. We start to lose mesh detail when the LOD index is biased towards the moving platform, in which case the railing is removed from LOD3 onwards, but we can only have one collision data in the mesh component, which will result in a collision mismatch, which is completely unacceptable for an FPS game. So, how do we solve this problem?
The solution is actually quite straightforward. We've expanded the collision settings and added data for each platform. By doing so, we can accommodate the game's unique performance limitations and quality requirements while maintaining consistent interactions.
After correcting the collision data, both PC and mobile platforms now have the correct collision that matches their rendered models. In fact, the property methods for each platform are also extended to other asset types and objects. For example, the Sky Light Cube has different texture and compression settings for each platform, and another example is a lightmap for the same Static Mesh component, as we bake separate lightmaps for different platforms in the same level.
For more details on the production of assets, Wang Lichuan, the head of technical art in our studio, introduced it from the perspective of technical art in a previous presentation. If you are interested, you can find more information there. (View details>>)
Okay, once we have all the assets in place, the question becomes how to efficiently build an immersive scene that meets quality and performance standards across platforms. We need to focus on the scene richness of different platforms and how to manage the different loading distances due to mobile hardware limitations. Not only that, but in relatively complex resource management and editing workflows, we need to identify errors before packaging into a build. In addition, some specific optimizations vary across different platforms.
The general idea is to manage level content in two ways. First, we have a basic shared layer that contains only shared assets. On top of that, we'll split platform-specific content appropriately and let them into their own exclusive layers. Another aspect is asset types, in the image above right, we have categorized asset types into different tiers for better management.
We kept the amount of work involved in the parts that required level artistry, which meant that most of the content was actually a shared layer, and made sure that the basic level layout was consistent with the level designer's intent across platforms. However, platform-specific content will still be fine-tuned or tweaked, such as adding richness to PC or optimizing for mobile devices. The video shows a hand-to-hand comparison of the screenshots of the level.
For content generated by PCG or other automated tools, since there is no human involvement, we completely separated the sets of levels and output them to each platform.
In a large project, with the cross-platform production process going on, it actually adds complexity to management. We inevitably encounter some human error and collaboration issues, such as placing the wrong assets, wrong platform references, texture waste, etc. To effectively eliminate these errors, we provide a suite of asset-related troubleshooting tools to help with error checking, performance recommendations, and statistics. The video shows how we can find unnecessary texture waste by analyzing texture similarity.
After all the editing and configuration work is done, we can't run the level directly. In order to meet our regulations and performance requirements, we have converted all levels in the edited state to runtime levels. To achieve this, we've layered the runtime levels in more detail, as shown in the image, we have different level divisions and streaming configurations between PC and mobile.
Another typical problem in shooters is aiming, in the case of PC we switch levels visibility. But for mobile, we took a balanced approach, which was to only stream the levels within the frustum when aiming.
Ultimately, our goal is to have the same assets, the same levels, but with different runtime levels to achieve consistent performance on each platform!
Let's take a look at what the build runtime level handles.
First, you need to clean up unused Actors, which are usually debugging Actors. Second, in order to improve the performance of the dedicated server, the physical data is extracted and merged. Three, all vegetation is converted to each type of streaming asset. As a result, we can manage the memory cost and draw calls of vegetation in detail. Fourth, the terrain is also converted to CDLOD. Fifth, according to the position and boundary size, the grid is automatically divided into different level grids. Finally, there are some other gameplay-related data that needs to be processed.
We've integrated the entire build process into an automated workflow. It analyzes raw levels, reassigns assets, generates corresponding asset types, extracts key data, performs specific optimizations, and automatically collects basic performance statistics. Model batching is the only part that is specifically tailored for mobile platforms, while the other processes are consistent across platforms, with only different configurations.
This is a brief introduction to the core strategy for batch processing on mobile platforms. First, we need to determine the feasibility of batching a set of models, including checking materials with parameters, texture formats, lightmaps, shadow casting, and IBL. Batching also needs to verify that it is reasonable, such as that the number of vertices exceeds the limit, that the texture resolution is sufficient for the size of the budget texture set, and that the distance is reasonable. Our final batch results show a good sense of local space and significantly reduce draw calls.
Now let's move on to the runtime support section, where we'll share some cross-platform rendering strategies and features, as well as how to unify game development, and some things about performance tracking tools.
Let's take a look at the rendering part first.
As mentioned earlier, in order to take advantage of the characteristics of the different platforms, we decided to run different render pipelines. However, we still strive to be consistent on some key features. This chart shows specific examples and breakdowns of rendering scenarios, highlighting similarities and differences. While ensuring overall consistency, we also differentiate the underlying technology to meet a wide range of quality and performance requirements.
When it comes to lighting, we ensure consistency in lighting components to achieve similar lighting perception across platforms. To achieve this, we used some underlying techniques to implement different approaches for the same lighting component. For example, sky occlusion is provided uniformly by our cross-platform global illumination solution. However, for more detailed ambient occlusion (AO), mobile devices rely more on texture baking, while PCs take advantage of GTAO and Distance Field AO. By maintaining consistency across lighting components, our lighting artists don't need to worry about cross-platform differences.
For the Global Illumination (GI) section, our algorithm is volume-based, but the voxels in the volume do not store any lighting information. Instead, they store sparsely distributed probes and off-line fitted weights.
To reduce overall complexity, common denominators between PCs and mobile devices include:
- Same offline probe distribution algorithm (but with different densities).
- The same data fitting algorithm.
- Same block-based streaming logic.
The difference lies mainly in the different optimization directions.
Mobile devices have a lower data density, so offline data is read by the CPU and assembled into 3D textures, and the GPU only performs simple sampling to minimize GPU load. As a result, optimizations on mobile devices focus more on CPU reads, enhancing cache-friendliness, and data compression to reduce memory costs and disk storage.
The PC has a higher data density, so the data is loaded directly into the graphics memory, where the GPU reads the fitted weights in real time and dynamically decompresses the lighting data there. Optimizations on PC are more dependent on the data storage format, and we use a tree structure and sparse storage to minimize runtime graphics memory. In addition, more asynchronous mechanisms have been added to prevent stuttering when sending data from the CPU to the GPU.
Once we've achieved lighting consistency, we can encapsulate some uniform parametric interfaces at a higher level to implement Time Lapse (TOD). Here, we use a sequence manager to drive all the TOD parameters. Most of the common parameters are implemented in the base sequencer, while minor platform differences are driven by subsequences.
Another important part is the terrain. When we implement some advanced technologies on our PC for better quality and performance, we always take into account the adaptability of mobile devices and finally succeed in bringing some of these technologies to mobile platforms.
We have made a number of optimizations for mobile platforms to achieve a quality similar to that of a PC.
- The Adaptive Dynamic Texture Array brings 32 materials to the entire world, but only runs at the memory cost of 8 materials.
- We implemented cliff rendering using random three-plane blending, using VT technology for the first time and bringing it to mobile devices for the first time, dramatically improving the quality of the terrain.
- To achieve per-meter humidity and hue on mobile devices, we've added clip map texture streaming, which reduces the memory cost of large textures from 133MB to less than 2MB, almost to 1%, and helps us increase terrain diversity on mobile devices.
- For the distant view of the terrain, we also implemented the terrain with 380K vertex detail at a performance cost of only 80K vertices without losing any quality.
For more details, check out the presentation by Yunhan Xu at Unreal Fest 2023.
Ok, the gameplay part. We have also built a number of cross-platform solutions for gameplay development, but today we are mainly talking about the combat and user interface parts.
In core combat development, we still have an integrated approach, available for both PC and mobile platforms. The overall combat characteristics are divided into three levels. The core trait layer is platform-agnostic and includes gameplay frameworks, network models, movement synchronization, weapon shooting, and other combat details. The flexible feature tier provides extensible, cross-platform configurations, and we can dynamically determine which features to enable based on performance budget and quality level. The platform feature layer handles platform-specific adaptations, such as input controller or HUD interactions.
In our project, the basic skeleton is shared between the PC and the mobile device.
As you can see, the complexity of the skeletal physics simulation is also different, and we have added more sub-bones and dynamic details on the PC. Animations are created based on the same parent skeleton of the character, ensuring that most animation clips are consistent and reusable across platforms. However, animations on PC have higher animation frame rates and better compression settings.
In terms of animation features, most of them were developed for both PC and mobile platforms. Considering the performance limitations, we also offer some alternatives on mobile platforms to achieve the same effect by sacrificing some quality. For example, in the case of replay animation, we provide full animation clips on PC, but using IK on mobile devices to simulate the same effect, it is actually difficult to distinguish the difference.
To dynamically determine which features to enable based on the performance budget and achieve the best overall performance on each device, we used a planning algorithm. The implementation approach involves defining the "value" of each feature using a planning algorithm that takes into account the static quality of the feature and the dynamic runtime conditions. Then, the best set of features is selected within the existing constraints, including performance constraints and feature-specific constraints. The overall planning algorithm is a bit more complex, but the basic idea is similar to the backpack algorithm. With a finite number of iterations, we aim to find the best combination of features that meet the constraints.
Let's take a look at the final planning results: in the video on the left, we can see the movement quality of multiple players with different budget configurations. As the budget increases, the likelihood of unrealistic movements decreases for the player. Characters at a distance don't sacrifice features such as floor detection in their movement behavior. Naturally, PCs can enable more and higher-quality features, while mobile devices can only enable limited features for a handful of characters.
The video on the right shows the final result in the game. With 64 players in the game, there is considerable performance pressure. It effectively balances performance budget and game presentation for smooth results.
For the user interface part, there is also a lot of cross-platform work. As a result, most of our UI pages are shared through the Adaptive UI framework, while some important systems are developed separately.
To improve the efficiency of sharing UI pages, we extract platform-specific parameters to the framework layer. First, the DPI curve is separated out to better fit the screen size. PCs need to handle a wider range of display scales, so we've added scale constraints to prevent anomalous display issues. In the visual design phase, we use one platform as a benchmark and then make adjustments to other platforms under certain rules. Different platforms have different approaches to UI navigation, so we extract and separate them into the navigation layer. We've also provided a variety of quick tweaking options, such as padding, scaling, resizing, and node cropping, to further fine-tune the final UI.
In order to better share UI resources, which is our original resource standard, we chose 2K as the base resolution. During the cooking process, the original 2K resources are compressed to 1080P for mobile devices. But for PC, the resource is dynamically scaled down to 66.67% at runtime by DPI to achieve the same effect as 4K resolution. A small tip is to enable mipmapping on your PC to avoid pixel mismatches to avoid flickering. Please refer to the video on the right for comparison.
Okay, the last part, performance.
To better monitor and analyze performance-related issues, we tried various tools, but none of them seemed to meet our requirements. What we need is a cross-platform tool that doesn't require any pre-instrumentation and supports long and extensive data collection without impacting runtime performance. That's why we've developed our own cross-platform performance monitoring and analysis tool called Metaperf.
The entire cycle of performance capture consists of 3 stages. During the recording phase, we implemented high-performance samplers for all platforms: by combining the frame time and custom data for each frame, we could get the various game states for each frame, and then, by processing the timestamps of each frame and merging all the call chains in the sample, we could get a full call tree for one frame. In the final analysis phase: we implemented an automated, rule-based analysis monitoring service. This includes various automated analysis methods, such as the Pearson correlation coefficient. By comparing the correlation between the two curves, we can determine the correlation of each function in the performance metric.
The Metaperf tool has been widely used in our projects and has collected more than 10,000 game records. Helped us identify and resolve more than 1000 performance-related issues.
Well, to sum it up. There is no right solution, only the right solution, and what we share is just one possible way to achieve high-quality and high-performance cross-platform development. You should find a solution based on the product and the team. Tools are very important for development efficiency. Hopefully, this exchange will give you some inspiration.
Special thanks to the entire Delta Operations development team, almost everyone provided suggestions and feedback that allowed us to continuously iterate and optimize the process to make it even better.