On April 18, the 2022 N.GAME NetEase Game Developer Summit (hereinafter referred to as the Summit) was held online.
"Real world" may not be a new word in China's game industry, but "the real world in high-speed motion" is a blank field that few people in China have stepped into. Maybe in the immersive racing game, you see a plain road reflection, that is, the technical barrier built by foreign main engine manufacturers for more than ten years.
At the technology-driven sharing meeting on the second day, Zhou Qian, senior technical manager of NetEase Games, shared with everyone how to tackle the tough problems and achieve real lighting in high-speed environments.
The following is the full text of the research and development sharing of the mobile game, in order to improve the reading experience, the content has been deleted.
Hello everyone, I am Zhou Qian, a technical expert from NetEase Games Big Talk Division, today I invite you to join me in the world of extreme speed light and shadow to share one of our lighting solutions in high-quality racing games.
First of all, let's introduce our game product, called Racing Master, Chinese "Peak Speed". The goal of making such a game is to create a high-quality, high-realistic, high-quality racing game.
As you can see from the video above, there are very rich scenes in the game as well as bright colors. Motorsport systems (including movement, drift, acceleration, etc.) achieve the ultimate realistic effect through physical realism and physical calculation techniques.
Players can make very rich customizations to the vehicle, including modifications, painting, and so on. The lobby interface features a vehicle display system for enjoying very elaborately modeled racing cars, including extremely high-resolution lights, paint, and realistic global illumination in night scenes.
Through the above introduction, I believe that everyone has a general idea of the artistic quality of our game. In order to achieve such artistic quality, one of the biggest difficulties we faced was "real-time global illumination".
Real-time global illumination is divided into two parts, namely: direct lighting and indirect lighting. To achieve this on mobile, we face a lot of challenges, including: broadband, performance, compatibility, and so on. In addition, there are many constraints, such as the need to use Forward pipelines to reduce broadband in addition to reducing DP and Pass.
Back in the game, the two problems that need to be solved by direct lighting and indirect lighting extending in the game are "real-time multi-lighting" and "real-time environment capture". So our topic today begins with these two questions.
01
Real-time multi-light sources
1. Two previous solutions for multi-light sources
"Multi-light sources" are very common in games, such as in night scenes, there will be many street lights, headlights, and vehicle tempering, etc., which will illuminate the surrounding objects.
Next, let's look at an effect demonstration.
This is how our car behaves after passing through a row of street lights, in addition to the street lights, the car also has a front light. Numerous dynamic light sources are difficult to achieve under the Forward tube line.
As a result, many people have improved the Forward pipeline to support multiple lights. Here are two schemes, called Tile Shading (also known as Forward+) and Clustered Shading.
The first is Tile Shading, whose idea is to divide the screen space into multiple grids, and then each grid cooperates with the depth Buffer to perform the light intersection calculation. This way we can know which lights will affect each cell, reducing the number of lights that need to be counted per pixel.
Although it can solve the problem of multiple lights, it requires a buffer with pre-drawn depth, that is, a PreZ Pass. But if the game scene is very rich, PreZ Pass will bring a lot of Draw Calls, which is unacceptable to us.
In addition, Tile Shading's intersection calculation must be done inside the Composite Shader, but for many mobile phones, the support for Composite Shader is not friendly.
Then there is Clustered Shading, whose idea is to further divide the depth space on the basis of Forward+, dividing the cone into multiple frustum blocks, and then performing a light intersection calculation on each block.
But similarly, Clustered Shading also needs a PreZ Pass, and its intersection calculations also need to be placed in the Compute Shader.
2. Grid Shading's thinking
In order to solve these problems on the mobile side, we propose a new multi-light scheme called "Grid Shading".
Grid Shading's idea is to mesh the world space along the XY axis with aligned axes; then use a light index map, where each pixel represents a grid and contains the number of lights received within the grid.
Each pixel with RGBA four channels, can record four lights. If the number of lights exceeds four, the contribution of these lights needs to be sorted.
Contribution, which is the light intensity of a light. After sorting, only the four lights with the greatest contribution are retained in this pixel. The lighting information is uploaded via UniformBuffer.
In addition, we can expand multiple layers in the Z-axis direction, so that we can achieve a wider coverage.
As you can see from grid shading's application in the scene, a 64x64 mesh is generated in the XY direction, and we expand two layers up and down in the horizontal height of the camera. The resulting total mesh size is 64x64x4.
However, because the mesh is flattened, it can only cover a more horizontal range, and when using this scheme, the player's field of view must also be limited, and the field of view must be horizontal.
However, in the racing game, the track is basically tiled, and the player's field of vision is basically in the horizontal direction, so the scene objects that need to be illuminated, including racing, road surface, etc., are just within the coverage. As a result, the resulting mesh basically meets our needs.
3. The process of Grid Shading
Grid Shading's process is as follows.
First of all, it is necessary to traverse the lights, calculate the range of the lattice according to the enclosing box of the light, and then perform a cross-crossing calculation on each lattice at the CPU layer.
The light contribution is then calculated and sorted, the results are populated into the light index map, and the light information is uploaded.
Finally, during the drawing phase, each pixel calculates the grid in which it is located based on its position in the GPU world coordinates. And in the light index chart, find out the light number it needs to calculate, and calculate the light of the light.
One of the most difficult points here is the intersection calculation of the lattice. The shape of each lattice is a cube, which can be approximated as a encircling ball. If the opponent is a point light source, the light range also happens to be a sphere, and the intersection calculation between the sphere and the sphere is very simple.
But if the other party is a spotlight, and the range of the spotlight is a cone, how to do the intersection calculation between the cone and the sphere? Usually, we generally extract the encircling ball of the cone, and then calculate the intersection of the encircling ball with the encircling ball of the lattice.
Although this comparison is simple, the results are very imprecise, because the size of the cone surrounding the ball is very different from the size of the cone itself. For this, a more precise method of calculating cones and encircling balls is needed. Here's how.
First, if the apex of the cone is placed at the center of a large ball, the extent of the cone can be seen as within a spherical corner of the large ball. Then, according to the position relationship between the large ball and the lattice surrounding the ball, the intersection situation can be divided into four types.
The first case is very simple, i.e. the lattice encircling ball contains the apex of the cone, according to which it is clear to see that the enveloping ball intersects the cone.
The second case is that the encircling ball and the large cone ball are separated from each other, then according to this case, it can be seen that the lattice encircling ball and the cone cannot intersect.
The remaining two scenarios are more complicated. In the third case, the lattice encircling ball has no more than half the volume inside the large ball. At this time, it is necessary to convert the intersecting part into a spherical angle, and then use the spherical angle to intersect with the spherical angle of the cone to judge.
In the fourth case, the lattice surrounds the ball with more than half of its size inside the large ball. Then we can use the center of the ball and the tangent direction formed by the grid surrounding the sphere of the ball to form a spherical angle. The intersection of this spherical angle with the spherical angle of the cone is then used to determine.
Through the discussion of the above four situations, it is possible to accurately judge the intersection of the spotlight and the grid.
4.Application and comparison of Grid Shading
Next, there's the application of Grid Shading in real-world scenarios.
The picture on the right is a street lamp placed in the race scene and the front light of the vehicle itself; the picture on the left is a light index map of the scene.
What do you think of this index map? Through observation, it was found that the light index map is divided into 4 layers from top to bottom, corresponding to the grid is 4 layers of grids of different heights. (the "4" layers of the aforementioned 64x64x4 mesh) In addition, each pixel is a grid.
If there is a color in the pixel, it means that the grid is affected by the light. As you can see from the index graph, the blue area that appears is the extent covered by the spotlight. The range is gradually increased from top to bottom, so it corresponds to the shape of a large cone on the spotlight. Prove that the results of the index graph correspond exactly to the scene.
Therefore, when we actually draw, we can sample this index map to determine which lights the pixels correspond to.
Finally, let's compare the difference between Grid Shading and the other two schemes.
First of all, in the PreZ Pass phase, this stage is completely unnecessary for Grid Shading, and the other two schemes cannot avoid this stage. This saves us a lot of Draw Calls and Passes.
In terms of intersection calculation, Grid Shading can be completely placed at the CPU level, and the calculation process is very simple, and the result of cone intersection is also very accurate. However, the other two schemes not only cannot be placed in the CPU layer for computation, but also the calculation process is more complicated. In the face of spotlight, the calculation results are also not precise enough.
From the perspective of dividing granularity, Grid Shading is a very fine scheme, clustered Shading is relatively fine because of the further division of the Z axis. But Tile Shading it's a screen division, so the granularity of the division is very coarse.
So you can see that in the above directions, Grid Shading is very advantageous, and its performance on the mobile side is very acceptable. Even on the terminal, the overhead during the calculation process does not exceed 1ms.
The problem with Grid Shading is that it requires a tiled horizontal field of view. But for racing games, the player's field of view is also exactly flat and horizontal. So, this limitation doesn't have much of an impact on us.
Therefore, Grid Shading can be said to be a multi-light scheme that is very suitable for racing games.
02
Real-time environment capture
1. Double parabolic mapping scheme
Let's start with an effect demonstration.
There are a lot of highlighted objects in the demo scene, such as fireworks. Then you can see that at night or in a scene with relatively low lights, the impact of such a bright object on the scene is even greater than that of direct lighting.
The reaction in the demonstration is that the car and the road surface can be illuminated by objects such as fireworks, and the car is also affected by indirect light from the road surface and the bounce of surrounding objects. So in order to achieve the lighting effect of this environment, the most important step is real-time environment capture.
On the mobile side, a scheme called "double parabolic mapping" is usually used.
Its idea is to map the 360° environment through two parabolic planes to the upper and lower directions, and to express the information of the entire scene through two diagrams.
The one on the right is the two double parabolic maps we captured in the game. Both the car and the track must be sampled for environmental information.
Why use this double parabolic mapping scheme? Below is our comparison of several different environment capture schemes, and I believe that the results can be obtained by comparing.
Generally speaking, there are three schemes for full scene capture, namely "spherical mapping", "cube mapping" and "double parabolic mapping".
In terms of the number of render targets, the targets of the three renderings are 1, 6 and 2 respectively. The more renders you have, the more Passes and Draw Calls you need.
At the aberration level, the spherical map will have a very large distortion, and the distortion at the edge is larger; the cube map is completely free of distortion, and the double parabolic map, although also distorted, is relatively small and acceptable.
In terms of mapping quality, the map quality of the spherical map at the edge is very poor and has singularities; the cube map has the most information, so the map quality is the highest; while the map quality of the double parabolic map is average, but it is still acceptable on the mobile end.
From the perspective of computational complexity, we need to map the vertices. Therefore, the map transformation of the spherical map is the most complex because it requires the use of square operations, while the cube map is relatively simple because it is only a simple perspective map; similarly, the map transformation of the double parabolic map is also relatively simple.
Well, taken together, dual parabolic mapping is a very suitable environment capture scheme for mobile ends.
2. Capture direction selection
In the choice of capture direction, we can choose to snap forward and backward, left and right, or up and down. If you choose to capture front and back or left and right, because the scene is tiled, the track will have a triangular crop in this dividing direction. Finally, there will be cracks in the synthesis of the environment map, which is unacceptable to us.
What if you use the up and down direction to capture? Although there is also a triangular crop, the cropped position is so far away that it is difficult for the player to notice. The resulting ambient lighting is also very complete.
In addition to the selection of the snap direction, you also need to select the location of the snap point.
First of all, let's look at a picture, the one on the left is the reflection effect of the slippery road. Can you see that there is anything wrong with this road reflection? It's easy to notice that the pavement reflection doesn't correspond to the location of the actual scene. Looking at this picture again, the picture on the right looks much better. Why is there such a difference between the two pictures?
We'll show you the camera position, and in the image on the left, we can see that the front-facing camera is the game view camera, and the up-facing and down-facing camera is the environment capture camera. The capture camera and the game view camera are not in the same position, which leads to misalignment in the picture.
We can see the schematic diagram in the upper right corner. When our capture points are not vertically consistent with the camera, they differ in the landscape for a reflection direction. If the capture point is in the same vertical direction as the camera, the captured information will only be different in portrait orientation, but aligned in landscape orientation.
You can also see the picture on the left, although there are differences in the vertical direction, but it is difficult for players to notice. However, if there is a difference in horizontal orientation, it will be very easy for players to observe this phenomenon.
However, this brings up a new problem, if you want to ensure that the vehicle reflects correctly, you must place the position of the capture camera near the car. But in the game, there is a certain difference between the position of the game perspective camera and the vehicle itself.
Therefore, we cannot guarantee that the ground reflection and the vehicle reflection are in a state of accuracy at the same time, and we can only choose one of the two. However, it is difficult for the player to notice the inaccuracies of the vehicle's reflections, but it is easier to notice the inaccuracies of the ground reflections.
That's why we put the capture camera in the same position as the game view camera to ensure the accuracy of ground reflections.
After capturing the scene, you need to sample both captures in IBL to generate an environment map. However, IBL has a requirement that in the case of relatively high roughness, it needs to filter the environmental map.
Originally, we directly generated a Mipmap on the capture results of the double parabolic surface to approximate the environment map after this filter. But this poses a problem, as shown in the following figure.
In the picture is a rough sphere with a clear dividing line between the upper and lower hemispheres.
This is because when capturing, the upward hemisphere is exposed to a higher intensity of light and a weaker downward hemisphere. When filtering, Mipmap can only mipmap one map, and there is no way to mix the entire environment. Therefore, the brightness of the upper and lower hemispheres is not uniform.
How to solve this problem? We come back to the spherical mapping scheme. Because the spherical map is a whole sticker, when you generate a Mipmap on it, you can filter the whole field. Thus, the results captured by the double parabolic surface can be synthesized into a post map by spherical mapping.
In addition, in order to reduce the binding and sampling of textures, you can also place the two snaps of the double parabola on different mips of the same texture, which can reduce some of the overhead.
3. Double parabolic capture process
Next, let's look at the process of double parabolic capture.
First, in order to reduce Pass and Draw Call, the capture of the upper and lower hemispheres will be divided into two stages. One frame captures the upper hemisphere, one frame captures the lower hemisphere, and the two frames alternate. This allows only one hemisphere to be sampled per frame.
After sampling the environment, it is then composited into a sticker in the form of a spherical map, and then the Mipmap is generated. The final drawing stage, apply it to the drawing of the scene pixels.
Well, in the outdoor scene, such a performance is very good. However, after we drove the car into the tunnel, a new problem arose.
As shown in the picture on the right, this is a white 918, but after entering the tunnel, it becomes a black car. Why? The reason is that the scene we captured in the tunnel was very dark, and it lacked static light information.
At this time, it is necessary to obtain static light information. So where exactly is the static light information stored? It exists in our Light Probe, so we need to get more information from the Light Probe to render.
In general, indirect light is divided into two parts: Diffuse and Specular. Diffuse is represented in the game by a light probe that finds some coefficients. It is a pre-baked post-baked message containing static light. Specular, on the other hand, captures content in real time.
When the roughness becomes larger, it is necessary to move Specular closer to the irradiance of Diffuse. So how? UE4 has actually done a similar process, but that's just a static solution. We've improved this process to apply to motion capture.
The calculation is divided into two phases. First, in the Verrtex phase, it is necessary to take its most advanced Mips for the content of this spherical mapping, and multiply the content points of this pixel (0.3, 0.59, 0.11). This is the brightness weight value of each channel of the RGB.
Through the above, you can get to the average brightness of the environment. In the Pixel phase, diffuse irradiance is divided by the average brightness and coarseness is interpolated between 1.0 and the value just calculated. This interpolation is then multiplied by Specular, which increases the brightness of Specular to achieve the same effect as the environment.
In this way, the original color of the car in the tunnel can also come back, and its effect is more consistent with the surrounding environment. This is one manifestation of global illumination.
After talking about real-time environment capture and real-time multi-lighting, combine the two to see how it actually works in the game.
Finally, let's make a technical outlook.
Our solution can be used for scenes where real-time ambient lighting changes are more drastic, and it has good adaptability to high-frequency changes in light sources. In addition to racing games, it can also be used for different game types, such as MMORPG, FPS and so on.
In the future, we will also extend this solution to the big world and the day and night changes and weather systems. There are also plans for players to customize the track, which means that it is necessary to capture the lighting of the ambient indirect light Diffuse in real time, which is also a direction we are currently working on.
OK, that's all I'm talking about, thank you!