April 1 marks the 19th anniversary of Zhang Guorong's death. On this day every year, countless fans and fans around the world will remember "brother" Zhang Guorong in their own way. This year, supported by the intelligent image restoration technology of Tencent Cloud Multimedia Lab, Tencent Music Entertainment Group and PolyGram, a subsidiary of Universal Music, and jointly presented the "Zhang Guorong 2000 "Hot Love" Concert Super Clear Restoration Edition" with Tencent Cloud, WeChat Video Account, Tencent Video and other teams.
4K version of the restoration concert trailer.
The "Hot Love" concert is a classic song of Zhang Guorong's life, with him personally serving as the artistic director, and the well-known fashion designer Jean-Paul Gautier is responsible for the costume design. This year, thanks to the blessing of digital restoration technology, this "era symbol" that has been sealed for more than 21 years was unlocked for the first time. The "Ultra-clear Immersion Repair Engine" used in the "Hot Love" concert restoration is based on tencent cloud multimedia laboratory's years of multimedia communication and processing research and development experience, and the high-definition repair uses the "engine's" intelligent analysis, picture quality repair, picture quality enhancement, and intelligent coding four modules to solve the scratches and noise problems caused by the large time span. But the overall environment of the concert is dark, the details are not rich enough, the scene switching is fast, the lighting environment is complex, the color is rich, which brings challenges to the algorithm, this time in the face of the lack of clarity, can not be accurately modeled, need to try different algorithms to achieve the best repair effect and many other technical difficulties, in an interview with the Beijing News, Tencent Cloud Multimedia Lab Director Li Songnan, Tencent Cloud Multimedia Lab expert researcher Xia Zhen introduced.
Repair the old as the old: everyone has a standard in mind
Beijing News: When did the idea of launching the super-Clean restoration version of Zhang Guorong's 2000 "Hot Love" concert come about?
Xia Zhen: We also had a relatively fortuitous opportunity to contact Universal Music's Poly Gold Records, learned that they just had the master tape of Zhang Guorong's "Hot Love" concert, and felt that it was quite compatible with our technology, so we reached the intention of cooperation. In early March, we officially started the restoration.
Beijing News: In Zhang Guorong's many concerts, why choose "Hot Love"?
Li Songnan: Because "Hot Love" was the last concert of "Brother" before he died, at that time, PolyGram Records also wanted to see if it could repair some of Zhang Guorong's previous concerts or clips, because our laboratory had been doing the repair of intelligent images before, so the two sides quickly reached a willingness to cooperate.
It is also a special honor to launch a restored version on the special day of April 1. On this day every year, everyone will commemorate him with film clips or other ways, and we will feel particularly meaningful and valuable to do this in the way of restoring concerts this time.
From the comparison chart, you can see the difference in image quality before and after the restoration.
Beijing News: Will the live broadcast of the restored version of the concert involve copyright and other related issues?
Li Songnan: This cooperation with PolyGram is based on a kind of remembrance of Mr. Zhang Guorong, so there is no commercial thing in it. In addition, we broadcast this concert live on many platforms, although these live broadcast channels used to open a tip function for star artists, but Zhang Guorong's live broadcast was closed.
Beijing News: How long did the repair process take, and which part of the repair took a long time?
Xia Zhen: If you look at the algorithm, the entire intelligent repair is not very long in the algorithm, but because we still want to make the concert "repair the old as old", retain the artistic effect of the whole concert, and try different algorithms and effects in the middle, we spent a lot of time to make a balance.
Beijing News: During the whole repair process, what do you think is more challenging?
Xia Zhen: With the help of the laboratory's repair engine and other technical accumulation, the clarity of the concert video can be restored to a great extent, but for the color and picture switching in the concert video, plus the problem that the clarity of the video material is not high, how to achieve "repair the old as old" and bring the audience an immersive experience integrated into it, which is a big challenge for the team.
The restored concert can present the effect from mastering quality (top) to post-repair (bottom), going through deinterlacing, noise removal and compression distortion, enhancement and supersizing.
Beijing News: In the field of concert restoration, what kind of standards should the "old" that repairs the old as the old meet?
Li Songnan: "Repair the old as old" is a goal that we want to achieve when we do the restoration of old classic images in the past, this understanding should be different from person to person, no one in the industry has proposed a standard, and the concert 'repair as old as old' is just a purpose we want to express.
First of all, our team will have a general judgment, first to understand and plan this concert in terms of art, how to follow the atmosphere that the whole stage wants to express at that time, we will try to adjust the algorithm to that aspect. After the restoration, we will also compare with the old version to see if the expression of the whole art and the overall senses can reach the feeling of "old" at that time, because indeed everyone will have their own views on "old".
Technical highlights: 6 times higher resolution and noise reduction in the scene
Beijing News: This time there is less "data" than movies and TV series, what does this data specifically refer to, and why is it said that it is less?
Xia Zhen: The "data" we are talking about, so before doing the restoration of TV series such as "Tianxia Renjia" and "Kangxi Dynasty", relatively speaking, the scenes of film and television dramas are relatively fixed, the movement of the lens and the whole picture are relatively stable, and the artistic expression form and overall color style from beginning to end are similar. But the concert is not the same, the lights in the concert and the picture lens will have a special number of switching, in the previous restoration rarely encountered such a complex scene, we say that the data is relatively small, usually refers to the scene we have encountered less before.
The changing lights at the concert site brought challenges to the restoration.
Beijing News: How much will be improved in the picture quality of this restoration, such as what is the resolution before and after the restoration, and have you seen some details that you have not seen before after the restoration?
Li Songnan: From the subjective feeling of the overall picture, after the repair, it will be clearer and more realistic than the previous picture. For example, we can see that the clarity of Zhang Guorong's face is very high, including the contours, pores, and beard of his face will be much clearer than the old version. Before the copyright company to give the master resolution is very low, about only 720×480 resolution, now after doing the repair close to 4K resolution, the resolution is about 6 times higher, the fluency of the entire picture has also been improved, in the frame rate we have also done some optimization.
Xia Zhen: When we do human face repair, there is a big face, just the lighting design of the whole concert is also red, when the light hits Zhang Guorong's face, the color of his entire face is different from the color of our normal face, resulting in a face model and the results of the repair will not have a very good effect, then we will specifically optimize these problems.
Beijing News: From the perspective of the audience, high-definition repair pays more attention to face repair, how to achieve the best effect of this repair?
Li Songnan: Now the industry is basically difficult to find video-based face repair, most of the face repair is only based on pictures or single photos, if you want to use the video will encounter more complex problems. For example, the angle of the face is not the same, if it is a side face or a very small face and a large face, the face of various angles and postures needs to be able to solve these problems in the algorithm processing.
There is a difficulty in the repair of this concert, that is, there are many lights in the concert, the color of the lights will be changeable, such as the entire color has red, blue, green, etc., after hitting the face, our current algorithm will have some problems that cannot be handled, and it also provides us with a lot of valuable experience in the work or optimization behind.
Beijing News: What specific scenarios have been built by the repair?
Li Songnan: For example, for the face, we will repair the face for some scenes, and for large scenes, we will use GAN (generative confrontation network) technology, because there are many large scenes in the previous mastering, and the lens will become very blurry when it is pulled to the distant view, and after passing the GAN technology, the clarity and texture of the scene will be particularly clear. In addition, we will also use some different algorithms in different scenes, for example, the scene of the whole concert has a lot of noise, and we also have a noise reduction process for this scene.
Beijing News: Compared with traditional manual restoration, what are the advantages of this technology?
Li Songnan: The whole concert has two hours, compared to the previous traditional manual repair may take one or two months, if from the perspective of algorithm, our algorithm processing is relatively fast. First of all, the intelligent analysis module using a variety of intelligent evaluation algorithms will conduct all-round and multi-dimensional analysis of video content, mainly to evaluate the motion and texture complexity of the video, the degree of compression distortion, etc. After obtaining the analysis results of different dimensions, the quality repair module can use a variety of intelligent algorithms to fix various major factors that damage the quality of the picture in a targeted manner, including but not limited to scratches, vertical lines, snowflake points, noise, compression distortion, jitter and other issues. The algorithms in this are basically Tencent's self-developed, such as scratch algorithms, which have not been encountered in public services.
Xia Zhen: The area and number of scratches and snowflake spots are inversely proportional to the viewing experience. The larger the area, the worse the experience, and vice versa. At the same time, there are many difficulties in scratch repair. On the one hand, scratch repair is irreversible and costly; on the other hand, there is a lack of a huge scratch public database, which is difficult to optimize the algorithm and very expensive to build. To this end, the laboratory has specially designed a targeted multi-directional data augmentation scheme. Schemes include scratch cropping and merging, random size, random blur degree, local and global random brightness adjustment, random transparency adjustment, etc.
Then, the image quality enhancement module will select the best video enhancement algorithm to improve the multi-dimensional image quality of the picture, improve the resolution and frame rate of the video in space and time, and improve the detail clarity and color richness in the airspace. There is a need to better harness and balance the two intelligent algorithms of CNN (convolutional neural network) and GAN (generative adversarial network).
Fix video comparisons before and after similarities and differences.
Beijing News: In order to create the effect of "face-to-face" live concerts, what other attempts and efforts have been made technically?
Xia Zhen: The lab team also made a special algorithm tuning for the eye details of Zhang Guorong's concert. A still picture of Zhang Guorong in the center of the stage is taken from the restored video, and when enlarged, it seems that the outline of the image refracted on the stage can be seen through his eyes.
Beijing News: Will this technology be closer to the public in the future?
Xia Zhen: "Ultra-clear Immersion Repair Engine" is mainly divided into four parts: intelligent analysis, image quality restoration, image quality enhancement, and intelligent coding, of which image quality repair is to repair this kind of old video, mainly to remove some of them missing frames. Image quality enhancement is to further improve its quality on the basis of the existing video, including improving the resolution, frame rate, dynamic range, and even to develop in the direction of immersion, to do some dimensional conversion work. I don't think the application prospects of repairing this side are very small now, because many of the businesses we have been exposed to before, whether it is media, pan-entertainment or the film and television industry, have indeed accumulated a lot of old materials, especially TV stations, or there is still a big market.
Beijing News: In addition to these professional skills, what aspects of the audience need to pay attention to sensory experience?
Li Songnan: The senses are indeed different from person to person, because in the whole process of the concert, we will not have a comparison between before and after the repair, if you want the audience to feel a good effect, feel that this is a great concert, then some details of the whole concert such as the state of "brother", can make everyone feel that it has been closer to the current concert, then it is to meet expectations.
Because the quality of the mastering tape was relatively poor, from our point of view, we can clearly feel that after the restoration, including some details of the "brother's" face, his eyes, eyes, and the hair strands of the fake long hair he wore during the whole concert, the clarity will be much better than before. We hope that in the process of watching this concert, we can clearly look back at the whole process of the concert of "brother" through the current technology.
Liu Zhen, a senior reporter of the Beijing News
Senior Editor Tian Kaini Proofreader Liu Jun