Soon, a black dot in the distance turned into a clear locomotive, and all kinds of passengers poured in.
The train slowed down, came to a slow stop along the platform, the door of the carriage opened, and Madame Auguste Lumiere led two children dressed in white into the carriage... Immediately after, a young girl in a white winter coat walked over, accidentally saw the camera, she showed a shy expression, and quietly dodged the camera...
Without complicated shooting techniques and without a rich cinematic language, a simple in-depth shot of a train entering a station realistically presents the scene of French passengers waiting for the train in the 1890s.
The first film in the world directed by the Lumiere brothers, a black-and-white silent film of only the 50s, which brought a strong sense of freshness to people at that time, and still has a strong historical charm after 4K restoration.
After 20 years of development, film restoration technology has freed a large number of classic, important and precious film films from the dust. Film history classics such as "The Pianist of the Sea", "The Radio Wave That Never Disappears", "The True Colors of Heroes", and "The True Biography of Ah Fei" have re-screened the silver screen with a new look, bringing back the youth memories of a generation and also showing the unique charm of old movies.
Stills from the Never-Go Away 4k Restoration
4K restoration can not only retain the unique texture of film films, but also adapt to the digitization of films, which can be said to be a good choice for the transition from the film era to the digital age. However, behind the "old look and new look" of these classic movies is a huge cost investment, and film restoration has always been a big industry.
It takes at least two or three months, or even half a year, for a film to complete a 4K restoration. And "long cycle" means "high cost", the film "Decisive Moment" from the restoration, investment to the release of the cost of tens of millions, James . Cameron's 3D version of Titanic cost more than 60 million yuan.
In addition to the high cost of capital, manpower shortage is also a major shackle to 4K repair. A 10-minute footage in the Founding Ceremony involved 600 people who not only had a deep understanding of the art of cinema, but also the physical properties, chemistry, software restoration techniques, and film effects of film.
For a vast sea of film films, restoration work is a "rescue job" in a race against time.
Recently, Watermelon Video and Volcano Engine jointly released the "Classic Video 4K Restoration Plan", saying that in the next year, it will cooperate with CCTV Animation and Shanghai Fine Arts Film Studio to jointly use 4K technology to restore 100 well-known classic animations such as "Shuker and Beta" and "Journey to the West".
Nezha Legend (2003), Go Boy (2005), Go Boy (2), Big Head Son and Little Head Dad (1995), Little Carp Adventure (2007), I Am A Song Maniac (2001), Hulu Brothers (1986), Three Monks, Little Tadpole Finds Mom, Black Cat Sheriff Episodes 1-5, Nine Colored Deer, Shuk and Beta 1-13, Brainless and Unhappy Episodes 1-26, Dirty King Adventures 1-13, Journey to the West, Rubik's Cube Mansion Episodes 1-10, Monkey Fishing Moon, The Great Thief Episodes 1-8, Hulu Xiaokong Episodes 1-6, Mr. NanGuo, Cao Chong Xiangxiang, Big Ears Tutu (Season 1), Ginseng Doll, Mirror Flower Edge Episodes 1-4, Little Carp Jumping Dragon Gate, Mr. Dong Guo, Clam Fight, Old Wolf Invitation, Gollum Coming, Laoshan Daoist, Midnight Chicken Crow, Zodiac 1-13 Episodes, Little Tiger Return, Proud General, Super Soap, Jigong Fighting Cricket, Ginseng Kingdom, Avanti Story 1-13 episodes, Monkey Mountain, Big Hero Di Qing 1-52 episodes, Wolf Coming and other 100 films. (After the repaired content, users can watch it in the watermelon video for free)
At the press conference, in view of the shortcomings of the current 4K movie repair cost, long cycle, manpower shortage, etc., Zhao Shijie, a researcher at the Volcano Engine Multimedia Laboratory, gave a "smart processing" solution for the volcano engine. He said that through self-developed algorithms such as super resolution, intelligent interpolation, color enhancement, and noise reduction, AI can greatly improve the repair efficiency and quality of old films.
<h3>4K restoration, restore the essence of art</h3>
At the press conference, Nezha, Huluwa, and the black cat sheriff, one by one, the classic images appeared on the big screen with super high resolution, and the details of each picture were perfectly presented, and the light and shadow levels became smooth, bringing a different childhood feeling to the audience.
The so-called film restoration is actually the process of re-copying the old film film to the digital carrier, and restoring and optimizing the original appearance of the film through technical processing such as repair, noise reduction, light filling, and color grading.
2k technology is the main means of early film restoration, used to remove film impurities, noise, so that the picture to restore the original visual texture; 4k technology in the resolution requirements of higher, more refined, in addition to clarity and fluency, pay attention to the original tone of the film and light and shadow level, for the audience to create an immersive feeling.
Released in 2014, "Stage Sisters" was the first 4K restoration film in China, and this full-color restoration film with 4K scanning and 4K output was refreshing to the audience at that time. Later, with the improvement of restoration technology, the restored version of "That Man That Mountain That Dog", "Battle of Waterloo" and "Decisive Moment" released in 2019 have successively become phenomenal works in the film market. Since this year, the 4K restored version of the film has begun to enter the public's movie life, and the film restoration technology has officially entered the "4K era" from the "2K era".
Despite AI, 4K movie fixing is still a tough and onerous task. Zhao Shijie introduced that due to the shooting conditions and factors such as film preservation and damage during use, old films generally have low definition, low fluency, color distortion, flaws and other different degrees of picture quality problems.
In the past, old films mainly used film as a medium for shooting and storage. The material of early film was a more flammable nitric acid sheet base than paper, and later developed acetic acid sheet base and polyester sheet base instead, but no matter what kind of film base, it is difficult to preserve at room temperature, temperature, humidity and handling, and the use of external factors of broadcasting are easy to cause film damage.
Source: Beijing Business Daily
At present, the China Film Archive has preserved nearly 30,000 film materials, which are generally dust, dirt, mildew, fading, image shaking, scratches, flickering, noise, discoloration and other problems due to their age, unsatisfactory preservation conditions or excessive reprinting and playback.
The quality of the film directly affects the difficulty and time cost of manual repair. Under normal circumstances, a skilled restorer can repair up to 200 frames a day, but if the film is not well preserved, dirty, cracked, discolored and other problems are serious, the restorer may only be able to repair one second a day (24 frames), and if the picture involves complex scenes such as night scenes, rain scenes, smoke plays, and special effects scenes, the repair cycle is multiplied. In the 4K restored version of "The Founding Ceremony" released in 2019, a clip shot before 1945 and 1945 is only 10 minutes long, taking up 70% of the entire restoration team's time.
Due to the different film quality and time taken, the cost of restoration is not the same, but a 90-minute standard 4K film repair usually costs more than one million yuan, and from physical repair, digital conversion, to sound and picture synthesis, color grading, the entire restoration process is no less difficult than making a new film.
In fact, putting aside the cost problem, the biggest problem of film restoration is how to retain the artistic style and beauty of the film, and the 4K restored version of the film was once controversial because of subverting the beauty of film. An old movie, after a long period of erosion, has long become dim and blurry, through AI technology to repair it bright and beautiful is not difficult, the difficulty is how to restore the original feeling of the old movie. Therefore, how to give AI "artistic accomplishment" is the core problem that the volcano engine wants to overcome.
<h3>Four intelligent algorithms, with "strength" to make feelings land</h3>
"Whether it's fixing cartoons or fixing old movies, in the final analysis, it's not just about improving its clarity, it's about fixing the memories behind those things, and it's the value of restoration that resonates and sparks across generations through those memories." Ren Lifeng, president of Watermelon Video, said that the classic works condense the wisdom and spirit of old artists, and we need to truly restore and present them.
Among the first batch of restored animation works, the classic ink-cut paper-cut animation "Hulu Brothers" is also among them, which is a highly mythological cartoon adapted from the folk literature "Ten Brothers", which is a common memory of the 70s and 80s. In order to retain the unique artistic sense of ink and paper-cutting, Zhao Shijie admitted that they repeatedly discussed the plan with the producer when restoring "Hulu Brothers", and after repeated testing and tuning, they achieved the desired effect.
In terms of algorithms, in order to prevent "accidental injury" to the hazy artistic effect of ink painting, they did not set a high intensity on the algorithm of eliminating defects, but "deliberately" missed a part of the defects and handed them over to human assistance. For old films with serious damage, it takes a lot of manpower to completely eliminate flaws. Volcano engine repair data shows that the algorithm can directly eliminate more than 95% of the defects, and the rest are manually labeled and then adjusted to the algorithm for secondary optimization.
It is understood that the 4K restoration uses some of the technical capabilities of the volcano engine intelligent processing products to enhance video quality through super resolution, intelligent interpolation, intelligent noise reduction, color enhancement and other algorithms. The repair process is roughly as follows, first using the video noise reduction algorithm for pre-processing, and then further through several different types of super-resolution algorithm enhancement to expand the frame to 4K resolution, and generate finer details, and finally through the interpolation algorithm network and HDR remake algorithm, a video that was originally full of noise and compression damage problems is enhanced into a subjective quality comfortable 4K 60 frame HDR program. The following is a brief introduction to the features and benefits:
Intelligent supersoning: Reconstruct the missing details according to the existing images and video information to solve the problems of blurring, poor clarity and low resolution that are common in old films.
Super-Resolution (Super-Resolution) has a very mature solution for its wide application in visual processing. The technological innovation of the volcano engine is mainly reflected in time domain modeling and adaptive processing. The former automatically generates to recover additional details, while the latter performs "area sharding" processing of the video or image to ensure the style and aesthetics of different video/image areas.
Intelligent interpolation: Using deep learning algorithms, low frame rate videos are turned into high frame rate videos through interpolation, making the video smoother and smoother.
Old cartoons are prone to stuttering and poor fluency, mainly because of the small number of frames in the film. The effect of intelligent interpolation technology is to generate intermediate frames by analyzing the dynamics and content of the front and back frames, thereby improving the video frame rate and fluency. Regarding the problem of less animation texture, the solution at this stage is difficult to judge the corresponding motion block of the front and back frames, for this problem, the volcano engine uses block optical flow optimization to improve the accuracy of interpolation frames.
Video Noise Reduction: Eliminates noise and flicker during film storage and transcription.
Common video noise reduction algorithm in the elimination of noise at the same time, easy to damage the texture area, volcano engine scheme through the intelligent analysis of texture, noise, in the denoising at the same time as far as possible to retain the original texture of the video is not affected.
Scratch repair: Identify and repair scratch damage from film in videos.
For common film scratches, considering the difficulty and effect of repair, the volcano engine adopts a combination of "algorithm + manual" solution. From the existing research, if pure AI technology is used for repair, dense and tiny scratches are difficult to be found in full, and it is easy to "accidentally injure"; relatively speaking, the effect of manual repair will be better, but it requires more time and funds, and the efficiency is reduced. The solution of the volcano engine is to first use the algorithm to initially repair small scratches, manual labeling algorithm and then "check for omissions and fill in the gaps", repair large scratches, experiments have proved that the algorithm re-repairs the results through manual labeling, and the efficiency and effect achieved are optimal, and this also reflects the volcano engine "although there is advanced audio and video technology, but not only technology" concept.
Sawtooth Restoration: An independent algorithm is designed for the aliased lines and spectral confusion that occur when the film is downsampled.
The key problem to be solved by sawtooth restoration is sawtooth positioning. Unlike scratches, not every image has jagged lines. After many trials and experiments, the Volcano Engine Multimedia Laboratory team found that the aliasing effect is usually introduced by the inaccurate digital scanning, and the current industry's remediation algorithms and tools do not have corresponding solutions. Therefore, the volcano engine has designed a set of optimization algorithms in a targeted manner, which has greatly improved the repair effect of the sawtooth.
SDR To HDR: For lower-quality videos, it analyzes from the dimensions of color, contrast, detail and picture level, adaptively processes low-quality problems to improve the quality of the source video and the quality of the transcoded video.
Whether in industry or academia, the demand for HDR videos to recover higher dynamic range and color breadth from existing SDR videos has been growing. Generally speaking, if the standard dynamic range is not much improved, it is likely that there will be too many pixels, low quality, and the impact of richer layers and details. High Dynamic Range (HDR) provides richer detail, wider color gamut, and more natural color transitions to deliver higher quality images compared to SDR.
<h3>Volcano engine, innate video capability</h3>
Volcano Engine is an enterprise-level technical service platform under ByteDance, which has built a video cloud product matrix integrating video on demand, veImageX, enterprise live broadcast, video live broadcast, real-time audio and video, cloud editing and intelligent processing around audio and video technology. Video cloud belongs to the volcano engine middle office level service, its biggest advantage is to have vibrato, watermelon video, today's headlines and other practical venues, can continue to iterate in a rich scene, polish the audio and video capabilities.
Keith, head of video cloud products at Volcano Engine, said in an interview with the media, "We will continue to explore the extreme of the video playback experience in the Douyin and watermelon scenes, and solve the large-scale problems that arise in this process." And the solution to these problems, we will precipitate it into a methodology, and then integrate it into the product of the volcano engine video cloud. ”
After the development of all audio and video products to a certain scale, its breakthrough point must be the processing of "details", as early as 2016, ByteDance launched a large-scale test platform A/B test, providing a platform for inspection and optimization of ByteDance algorithms and products. After the A/B test connects the whole line of services such as Douyin and Watermelon Video, tens of thousands of tests are carried out every day, and the number of new experiments in a single day exceeds 1500, covering more than 500 large and small services. The algorithms used in the 4K repair technology such as intelligent interpolation, intelligent super resolution, and intelligent noise reduction have been tested by A/B and multi-platform and multi-scene.
Standards are hailed as the jewel in the crown of the high-tech industry. Whoever has more patents in the standard has a greater right to speak and take the initiative in the industry. Because of this, major technology giants attach great importance to standard setting and regard related patented technologies as core assets.
In the international standard H.266/VVC contributor list, ByteDance ranks third, behind Qualcomm and Huawei.
As an Internet company that did not participate in the development of the previous generation of video codec standards, ByteDance has achieved a leap from a standard follower to a formulator. In the process of formulating a new generation of video codec standards, more than 100 technical proposals initiated by ByteDance have been adopted by the H.266/VVC standard, forming a series of original technologies.
In addition to the important contribution of the standardization work, the Volcano Engine Multimedia Laboratory team has also made positive contributions to the commercialization of H.266/VVC.
As early as June 2019, Volcano Engine has completed the first version of the self-developed encoder BVC that satisfies the application of on-demand scenarios, which can reduce the average bitrate by 33% compared with the x265 encoder, under the same computing resources, for a large number of 1080p HD videos. After that, BVC continued to iterate, and the performance was greatly improved. In addition to the self-developed encoder, the self-developed decoder has also been completed, and high-definition and ultra-high-definition videos can be played smoothly in real time on high-end mobile phones.
After the H.266 standard, Volcano Engine is still further exploring video coding technology, focusing on two directions, one is video compression based on emerging deep learning (including the combination of deep learning and traditional hybrid video coding framework), and the other is based on the traditional hybrid video coding framework technology to continue mining. Although it is only just the beginning, a breakthrough has been made:
Adaptive filter DAM algorithm based on deep learning obtains performance gain of more than 15%;
Based on the hybrid video coding framework, multiple technologies are integrated to achieve performance gains of more than 13%.
Among them, the adaptive filter (DAM) algorithm, the backbone is a deep convolutional network based on the stacking of residual elements, supplemented by adaptive model selection to maximize the adaptation of the characteristics of complex natural video. The so-called residual unit refers to the introduction of jumping layer connection, allowing the network to focus on the residuals of the change, which is similar to the residual difference between the video image frames, such as the duel between the martial arts masters, "sword light flash", most of the content of each frame of the image is the same, the flashing sword light forms a residual difference, which is also the focus of video encoding and compression.
Experimental results show that compared with the latest H.266/VVC standard, ByteDance's DAM solution can bring significant improvement to video encoding performance, luminance signal Y can achieve 10.28% performance gain, two chromaticity signals U and V, performance gain also reached 28.22% and 27.97%, while the video quality has been optimized, at least 13% of the data volume can be reduced.
The research results of the Volcano Engine Multimedia Lab will also be put into use through the upgrading of BVC encoders, including video content processing of apps such as Douyin, Watermelon Video, and Today's Headlines, as well as infrastructure fields such as cloud computing and cloud games, bringing users a higher picture quality and smoother video experience.
<h3>brief summary</h3>
Volcano Engine is positioned as a window for ByteDance to provide technical services to the outside world, providing technical support from the four directions of unified basic services, technology middle platform, intelligent applications and industry solutions. Over the past nine years, ByteDance has precipitated a large number of growth methods, tools and technical capabilities, which have been organically combined on the volcano engine to form more than 60 items, collectively known as "smart growth technology". With the rapid development of digital transformation and the enterprise service industry, the volcano engine "intelligent growth technology" needs to go to the market, be polished and tested.
Volcano Engine has launched a large-scale restoration of classic 4k medium video, expanding the scope of application of its audio and video capabilities, and also contributing to the promotion of Cultural Heritage Protection in China. The restoration of old films is a race against time, and many precious and important classic film heritages are destroyed without repair. In 2006, the China Film Archive took the lead in launching the "Film Archive Film Digital Repair Project" project, took the lead in discovering, collecting, saving and preserving Chinese film films, and has repaired more than 500 domestic films with more than 2K so far, but in order to win this "competition", improving AI technology capabilities is the fundamental solution.
Lei Feng network Lei Feng network Lei Feng net