How to make blurry old films into high definition?
The answer to AI is a super-resolution algorithm.
Now, in the field of video scoring, there is a powerful algorithm that has won the excellent result of the NTIRE 2021 Triple Crown and one runner-up, and has reached CVPR 2022.
It's called BasicVSR++ and is a further improvement on the video superscript SOTA model Basic VSR.
BasicVSR also won the NTIRE title and was named to CVPR 2021.
Now, this BasicVSR+++ not only greatly exceeds its predecessors in performance with essentially the same amount of parameters, but also increases the PSNR (peak signal-to-noise ratio, image quality evaluation index) by 0.82dB, and can also be applied to more video restoration tasks (such as compressed video enhancement).
Enhanced Basic VSR
BasicVSR uses propagation + feature alignment to extract valid information from the entire input video for overdividing.
However, its basic design also limits the power of information aggregation, such as difficulty in recovering fine details, especially when dealing with complex occlusion areas.
As a result, the enhanced version of BasicVSR++ has been revamped in terms of propagation and alignment, using second-order grid propagation and flow-guided deformable alignment designs to improve the ability to aggregate information in the network and improve the robustness and effectiveness of the occluded area.
Among them, the second-order grid allows information to be propagated forward to backward from different space-time locations, making the propagation of features more effective.
Optical flow guidance distortable alignment allows frames to be more robustly feature-aligned.
This alignment is primarily due to the fact that simple deformation alignment training is not stable, although the diversity of offsets in deformable convolution (DCN) networks makes deformation alignment better than optical alignment.
The specific architecture of BasicVSR+++ is as follows:
Given the input video, the residual module is first used to extract features for each frame; then these features are propagated in the second-order network propagation, where the alignment part is aligned with optical flow-guided deformation; after the information propagation is completed, the aggregate features generate an output image.
The best performance of all 16 algorithms of its kind
The authors compared the performance, parameter amounts, and time-consuming of 16 different video supersing algorithms, and the results were that BasicVSR++ achieved the best performance in two degradation ways across all datasets (red for the best score and blue for the suboptimal score).
In particular, compared with the large-capacity sliding window algorithm EDSR, BasicVSR++ has achieved a performance improvement of 1.3dB, while the amount of parameters is 65% less;
Compared with the latest technology IconVSR, BasicVSR++ has a 1dB performance improvement with fewer parameters.
The lighter version of BasicVSR++ (S) also has a 0.82dB improvement compared to the predecessor BasicVSR, with significant benefits.
In specific effects, whether on the REDS4, Vimeo-90K-T or Vid4 datasets, BasicVSR++ can restore images in extreme detail, and the results are the best.
At present, the code of BasicVSR++ has been open sourced, and interested students can try it.
About the author
Kelvin C.K. Chan is from nanyang technological university, school of computer science and engineering, ph.D. in the third year, graduated from the University of Hong Kong Chinese.
His current research interests are image/video restoration, and he has published a total of 5 top papers.
The corresponding author is his mentor, Chen Change Loy, associate professor at nanyang technological university school of computer science, and deputy director of S-Lab, sensetime-Nanyang technological university joint laboratory.
They are also the original writers of BasicVSR.
The remaining two authors of BasicVSR++ are Zhou Shangchen, a second-year doctoral student at the university, and Xu Xiangyu, a researcher at the university.