Super resolution can overcome or compensate for the problems of image blur and low image quality caused by the limitations of the image acquisition system and the acquisition environment itself, improve the image resolution, and provide important support for the subsequent processing of images such as feature extraction and information recognition.
In December 2010, Su, a professor in the Department of Electronic Engineering at Tsinghua University, received an unusual phone call from a police officer from the Inner Mongolia Autonomous Region's Jungger Criminal Police Team, holding a blurred image of a criminal suspect's face and asking Su Guangda for help.
"This image was taken by a roadside surveillance camera and is so low resolution that it is completely unrecognizable to the naked eye." Su Guangda recalled that at that time, he used super-resolution technology to reconstruct the blurry image into the software they developed, and the Jungger police quickly locked on the suspect based on the reconstructed high-definition image and solved the homigre.
In fact, this case is not unique. When the police solve the case, they can get the relevant images of the suspect through the surveillance camera to speed up the case. However, the details of the photos taken by surveillance cameras are often blurred when zoomed in, which greatly reduces the efficiency of the police in obtaining critical evidence.
With the development of super-resolution technology, these problems have ushered in a turning point and are gradually being solved.
This article will introduce the mainstream algorithms of super-resolution and their advantages and disadvantages from the concept, and show its application in different scenarios such as public security, medical diagnosis, satellite remote sensing, and digital entertainment.
What is Super Resolution?
Super-resolution (SR), in simple terms, is the process of upscaling a low-resolution (LR) image to a high resolution (HR) by an algorithm. Compared with low-resolution images, high-resolution images have a higher pixel density and richer texture detail, making them more reliable.
Source: SUPIR official website
This technology can overcome or compensate for the problems of image blur and low image quality caused by the limitations of the image acquisition system or the acquisition environment itself, and provides important support for the subsequent processing of images such as feature extraction and information recognition.
Super-resolution algorithm classification
At present, there are three main types of super-resolution methods: interpolation-based methods, reconstruction-based methods, and learning-based methods.
* Interpolation-based approach
The interpolation method increases the size of the image by inserting new pixels around the original pixels of the image, and assigns values to these pixels, so as to restore the image content and achieve the effect of improving the image resolution.
* Pixel: The most basic unit element that makes up the image, that is, the dots, each pixel has its own color value, the more pixels on the unit area, the clearer the picture.
* Refactoring-based approach
The super-resolution algorithm based on reconstruction usually uses multiple low-resolution images taken in the same scene as input, and then analyzes the frequency domain or spatial domain relationship of these images, and guides and constrains the reconstruction process by introducing prior information, and then reconstructs a single high-resolution image.
*Frequency domain: refers to the characteristics of the signal in the frequency domain.
*Airspace: refers to the spatial distribution of signals.
*Prior information: This kind of information is available "before the experiment" and can generally be understood as domain knowledge.
* Learning-based approach
The learning-based super-resolution method usually uses a large amount of training data to predict the high-frequency detail information lost in the low-resolution image by learning the mapping relationship between the low-resolution image and the high-resolution image, so as to generate a super-resolution image.
Methods based on shallow learning mainly include machine learning, manifold learning, sample learning, and sparse coding, which can be used in cases with small data volumes.
基于深度学习的方法可以分为基于卷积神经网络的 SR 方法、基于残差网络 (residual network, ResNet) 的 SR 方法和基于生成对抗网络 (generative adversarial networks, GAN) 的 SR 方法。
Network structure of super-resolution reconstruction algorithm based on deep learning
Source: France.com
The advantages and disadvantages of the above three methods are as follows:
Source: HyperAI
Today, deep learning has become the mainstream method in the field of super-resolution.
In 2014, researchers applied deep learning to the field of image super-resolution reconstruction for the first time, and proposed the SRCNN (Super-Resolution Convolutional Neural Network) network model. Since then, the field of super-resolution reconstruction has set off a wave of deep learning.
SRCNN network structure
SRCNN is the first model to apply deep learning methods to image super-resolution, and only 3 convolutional layers are used to achieve PSNR values that far exceed those of traditional methods.
Specifically, a low-resolution image is input, the image is enlarged to the target size by bicubic interpolation, and then a 3-layer convolutional neural network is used to fit the nonlinear mapping between the low-resolution image and the high-resolution image, and finally the reconstructed high-resolution image is output.
*PSNR Value: Peak signal-to-noise ratio, the higher the value, the better the output HR image quality.
With its simplicity and efficiency, SRCNN is an important milestone in the field of image super-resolution. Since then, deep learning-based super-resolution techniques have developed rapidly, from early convolutional neural network (CNN)-based super-resolution techniques to more recent generative adversarial network-based techniques.
Deep Learning + Super Resolution: Diverse Applications from Public Security to Digital Entertainment
Demand drives technology development, and technology iteration helps applications land. Today, super-resolution technology is widely used in public security, medical diagnosis, satellite remote sensing, and entertainment media.
* Public security
Public surveillance footage is often blurry and low-resolution due to factors such as weather and distance. The application of super-resolution technology can help the police extract clear key information such as faces and license plate numbers, which is helpful in case detection.
The researchers used ESRGAN and BSRGAN networks to analyze portraits and natural scenes in different environments, and explored the feasibility of super-resolution technology in public security and courtrooms.
First, the researchers reproduced, trained, and tested the ESRGAN and BSRGAN models to obtain the optimal model parameters. Then, the trained model was used to perform super-resolution reconstruction on the collected low-quality images such as portraits and natural scenes, and the reconstruction results of ESRGAN and BSRGAN were obtained.
Comparison of portrait low-quality image reconstruction
The researchers compared the reconstructed images of ESRGAN and BSRGAN with the original images. The results show that the reconstructed portraits in frontal, oblique and complex scenes have a great improvement in visual quality and fidelity.
Comparison of low-quality image reconstruction in natural scenes
In comparison with natural scenes, BSRGAN reconstructions are better than ESRGAN, effectively removing complex noise unknown to the original low-quality images, and can produce sharp edges and fine details.
* Medical diagnostics
Due to the limitations of imaging equipment and the complex clinical environment, the images obtained in the medical field often have the problem of insufficient resolution, which directly affects the accurate diagnosis and treatment decisions of doctors.
Improving the structure of the SRGAN
Using the Generative Adversarial Network (SRGAN) for super-resolution reconstruction of natural images as the basic method, the researchers changed the structure of the network by reducing 2 input channels and removing 1 residual block, improved the network loss function, and added a fuzzy processing dataset to reconstruct a 4x magnified medical ultrasound image with clear edges and no artifacts.
Comparison of regions of interest in Case1 reconstruction results
Case2 Comparison of regions of interest in reconstruction results
The researchers compared the improved SRGAN to 3 other algorithms. The results show that the reconstruction results of the improved SRGAN are overall smooth, and the texture edges are sharper.
* Satellite remote sensing
In recent years, remote sensing satellite imagery has been widely used in environmental monitoring, resource exploration, disaster warning, and military fields. However, factors such as atmospheric variation, transmission noise, motion blur, and undersampled optical sensors severely limit the clarity of remote sensing satellite images. Super-resolution technology can improve the quality and usability of satellite remote sensing data by processing and upscaling low-resolution images.
The process of super-resolution reconstruction algorithm of remote sensing images based on local group target assistance
The researchers introduced the detailed feature information of the target area of the local cluster group of remote sensing images into the sampling reconstruction of complete remote sensing images, extracted image features of different scales through multi-level neural networks, and fused and reconstructed these features through residual learning. With the help of the pixel information of local images, the details of global remote sensing images can be significantly improved, and the resolution of the target area of the cluster can be optimized.
Visual comparison with the baseline method
The comparison chart shown in the study shows that the proposed method is significantly better than other existing methods in terms of visualization effect, and is suitable for urban and field scenes, and shows good results.
* Digital Entertainment
An anime is made up of multiple still images connected together, and the resolution of the still image will affect the clarity of the final anime. However, existing hand-drawn or digitally drawn drawings do not guarantee a high resolution of the first draft, which is quite unfriendly to the user's visual experience. By applying super-resolution technology to convert these low-resolution images into higher-resolution images, more details and textures can be presented, which can make the images of anime works more vivid and realistic.
B 站就曾推出了一个名为 Real-CUGAN (Real Cascaded-U-Net-style Generative Adversarial Networks) 的动漫画质修复模型。
Firstly, the researchers used the model to cut the animation frames into pieces, and used the image quality scoring model to score and filter the candidate blocks, and finally obtained a million-level training set of high-quality animation image blocks.
Then, through the multi-stage degradation algorithm, the high-definition image blocks are downsampled to obtain low-quality images, so that the model can learn and optimize the reconstruction process from low-quality images to high-quality images. After the training is completed, the model can perform high-definition processing on the real two-dimensional low-quality images.
The comparison chart of different algorithms is as follows:
Author: XIX
Editor: Li Baozhu, Sanyang
Source: HyperAI
Original title: Super Resolution Special | 3 methods, 4 tutorials, 10 datasets, and one article Get core knowledge points
Edit: Hanamaki