Usually, the image we see in the display is an RGB tri-color image, and the three colors of red, green and blue are superimposed to form a pixel color. If we expand the tri-color image into ten colors, hundred colors, and thousand colors, then our control and rendering of the image color will be extremely fine. An image with so many bands is a spectral image. The spectral image can obtain the spatial characteristics of the imaged target like an RGB image, while reflecting the internal physical structure and chemical composition of the target. Therefore, spectral imaging technology is widely used in remote sensing, medical testing, food testing and other fields.
Traditional spectral imaging techniques acquire images by scanning, with long acquisition time and large system size. With the development of computational optics technology, researchers tend to perform some coding during spectral image acquisition and use iterative optimization algorithms for spectral reconstruction. However, this approach creates a computational burden, and a single spectral reconstruction typically takes minutes or even hours. In recent years, deep learning has shown great potential in scientific and technological applications, and spectral imaging is no exception. Applying deep learning to spectral imaging can reconstruct images in seconds while combining high resolution with a concise system.
Traditional spectral imaging methods acquire images by scanning, and the slower speed (point scan, line scan, band scan, band scan from left to right) | References[1]
Spectral image reconstruction method after applying deep learning | References[1]
Recently, Hao Xiang's team of researchers from the School of Optoelectronics of Zhejiang University published a review paper at Light: Science & Applications under the title of "Spectral Imaging with Deep Learning", reviewing the latest progress of spectral imaging technology application deep learning, sorting out the spectral imaging technology based on deep learning, expounding and summarizing the principles of various technical routes, and sorting out the current spectral imaging data sets. Outline possible future trends and challenges.
Screenshot of the website
In review articles, effective classification is essential. The authors believe that for spectral imaging based on deep learning, they have found the ideal classification method. "Based on the basic properties of light, our team divides various known spectral imaging methods into three categories: amplitude coding, phase coding, and wavelength coding."
Amplitude-encoded spectral imaging
The spectral coding process in the coding aperture system | References[1], author Sinicized
Amplitude-encoded spectral imaging is performed through a coded aperture system (CASSI), which encodes objects using coded apertures (amplitude masks) and grating elements, and then performs spectral reconstruction through compression perception recovery algorithms. Amplitude-encoded spectral imaging based on deep learning replaces the iterative algorithm of compression perception recovery with a deep neural network, and performs efficient spectral reconstruction by methods such as codec co-optimization, iterative expansion neural network, and non-training network.
Phase-encoded spectral imaging
A phase-encoded spectral imaging system | References[1], author Sinicized
Phase-encoded spectral imaging is performed by diffraction Optical Element (DOE), which achieves phase coding by designing a two-dimensional height profile of the DOE to achieve specific phase delays at different locations. Phase encoding affects different spectral components through Fresnel diffraction, and then the original spectral image can be reconstructed by algorithm by modeling the diffraction process. Due to the complexity of diffraction calculation after phase coding, it is difficult for traditional iterative algorithms to effectively restore spectral images, which has been solved after the emergence of deep learning, and the spectral recovery of phase coding is mainly carried out through deep neural networks. Compared with amplitude coding, phase-coded spectral imaging has the advantages of small light energy loss and compact system.
Wavelength-encoded spectral imaging
Wavelength-encoded spectral imaging encodes the image directly in the spectral dimension and can be performed with an optical filter. RGB images can be seen as a kind of spectral coding. At present, the mainstream wavelength coding method uses the existing RGB or design optical filter, and the spectral reconstruction after encoding uses deep learning technology.
Direct spectral reconstruction based on RGB images is a very fiery direction. With the NTIRE 2018 and NTIRE 2020 spectral reconstruction competitions, many deep learning technology teams have participated, greatly expanding the existing deep learning spectral recovery technology. The researchers analyzed the RGB spectral reconstruction and filter design spectral reconstruction involved in deep learning, and divided the reconstruction methods into point reconstruction and block reconstruction, so as to introduce each reconstruction method.
The codec collaborative design spectral imaging technology proposed by this research group | References[1]
Wavelength coding based on custom optical filters is an emerging spectral imaging technique in recent years. By designing a wide-spectrum filter, you can achieve greater coding freedom than RGB filters. Combined with effective deep learning techniques, compact, fast, and accurate spectral recovery can be achieved.
Thanks
Thanks to the corresponding author hao xiang for his review and suggestions for this article.
bibliography
[1] Huang, L., Luo, R., Liu, X. et al. Spectral imaging with deep learning. Light Sci Appl 11, 61 (2022). https://doi.org/10.1038/s41377-022-00743-6
[2] https://www.eurekalert.org/news-releases/947692
Author: Matrix Star
Edit: Crispy fish
Typography: Yin Ningliu
Source: Provided by the team
Research team
Corresponding author Hao Xiang: Doctoral supervisor of Zhejiang University, assistant dean of Jiaxing Research Institute of Zhejiang University. He received his Ph.D. in Engineering from Zhejiang University in 2014 and later worked as an associate researcher in the Department of Cell Biology, Yale University School of Medicine, focusing on optical microscopy/super-resolution microscopy, hyperspectral techniques and computational optics.
The homepage of the research group is https://person.zju.edu.cn/en/nanoscopy
Thesis information
Published the journal Light Science & Application
Published March 16, 2022
The title of the paper is Spectrum imaging with deep learning
(DOI:10.1038/s41377-022-00743-6)
The areas of the article are deep learning, spectral imaging, computational optics
If you want to know more about industry and technology venture capital news,
Welcome to pay attention to the fruit shell hard technology