Fluorescence microscope is an indispensable and important research tool in the field of life sciences, its principle is to use ultraviolet light as the light source, irradiate the object to make it fluoresce, and then observe the shape and location of the object under the microscope, which can be used to study the absorption and transport of intracellular substances, the distribution and localization of chemical substances, etc.
However, high-intensity exposure to excitation light can directly or indirectly affect cells through photochemical processes. In long-term live-cell experiments, fluorescence observation is best performed with minimal light exposure. At the same time, lower exposures result in weaker fluorescence signals, reducing the signal-to-noise ratio (SNR) of images and making quantitative image analysis more difficult.
As a result, fluorescence microscopy-based image restoration (FMIR) has received a lot of attention in the life sciences as it aims to obtain images with high signal-to-noise ratios from images with low signal-to-noise ratios, helping to reveal important imaging information at the nanoscale.
At present, benefiting from the rapid development of artificial intelligence technology, many deep learning-based FMIRs have broken through the physical limits of fluorescence microscopy and made significant progress, but mainstream models still have challenges such as poor generalization ability and strong data dependence.
对此，来自复旦大学计算机科学技术学院的研究团队在 Nature Methods 上发表了题为「Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration」 的论文，提出的跨任务、多维度图像增强基础 AI 模型 UniFMIR，不但实现了对现有荧光显微成像极限的突破，而且为荧光显微镜图像增强提供了一个通用的解决方案。
Research Highlights:
* The UniFMIR model greatly improves the performance in the five task directions of "image super-resolution, isotropic reconstruction, 3D denoising, surface projection, and volume reconstruction".
* Push the limits of existing fluorescence microscopy
* Simple parameter fine-tuning can be applied to different tasks, imaging modalities, and biological structures
Address:
https://www.nature.com/articles/s41592-024-02244-3
Follow the official account, and reply to "Microscopy" in the background to get the full PDF
Dataset: 30 GB, 196,418 pairs of training samples
The researchers collected a large training dataset (approximately 30 GB) from 14 public datasets, including 196,418 pairs of training samples, covering a wide range of imaging modalities, biological samples, and image restoration tasks. At the same time, the researchers also grouped the datasets for different fluorescence microscopy-based image restoration tasks and imaging methods.
Overview of dataset details
Because these datasets vary widely in format, domain, and range of values, the researchers processed the images for subsequent training and validation across datasets. Specifically, it writes input and GT images from existing datasets with different storage formats, including 'TIF', 'npz', 'png', and 'nii.gz') to a '.npz' file. In addition, the images were normalized by following the data processing methods in CARE4 to unify the numerical distributions of the different data sets.
Model architecture: multi-head and multi-tail structure
The UniFMIR model constructed by the researchers uses a multihead, multitail structure, as shown in the figure below:
UniFMIR 架构
Specifically, UniFMIR consists of a multi-head module, a feature enhancement module, and a multi-tail module.
Among them, the multi-head module and the multi-tail module use different branches to extract the shallow features of specific tasks, and obtain accurate results for different image inpainting problems.
The Feature Enhancement Module uses an advanced Swin Transformer structure to enhance feature representations and reconstruct universally valid features for high-quality image restoration based on fluorescence microscopy. Different fluorescence microscopy-based image restoration operations cover different head and caudal branches, but share the same feature enhancement module.
The UniFMIR model was implemented based on PyTorch and optimized using adaptive moment estimation (Adam), where β1 = 0.9 and β2 = 0.999, with a total of 500 epochs trained. The initial learning rate starts at 5 × 10-5 and halves after 200 epochs. All experiments were conducted on a machine equipped with an Nvidia GeForce RTX 3090 GPU with 24GB of RAM.
In the pre-training phase, the researchers input all the training data into the model and use the corresponding data to optimize the different head and tail branches to perform different tasks. The middle feature enhancement branch is optimized using all the training data.
In the fine-tuning phase, the researchers set the batch size/patch size to 4/128, 32/64, 32/64, 4/64, and 1/16 for processing image super-resolution, isotropic reconstruction, 3D denoising, surface projection, and volume reconstruction tasks, respectively, to produce better learning results.
By pre-training the model on a large-scale dataset collected and fine-tuning the model parameters using data from different image enhancement tasks, UniFMIR exhibits better augmentation performance and generalization than proprietary models.
The result: Dramatically improved performance in 5 major tasks
The results show that UniFMIR, the AI foundation model for image enhancement in fluorescence microscopy, has greatly improved its performance in five task directions: image super-resolution, isotropic reconstruction, 3D denoising, surface projection, and volume reconstruction.
Super Resolution (SR)
The study first validated the potential of the UniFMIR method to address SR problems involving images of increasing structural complexity, including capsular niches (CCPs), endoplasmic reticulum (ERs), microtubules (MTs), and fibrillar actin (F-actin), obtained by multimodal structured illumination microscopy systems.
UniFMIR successfully inferred SR SIM images from widefield (WF) images with high fluorescence levels on the diffraction-limited scale and showing clear structural details.
Compared to two deep learning-based fluorescence microscopy SR models (XTC15 and DFCAN5) and a single-image super-resolution model (ENLCN36), UniFMIR is able to correctly reconstruct most of the microtubule images without losing or merging them, even when the microtubules are densely distributed and in close proximity to each other. For diverse subcellular structures, UniFMIR also restored hollow, ring-shaped CCPs and staggered F-actin fibers with high fidelity.
Comparison of PSNR↑/SSIM↑/NRMSE↓ of different datasets n=100
The researchers also quantified the achieved SR accuracy using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), standardized root mean square error (NRMSE), resolution estimation for decoration analysis, Fourier ring correlation (FRC), SQUIRREL analysis, and segmentation metrics, as shown in the figure above.
When evaluating the fluorescence intensity and structure of SR SIM images, higher PSNR/SSIM values and lower NRMSE values indicate better SR, and UniFMIR clearly stands out on both of these metrics.
Homosexual Homosexual Weights
Isotropy means that the physical and chemical properties of an object do not change depending on the direction, such as all gases, liquids (except liquid crystals) and amorphous objects show isotropy. On the other hand, anisotropy refers to the fact that all or part of the chemical, physical and other properties of a substance will change with the change of direction, showing different properties in different directions.
The investigators applied UniFMIR to anisotropic raw data from mouse liver volume imaging to predict isotropic axial slices and compared them to two deep learning-based isotropic reconstruction models (CARE and 3D U-Net models).
The results show that UniFMIR produces more accurate isotropic reconstruction results of pixel distribution.
3D denoising
The researchers further benchmarked the performance of UniFMIR in the denoising task of live-cell images on Planaria and Tribolium datasets.
Visual results of a 3D image denoising task performed on a flatworm
Compared to the two U-Net-based denoising models, CARE and GVTNets, the UniFMIR model significantly suppressed the noise of low signal-to-noise ratio fluorescence microscopy images at different laser powers/exposure times, and clearly depicted the flatworm (S. mediterranea) and red dung beetle volumes with labeled nuclei, which facilitated the observation of embryonic development.
Surface Projection
To better analyze and study the behavior of developing epithelial cells in Drosophila melanogaster, surface projection helps to project a 3D volume into a two-dimensional surface image. Current deep learning models (ARE and GVTNets) divide this image restoration problem into two sub-problems, 3D-to-2D surface projection and 2D image denoising, and use two task-specific networks that follow the same encoder-decoder framework as U-Net to solve them.
Comparison of PSNR↑/SSIM↑/NRMSE↓ of UniFMIR applied to surface projection n = 26
The method proposed in this study further examines UniFMIR in more complex composite fluorescence microscopy image restoration tasks. Compared to ARE and GVTNets, UniFMIR achieves higher projection reconstruction accuracy in terms of PSNR/SSIM/NRMSE metrics.
Volume reconstruction
In the experiment, the researchers also validated the ability of UniFMIR to perform volumetric reconstruction on the data provided by VCD-Net. The reconstructed 3D volume of each view can identify the trajectory of the imaged object, as shown in the figure below, which helps to uncover the underlying mechanisms of many complex living cell dynamics involving various subcellular structures.
Apply UniFMIR to volumetric reconstruction
In summary, fluorescence microscopes loaded with UniFMIR could become a "magic weapon" in life science laboratories. Scientists can more clearly observe the microstructures and complex processes inside living cells, accelerating scientific discovery and medical innovation in the fields of life sciences, medical research, and disease diagnosis around the world.
At the same time, in the fields of semiconductor manufacturing and new material research and development, this achievement can be used to improve the quality of observation and analysis of material microstructure, so as to optimize the manufacturing process and improve product quality. In the future, scientists in the life sciences laboratory can continue to enhance UniFMIR's image reconstruction capabilities by further expanding the data volume and richness of the training data.
AI-driven new paradigm of image processing in the life sciences
Nowadays, advances in microscopy are creating large amounts of imaging data, and how to efficiently perform image processing is an important part of research in the biomedical field. As artificial intelligence continues to make disruptive breakthroughs in life science research, a new paradigm of AI-driven image processing is here.
In 2020, a professor of bioengineering at Rice University in Houston, Texas, partnered with MD Anderson Cancer Center to develop a computational microscope called DeepDOF, which is AI-based and can achieve more than 5 times the DOF of traditional microscopes while maintaining resolution, dramatically reducing the time required for image processing.
Locating AFM and X-ray structures (source: Weill Cornell Medical College)
In 2021, the Weill Cornell Medicine research team developed a computational technique that greatly improves the resolution of AFM by applying a localized image reconstruction algorithm to peak positions in atomic force microscopy (AFM) and traditional AFM data, increasing resolution beyond the limits set by the tip radius and resolving individual amino acid residues on protein surfaces under native and dynamic conditions. The method reveals atomic-level details of proteins and other biological structures under normal physiological conditions, opening a new window into cell biology, virology, and other microscopic processes.
In April 2024, a paper from the Massachusetts Institute of Technology, the Broad Institute of MIT, and Harvard University, as well as the Massachusetts General Hospital, introduced a new artificial intelligence tool that could capture uncertainty in medical images. The system, known as Tyche (named after the Greek god of opportunity), offers multiple plausible segments, each highlighting a slightly different area in the medical image. The user can specify how many options Tyche outputs and choose the one that best suits their purpose.
In summary, AI can be used to enhance, segment, register, and reconstruct biomedical images to improve image quality and extract useful information, giving microscopes a pair of "sharp eyes". In the future, with the help of AI, microscopes will be able to see more clearly, while processing data faster, more automatically, and more accurately, making scientific research more efficient and easy.
Resources:
1.https://www.nature.com/articles/s41592-024-02244-3
2.https://news.fudan.edu.cn/2024/0413/c5a140009/page.htm
3.https://new.qq.com/rain/a/20240417A06LF900
4.http://www.phirda.com/artilce_28453.html?cId=1
5.https://www.ebiotrade.com/newsf/2024-4/20240412015712482.htm