PUB - Publikationen an der Universität Bielefeld

Super-resolution structured illumination microscopy (SR-SIM) provides an up to two-fold enhanced spatial resolution of fluorescently labeled samples compared to conventional widefield microscopy. SR-SIM images are reconstructed by combining several raw SIM images, which were exposed to patterned illumination with high modulation contrast, in the frequency domain. Noisy raw image data, e.g. due to low laser excitation power or short exposure times, lead to reconstruction artifacts. This thesis demonstrates different deep learning based SR-SIM image denoising and reconstruction methods to produce high quality reconstructed SR-SIM images. Deep learning based methods have shown remarkable performance and are considered as excellent candidates for several image restoration tasks. Accordingly, in the first part of this work, convolutional neural networks (CNNs) and Swin Transformer based methods were proposed for restoring SR-SIM images with low signal-to-noise ratio (SNR). The experimental results show that our proposed Transformer based SwinT-fairSIM method (i.e., a combination of computational reconstruction and subsequent denoising via deep learning) outperforms other CNN-based and end-to-end deep-learning based methods. The proposed SIM image reconstruction methods prove to be very robust against reconstruction artifacts and generalize very well over various noise levels and microscope settings. The deep learning methods were successful in denoising noisy SR-SIM images and provide superior results in terms of peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) values compared to existing conventional methods.

However, the analysis of the resulting images in the Fourier spectra shows some discrepancies in the high frequency region. To address this problem, two new combinations of frequency based loss functions were proposed for denoising low SNR images. The first loss function is the combination of frequency and pixel-wise based losses. The second combination combines frequency domain with feature-based losses. These new loss functions were compared with various traditional loss functions. The results show that the networks trained with the novel losses are superior in terms of average PSNR and SSIM values. Moreover, critical structures of biological cells are better preserved in the denoised outputs. The frequency based loss functions prove to be useful for improving the image quality in the spatial domain, while slightly improving the Fourier spectra of the resulting images.

To extend this research, complex-valued convolutional neural networks (CV-CNNs) based methods were designed to denoise and reconstruct the SR-SIM images in the frequency domain rather than the spatial domain. Recently, CV-CNNs have shown remarkable performance in several computer vision tasks such as image classification and segmentation. However, the potential of CV-CNNs for image restoration problems in the frequency domain has not been fully investigated. The main objective of this experiment is to explore the potential of CV-CNNs for image restoration in the frequency domain and to preserve the high frequency region in the Fourier space. In this experiment, two novel CV-CNN based models equipped with complex-valued attention gates are proposed for denoising and super-resolution problems. The proposed CV-CNN models outperform their real-valued counterparts for denoising SR-SIM and conventional image datasets. Additionally, the experimental results reveal that the CV-CNN based models preserve the frequency spectrum better than their real-valued counterparts. Based on these findings, it is concluded that CV-CNN based methods provide a plausible and beneficial deep learning approach for image restoration in the frequency domain.

Finally, the generalization capabilities of deep learning methods for the denoising task on real fluorescence microscopy data were fully explored. As an additional contribution, two types of transfer learning strategies were benchmarked, i.e., direct transfer and fine-tuning. Direct transfer does not prove to be a viable strategy, but fine-tuning achieves results comparable to conventional training from scratch, while saving computational time and potentially reducing the amount of training data required. As a final contribution, four datasets of raw SIM images and already reconstructed SR-SIM images were published. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. These datasets are structured in such a way that they can be easily used by the research community for research on denoising, super-resolution, and transfer learning strategies.