Enhancing image resolution of confocal fluorescence microscopy with deep learning

1.
Key Laboratory of Optoelectronic Devices and Systems of Ministry of Education and Guangdong Province, College of Physics and Optoelectronic Engineering, Shenzhen University, Shenzhen, 518060, China
2.
Department of Electrical and Electronic Engineering, University of Hong Kong, Pokfulam, Hong Kong, SAR, China

Funds: We thank Dr. Min Zhang from Department of Biochemistry at Free University of Berlin and Dr. Iullia Golovynska from Shenzhen University for their helpful discussions.

摘要

- /
- /
- /
Abstract: Super-resolution optical imaging is crucial to the study of cellular processes. Current super-resolution fluorescence microscopy is restricted by the need of special fluorophores or sophisticated optical systems, or long acquisition and computational times. In this work, we present a deep-learning-based super-resolution technique of confocal microscopy. We devise a two-channel attention network (TCAN), which takes advantage of both spatial representations and frequency contents to learn a more precise mapping from low-resolution images to high-resolution ones. This scheme is robust against changes in the pixel size and the imaging setup, enabling the optimal model to generalize to different fluorescence microscopy modalities unseen in the training set. Our algorithm is validated on diverse biological structures and dual-color confocal images of actin-microtubules, improving the resolution from ~ 230 nm to ~ 110 nm. Last but not least, we demonstrate live-cell super-resolution imaging by revealing the detailed structures and dynamic instability of microtubules.
- Super-resolution fluorescence microscopy /
- Image resolution enhancement /
- Deep learning /
- Generative adversarial network

HTML全文

参考文献(38)

[1]	Sage D, Kirshner H, Pengo T, Stuurman N, Min J, Manley S, et al. Quantitative evaluation of software package for single-molecule localization microscopy. Nat Methods. 2015;12(8):717–24.
[2]	Rust M, Bates M, Zhuang X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat Methods. 2006;3(10):793–6.
[3]	Gustafsson MGL. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. J Microsc. 2000;198:82–7.
[4]	Agarwal K, Macháň R. Multiple signal classification algorithm for super-resolution fluorescence microscopy. Nat Commun. 2016;7:13752.
[5]	Hell SW, Wichmann J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt Lett. 1994;19:780–2.
[6]	Wang C, Taki M, Sato Y, Tamura Y, Yaginuma H, Okada Y, et al. A photostable fluorescent marker for the superresolution live imaging of the dynamic structure of the mitochondrial cristae. Proc Natl Acad Sci U S A. 2019;116(32):15817–22.
[7]	Vicidomini G, Bianchini P, Diaspro A. STED super-resolved microscopy. Nat Methods. 2018;15(3):173–82.
[8]	Yang Z, Sharma A, Qi J, Peng X, Lee DY, Hu R, Lin D, Qu J. J Seung Kim, “Super-resolution fluorescent materials: an insight into design and bioimaging applications.” Chem Soc Rev. 2016;45:4651–67.
[9]	Wang H, Rivenson Y, Jin Y, Wei Z, Gao R, Günaydɪn H, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods. 2019;16:103–10.
[10]	Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods. 2021;18:194–202.
[11]	Patterson J. A Gibson, deep learning: a Practitioner’s approach: O’Reilly Media; 2017.
[12]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nat. 2015;521(7533):436–44.
[13]	He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE; 2016. p. 770–8.
[14]	Zhang K, Zuo W, Chen Y, Meng D, Zhang L. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans Image Process. 2017;26(7):3142–55.
[15]	Ouyang W, Aristov A, Lelek M, Hao X, Zimmer C. Deep learning massively accelerates super-resolution localization microscopy. Nat Biotechnol. 2018;36:460–8.
[16]	Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell. 2018;172:1122–31.
[17]	Wang N, Yan W, Qu Y, Ma S, Li SZ, Qiu M. Intelligent designs in nanophotonics: from optimization towards inverse creation. PhotoniX. 2021;2:22.
[18]	Wang K, Zhang MM, Tang J, Wang L, Hu L, Wu X, et al. Deep learning wavefront sensing and aberration correction in atmospheric turbulence. PhotoniX. 2021;2:8.
[19]	O Ronneberger, P Fischer, T Brox, “U-net: convolutional networks for biomedical image segmentation,” arXiv: 1505.04597 (2015).
[20]	Maas AL, Hannun AY, Ng AY. Rectifier nonlinearities improve neural network acoustic model. In: 30th International Conference on Machine Learning (ICML). Atlanta: IMLS; 2013. p. 6–11.
[21]	Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hawaii: IEEE; 2017. p. 2261–9.
[22]	D Hendrycks, K Gimpel, “Gaussian error linear units (GELUs),” arXiv: 1606.08415 (2016).
[23]	Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. In: 14th International Conference on Artificial Intelligence and Statistics (AISTATS). Fort Lauderdale: Society for Artificial Intelligence and Statistics; 2011.
[24]	Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13:600–12.
[25]	DP Kingma, J Ba, “Adam: a method for stochastic optimization,” arXiv: 1412.6980 (2014).
[26]	Paszke A. Automatic differentiation in PyTorch. In: in 31st Conference on Neural Information Processing Systems. Long Beach: NeurIPS Foundation; 2017.
[27]	Pan SJ, Yang Q. A survey on transfer learning. IEEE T Knowl Data En. 2010;22(10):1345–59.
[28]	Castello M, Tortarolo G, Buttafava M, Deguchi T, Villa F, Koho S, Pesce L, Oneto M, Pelicci S, Lanzanó L, Bianchini P, Sheppard CJR, Diaspro A, Tosi A, Vicidomini G. A robust and versatile platform for image scanning microscopy enabling super-resolution FLIM. Nat Methods. 2019;16:175–8.
[29]	Descloux A, Grußmayer KS, Radenovic A. Parameter-free image resolution estimation based on decorrelation analysis. Nat Methods. 2019;16:918–24.
[30]	Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hawaii: IEEE; 2017. p. 5967–76.
[31]	IJ Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, Y Bengio, “Generative adversarial networks,” arXiv: 1406.2661 (2014).
[32]	Girshick R. Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV). Santiago: IEEE; 2015. p. 1440–8.
[33]	Zhao H, Gallo O, Frosio I, Kautz J. Loss functions for image restoration with neural networks. IEEE T Comput Imag. 2017;3(1):47–57.
[34]	Guo Y, Li D, Zhang S, Yang Y, Liu J-J, Wang X, Liu C, Milkie DE, Moore RP, Tulu US, Kiehart DP, Hu J, Schwartz JL, Betzig E, Li D. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell. 2018;175:1430–42.
[35]	Gudimchuk NB, McIntosh JR. Regulation of microtubule dynamics, mechanics and function through the growing tip. Nat. Rev. Mol. Cell Bio. 2021;22:777–95.
[36]	Bálint Š IV, Vilanova ÁSÁ, Lakadamyali M. Correlative live-cell and superresolution microscopy reveals cargo transport dynamics at microtubule intersections. Proc Natl Acad Sci U S A. 2013;110(9):3375–80.
[37]	Huang X, Fan J, Li L, Liu H, Wu R, Wu Y, et al. Fast, long-term, super-resolution imaging with hessian structured illumination microscopy. Nat Biotechnol. 2018;36(5):451–9.
[38]	Dogterom M, Koenderink GH. Actin-microtubule crosstalk in cell biology. Nat Rev Mol Cell Bio. 2019;20:38–54.