Deep-learning-enabled temporally super-resolved multiplexed fringe projection profilometry: high-speed kHz 3D imaging with low-speed camera

Wenwu Chen^{1, 2, 3},
Shijie Feng^{1, 2, 3},
Wei Yin^{1, 2, 3},
Yixuan Li^{1, 2, 3},
Jiaming Qian^{1, 2, 3},
Qian Chen³,
Chao Zuo^{1, 2, 3}

1.
Smart Computational Imaging Laboratory (SCILab), School of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu Province, China
2.
Smart Computational Imaging Research Institute (SCIRI) of Nanjing University of Science and Technology, Nanjing, 210019, Jiangsu Province, China
3.
Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu Province, China

Funds: This work was supported by National Key Research and Development Program of China (2022YFB2804603), National Natural Science Foundation of China (62075096, 62005121, U21B2033), Leading Technology of Jiangsu Basic Research Plan (BK20192003), “333 Engineering” Research Project of Jiangsu Province (BRA2016407), Fundamental Research Funds for the Central Universities (30921011208, 30919011222, 30920032101), Fundamental Research Funds for the Central Universities (2023102001, 2024202002).

摘要

- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
- /
Abstract: Recent advances in imaging sensors and digital light projection technology have facilitated rapid progress in 3D optical sensing, enabling 3D surfaces of complex-shaped objects to be captured with high resolution and accuracy. Nevertheless, due to the inherent synchronous pattern projection and image acquisition mechanism, the temporal resolution of conventional structured light or fringe projection profilometry (FPP) based 3D imaging methods is still limited to the native detector frame rates. In this work, we demonstrate a new 3D imaging method, termed deep-learning-enabled multiplexed FPP (DLMFPP), that allows to achieve high-resolution and high-speed 3D imaging at near-one-order of magnitude-higher 3D frame rate with conventional low-speed cameras. By encoding temporal information in one multiplexed fringe pattern, DLMFPP harnesses deep neural networks embedded with Fourier transform, phase-shifting and ensemble learning to decompose the pattern and analyze separate fringes, furnishing a high signal-to-noise ratio and a ready-to-implement solution over conventional computational imaging techniques. We demonstrate this method by measuring different types of transient scenes, including rotating fan blades and bullet fired from a toy gun, at kHz using cameras of around 100 Hz. Experiential results establish that DLMFPP allows slow-scan cameras with their known advantages in terms of cost and spatial resolution to be used for high-speed 3D imaging tasks.

HTML全文

下载: 全尺寸图片幻灯片

参考文献(43)

[1]	Malamas EN, Petrakis EGM, Zervakis M, Petit L, Legat JD. A survey on industrial vision systems, applications and tools. Image Vision Comput. 2003;21(2):171–88.
[2]	Ford KR, Myer GD, Hewett TE. Reliability of landing 3D motion analysis: implications for longitudinal analyses. Med Sci Sports Exerc. 2007;39(11):2021.
[3]	Tiwari V, Sutton MA, McNeill SR. Assessment of High Speed Imaging Systems for 2D and 3D Deformation Measurements: Methodology Development and Validation. Exp Mech. 2007;47(4):561–79.
[4]	Gorthi SS, Rastogi P. Fringe projection techniques: whither we are? Optics Lasers Eng. 2010;48(2):133–40.
[5]	Li B, Wang Y, Dai J, Lohry W, Zhang S. Some recent advances on superfast 3D shape measurement with digital binary defocusing techniques. Optics Lasers Eng. 2014;54:236–46.
[6]	Zuo C, Chen Q, Feng S, Feng F, Gu G, Sui X. Optimized pulse width modulation pattern strategy for three-dimensional profilometry with projector defocusing. Appl Opt. 2012;51(19):4477–90.
[7]	Heist S, Lutzke P, Schmidt I, Dietrich P, Kühmstedt P, Tünnermann A, et al. High-speed three-dimensional shape measurement using GOBO projection. Opt Lasers Eng. 2016;87:90–6.
[8]	Heist S, Mann A, Kühmstedt P, Schreiber P, Notni G. Array projection of aperiodic sinusoidal fringes for high-speed three-dimensional shape measurement. Opt Eng. 2014;53(11):112208.
[9]	Caspar S, Honegger M, Rinner S, Lambelet P, Bach C, Ettemeyer A. High speed fringe projection for fast 3D inspection. In: Optical Measurement Systems for Industrial Inspection VII. vol. 8082. SPIE; 2011. p. 298–304.
[10]	Feng S, Zuo C, Tao T, Hu Y, Zhang M, Chen Q, et al. Robust dynamic 3-D measurements with motion-compensated phase-shifting profilometry. Optics Lasers Eng. 2018;103:127–38.
[11]	Liu K, Wang Y, Lau DL, Hao Q, Hassebrook LG. Dual-frequency pattern scheme for high-speed 3-D shape measurement. Opt Express. 2010;18(5):5229–44.
[12]	Zuo C, Chen Q, Gu G, Feng S, Feng F, Li R, et al. High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection. Optics Lasers Eng. 2013;51(8):953–60.
[13]	Tao T, Chen Q, Da J, Feng S, Hu Y, Zuo C. Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system. Opt Express. 2016;24(18):20253–69.
[14]	Zuo C, Tao T, Feng S, Huang L, Asundi A, Chen Q. Micro Fourier transform profilometry (μFTP): 3D shape measurement at 10,000 frames per second. Optics Lasers Eng. 2018;102:70–91.
[15]	Takeda M, Mutoh K. Fourier transform profilometry for the automatic measurement of 3-D object shapes. Appl Opt. 1983;22(24):3977.
[16]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.
[17]	Schmidhuber J. Deep learning in neural networks: An overview. Neural Netw. 2015;61:85–117.
[18]	Zuo C, Qian J, Feng S, Yin W, Li Y, Fan P, et al. Deep learning in optical metrology: a review. Light-Sci Appl. 2022;11(1):39.
[19]	Feng S, Chen Q, Gu G, Tao T, Zhang L, Hu Y, et al. Fringe pattern analysis using deep learning. Adv Photon. 2019;1(02):1.
[20]	Qian J, Feng S, Li Y, Tao T, Han J, Chen Q, et al. Single-shot absolute 3D shape measurement with deep-learning-based color fringe projection profilometry. Opt Lett. 2020;45(7):1842–5.
[21]	Qian J, Feng S, Tao T, Hu Y, Li Y, Chen Q, et al. Deep-learning-enabled geometric constraints and phase unwrapping for single-shot absolute 3D shape measurement. Apl Photon. 2020;5(4):046105.
[22]	Li Y, Qian J, Feng S, Chen Q, Zuo C. Deep-learning-enabled dual-frequency composite fringe projection profilometry for single-shot absolute 3D shape measurement. Opto-Electron Adv. 2022;5(5):210021.
[23]	Li Y, Qian J, Feng S, Chen Q, Zuo C. Composite fringe projection deep learning profilometry for single-shot absolute 3D shape measurement. Opt Express. 2022;30(3):3424–42.
[24]	Barbastathis G, Ozcan A, Situ G. On the use of deep learning for computational imaging. Optica. 2019;6(8):921–43.
[25]	Shaked NT, Micó V, Trusiak M, Kuś A, Mirsky SK. Off-axis digital holographic multiplexing for rapid wavefront acquisition and processing. Adv Opt Photon. 2020;12(3):556.
[26]	Zuo C, Feng S, Huang L, Tao T, Yin W, Chen Q. Phase shifting algorithms for fringe projection profilometry: A review. Opt Lasers Eng. 2018;109:23–59.
[27]	Feng S, Xiao Y, Yin W, Hu Y, Li Y, Zuo C, et al. Fringe-pattern analysis with ensemble deep learning. Adv Photon Nexus. 2023;2(3):036010.
[28]	Gao L, Liang J, Li C, Wang LV. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature. 2014;516(7529):74–7.
[29]	Yuan X, Brady DJ, Katsaggelos AK. Snapshot compressive imaging: theory, algorithms, and applications. IEEE Signal Proc Mag. 2021;38(2):65–88.
[30]	He Y, Yao Y, Qi D, He Y, Huang Z, Ding P, et al. Temporal compressive super-resolution microscopy at frame rate of 1200 frames per second and spatial resolution of 100 nm. Adv Photon. 2023;5(2):026003.
[31]	Qiao C, Li D, Liu Y, Zhang S, Liu K, Liu C, et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat Biotechnol. 2023;41(3):367–77.
[32]	Yin W, Che Y, Li X, Li M, Hu Y, Feng S, et al. Physics-informed deep learning for fringe pattern analysis. Opto-Electron Adv. 2024;7(1):230034–1.
[33]	Weise T, Leibe B, Van Gool L. Fast 3D Scanning with Automatic Motion Compensation. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis: IEEE; 2007. pp. 1–8.
[34]	Ibtehaz N, Rahman MS. MultiResUNet: rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw. 2020;121:74–87.
[35]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer; 2015. p. 234–241.
[36]	Zhang Z, Zhang B, Yuan X, Zheng S, Su X, Suo J, et al. From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth. PhotoniX. 2022;3(1):19.
[37]	Kellman MR, Bostan E, Repina NA, Waller L. Physics-based learned design: optimized coded-illumination for quantitative phase imaging. IEEE Trans Comput Imaging. 2019;5(3):344–53.
[38]	Wang F, Bian Y, Wang H, Lyu M, Pedrini G, Osten W, et al. Phase imaging with an untrained neural network. Light Sci Appl. 2020;9(1):77.
[39]	Bostan E, Heckel R, Chen M, Kellman M, Waller L. Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network. Optica. 2020;7(6):559–62.
[40]	Saba A, Gigli C, Ayoub AB, Psaltis D. Physics-informed neural networks for diffraction tomography. Adv Photon. 2022;4(6):066001.
[41]	Lin X, Rivenson Y, Yardimci NT, Veli M, Luo Y, Jarrahi M, et al. All-optical machine learning using diffractive deep neural networks. Science. 2018;361(6406):1004–8.
[42]	Liu J, Wu Q, Sui X, Chen Q, Gu G, Wang L, et al. Research progress in optical neural networks: theory, applications and developments. PhotoniX. 2021;2:1–39.
[43]	Luo Y, Zhao Y, Li J, Çetintaş E, Rivenson Y, Jarrahi M, et al. Computational imaging without a computer: seeing through random diffusers at the speed of light. ELight. 2022;2(1):4.