Conversion of 2D picture to color 3D holography using end-to-end convolutional neural network

Chenliang Chang; Chenzhou Zhao; Bo Dai; Qi Wang; Jun Xia; Songlin Zhuang; Dawei Zhang

doi:10.1186/s43074-025-00186-3

Article Contents

Article Navigation > PhotoniX > > Accepted Manuscript

Chenliang Chang, Chenzhou Zhao, Bo Dai, Qi Wang, Jun Xia, Songlin Zhuang, Dawei Zhang. Conversion of 2D picture to color 3D holography using end-to-end convolutional neural network[J]. PhotoniX. doi: 10.1186/s43074-025-00186-3

Citation:

PDF( 2544 KB)

Conversion of 2D picture to color 3D holography using end-to-end convolutional neural network

doi: 10.1186/s43074-025-00186-3

1.
School of Optical-Electrical and Computer Engineering, Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Laboratory of Modern Optics System, University of Shanghai for Science and Technology, Shanghai, 200093, China
2.
Joint International Research Laboratory of Information Display and Visualization, School of Electronic Science and Engineering, Southeast University, Nanjing, 210096, China

Funds: This work is supported by Science and Technology Commission of Shanghai Municipality (24511106500); Youth Innovation Promotion Association, Chinese Academy of Sciences (2022232); National Natural Science Foundation of China (62075040); National Key Research and Development Program of China (2021YFF0701100).

Received Date: 2025-03-15
Accepted Date: 2025-08-06
Rev Recd Date: 2025-06-04

Available Online: 2025-09-25

Abstract

Abstract

In the field of holographic 3D display, generating a three-dimensional (3D) computer-generated hologram (CGH) from a single two-dimensional (2D) image has been a significant challenge due to the high-dimensionality of the problem. In this paper, we introduce an end-to-end Convolutional Neural Network (CNN) framework, trained using a large dataset, which directly infers a full-color 3D CGH from a single 2D picture. The proposed method bypasses the need for depth or any other 3D information, facilitating the transformation of readily available 2D images into 3D holograms. We demonstrate that our end-to-end CNN can successfully convert either computer graphics (CG) generated 2D image or real-world captured 2D image into high-quality phase-only hologram, and experimentally achieving the effect of full-color 3D holographic display. Our work extends the horizons of lower-dimensional to higher-dimensional holographic wavefront information conversion, and therefore has potentials to advanced applications such as 3D display technology and metaverse development.
- Engineering",
- Holographic display')">" data-track="click" data-track-action="view keyword" data-track-label="link">Holographic display,
- Engineering",
- Computer-generated holography')">" data-track="click" data-track-action="view keyword" data-track-label="link">Computer-generated holography,
- Engineering",
- 3D display')">" data-track="click" data-track-action="view keyword" data-track-label="link">3D display,
- Engineering",
- Deep learning')">" data-track="click" data-track-action="view keyword" data-track-label="link">Deep learning

FullText(HTML)

References(54)

References

[1]	Blinder D, Birnbaum T, Ito T, Shimobaba T. The state-of-the-art in computer generated holography for 3D display. Light Adv Manuf. 2022;3:1.
[2]	Shi L, Ryu D, Matusik W. Ergonomic-Centric Holography: Optimizing Realism, Immersion, and Comfort for Holographic Display. Laser Photonics Rev. 2024;18:2300651.
[3]	Pi D, Liu J, Wang Y. Review of computer-generated hologram algorithms for color dynamic holographic three-dimensional display. Light Sci Appl. 2022;11:231.
[4]	Chang C, et al. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica. 2020;7:1563–78.
[5]	Maimone A, Georgiou A, Kollin JS. Holographic near-eye displays for virtual and augmented reality. ACM Trans Graph. 2017;36:1–16.
[6]	Jiao SM, Zhuang ZY, Zou WB. Fast computer generated hologram calculation with a mini look-up table incorporated with radial symmetric interpolation. Opt Express. 2017;25:112–23.
[7]	Tsang PWM, Poon TC, Wu YM. Review of fast methods for point-based computer-generated holography. Photonics Res. 2018;6:837–46.
[8]	Gilles A, Gioia P. Real-time layer-based computer-generated hologram calculation for the Fourier transform optical system. Appl Opt. 2018;57:8508–17.
[9]	Zhao Y, et al. Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method. Opt Express. 2015;23:25440–9.
[10]	Chen JS, Chu DP. Improved layer-based method for rapid hologram generation and real-time interactive holographic display applications. Opt Express. 2015;23:18143–55.
[11]	Zhang H, Cao L, Jin G. Computer-generated hologram with occlusion effect using layer-based processing. Appl Opt. 2017;56:F138–43.
[12]	Makey G, et al. Breaking crosstalk limits to dynamic holography using orthogonality of high-dimensional random vectors. Nat Photon. 2019;13:251–6.
[13]	Zhang YP, et al. Fast generation of full analytical polygon-based computer-generated holograms. Opt Express. 2018;26:19206–24.
[14]	Wang F, et al. Acceleration of polygon-based computer-generated holograms using look-up tables and reduction of the table size via principal component analysis. Opt Express. 2021;29:35442–55.
[15]	Ding S, Cao S, Zheng YF, Ewing RL. From image pair to a computer generated hologram for a real-world scene. Appl Opt. 2016;55:7583–92.
[16]	Lucente ME. Interactive computation of holograms using a look-up table. J Electron Imaging. 1993;2:28.
[17]	Kim SC, Kim ES. Effective generation of digital holograms of three-dimensional objects using a novel look-up table method. Appl Opt. 2008;47:D55–62.
[18]	Lucente M. Interactive three-dimensional holographic displays: seeing the future in depth. Comput Graph. 1997;31:63–7.
[19]	Shimobaba T, Masuda N, Ito T. Simple and fast calculation algorithm for computer-generated hologram with wavefront recording plane. Opt Lett. 2009;34:3133–5.
[20]	Shimobaba T, et al. Rapid calculation algorithm of Fresnel computer-generated-hologram using look-up table and wavefront-recording plane methods for three-dimensional display. Opt Express. 2010;18:19504–9.
[21]	Symeonidou A, Blinder D, Munteanu A, Schelkens P. Computer-generated holograms by multiple wavefront recording plane method with occlusion culling. Opt Express. 2015;23:22149–61.
[22]	Arai D, et al. Acceleration of computer-generated holograms using tilted wavefront recording plane method. Opt Express. 2015;23:1740–7.
[23]	Chang C, et al. From picture to 3D hologram: end-to-end learning of real-time 3D photorealistic hologram generation from 2D image input. Opt Lett. 2023;48:851–4.
[24]	Shi L, Li B, Kim C, Kellnhofer P, Matusik W. Towards real-time photorealistic 3D holography with deep neural networks. Nature. 2021;591:234–9.
[25]	Peng Y, Choi S, Padmanaban N, Wetzstein G. Neural holography with camera-in-the-loop training. Acm T Graphic. 2020;39:1–14.
[26]	Choi S, Gopakumar M, Peng Y, Kim J, Wetzstein G. Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays. Acm T Graphic. 2021;40:1–12.
[27]	Lee J, et al. Deep neural network for multi-depth hologram generation and its training strategy. Opt Express. 2020;28:27137–54.
[28]	Horisaki R, Nishizaki Y, Kitaguchi K, Saito M, Tanida J. Three-dimensional deeply generated holography. Appl Opt. 2021;60:A323–8.
[29]	Zheng H, Hu J, Zhou C, Wang X. Computing 3D Phase-Type Holograms Based on Deep Learning Method. Photonics. 2021;8:280.
[30]	Liu K, Wu J, He Z, Cao L. 4K-DMDNet: diffraction model-driven network for 4K computer-generated holography. Opto-electron Adv. 2023;6:220135–213.
[31]	Shui X, et al. Diffraction model-informed neural network for unsupervised layer-based computer-generated holography. Opt Express. 2022;30:44814–26.
[32]	Jin Z, et al. Vision transformer empowered physics-driven deep learning for omnidirectional three- dimensional holography. Opt Express. 2024;32:14394–404.
[33]	Shi L, Li B, Matusik W. End-to-end learning of 3D phase-only holograms for holographic display. Light Sci Appl. 2022;11:247.
[34]	Marrugo AG, Gao F, Zhang S. State-of-the-art active optical techniques for three-dimensional surface metrology: a review [Invited]. J Opt Soc Am A. 2020;37:B60–77.
[35]	Wang D, et al. Holographic capture and projection system of real object based on tunable zoom lens. PhotoniX. 2020;1:6.
[36]	Chang C, et al. Deep-learning-based computer-generated hologram from a stereo image pair. Opt Lett. 2022;47:1482–5.
[37]	Ishii Y, et al. Multi-depth hologram generation from two-dimensional images by deep learning. Opt Lasers Eng. 2023;170:0143–8166.
[38]	Liu N, et al. DGE-CNN: 2D-to-3D holographic display based on a depth gradient extracting module and ZCNN network. Opt Express. 2023;31:23867–76.
[39]	Yan X, et al. Generation of Multiple-Depth 3D Computer-Generated Holograms from 2D-Image-Datasets Trained CNN. Adv Sci. 2025;12:2198–3844.
[40]	Ranftl R, Bochkovskiy A, Koltun V. Vision Transformers for Dense Prediction. arXiv. 2021;arXiv:2103.13413. http://arxiv.org/abs/2103.13413.
[41]	Lin TY. et al. Microsoft COCO: Common Objects in Context. In Proc ECCV. 2014;740.
[42]	Zhou B. et al. Scene Parsing through ADE20K Dataset. In Proc CVPR. 2017;631–641.
[43]	Chen W, Fu Z, Yang D, Deng J. Single-Image Depth Perception in the Wild. 2016;arXiv:1604.03901 http://arxiv.org/abs/1604.03901
[44]	Vasiljevic I. et al. DIODE: A Dense Indoor and Outdoor DEpth Dataset. 2019;arXiv:1908.00463. http://arxiv.org/abs/1908.00463.
[45]	Saxena A, Sun M, Ng AY. Make3D: Learning 3D Scene Structure from a Single Still Image. In IEEE Trans Pattern Anal Mach Intell. 2009;31:824.
[46]	Silberman N, Hoiem D, Kohli P, Fergus R. Indoor segmentation and support inference from rgbd images. In Proc ECCV. 2012;746–760.
[47]	Mayer N. et al. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In Proc CVPR. 2016;4040–4048.
[48]	Qi Y, Chang C, Xia J. Speckleless holographic display by complex modulation based on double-phase method. Opt Express. 2016;24:30368–78.
[49]	Available online: www.bigbuckbunny.org.
[50]	Available online: http://sintel.is.tue.mpg.de.
[51]	Choi S. et al. Time-multiplexed neural holography: A flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators. ACM SIGGRAPH Conf Proc. 2022.
[52]	Yoo D, Jo Y, Nam SW, Chen C, Lee B. Optimization of computer-generated holograms featuring phase randomness control. Opt Lett. 2021;46:4769–72.
[53]	Lee B, et al. Wide-angle speckleless DMD holographic display using structured illumination with temporal multiplexing. Opt Lett. 2020;45:2148–51.
[54]	Choi S, Sitzmann V, Rubinstein M, Wetzstein G. Real-time neural holography for 4K spatial light modulators (ACM, Vancouver, Canada, 2022). ACM SIGGRAPH Conf Proc 2022;32.