Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality

The representation of consistent mixed reality (XR) environments requires\nadequate real and virtual illumination composition in real-time. Estimating the\nlighting of a real scenario is still a challenge. Due to the ill-posed nature\nof the problem, classical inverse-rendering techniques tackle the problem for\nsimple lighting setups. However, those assumptions do not satisfy the current\nstate-of-art in computer graphics and XR applications. While many recent works\nsolve the problem using machine learning techniques to estimate the environment\nlight and scene's materials, most of them are limited to geometry or previous\nknowledge. This paper presents a CNN-based model to estimate complex lighting\nfor mixed reality environments with no previous information about the scene. We\nmodel the environment illumination using a set of spherical harmonics (SH)\nenvironment lighting, capable of efficiently represent area lighting. We\npropose a new CNN architecture that inputs an RGB image and recognizes, in\nreal-time, the environment lighting. Unlike previous CNN-based lighting\nestimation methods, we propose using a highly optimized deep neural network\narchitecture, with a reduced number of parameters, that can learn high complex\nlighting scenarios from real-world high-dynamic-range (HDR) environment images.\nWe show in the experiments that the CNN architecture can predict the\nenvironment lighting with an average mean squared error (MSE) of \\num{7.85e-04}\nwhen comparing SH lighting coefficients. We validate our model in a variety of\nmixed reality scenarios. Furthermore, we present qualitative results comparing\nrelights of real-world scenes.\n