NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

We present a method that takes as input a set of images of a scene\nilluminated by unconstrained known lighting, and produces as output a 3D\nrepresentation that can be rendered from novel viewpoints under arbitrary\nlighting conditions. Our method represents the scene as a continuous volumetric\nfunction parameterized as MLPs whose inputs are a 3D location and whose outputs\nare the following scene properties at that input location: volume density,\nsurface normal, material parameters, distance to the first surface intersection\nin any direction, and visibility of the external environment in any direction.\nTogether, these allow us to render novel views of the object under arbitrary\nlighting, including indirect illumination effects. The predicted visibility and\nsurface intersection fields are critical to our model's ability to simulate\ndirect and indirect illumination during training, because the brute-force\ntechniques used by prior work are intractable for lighting conditions outside\nof controlled setups with a single light. Our method outperforms alternative\napproaches for recovering relightable 3D scene representations, and performs\nwell in complex lighting settings that have posed a significant challenge to\nprior work.\n