IllumiNeRF: 3D Relighting Without Inverse Rendering

Summary

This paper introduces a new paradigm for the relightable 3D reconstruction: leverage a Relighting Diffusion Model to generate relit images and distill them into a latent NeRF. The proposed paradigm demonstrates superior performance compared to existing methods, which are based on inverse rendering, on both sythetic and real-world datasets.

Strengths

- The writing is clear and the content is well-structured. The authors provide comprehensive implementation details. - The paper addresses a long-standing challenge by introducing a novel paradigm, which diverges from conventional inverse rendering methods. This approach offers a promising alternative for relighting. - Thorough numerical experiments showcase the effectiveness of the proposed method. The results are convincing.

Weaknesses

One major weakness of this method is its efficiency. Unlike traditional methods that can directly relight a 3D scene once the inverse rendering is performed, the proposed approach requires optimizing a NeRF for each new target lighting condition. Furthermore, the computation of the latent NeRF involves a significant time investment (0.75 hours using 16 A100 GPUs). It would be beneficial to discuss potential solutions for this issue.

Questions

- How many samples ($S$) were used during the experiments presented in Table 1 and Table 2? - As this method models the probability of relit images and fixes $Z=0$ during inference, it would be interesting to investigate how the performance varies by sampling different $Z$ values. - Typo in Line $446$

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations have been well discussed.

Summary

The paper proposes a method for novel view synthesis with novel lighting given multi-view observation of an object. The pipeline is composed of a learned single-image-based relighting model based on image diffusion models, and a latent NeRF model that reconstructs the appearance of the relit object by reconciling relit images across views and samples. The relighting diffusion model (RDM) is conditioned on radiance cues generated from the geometry and target lighting under samples of roughness values, using a finetuned ControlNet architecture. The method is evaluated on both synthetic and real-world scenes, against baseline methods (mostly object-centric inverse rendering models), with both qualitative and quantitative results. Ablation study is also provided on design choices (e.g. latent/no latent NeRF, and number of diffusion samples).

Strengths

[1] An alternative approach to object-centric relighting without optimization-based inverse rendering. The main novelty of the method which distinguishes it from previous methods of the task is, the method leverages a learned image-based diffusion model to directly predict the relit images, and then reconstructs 3D representation of the relit object for novel view synthesis. This approach opens up a new venue for solving the task besides baseline methods. The diffusion model is conditioned on 'radiance cues' as an input lighting proxy, which itself is not new in a learning-based inverse rendering setting, but the method is able to retarget the lighting proxy as condition for a diffusion model, which might inspire future works using diffusion models to hallucinate lighting. [2] The design of the Latent NeRF to reconcile sampled relit results from the diffusion model. The paper demonstrates that the latent NeRF is able to better reconcile sampled results from a diffusion model while preserving the details and decent specular highlights. [3] Extensive evaluation. The paper is able to include extensive evaluation on both real-world and synthetic datasets on various metrics.

Weaknesses

[1] Effectiveness of the latent NeRF. The paper claims that the latent NeRF is able to reconcile difference in the sampled results from RDM. However, insufficient analysis is provided for justification. For instance, in Fig. 4, results in (a) seem to present more specularity than what is actually present in the hotdog scene (in other words, RDM seems to produce unrealistic relit images), and results from the latent NeRF in (b) seems to be an averaged version of the samples, with most of the specularity averaged out, arriving at a mostly diffuse appearance except for areas where specularity all agree among sampled. As a result, in Fig. 6, when compared to inverse rendering methods like Neural-PBIR, despite that the proposed method is able to produce better specularity in highlighted areas (thanks to explicitly conditioning on radiance cues), the results of the proposed method look 'dull', where shadows and high-frequency details seems to be lacking. This casts questions into the performance of both RDM to produce consistent and photorealistic relit images, as well as the ability of the latent NeRF to successfully reconcile inconsistency in the sampled results without sacrificing details and shadows. [2] Claim on better secularity in the results is questionable. The paper explains that, when compared to Neural-PBIR the proposed method is in second place numerically because the approximate lighting ground truth from the light probe unfairly favors Neural-PBIR despite the proposed method produced more convincing specularity. However this is largely unjustified, as (1) it is not clear how much weight is on the specular pixels in the overall metrics (2) whether the proposed method actually produces better specularity in the absence of true ground truth of relit objects. Moreover as mentioned in [1], the proposed method seems to lose more details (e.g. in the spikes of the cactus, or details in the ball). In this case, even if the methods excels in specularity, this is not the single most important property that is accounted for in those metrics, or visually, when evaluating the quality of the results.

Questions

[1] Quality of the sampled results from RDM. Why are some results look overly specular (e.g. the bread in the hotdog scene in Fig. 4). And if more examples can be provided on results from RDM? [2] Reconciliation by the latent NeRF. Does the Latent NeRF cost details and specularity in sampled results from RDM when reconciling the results into a NeRF representation? Also if there are view-dependent effects in the sampled results, does the latent NeRF properly preserve those effects? Again, more examples are to be provided in order to further examine the behavior of the two modules beyond one single scene (and especially for synthetic scenes, where 'true' ground truth of relit objects is available). There are additional results on Standford-ORB scenes in the Supp but analysis is needed in the main paper to support the method using those results. [3] Additional ablation. Is it helpful to use more or fewer roughness levels in generating radiance cues? [4] In designing baselines, why not comparing against ALL methods in both synthetic and real-world settings?

Rating

4

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

NA

Summary

The paper proposes a method to relight NeRF representations from a set of multi-view input images. The paper first trains a NeRF model and recovers geometry using UniSDF from the input images. Then, from the mesh, they obtain radiance cues which are used to condition a diffusion model to sample relit images of the scene under a target lighting. Since these relit images are samples from a distribution of plausible relit images, the sampled results are used to train a latent code conditioned NeRF model, and the lighting of the scene can be manipulated by walking through the latent space. The core contribution of the paper is to use the relit images rather than trying to explicitly model and learn the lighting and material properties.

Strengths

1. The paper shows that instead of recovering the material properties, it is enough to relight the training images and optimize the appearance to reach state-of-the-art results. The comparisons demonstrate that the results are state-of-the-art, which is important for improving the task of object relighting. 2. The diffusion model trained is described in detail and seems novel for the task of relighting. Details on the radiance cues and the architecture and training process of the diffusion model are described in detail. The model can be trained on synthetic data and applied to real-world data. 3. The lighting is multi-view consistent with the help of the latent-NeRF model 4. The presentation of the paper is clear, description of the method is high-quality, and comparisons are detailed

Weaknesses

1. The significance of using diffusion models to relight images, which are then used to train a NeRF, is limited to the context of application to NeRFs. Concurrent works have very similar methods in the context of Gaussian splatting (https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/) - what is the benefit of this approach over those? Some discussion should be made on this. 2. The pipeline seems brittle, requiring many steps to go right and limited by the sum of limitations of many different technologies. The quality of relighting is not only limited to the quality of the diffusion model output but also depends on the quality of the UniSDF mesh reconstruction, which is required to create the radiance cues. This means that a good capture is a necessary prerequisite to relight with this method, and one could simply relight the extracted mesh to achieve a similar effect. 3. I find the abstract a bit misleading: “we first relight each input image using an image diffusion model conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting.” This description doesn’t accurately represent what actually happens, as it leaves out the most crucial part of the pipeline (reconstruction + conditioning) and instead suggests that the NeRF can be directly reconstructed from the relit images (Compare figure 2 of the paper). 4. The choice of UniSDF is neither justified nor are alternatives discussed. The part describing this aspect of the approach is relegated to the appendix. Consequently, there is no ablation study to determine if other reconstruction methods might work better. 5. With respect to figure 6: It makes sense to highlight the same area for all depicted approaches. For instance, the blocks and the ball have no highlights, but other approaches seem to perform better in these areas. Additionally, the normalization technique used to match the ground truth images is not ablated. Does this make a difference in the reconstructed quality? I notice that many of the other methods are darker than I’d expect and I wonder if this introduces some artifacts which hurt these methods and promote the proposed method. 6. It is unclear why the paper limits itself only to objects. UniSDF clearly shows good performance for scenes. The authors don’t discuss this limitation in the paper. 7. Changing the target light even slightly requires a wait of 45 minutes for a final rendering. Other approaches, as referenced above, seem to not require that much time.

Questions

Many questions are discussed in the weaknesses, specifically: 1. How sensitive is the paper to the UniSDF reconstruction pipeline? Are there better methods which exist? Why does the paper only work for objects? 2. How does the normalization to ground-truth images for visualization of relit images actually affect the quality?

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Some limitations are discussed in the paper, but there are additional ones which should be discussed in more detail, such as the long reconstruction time to get new lighting conditions.

Summary

The paper proposes a new method called illumiNeRF for 3D relighting — given a set of posed input images under an unknown lighting, illumiNeRF produces novel views relit under a target lighting. Most of existing methods use inverse rendering to first recover the material and lighting in the given images, and apply new lighting to the object. The authors point out there is inherent ambiguity in decomposing materials and lighting. Instead, they propose a probabilistic model of representing multiple possible materials as latents, and train a latent NeRF using a relighting diffusion model. The authors compare their method with existing baselines and outperform most of them.

Strengths

- The problem formulation in sec 3.1 is explained clearly. The authors point out the inherent ambiguity in inverse rendering problems when decomposing materials and lighting. - The probabilistic model of representing multiple possible materials as latents in NeRF is very interesting.

Weaknesses

- Although during training, multiple materials are considered as different Z when optimizing the latent-NeRF, Z is manually set to 0 at test time, which corresponds to a specific explanation. However, Z=0 does not seem to represent the most likely material because the authors assume a uniform prior over Z according to line 163. - From Figure 6, the results lose a lot of details compared to other baselines like Neural-PBIR. For example, the method lost details of the spikes in the cactus sample. In table 2, Neural-PBIR also outperforms the proposed method.

Questions

It would be great to show 3D renderings with different latent explanations in Figure 4(b).

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Discussion

Dear reviewer q4ex We received diverse ratings for this submission, and you were the only one with a negative initial review. Please review the rebuttal and the comments from other reviewers and join the discussion as soon as possible. Thank you! Your AC