Latent Intrinsics Emerge from Training to Relight

Summary

This paper proposes a fully data-driven relighting method applicable to images and real scenes. The approach requires only paired images of the same scene under different illumination as inputs. The trained model can also produce albedo-like maps even though it is not trained without such supervision. The experimental results show that it significantly outperforms unsupervised baselines while being competitive with baselines that use supervision.

Strengths

1. The idea and the overall achieved goal are quite novel. Training models that can do relighting without any supervision on ground-truth intrinsic properties (albedo, depth, normal, lighting, etc.) is quite challenging and useful. 2. The presentation is clear. The overall methodology is quite straightforward, with all the details (inputs, outputs, intermediate steps, losses) well-explained and self-contained. 3. The experiments are sufficient.

Weaknesses

1. The paper proposes several novel regularizations, e.g. Eq 6 and Eq 9, which are not ablated. It would be good to see how much these losses improve the model's accuracy.

Questions

Please see the weaknesses.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations and potential negative societal impact are discussed properly.

Summary

This paper proposes a fully data driven method to perform scene relighting that does not require any groundtruth lighting supervision. As such, it allows for scene relighting by training only on real paired images of the same scene under different illuminations rather than requiring synthetic data that contains the groundtruth lighting environments, which improves the generalizability and performance on real-world scenes. The method involves learning both intrinsic and extrinsic features from each scene, where the intrinsics represent the albedo and geometry and the extrinsics represent the lighting. The authors design a fairly simple but well thought out pipeline with several loss functions that encourage the intrinsics to be consistent for the same scene under different illumination, self reconstruction losses that reconstruct the original image using its own intrinsic and extrinsic features, and relighting loss functions that involve swapping extrinsic features between image pairs to perform scene relighting. The authors also design a constrained scaling mechanism that prevents too much information outside of the lighting information in the target image from being transferred to the source image. Experiments demonstrate that the relighting performance greatly outperforms other unsupervised methods that don't require groundtruth lighting and is comparable or slightly better depending on the metric than the existing state of the art supervised scene relighting methods. Ablations for the constrained scaling are also provided, which is helpful to understand the importance of this technical contribution.

Strengths

Experiments in the paper are quite thorough and demonstrate state-of-the-art performance for scene relighting, most notably being comparable to methods that require full light supervision and greatly outperforming unsupervised methods. Ablation studies on the constrained scaling are also helpful for understanding this contribution. The model and pipeline itself is quite simple and straightforward yet effective and easy to follow. The loss functions align with intuition and are carefully designed. The ability to estimate albedo from a scene is an important feature that has many implications for downstream tasks, and the authors achieve state-of-the-art performance for albedo estimation.

Weaknesses

One experiment that would be interesting to see is the level of albedo consistency. Since the authors design a loss function that encourages intrinsic features of the same scene under different illuminations to be similar, it would be good to see how well that design performs empirically, both quantitatively and qualitatively. Some citations are missing for the image-based relighting domain, namely: 1. Towards High Fidelity Face Relighting with Realistic Shadows (CVPR 2021) 2. Learning to Relight Portraits for Background Replacement (SIGGRAPH 2021) 3. Face Relighting with Geometrically Consistent Shadows (CVPR 2022) 4. Lumos: Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation (SIGGRAPH Asia 2022) 5. DiFaReli: Diffusion Face Relighting (ICCV 2023)

Questions

As I mentioned above, the experiments in this paper are quite thorough and I'm mostly interested to see whether the estimated albedo is consistent for the same scene under different illuminations. It would be good to compare this with other methods as well. Otherwise, please add the missing citations for the image-based relighting domain as there are many recent methods in that area.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations section is well thought out and touches on many important points. Relighting often does not have significant negative societal impacts since only the illumination is edited, but as with any other editing work there is always the potential to generate fake content. Perhaps this could be mentioned in the potential negative societal impact section.

Summary

The paper proposes a 2D relighting pipeline based on latent space manipulation, which is purely data-driven without explicit intrinsic representations such as geometry and materials. Given a single image input, the proposed model recovers latent variables representing scene intrinsic properties and latent variables representing lighting, enabling applications like 2D relighting based on a single reference image and albedo estimation. The proposed methods are validated on a held-out dataset, demonstrating its generalization capability and precision in real-world scenarios.

Strengths

1. The paper is well-written and easy to understand. 2. The idea of relighting images with latent latent intrinsic and without explicit explicit intrinsic representations is interesting and novel. The paper shows impressive results on the test datasets and outperforms the baselines for relighting and albedo estimation. 3. The proposed method doesn't need explicit lighting supervision and is purely data-driven, which indicates the method may have the potential to be scaled up in the future.

Weaknesses

1. using latent intrinsic to relight means the users lose the ability to control the relighting in a fine-grained way. For example, the user can't precisely control the lighting intensity and directions. 2. The proposed method doesn't show enough generalization ability. In Sec. 4.2, the authors first mention that they train the model on the Multi-illumination training dataset and then test it on the Multi-illumination test dataset. Then, when the authors validate the method on StyLitGAN images, the authors mention that they train the model again on StyLitGAN images. (as said in L213). It seems to mean that the trained model can't be used to test out of the distribution images directly. To show enough generalization ability, the model should be able to test on unseen real images directly after training ends.

Questions

What if the authors use the trained model to test some unseen images on the internet? Will the model still show good relighting ability? Can the checkpoint trained on the Multi-illumination dataset be used to infer StyLitGAN images? Some results would be appreciated.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations have been well discussed by the authors in the paper.