StyleGAN knows Normal, Depth, Albedo, and More

Summary

This paper proposes a novel method for obtaining "intrinsic images" for an RGB image, where in this context intrinsic images mean mainly depth, normal, segmentation, albedo, and shading maps. To this end, the authors investigate how a pre-trained StyleGAN generator model can be used to output the respective intrinsic image instead of an RGB image by only adding a constant in style weight space. More specifically, the proposed method optimizes an offset vector in the StyleGAN "w+" style space for a given editing constraint. The offset vectors are optimised via gradient descent where a comparably small dataset of generated images together with respective intrinsic images, predicted by SOTA models, is used as guidance information. The authors compare the proposed method to SOTA methods and find that it performs comparably.

Strengths

- The proposed approach is an interesting and valuable method for investigating what information is already contained in generative models such as StyleGAN. The finding that adding an offset vector in style weight space to output one of the other data modalities instead of RGB is surprising and very interesting. It highlights how much "semantic and 3D scene understanding" can be found in 2D generative models. - The authors investigate the performance of the same proposed method with multiple different data modalities highlighting the generality of the approach. - I appreciate that the authors also show an example decomposition that clearly does not work with the given method (L. 178). As only a style weight space vector is added, non-pixel-based transformations are expected to not work well / fail. - The authors compare against relevant baselines quantitatively and qualitatively and show helpful visual comparisons.

Weaknesses

- Performance across different datasets / Dependence on small labeled dataset: The proposed method requires a sub-set of "labeled" generated image and intrinsic image pairs, where the latter is provided by a SOTA method. Have the authors investigate how well the method performs if this "fine-tuning" is done on a different dataset, or on only a sub-set of the covered "modes"? More general, investigating different factors of this training stage with the resulting end performance, e.g. in form of a graph showing number of training images against performances, would be very interesting. - Labeled dataset: In similar spirit, how many labeled images (provided by the SOTA method) are used? (L. 124 - 126 is vague on this, and it would be beneficial to state clear numbers.) - Evaluation: The authors argue that no ground truth is available, hence SOTA predictions are used as GT. But couldn’t the test set of these SOTA methods be used as GT? For the proposed method, the input image would be needed to be inverted in StyleGAN space, but this has been extensively done before in the literature. Then, the proposed method could directly be compared to the SOTA prediction model itself. - Depth Comparison: It is unclear to me why the proposed method performs better in the L1 depth metric (Table 1) while it looks qualitatively worse than the other methods. Could the authors expand on this? I am not sure if this is caused by some scale/shift misalignments; as these are monocular predictors, the scale and shift is arbitrary, so this needs to be taken into account during evaluation. - Caption Fig 6: “Note that our quantitative comparison in Table 2 to a SOTA segmenter [17] likely understates how well StyleGAN can segment;” -> Can the authors expand on this? This is not clear to me. The authors give the two-lamp example, however, here all methods were able to segment the two lamps.

Questions

- How sensitive is the proposed system to the number of image "labeled" image pairs used for training? - Could the GT image and data modality pairs of the test set of the SOTA predictors be used as GT for evaluation? - Why is the proposed system better in the L1 depth metric while it looks worse qualitatively? - Is the same offset style vector d(c) used across all images for the same intrinsic image modality? - How many images of the "labeled" datasets are used for optimising the offset style vectors? For more context on these questions, please see the "Weakness" paragraph. The manuscript contains the following typos / other formatting errors: Typos: Caption Fig 1: “StyleGAN has easily accessed and accurate representations of intrinsic images” is unclear and grammatically wrong. What does this mean? Caption Fig 1: ““knows” fine detail in normal” -> “knows” fine detail in normal maps L. 49: “Showing that intrinsic images so extracted” -> “Showing that these intrinsic images extracted from StyleGAN … L. 54: “and produce blurry outputs” -> and producing blurry outputs L. 91: “significant recent methods” -> Important recent methods L. 92: “Learned methods” -> Learning-based methods Caption Fig 2: “We demonstrate a StyleGAN that is trained to generate images encode accessible scene property maps” -> […] is trained to generate images encodes accessible scene property maps Caption Fig 2: “Our approach explores [….], generate” -> Our approach explores […], generates L. 123: “a select number” -> a selected number L. 198: “We have demonstrated that StyleGAN has easily extracted representations” -> We have demonstrated […] has easily extractable representations L. 200: “be true of other generative models” -> be true for other generative models Other: Figure 1 and Figure 2 are not referenced in the text. Please add a reference to the main text.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- The authors show and discuss and example data modality / transformation that is cannot be tackled with the proposed approach due to the pixel-aligned features (L. 178). - The authors investigate the performance of the proposed system for different lighting conditions of the images (L. 184). - In the discussion (Sec. 7), the authors discuss unclear points that would require additional investigation. This includes to what extent these findings translate to other generative models and if more intrinsic image modalities next to the studies ones are contained in StyleGAN's weight space. - The authors do not discuss potential negative societal impact.

Summary

The paper proposed the method of extracting intrinsic images (Normal, Depth, Albedo, Shading, and Segment) from a pre-trained StyleGAN model without any pre-training or finetuning. It is empirically shown to be better than the SOTA for getting images intrinsic and robust to relighting effects, unlike SOTA.

Strengths

Showing that the StyleGAN W-space also learned the internal representation of intrinsic scene properties without being explicitly trained for the same. Proposed a simple, effective, and generalizable method for extracting intrinsic information without explicit training StyleGAN. Optimization based method is used to search the desired latent code for intrinsic image generation. Extracted intrinsic images as robust to lighting changes as compared to SOTA for extracting intrinsic images.

Weaknesses

This work can be considered as StyleGAN W-space exploration for intrinsic image generation. As mentioned in the paper, pre-trained StyleGAN is used, but all the quantitative and qualitative results are only shown for the LSUN Bedroom dataset. Does it applicable to other datasets, also such as the LSUN Church dataset, Animal Faces, FFHQ face, etc.? As per the method proposed in the paper, pre-trained intrinsic SOTA models are used for the optimization, so exploring W-spaces for these datasets directly depends on how well SOTA generalizes for other datasets. For real images, if we can use the SOTA models for intrinsic extraction directly, why is there a need for GAN inversion, as discussed in the paper? One of the purposes of GAN inversion is to give control over image editing of real images, and as per the problem statement in the paper, there is no such task of editing images based on intrinsic parameters. The proposed work uses the SOTA model for optimization, then it is possible for the SOTA model to make incorrect predictions for some images due to domain shifts in train-test data, and it can carry forward to latent space optimization as well as in quantitative and qualitative evaluation. Minor mistakes: On line 117, it should be w+’ not w’.

Questions

Refer above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The proposed method does not work on real images. The authors pointed out that this is due to GAN inversion, which does not preserve the parametrization of the W-space. With improvement in the GAN inversion method, the proposed approach can work on real images also.

Summary

The paper uses StyleGAN latent space properties to extract properties like normals, depth, segmentation, albedo and shading. The method does not train a new model to achieve this. The method relies on the properties of w+ space to predict these properties. In practice, the method uses the off-the-shelf predictors to use as pseudo labels and the signal is back-propagated to the w+ code. The authors evaluate their method on the LSUN bedroom dataset. The generated images and their corresponding predictions show that the StyleGAN is able to generate consistent results on bedrooms which suggests that the style codes can be modified to generate results in other domains not seen by the network.

Strengths

1) The paper explores a method to generate multiple properties of an image just using the w+ space of the StyleGAN. The paper shows that these properties like normals, depth, segmentation, albedo and shading can be produced by the latent space of the StyleGAN which is primally used for style transfer tasks. The results in the paper shows that the method produces consistent results on bedrooms. 2) The method works good on StyleGAN generated bedroom images. The paper shows multiple figures demonstrating consistency of the method and the qualitative evaluation. 3) The paper evaluates the method using the metrics of the corresponding tasks. The results show that the scores are not random and the method produces meaningful predictions of the properties.

Weaknesses

1) Firstly I feel that the abstract of the paper can be improved. To me it felt more like an introduction than an abstract. I would ask the authors to revisit the abstract and write it in the conventional way. What problem are you solving ? Why is it important/ what is the motivation? What is your solution? What results you show in the paper. In the current form it is not informative to me. 2) While the paper has proposed to solve some discriminative tasks using StyleGAN, there are other papers like DatasetGAN[1], Lables4Free [2] etc for segmentation and others related like [3] that already show that the features of the StyleGAN can be modified to achieve the goal or a dataset derived from StyleGAN can be used for the training. It is not clear if the method is generalizable especially when the previous methods have shown to exhibit these properties. How general is the method compared to these methods for example in segmentation domain? I would suggest adding a comparison. 3) I am not sure about the evaluations done in the paper. While pseudo labels for evaluation is a way to evaluate the results, another (more significant) way would be to train a supervised network (see Labels4free) based on the predictions of the StyleGAN. Then it can be tested on the real images rather than the generated ones. What about the inversion of a real photograph? Is the method generalizable to the real domain? The current evaluations do not show that. 4) The authors only test on LSUN bedrooms dataset. It is difficult to access the contribution of the paper if its evaluated on only one dataset especially when other pre-trained StyleGANs are available. Why did the authors only evaluation on this particular dataset? [1] Abdal, Rameen, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. "Labels4free: Unsupervised segmentation using stylegan." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13970-13979. 2021. [2] Zhang, Yuxuan, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. "Datasetgan: Efficient labeled data factory with minimal human effort." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155. 2021. [3] IMAGE GANS MEET DIFFERENTIABLE RENDERING FOR INVERSE GRAPHICS AND INTERPRETABLE 3D NEURAL RENDERING, ICLR 2021

Questions

Please look at the questions in the Weakness and the Limitation sections.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have not discussed the limitations of the work clearly. When does the method fail? Does the method work without the truncation of the latent space where the samples can be diverse but not high quality? How do the off-the-shelf predictors behave in this scenario? What about other datasets for example ImageNet, (StyleGAN-XL)?

Summary

The paper finds that for a GAN generator that is solely trained on images, intrinsic information such as depth, albedo, normal, etc. are implicitly learned during GAN’s training, even though such information is not provided as supervision. More surprisingly, such intrinsic information can be easily retrieved by calculating a direction offset of the latent codes via optimization an L1 loss on monocular predictor’s predictions on a set of generated images. This operation is not per-scene, means the offset is generalizable to the entire latent space.

Strengths

- Originality: To my knowledge, the work is an original finding. It is the first work to explicitly show that intrinsic information is implicitly learned during an image-trained StyleGAN. It is also novel in finding that the intrinsic can be retrieved by a very straightforward optimization that results in a constant additive direction in the latent space, without any finetuning of the generator itself. - Quality: The work is technically somewhat sound. The finding presented is appropriate and steps logically. Claims are supported via qualitative results, and quantitative experiments compared against SOTA monocular predictors. The authors have discussed the limitations of their method honestly and in much detail---I really like the “Discussion” part, which contains several questions which I raised too while reading the paper, and Figure.7, which shows a clear example of the limitation of the intrinsic nature of the presented claim. However, see my 1st comment in weaknesses for a factor that makes me hesitate to accept this paper. - Clarity: The paper is overall very clearly written. The claims and approaches are clearly presented and are also easily understandable, with many of my main concerns while skimming through the paper addressed in significant places. - Significance: I think this paper presented a fascinating finding. Researchers have been suspecting that the intrinsic information should be learned implicitly in the generator, but it is definitely surprising to see that the operation is as simple as a constant additive offset to the latent space that scales across scenes, without any further training. I believe the paper is a major result for the community.

Weaknesses

- I think the main issue with this paper comes from its narrow evaluation criteria. All experiments conducted in the paper are only based on bedroom images which may have a bias to support the claim of this paper. As the authors have pointed out, perhaps StyleGAN ‘knows’ intrinsic images because they are an efficient representation of what needs to be known to synthesize an image. It could be difficult to see the finding scale to other datasets, especially in the wild image datasets where there is no clear principal component. - In addition, the paper seems to be a solely ‘scientific finding’ paper, I have difficulty thinking about what aspect of this finding will be useful in terms of the application side since there is clearly still a performance gap with (even non-SOTA) predictors, and it needs SOTA monocular predictor to guidance the finding of the direction offset at the first place.

Questions

The experiments use SOTA monocular predictors as the “pseudo-ground truth”. However, wouldn’t it be a more appropriate way to evaluate some synthetic dataset with ground truth? This way, we can have a better idea of the gap between StyleGAN intrinsic & monocular estimators & ground truths.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have discussed the limitations of their method honestly and in very much detail.