Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images

Summary

The paper proposes an encoder-based approach to GAN-inversion for full 3D head reconstruction from a single image. To address the challenge of reconstructing the entire 360° head, the method employs two encoders: one for high-fidelity reconstruction of the input image (typically near-frontal) and another for generating realistic invisible parts of the head (e.g., back of the head). Finally, the paper presents a method to combine the representations learned from these two encoders to reconstruct a complete 3D head.

Strengths

+ The motivation for the problem and the proposed method seem reasonable. + Training a Discriminator directly on Triplane instead of the final images is an interesting and novel approach, differing from other GAN inversion work. This approach is also more relevant for full 360° reconstruction, rather than just near-frontal scenes. + The paper includes a good amount of qualitative comparison, ablation studies, and quantitative comparison.

Weaknesses

+ The head geometry after editing (Figure 8) appears to be quite flat compared to other results presented in the paper. + I would appreciate it if the authors could provide visualizations of the geometry of the generated results, in addition to the final generated RGB images.

Questions

+ In line 163-164, the paper mentioned: "This discrepancy may stem from the real samples originating from the generator, lacking the detailed feature characteristic of real-world images". Does this means that the reconstruction quality of the unobserved part is bounded by the capacity of Panohead, compared to Encoder 1 that is trained with real-world images? + How does the proposed method compare to the optimization-based inversion results presented in Panohead?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

+ The authors have acknowledged and discussed the limitations of the societal impact of their work. + I would appreciate it if the authors could provide more details on potential directions for improving the quality of the results or on other future directions.

Summary

This paper introduces an encoder-based method to do the GAN inversion task, especially for the full head inversion. The occlusion aware discriminator is interesting and reasonable. The results are good.

Strengths

1. The idea is interesting, I like the occlusion-aware discriminator. Meanwhile, training the discriminator in the triplane domain is also reasonable. 2. The results are good.

Weaknesses

There is no significant weakness in this paper. I have a question about the training process. Are these two encoders trained on the face first and then on the occluded parts? Is the occlusion-aware encoder initialized with the facial encoder?

Questions

see the weakness

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

see the weakness

Summary

This paper addresses the challenge of 3D GAN inversion from a single image, focusing on a method called PanoHead designed for generating 360 views of human heads. Unlike optimization-based approaches, this work adopts an encoder-based technique to reduce inversion times. The authors observe that varying encoder training objectives result in different advantages and disadvantages in the inversion output. Consequently, the authors propose using two encoders and introduce a stitching technique in the triplane feature space to combine the strengths of each encoder.

Strengths

1. The paper is well-written and easy to follow. 2. The ablation study effectively demonstrates the impact of each design choice, offering valuable insights. 3. The technique for stitching occluded and visible region features could potentially be useful in a broader context. e.g., in applications beyond human head modeling.

Weaknesses

1. The claim on lines 168-169, stating that triplane features associated with visible regions (e.g., a frontal face) are excluded from discriminator training and thus do not receive gradients, is not entirely accurate. The method forms a mask by back-projecting the depth map onto the triplane coordinates, which limits the mask to points lying on the surface of the visible region. However, since the model uses triplane features for volume rendering, the features used for rendering the frontal view are not necessarily confined to the surface of the face. Features behind that surface may also contribute to color aggregation during volume rendering, potentially leaving some features used for rendering the visible regions uncovered by the mask. 2. Some details lack clarity, as noted in the questions below.

Questions

1. Line 177: How do you obtain the depth map D? Do you use a monocular depth estimation method, or do you extract it from the NeRF model? 2. Figure 5: Are E1 the encoder from section 3.2 and E2 the encoder from section 3.3? If so, why is the triplane feature of E1 masked by the occluded regions and E2 masked by the visible regions? According to the paper, the proposed method appears to use the visible regions of the triplane for E1 and stitch them with the occluded regions of E2. Is this a mistake in the figure? 3. Line 223: The FID is calculated between the rendered image from the encoded input and "1k randomly synthesized" images. Which method is used to generate these "1k randomly synthesized" images? Is it PanoHead? Are all methods compared against the same set of images synthesized by the same method?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors included discussions on both the limitation and broader impact.

Summary

This paper introduces a novel encoder-based method for 3D pano-head GAN inversion. Unlike previous 3D GANs focused on near-frontal views, this work designs a dual-encoder system tailored for both high-fidelity reconstruction and realistic generation from multiple viewpoints. The method incorporates a stitching mechanism within the triplane domain, ensuring optimal predictions from both encoders. This results in superior performance compared to existing methods, both qualitatively and quantitatively.

Strengths

- Investigating GAN inversion for 360-degree GANs diverges from prior research as it necessitates accounting for more complex viewing conditions. - The dual-encoder design coupled with an occlusion-aware triplane discriminator proves to be intuitive and beneficial. - The approach outperforms existing methods, both in qualitative and quantitative assessments.

Weaknesses

- The input images in the study are primarily front-view; it would be beneficial to include some from extreme viewpoints. - Further assessment of the model's generalization is required for more complex input images, such as those with heavy makeup. - The quantitative experiments currently focus on single-view cases; incorporating results from other viewpoints may provide a more comprehensive evaluation.

Questions

- As discussed in Weaknesses. - Will the choice between a CNN-based or Transformer-based encoder affect the reconstruction results? Current works like GOAE, use transformer-based encoder for better different-view input instead of just front-view.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

- Discussion of failure cases is necessary. - The formula symbols in the paper need to be clearly distinguished, such as E1 and E2. Clear differentiation of these symbols should be included in the main text.

Summary

This paper presents a framework for 3D GAN inversion aimed at reconstructing high-fidelity 3D head models from single 2D images. A dual encoder system that combines one encoder specialized in high-fidelity reconstruction of the input view with another focused on generating realistic representations of invisible parts. An occlusion-aware triplane discriminator that enhances both fidelity and realism.

Strengths

The qualitative results show improvements over existing methods, especially for novel viewpoints.

Weaknesses

1. The writing and language of the paper need substantial improvement. 2. The innovation of this paper may be a little lacking. This paper proposes a new method, but the insights shown in the method are limited and it is difficult to generate insights into other problems. 3. There is also a lack of in-depth research on the proposed method. 4. While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.

Questions

1. From Figure 4, it can be seen that Encoder 2 may be the main solution to the problem in this paper. Is there any way to introduce the features of Encoder 1 into Encoder 2? It seems that Encoder 1 is more like a subsidiary of Encoder 2. For this reason, introducing two encoders is feasible, but not elegant enough. 2. I'm curious about more training details. For example, how much data was used for training? 3. The evaluation is primarily on face datasets (FFHQ, LPFF, CelebA-HQ). It's unclear how well the method generalizes to more diverse head shapes or ethnicities. 4. In Encoder 2, is the relevant information about the person's head implicitly included? Is there any experiment to prove this?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.

Re: Rebuttal by Authors

I have read the rebuttal and thank you for the clarifications, they are helpful. I will take the responses into consideration and discuss with other reviewers before making the final decision on my rating.