3D-GOI: 3D GAN Omni-Inversion for Multifaceted and Multi-object Editing

Response to Reviewer YC1j

Thank you for your time and encouraging review! We address your questions below: ## Weakness: >1.Still lies in the hybrid optimization method. Due to the limitations of current segmentation models, achieving 100% accurate segmentation is challenging. Additionally, GIRAFFE exhibits issues with multiple attribute codes that are interdependent, relying solely on the encoder fails to obtain accurate codes. Our approach of involving more variety of codes lead to accumulated inaccurate reconstructions using only the encoder (as shown in **Appendix D.3 Figure 14**), necessitating additional optimization. Using only the optimizer significantly prolongs reconstruction time (approximately 10 minutes or more) and may fail to reconstruct images from multiple codes. Therefore, we currently consider this hybrid optimization structure as the optimal choice. >2.Lacks qualitative comparison. Please find the comparison results in **Figure 14 in Appendix D.3.** >3.Limitation and how many objects can this method handle. The limitations are outlined in **Appendix E**. In our experiments, we have tested with up to 6 objects. In theory, if each object in the image adheres to the distribution of the training set, our method can be used to invert and edit as many objects as needed, restricted by available computation capacity of course. >4.Editing result based on manipulating the latent space. We want to thank the reviewer for bringing up this more granular editing. We will definitely consider it as an improvement strategy for enhancing the generation capabilities of GIRAFFE in the future. ## Questions: >1.Computational cost compared with existing encoder-based 3D GAN inversion framework. Due to the inapplicability of existing methods to multi-object image editing, we conducted tests on the same task of synthesizing new viewpoints for faces. Compared to encoder-based methods, 3D-GOI with multiple encoders requires a total of 169M parameters and 165 GFLOPs, while E3DGE[1] has 90M parameters and 50 GFLOPs, and TriplaneNet[2] has 247M parameters and 112 GFLOPs. Inference times on a single A100 GPU are about 0.1 second for all three methods. | | TriplaneNet (single encoder)| E3DGE(single encoder) | 3D-GOI(multi encoders) | | ---------- | --------------------- | --------------------- | ---------------------- | | parameters | 247M | 90M | 169M | | FLOPs | 112G | 50G | 165G | >2.Why single optimizer to optimize all the codes in Sec.3.4 will fail? As mentioned in **Appendix E**, we observed that in GIRAFFE, different object attribute codes are not entirely independent. For example, the shape code of an object can affect color control, and all attribute codes slightly impact the background as shown in **Figure 4**. Failed cases in simultaneous optimization primarily manifest as poor background reconstruction and errors in object rotation, color, and shape features. To address this, we employ multiple optimizers to tackle these issues. >3.shape collapse This is indeed a shared challenge for most if not all existing 3D GAN generation methods. We think this is because most 3D GAN methods now use NeRF to implicitly represent the 3D space, while ensuring 3D consistency and high-fidelity image generation, may struggle with accurate modeling of visible shapes and depths. Our method, as a reverse method based on 3D GAN, therefore inevitably also faces limitations in addressing these inherent issues. We think explicit 3D representations like dense point clouds or recently proposed 3D Gaussians could potentially improve the modeling of 3D shapes while maintaining generation effectiveness. >4.Better MSE,LPIPS but worse ID loss. The MSE and LPIPS metrics assess pixel-level and perceptual losses for the entire image, including the background. On the other hand, the ID loss (utilizing pre-trained Arcface for human faces and ResNet-50 for cars) evaluates object attribute similarity. As shown in **Figure 14 in Appendix D.3**, E3DGE outperforms TriplaneNet in reversing object attributes, especially in multi-car scenarios, while sometimes exhibiting weaker background inversion. Reference [1]Yushi Lan, Xuyi Meng, Shuai Yang, Chen Change Loy, and Bo Dai. Self-supervised geometry-aware encoder for style-based 3d gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20940–20949, 2023. [2]Ananta R Bhattarai, Matthias Nießner, and Artem Sevastopolsky. Triplanenet: An encoder for eg3d inversion. arXiv preprint arXiv:2303.13497, 2023.

Response to Reviewer gkt1

Thank you for your time and encouraging review! We address your questions below: ## Weakness: Thank you for this valuable feedback. Indeed, encoders that can capture precise codes wouldn't require additional optimization steps. Unfortunately, all mainstream GAN inversion encoders fail to accurately predict all codes in settings with multiple objects and numerous codes, as shown in **Figure 14** . As we discussed in the limitations, we found that in GIRAFFE, the attribute codes of different objects are not entirely independent. For instance, the shape code of an object can influence its color, and all attribute codes slightly affect the background as shown in **Figure 4**. Therefore, as the number of objects (and hence codes) increases, their interrelationships become more complex. Meanwhile the accuracy of encoder predictions varies across different attributes; yet only with precise predictions for all attributes can we accurately reconstruct the image. Therefore, using just the segmented images of objects seem to become a dead end for obtaining precise codes. As such, we propose to use the round-robin optimization algorithm to accurately determine the code values. >1.poor generalization to unseen data. In our tests on unseen vehicle images, we found that 3D-GOI performs well in inversing and multifaceted editing vehicles but faces challenges in reconstructing the background, resulting in less ideal background reconstruction. This is due to the difference between the background distributions of the dataset and the real-world. We plan to incorporate methods like optimizing the generator parameters on parallel, similar to PTI, in the future work. >2.Differentiating and visualizaing between closely related attributes Thank you very much for your suggestion. Visualizing all the codes is challenging for us because codes for different attributes reside in separate latent spaces. Additionally, GIRAFFE couples them together using affine transformations and a NeRF neural network, making it much more complex than the typical GAN process of mapping from latent space to images. We will consider this as part of our future work. >3.inaccurate segmentation We tested an inaccurate segmentation model with an AP50 of 0.7 (where the current SOTA is 0.8 in InternImage-H) as shown in **Appendix D.6** and **Figure 19**. Despite the coarse codes being less accurate, our round robin optimization algorithm was still able to refine them effectively. ## Questions: >1.How can the Neural Inversion Encoder's accuracy be improved? Improving the accuracy of the Neural Inversion Encoders is challenging, as our current architecture has undergone multiple refinements, confirmed by experiments (as shown in **Table 1** and **Figures 14, 15**) to show our encoder outperform the baselines. Our current multi-encoder structure, although using multiple encoders, achieves the highest accuracy. >2.computational costs Under the same experimental setup, compared to encoder-based inversion methods, 3D-GOI with multiple encoders requires a total of 169M parameters and 165G FLOPs, while E3DGE[1] has 90M parameters and 50G FLOPs, and TriplaneNet[2] has 247M parameters and 112G FLOPs. All three methods achieve inference times of approximately 0.1 second on a single A100 GPU.Compared to optimization-based methods, 3D-GOI's optimization process takes 30 seconds, while SPI[3] requires 550 seconds, and PTI[4] requires 55 seconds. || SPI | PTI | 3D-GOI | |----------------- | ----- | ------ | ------ | | optimization time | 550s | 55s | 30s | || TriplaneNet (single encoder)| E3DGE(single encoder) | 3D-GOI(multi encoders) | |---------- | ----------- | --------------------- | ---------------------- | | parameters| 247M | 90M| 169M| | FLOPs | 112G | 50G| 165G | >3.other applications where 3D-GOI could be particularly useful. Yes, 3D muli-object editing has broad applications. It can enhance film production, video game development, product design, interior design, and architecture by enabling realistic visual content creation and edition. Artists and content creators can leverage this framework to craft personalized and innovative 3D art and media. Reference [1]Yushi Lan, Xuyi Meng, Shuai Yang, Chen Change Loy, and Bo Dai. Self-supervised geometry-aware encoder for style-based 3d gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20940–20949, 2023. [2]Ananta R Bhattarai, Matthias Nießner, and Artem Sevastopolsky. Triplanenet: An encoder for eg3d inversion. arXiv preprint arXiv:2303.13497, 2023. [3]Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Ying Shan, Cengiz Oztireli, et al. 3d gan inversion with facial symmetry prior. arXiv preprint arXiv:2211.16927, 2022. [4]Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.

Response to Reviewer f1rX

Thank you for your time and encouraging review! We address your questions below: ### Weakness： >1.This paper still follows the framework of GIRAFFE. Our work fundamentally differs from GIRAFFE which generates images by initializing random codes as input, while our method takes multi-object scene images as input and map them back to the GIRAFFE code space. Therefore, although some parts of our method resemble GIRAFFE, there are fundamental mechanism differences. >2.Second paragraph of Intro: there are some other methods based on VAEs (Sync2Gen, ICCV'21) and transformers (NeurIPS'21) Thank you for this suggestion. We have added the discussion of mentioned references in the introduction section. >3.The segmentation method part is pretty intuitive. Thank you for this suggestion. We will shorten the relevant text in the final version. >4.How do you combine the two training strategies? Our approach first trains the Neural Inversion Encoders (NIE) for each object. Afterwards, we input the coarse code generated by the frozen NIEs to the optimizer and use the round-robin optimization algorithm to refine them into accurate codes. >5.The paper should be written in a more concise way. Thank you for this suggestion. We will streamline Section 3, including related work, more details of the encoder design, and the experimental part in the main text. ### Questions: > 1.Time efficiency of this method compared with the baselines. Our method is very time-efficient compared to methods that need optimization. For instance, to invert a single image, SPI[1] requires 550 seconds of optimization, PTI[2] requires 55 seconds and our 3D-GOI only needs 30s. This is primarily because, unlike other approaches, we only optimize the codes but not the generator parameters. | | SPI | PTI | 3D-GOI | | ----------------- | ------| ------ | -------- | | optimization time | 550s | 55s | 30s | Reference [1]Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Ying Shan, Cengiz Oztireli, et al. 3d gan inversion with facial symmetry prior. arXiv preprint arXiv:2211.16927, 2022. [2]Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.