Domain Re-Modulation for Few-Shot Generative Domain Adaptation

Summary

This work studies generative domain adaptation (GDA), on the StyleGAN2 architecture. Methods in this setting are commonly evaluated in terms of quality, diversity, and cross-domain consistency. This work claims to also be the first to explore the abilities of memory (“retain knowledge from previously learned domains”) and domain association (“integrate multiple learned domains and synthesize hybrid domains”). To tackle all these aspects, a method called DoRM is proposed. DoRM learns a new mapping network and affine transformations – M&A - (components of StyleGAN2) for each new domain. Hybrid domains are generated by interpolating the style codes of the original mapping and affine layers with the new ones.

Strengths

1. The paper is clearly written and easy to follow. 2. The tasks studied in this paper are interesting and relevant. 3. The proposed method is intuitive and novel, and as demonstrated by some experiments it is effective in performing domain adaptation.

Weaknesses

The claimed contributions are incorrect, and the evaluation is severely lacking. Both issues stem from outright ignoring the role and contributions of the most relevant previous works – HyperDomainNet [1], DynaGAN [18], Domain Expansion [24]. Below I list several examples of severe problems with claims and evaluations. All three works consider settings that cover the supposably new aspects of domain adaptation that are discussed in this paper. Both DynaGAN and HyperDomainNet rely on similar modulation techniques and achieve *”memory”* the same way. In Domain Expansion, the original domain is preserved on a dedicated subspace. Also, both DynaGAN and Domain Expansion discuss *”association”* in length (using the terms “domain interpolation” and “domain composition”). So, clearly, this work’s claims (lines 64, 247) of being the first to consider this setting are false. The previous fact is even somewhat acknowledged in the Related Work section (lines 99-105). The three works are said to “fall short in integrating multiple domains”. However, this claim is not supported by any experiment. Not only this one claim is not supported, but the authors choose to not compare their method with the three most relevant works - all of which have code on Github and all have been published well before the submission deadline (October 2022-January 2023). Instead, the method is compared to the more conventional setting of “single-domain, entire generator” adaptation, which is negligent at best. Comparing the generation of “hybrid” domains to CDC [25] which was not designed for this purpose (and published in 2021) and not to DynaGAN and Domain Expansion is unreasonable. Similarly, claiming superiority over CDC in terms of storage required is irrelevant. DoRM trains a mapping network (~6M params) for each new domain, while HyperDomainNet trains ~6K params, and Domain Expansion requires no additional weights whatsoever! The inaccurate contribution claims and lack of comparison to relevant previous works is fatal as it prevents the community and practitioners from understanding the landscape of works in this field. Perhaps the paper was written around the end of 2022 and is simply not up to date. I urge the authors to consider significantly rewriting this paper before resubmitting or arXiving a preprint. Additional issues: 1. Measuring consistency between domains using ArcFace – a face recognition network trained on faces -- makes little sense to me. For example, why would it output anything meaningful on sketches? 2. Missing citation to StyleSpace [Wu et al.] who introduced the $\mathcal{S}$ used in this paper (line 120).

Questions

1. Please clarify in what way is the setting that is supposably suggested was not already covered by HyperDomainNet, DynaGAN and Domain Expansion? 2. Why did you not include comparisons to these methods despite acknowledging that they are highly similar and relevant?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

One limitation is mentioned. However, several questions remain: 1. Can hybrids be produced by combining more than two domains? 2. Are the produced hybrids less faithful to each individual domain and to what extent?

Summary

This paper focuses on the concept of few-shot generative domain adaptation. Drawing inspiration from the human memory mechanism, the authors introduce a novel approach called DoRM (Domain-Adaptive Mapping and Affine Modules) to adapt the generator to a new domain. By incorporating new mapping networks and affine modules into the frozen source generator, they aim to enhance its adaptability. Additionally, the paper introduces a consistency loss based on CLIP image features to ensure the preservation of domain-sharing attributes during adaptation. Notably, the authors explore the intriguing concept of domain association in generative models, which involves merging the generative abilities of multiple domain-specific generators to enable generation in a new domain. Through a combination of mapping networks and affine modules within a single generator, the proposed DoRM generator successfully achieves domain association and multi-domain generation concurrently. To validate the effectiveness of their approach, the authors conduct extensive qualitative and quantitative experiments.

Strengths

(1) This paper presents a novel task: domain association, which involves combining the generative abilities of different domain-specific generators to enable generation in a new domain. This topic represents an exciting avenue for further exploration in the field of generative models. Notably, Figure 4 demonstrates impressive synthesis quality in the hybrid domain, particularly in the baby-sketch domain. (2) The proposed generator network structure, DoRM, is skillfully designed to facilitate domain association in a straightforward yet effective manner. By incorporating lightweight modules, the method achieves domain adaptation while preserving the generative abilities in the learned domains. This approach is both efficient in terms of storage and yields promising results. (3) The paper is excellently written and presents its concepts in a clear and understandable manner, making it easy for readers to follow along with the proposed methodology. (4) Through extensive qualitative and quantitative experiments, the paper demonstrates the superior performance of the proposed method compared to state-of-the-art techniques. Particularly impressive are the synthesis results achieved through domain association using the proposed method.

Weaknesses

Please view the Questions below.

Questions

(1) It is worth noting that the proposed generator structure bears similarities to the one introduced in [1]. This overlap may somewhat diminish the novelty of the proposed method. To enhance the clarity of differentiation, it would be beneficial if the authors provide a more comprehensive explanation highlighting the distinct features and advancements of their approach in comparison to [1]. (2) In addition to the proposed loss regularization, the authors have included supplementary regularizations, referred to as DoRM++, based on [2], in the supplementary material's one-shot experiments. Although the synthesis results of DoRM++ outperform those of [2], it is important to note that the proposed DoRM approach already addresses overfitting concerns by freezing the discriminator's backbone. Consequently, the inclusion of these supplementary loss components may be considered redundant. Therefore, it would be beneficial to conduct further qualitative and quantitative comparisons between DORM and DORM++ to gain a comprehensive understanding. (3) Section 4.3 would greatly benefit from further elaboration on the training details and specific datasets employed in the domain association experiments. Providing more specific information in these areas would enhance the reproducibility and understanding of the experimental setup for readers. (4) In recent times, the diffusion model has made remarkable strides in image generation, particularly in the area of few-shot image generation. Therefore, there is a need for further discussion regarding the application of the diffusion model in the context of few-shot Generative Domain Adaptation and domain associations. [1] HyperDomainNet: Universal Domain Adaptation for Training Generative Adversarial Networks. NeurIPS’22 [2] Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks. NIPS 2022

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations are discussed in the paper.

Summary

This paper proposes two advanced criteria for few-shot Generalized Domain Adaptation (GDA) inspired by the way human brains acquire knowledge in new domains: memory and domain association. To fully realize the potential of few-shot GDA, an innovative generator structure called Domain Re-Modulation (DoRM) is introduced. This structure freezes the source generator and incorporates new lightweight mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA, resulting in a linearly combinable domain shift in the style space. By incorporating multiple M&A modules, the generator gains the ability to perform high-fidelity multi-domain and hybrid-domain generation.

Strengths

1. This paper presents a novel exploration into the use of memory and domain association in few-shot GDA, which has not been previously explored. These advanced properties greatly reduce memory usage and simplify deployment, while also producing impressive results. 2. The proposed DoRM structure and similarity-based structure loss are innovative and advanced. 3. The figures in this paper effectively demonstrate the major contributions and the overall organization is clear and easy to follow. 4. The experiments conducted in this paper are robust and include both one-shot and 10-shot settings. Extensive testing demonstrates that our proposed method outperforms the previous state-of-the-art method in all five properties: quality, synthesis diversity, cross-domain consistency, memory, and domain association.

Weaknesses

There are a few areas where this paper could be improved. 1. It would benefit from including some recent studies on one-shot image generation [1]. 2. The paper should include experiments of DoRM++ in 10-shot GDA and DoRM in one-shot GDA, as this ablation experiment would help to demonstrate the advancement of the mentioned losses and the impact of data size on GDA tasks. 3. The paper lacks a future outlook, and it would be helpful to explore more possible improvement directions, especially as the effect of domain association on some datasets is not always satisfactory. [1] StyO: Stylize Your Face in Only One-Shot

Questions

See weakness

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

NA

Summary

The paper introduces a novel approach for domain adaptation of StyleGAN2 called Domain Re-Modulation, which is a few-shot technique. The authors argue that this method draws inspiration from the workings of the human brain. To achieve the desired domain shift, the paper utilizes the stylespace of StyleGAN along with adversarial and consistency losses. Additionally, a structural similarity loss is incorporated, which is based on the auto-correlation map extracted from the CLIP encoder. The effectiveness of the proposed method is evaluated through both quantitative and qualitative analyses. Furthermore, the paper includes comparisons with other state-of-the-art StyleGAN-based methods, highlighting its advancements. The authors also conduct an ablation study to analyze the individual components employed in their approach.

Strengths

* The paper introduces the concept of hybrid domain generation, which involves combining the affine layers of StyleGAN to generate results from multiple domains using a single generator. The authors demonstrate that this approach yields consistent and generalizable outcomes across multiple domains. * Notably, the method proves effective even in scenarios with limited training data, such as 1-shot training. By leveraging the StyleGAN2 backbone, the proposed technique achieves reliable and coherent generation outcomes. * In terms of evaluation, the paper compares its approach with various StyleGAN-based methods. It provides both qualitative and quantitative results to assess the performance of the proposed method. Furthermore, the paper includes comparisons with related methods to highlight the advantages and advancements of the proposed domain generation approach.

Weaknesses

1. The paper lacks substantial novelty as there exist other works such as StyleCLIP and 3DAvatarGAN that demonstrate the capability of performing image editing and few-shot domain adaptation using the "s" space of StyleGAN. Although the paper incorporates a CLIP-based loss, it fails to provide a comparative analysis with papers falling under the same category, such as StyleGAN-NADA and Mind the GAP HyperStyle. The absence of such comparisons and explanations regarding the differences between the methods makes it challenging to determine the uniqueness of the paper. To assess how the current method surpasses these works, it is essential to consider the individual components and techniques utilized in each approach. By conducting a thorough comparison, the paper could highlight the specific strengths of the proposed method that outperform other works. This would provide a clearer understanding of the paper's novelty and identify the components that excel in comparison to existing methods. Hence, it is imperative for the paper to address these concerns by including a detailed comparison that elucidates how the current method improves upon or surpasses existing approaches, specifically identifying the components that outperform these methods. How is the current method better than these works? Which component of the method outperforms these methods? 2. Why did the authors use 2D generators especially when 3D StyleGAN based generators are available trained on the same data. Besides the concerns above, it would be interesting to see how the method will perform in the 3D -GAN domain. What are some additional challenges in that domain? 3. How is the editability of the generator after the domain adaptation. Since there are no edits performed, it is difficult to assess if the latent space properties of the generator are preserved or it just overfits the given Styles. There are other datasets like AFHQ Cats, Dogs, Cars that are tested for such few shot domain adaptation tasks in the StyleGAN domain. The method seems specific to the face domain. What about the results on these domains? How generalizable is the method to these domains? 4. The artistic domains are quite subjective. It is not fair to just evaluate quantitative metrics for such tasks. I would suggest conducting a user study. It would be better to embed real images in the generators and ask users about the similarity, consistency and identity preservation of the target images.

Questions

Refer to Weakness section

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors discuss the ethics and limitations.

Summary

This paper proposes a novel approach for few-shot generation using a lightweight GAN architecture and a new loss function. The method is capable of handling multi-domain and hybrid domain tasks with a single model. The experiments demonstrate the superior performance of the proposed method in terms of both qualitative and quantitative results.

Strengths

1. The paper addresses a novel task and presents a unique approach to handle it. 2. The experiments are comprehensive and well-organized, providing strong evidence for the effectiveness of the proposed method. 3. The proposed method outperforms previous methods in terms of both qualitative and quantitative results. 4. The exploration of the generative ability to merge two or three domains for few-shot generation is interesting and innovative.

Weaknesses

1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association. 2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method. 3. The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures. 4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4]. 5. This paper utilized adversarial training while freezing the discriminator's backbone. However, in my understanding, the discriminator may still overfit the training data due to the adversarial loss. This issue has also been observed in comparable works, like CDC, where the adversarial loss weight is only 1 while the consistent loss weight is 1000, but the overfitting problem persists as training progresses. How about of the proposed approach? [1] Hyperdomainnet: Universal domain adaptation for generative adversarial networks. NIPS 2022 [2] Dynagan: Dynamic few-shot adaptation of gans to multiple domains. SIGGRAPH Asia 2022 [3] Minegan: effective knowledge transfer from gans to target domains with few images. CVPR 2020 [4] Few-shot image generation with elastic weight consolidation. NIPS 2020

Questions

1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association. 2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method. 3. The generalization of the proposed method should be discussesd. 4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4].

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Authors have addressed some limitation of this paper. More Limitation should be further addressed: The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures.

Comment on rebuttal

Thanks for the rebuttal. I’m listing a few follow-up questions: 1. Can you please share the revised set of contributions, as you plan to present them in the paper? 2. Evaluation of hybrids - I agree that evaluating the fidelity of a hybrid domain with respect to each of the domains composing it is not trivial. Nevertheless, as this is one of the selling points of the paper, I would have expected the paper to include it. Domain Expansion includes such an experiment, for example. Also, until such an evaluation is performed, I suggest not to dismiss DynaGAN’s interpolation results. A midpoint in interpolation is clearly not a perfect hybrid but does include characteristics of both domains. It is not clear to me that the hybrids produced by this method would be superior. 3. Baselines - * Can you please explain why HyperDomainNet using a third-party StyleGAN implementation is a reason to not compare with it? Especially given that this specific implementation was abundantly used by previous works. * How were the Domain Expansion results in the attached Fig. 6 produced? I’m assuming it uses CLIP and text and thus is not exactly an apples-to-apples comparison. I think the best baseline would be applying CDC within the expansion framework. I acknowledge that this requires some additional coding, but from a practical standpoint, if the results of this baseline are better, I’m not sure what the contribution of this paper would be.

Response to the Rebuttal

Thank you for the detailed rebuttal and clarifications provided in response to my initial review. Your explanations have significantly addressed my concerns and have deepened my understanding of the paper's contributions. I appreciate your thoroughness and look forward to seeing the refined version of your work.

Please read the authors’ response and other reviewers’ comments, and give your final ratings

Dear Reviewers, Thanks again for your time and support. As the author-reviewer discussion is closed on Aug 21st 1pm EDT, please read the authors’ response to see whether your concerns have been addressed or not. If not, the authors may still have time to give more results or clarifications. Furthermore, You can read other reviewers’ comments and the corresponding responses, then give your final rating. Best, AC