Balanced Training for Sparse GANs

Summary

This work proposes a metric named balance ratio to represent the balance between the generator and discriminator in dynamic sparse training, and furthermore proposes balanced dynamic sparse training to balance the performance and computation cost.

Strengths

(++) There are not many previous works that tried to apply DST to GANs, so this research is very valuable, in my opinion. The insight of the problem of balance between the generator and discriminator is also accurate. (++) The motivation is clearly explained and discussed well.

Weaknesses

(----) The experiments are the main problem. The baselines (SNGAN, BigGAN) seem to be out of date. The involved datasets are all small, but larger images (e.g., FFHQ) are not utilized. Recently, GAN frameworks can even generate images larger than 1024x1024. Generating small images is not so challenging today, as powerful frameworks, e.g., diffusion models, have achieved great success. In my opinion, at least 256x256 images should be considered to show the value of the proposed method.

Questions

Could you please give a reasonable explanation for only involving small datasets? If there is a reason I did not notice, I will correspondingly update my score.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

4 excellent

Limitations

I did not find the discussion of limitations. I also did not notice the potential limitations of this work.

Summary

This paper presents a method for dynamic sparse training for GANs. In particular, the authors propose the balance ratio to study the balance status between the generator and discriminator. In addition, a balanced dynamic sparse training strategy is designed by applying BR to achieve a good trade-off between performance and computational cost. Experimental results proved the effectiveness of the proposed method.

Strengths

(1) The motivation of balancing sparse GAN training under resource budget is well presented. (2) The explanations and illustrations of the balance ratio and balanced dynamic sparse training are well-formulated and mostly clear and intuitive.

Weaknesses

(1) The motivation of applying DST to GAN is not that novel to the community and the authors listed STU-GAN as an example. In addition, the experimental improvements over STU-GAN is trivial according to Table 1-3. (2) The normalized training FLOPs in Table 3 fluctuate a lot and are not linearly consistent with the generator density. It contradicts the basic belief of balanced model size between the generator and discriminator for stable GAN training. (3) The DST strategy is in-time over-parameterization, what is the difference to a dynamic-ratio Dropout? (4) When the DST strategy is applied, the balance ratio cannot explain away other important factors for the GAN performance. For example, when spectral normalization is used in the discriminator, the GAN training can be very stable among a lot of practice even for unbalanced model size between the generator and discriminator.

Questions

No.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No.

Summary

The paper addresses the challenge of reducing the computational complexity of training GANs by leveraging dynamic sparse training (DST) techniques. The authors propose a novel metric called the balance ratio (BR) to quantify the balance between the sparse generator and discriminator during GAN training. They also introduce a method called balanced dynamic sparse training (ADAPT) to control the BR and achieve a balance between performance and computational cost. The paper begins by providing a thorough background on GANs, sparse training, and the challenges associated with applying DST to GANs. The motivation for the research is clearly explained, emphasizing the need for efficient training methods without sacrificing performance. The proposed metric, BR, is introduced and its significance in measuring the balance between the generator and discriminator is well-established. The methodology section is comprehensive, detailing the steps involved in ADAPT. The authors describe the specific modifications to the GAN training process, incorporating sparse training techniques and controlling the BR. The mathematical formulations and algorithms are clearly presented, making it easier to understand the implementation details. The paper is generally well-written and structured, with a logical flow of ideas. However, there are a few areas where the clarity could be improved. In some sections, the technical details and explanations are a bit dense, making it challenging for readers unfamiliar with the topic to follow along. Providing more intuitive explanations or examples could enhance the accessibility of the paper.

Strengths

1. The paper is well-motivated. 2. Using balance-ratio as a metric to understand generator and discriminator sparsity and affects is interesting and seems very useful. 3. The performance improvement over STATIC demonstrate the effectiveness of DDST.

Weaknesses

The technical details and explanations are a bit dense. Providing more intuitive explanations or examples could enhance the accessibility of the paper. Not required but this can be part of Appendix.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Summary

Motivated by the identified imbalance between the generator and discriminator during sparse GAN training, this work proposes a quantitative metric dubbed balance ratio as an indicator for the degree of balance in sparse GAN training. Leveraging this metric, this work further proposes the ADAPT framework to dynamically adjust the sparsity of the discriminator towards balanced GAN training. Experiments across various datasets validate the effectiveness of the proposed method in achieving a good trade-off between performance and computational cost.

Strengths

1. This work is well-written with a clear logical flow. The well-organized structure from observations to understandings and solutions is appreciated. 2. The proposed balance ratio can well indicate the balance between the generator and discriminator during GAN training, which can potentially serve as a useful metric for the community.

Weaknesses

1. The imbalance between the generator and discriminator has been extensively studied and it is not clear why the proposed quantitative metric (i.e., the balance ratio) can outperform previous indicators or solutions under the scenario of sparse GAN training. The authors are expected to provide a literature review and discuss the key advantage of the proposed method that makes it particularly suitable for sparse GAN training. 2. It is not clear what is the rationale for proposing the two variants ADAPT_relax and ADAPT_strict. What is the expectation of the performance ranking between the two according to the claim "a more interesting observation is that ADAPT_strict sometimes outperforms ADAPT_relax"? 3. For the experimental results, only limited/insufficient baselines are considered. The authors are expected to benchmark the proposed framework with other sparse GAN training methods, e.g., [1][2] cited below. [1] "Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective", T. Chen et al., NeurIPS'21. [2] "Don't be so dense: sparse-to-sparse gan training without sacrificing performance", T. Chen et al., IJCV'23. 4. Since only unstructured sparsity is considered, the reported FLOPs reduction cannot be turned into real-device speedup. The authors are expected to perform experiments under structured sparsity to validate whether the proposed method is still effective.

Questions

I have listed my questions in the weakness section. I am willing to adjust my scores if my concerns are properly addressed. Minor: Line 256 "while the density of the generator is dynamically adjusted with DDA" => Here "generator" should be the "discriminator"?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This work targets improving the training efficiency of GANs, thus not suffering from obvious negative societal impact.

Reviewer response

Thank the authors for their efforts in providing the rebuttal. Most of my concerns are properly addressed. I tend to accept this paper given its current shape and will further adjust my scores based on the discussion with other reviewers.