The GAN is dead; long live the GAN! A Modern GAN Baseline

Summary

A lot of GANs papers have shrunk in quality since diffusion arose as a strong method. The authors goes against the grain by focusing on modernizing GAN baseline in a principled manner and in doing so they obtain stable, convergent, diversity, and high-quality images comparable to diffusion models. The authors highlight the fact that a lot of the GAN literature is riled with empirical tricks with no theory to back them. They explain that StyleGAN without tricks is very close to the original DCGAN of 2015. Meanwhile a lot more progress as been done in diffusion models leveraging many architectural improves. They instead start with a Relativistic pairing GAN baseline which, unlike regular GANs, have been shown to have no local-minima basins where the loss must worsen before reaching the true global minimum. However, they show that, just like regular GANs, RpGANs are not guaranteed to converge to the global minimum. They show that, just like regular GANs, gradient penalty on real data (R1) or fake data (R2) guarantee convergence. They suggests using both together. They then show empirically that on StackMNIST, GANs with R1 alone fails, but having both R1, R2, and RpGAN lead to perfect mode coverage and strong stability. As further justification, they also leverage the fact that from a constructive approach, when deriving a maximum-margin GAN, one obtains a gradient penalty with both R1, R2. Then they use a very careful ablation approach by starting from StyleGAN2, removing all the complicated empirical tricks but keeping R1 penalty, which reduce performance a bit, then moving to RpGAN, adding R2 penalty, then modernizing the architecture, reaching slightly better performance at similar number of parameters, but without tricks. The describe extensively and carefully the effect of different architecture choices on performance. Experiments on CIFAR-10, Imagenet-32, and FFHQ-256 are shown were the method proposed shows extremely low FID, better than diffusion models and do so while keeping the parameter count low. Results are very good, but work is still needed to scale these methods to text-to-image settings. However, the current results are very promising and the method has been made very clean and carefully designed (similar to ConvNext).

Strengths

Experiments on CIFAR-10, Imagenet-32, and FFHQ-256 are shown were the method proposed shows extremely low FID, better than diffusion models and do so while keeping the parameter count low. The derivation of their architecture is done carefully in a thorough manner.

Weaknesses

Great results on image datasets, but it still need to be scaled to large-scale text-to-image setting. Many GPUs are used, even for CIFAR-10; this is also in line with diffusion models, but it seems a lot, are all these GPUs needed for CIFAR-10? Although cleaner, the new baseline still contains a lot of complicated hyperparameter tuning, which change from one dataset to another (in Table 7). It would be great to clean this set of hyperparameters, I suggested some potential ideas in the "questions".

Questions

- Any intuition to why worse performance with GELU and swish activations since these are very successful with transformers? Is this true for both G and D? - Have you consider a transformer architecture as alternative to convnext-style architecture? - You mention that normalization can harm GANs, but groupnorm is used in diffusion models and layernorm is used in language models, could this be useful for your approach? Has it been tried before? I saw the references you mentioned to say that normalization is bad for GANs, but I haven't looked at them individually. - You mention that PresGAN and DDGANs obtain good performances, do you think that some of their ideas could be useful for improve the GAN baseline? - What is EMA half-life? Is the EMA .999? - Why do you schedule Adam B2? This seems very atypical. - Assume $\alpha$ is the lr. Rather than schedule the R1/R2 $\gamma$ manually, why not make it $\alpha\gamma$ so that by decreasing the lr $\alpha$ over time, $\gamma$ is also reduces automatically (thus reducing the hyperparameter tuning). The same could be done with Beta2, by doing $1-((\alpha/alpha_{init})*(1-beta2))$, so moving the 1-.9 = 0.1 to 0.01 (1-.99) when decreasing the lr to 1/10 of its value over training time. I just feel like choices like this would greatly reduce the arbitrary hyperparameter tuning used here which is quite complex in Table 7.

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed the limitations.

Summary

This paper introduces R3GAN as a GAN baseline that simplifies and modernizes the architecture by replacing ad-hoc tricks with modern designs. R3GAN utilized a regularized relativistic GAN loss coupled with zero-centered gradient penalties on both real and generated data, to addresses mode dropping and non-convergence, providing local convergence guarantees. The proposed loss allows for the elimination of unnecessary tricks and the adoption of advanced architectures. R3GAN demonstrates improved stability and competitive performance against state-of-the-art models on multiple datasets such as StackedMNIST, CIFAR, ImageNet-32, and FFHQ-256.

Strengths

+ The writing of the paper is clear, making it easy to understand and easy to follow. + One contribution of this paper is the simplification and optimization of the StyleGAN network architecture, which, as demonstrated by experiments, achieves considerable performance without the need for many elaborate tricks. + The paper provides theoretical analysis to prove the convergence of the loss used.

Weaknesses

- The novelty of the method is somewhat limited, as both relativistic pairing GAN (RpGAN) and zero-centered gradient penalties (0-GPs) are previously proposed and validated approaches in the field of GANs. - Small-scale experiments to validate the significant improvements in stability and diversity provided by the used loss are insufficient; more complex datasets should be used to verify its effectiveness. As shown in Table 2, the improvement from config C compared to config B is not significant. Generally, stability can allow the model to converge better, thereby enhancing the final results, and diversity can be measured by metrics such as recall or coverage.

Questions

How scalable is the proposed model? It is suggested to validate the effectiveness of the method on higher-resolution datasets and more complex tasks. This would align with the motivation to serve as a modern baseline, especially in the current context where both model parameter size and training data scale are increasing.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have discussed the limitations and potential negative societal impact of their work.

Summary

The authors posit that the main reason GAN research has been slow in recent years is due to the most foundational StyleGAN2 not having undergone major architectural changes, essentially due to a lack of convergence guarantee in GAN objectives and being prone to mode collapse. This has limited the scope of architectural changes and further GAN research. Two identifiable goals of the paper are to mathematically derive a well-behaved regularized loss which guarantees GAN convergence and diversity, provide empirical evidence for the same, and now being free to make unconstrained design choices attempt to develop a GAN baseline which inherits features from modern vision architectures. The authors' methodology is inspired by two key works: (i) RpGAN [20], which addressed mode dropping in GANs with a better loss landscape, theoretically confirmed by Sun et al. [65] to contain only global minima; and (ii) Mescheder et al. [43], which provided evidence for local convergence with zero-centered gradient penalties in GAN objectives. By extending zero-centered gradient penalties to the RpGAN formulation, the authors address both mode dropping and unstable training. They offer theoretical proof for local convergence and present experimental evidence supporting global convergence, though no theoretical proof for the latter is provided. Design changes in Config D are motivated by ResNet [14] and ConvNeXt [41], selectively applying aspects consistent with the authors' observations. Following findings in [31, 29], they eliminate normalization in StyleGAN in B and introduce fix-up initialization [77] in D to prevent variance explosion. Changes in E are inspired by ResNeXt [74], incorporating group convolutions to increase bottleneck width and followed by bottleneck inversion for efficient use. This simple baseline is empirically found to outperform previous GAN methods and some recent diffusion based models on several unconditional and conditional generation tasks.

Strengths

1. The paper presents a combination of well-known techniques by extending the RpGAN formulation with zero-centered gradient penalties. 2. While the individual components like RpGAN, zero-centered gradient penalties, and modern vision architectures (ResNet, ConvNeXt) are not new, their integration into a cohesive GAN framework is a notable contribution. 3. The submission is technically sound with well-supported claims through theoretical analysis and experimental results. 4. The paper is clearly written and well-organized. 5. The results are significant, with improved performance over existing GANs and some diffusion models on various datasets.

Weaknesses

1. While experimental results are well presented, there seems to be a lack of information about the training setup, hyperparameters used, number of inference steps, etc pertaining to the diffusion based models in Tables 4, 5 and 6. It would be useful to include this information in the supplementary or the tables itself for a transparent comparison.

Questions

1. To clarify, on line 132 on page 4, “857” corresponds to StyleGAN backbone trained on WGAN-GP loss and “881” corresponds to the mini-batch standard deviation trick added to the same? Would be better to clarify the exact configuration and include the two cases in table 1 and figure 1.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

-

response

Thank you for addressing my questions and concerns. Having some of these short discussions in the appendix would indeed be useful to the reader. The extra imagenet-64 experiment will also help. I will increase my score by 1. This method is particularly good and helps bring GANs extremely close to diffusion performance which is quite important for the field of GANs to regain usefulness. The design choices are quite interesting. This reminds me a bit of the very useful experiments detailed discussions in the BigGAN paper which really helped pave the way to better GANs.