Library

Augmentation-Aware Self-Supervision for Data-Efficient GAN Training

Training generative adversarial networks (GANs) with limited data is challenging because the discriminator is prone to overfitting. Previously proposed differentiable augmentation demonstrates improved data efficiency of training GANs. However, the augmentation implicitly introduces undesired invariance to augmentation for the discriminator since it ignores the change of semantics in the label space caused by data transformation, which may limit the representation learning ability of the discriminator and ultimately affect the generative modeling performance of the generator. To mitigate the negative impact of invariance while inheriting the benefits of data augmentation, we propose a novel augmentation-aware self-supervised discriminator that predicts the augmentation parameter of the augmented data. Particularly, the prediction targets of real data and generated data are required to be distinguished since they are different during training. We further encourage the generator to adversarially learn from the self-supervised discriminator by generating augmentation-predictable real and not fake data. This formulation connects the learning objective of the generator and the arithmetic $-$ harmonic mean divergence under certain assumptions. We compare our method with state-of-the-art (SOTA) methods using the class-conditional BigGAN and unconditional StyleGAN2 architectures on data-limited CIFAR-10, CIFAR-100, FFHQ, LSUN-Cat, and five low-shot datasets. Experimental results demonstrate significant improvements of our method over SOTA methods in training data-efficient GANs.

cs.LGcs.AIcs.CVGenerative Adversarial Networks and Image SynthesisHuman Pose and Action RecognitionMusic and Audio ProcessingComputer ScienceEngineering
View graphSource

Paper

References (52)

01DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited DataTiantian Fang, Ruoyu Sun, A. Schwing2022 · Neural Information Processing Systems · 26 citations In Library
02FreGAN: Exploiting Frequency Components for Training GANs under Limited DataMengping Yang, Zhe Wang, Ziqiu Chi et al.2022 · Neural Information Processing Systems · 47 citations In Library
03Ensembling Off-the-shelf Models for GAN TrainingNupur Kumari, Richard Zhang, Eli Shechtman et al.2021 · Computer Vision and Pattern Recognition · 130 citations In Library
04Improving Transferability of Representations via Augmentation-Aware Self-SupervisionHankook Lee, Kibok Lee, Kimin Lee et al.2021 · Neural Information Processing Systems · 65 citations In Library
05Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited DataLiming Jiang, Bo Dai, Wayne Wu et al.2021 · Neural Information Processing Systems · 132 citations In Library
06Projected GANs Converge FasterAxel Sauer, Kashyap Chitta, Jens Muller et al.2021 · Neural Information Processing Systems · 310 citations In Library
07GenCo: Generative Co-training for Generative Adversarial Networks with Limited DataKaiwen Cui, Jiaxing Huang, Zhipeng Luo et al.2021 · AAAI Conference on Artificial Intelligence · 42 citations In Library
08Generative Adversarial NetworksI. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza et al.2021 · International Conference on Computing Communication and Networking Technologies · 30k citations In Library
09Conditional GANs with Auxiliary Discriminative ClassifierLiang Hou, Qi Cao, Huawei Shen et al.2021 · International Conference on Machine Learning · 53 citations In Library
10Alias-Free Generative Adversarial NetworksTero Karras, M. Aittala, S. Laine et al.2021 · Neural Information Processing Systems · 2.1k citations In Library
11Self-supervised GANs with Label AugmentationLiang Hou, Huawei Shen, Qi Cao et al.2021 · Neural Information Processing Systems · 21 citations In Library
12Data-Efficient Instance Generation from Instance DiscriminationCeyuan Yang, Yujun Shen, Yinghao Xu et al.2021 · Neural Information Processing Systems · 97 citations In Library

Scroll for more · 38 remaining

Similar papers

Differentiable Augmentation for Data-Efficient GAN TrainingShengyu Zhao, Zhijian Liu, Ji Lin et al.2020 · arXiv (Cornell University) · 694 citations In Library
Adversarial Semantic Augmentation for Training Generative Adversarial Networks under Limited DataMengping Yang, Zhe Wang, Ziqiu Chi et al.2025 · arXiv (Cornell University) · 0 citations In Library
Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited DataLiming Jiang, Bo Dai, Wayne Wu et al.2021 · arXiv (Cornell University) · 132 citations In Library
Regularizing Label-Augmented Generative Adversarial Networks Under Limited DataLiang Hou2023 · IEEE Access · 3 citations In Library
Training GANs with Stronger Augmentations via Contrastive DiscriminatorJongheon Jeong, Jinwoo Shin2021 · arXiv (Cornell University) · 70 citations In Library
Training Generative Adversarial Networks with Limited DataTero Karras, Miika Aittala, Janne Hellsten et al.2020 · Neural Information Processing Systems In Library
Training Generative Adversarial Networks with Limited DataTero Karras, Miika Aittala, Janne Hellsten et al.2020 · arXiv (Cornell University) · 2.3k citations In Library
Dynamic Generative Adaptation for Data-Efficient GANDivya Saxena, Jiannong Cao, Tarun Kulshrestha et al.2025 In Library
On Data Augmentation for GAN TrainingNgoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen et al.2021 · IEEE Transactions on Image Processing · 344 citations In Library
On Data Augmentation for GAN TrainingNgoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen et al.2020 · arXiv (Cornell University) · 38 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer NBv75/10 · confidence 5/52023-07-05

Summary

This paper introduces AugSelf-GAN, an enhanced approach for self-supervised GANs with label augmentation (SSGAN-LA). The authors present two major contributions. Firstly, they propose a discriminator that predicts continuous augmentation parameters, instead of classifying discrete augmentation methods. This allows the discriminator to learn a more fine-grained representation of the data distribution and to better distinguish between real and fake images. Secondly, the authors conduct a theoretical analysis that establishes a connection between the proposed method and the arithmetic-harmonic mean divergence. Experimental results demonstrate that AugSelf-GAN enhances GAN performance (as measured by FID and IS) in scenarios of limited data settings.

Strengths

- The proposed method demonstrates significant improvements in quantitative results despite its simplicity. - This paper offers comprehensive empirical and theoretical evidence to support the effectiveness of the proposed approach.

Weaknesses

- L138–139: The authors claim that SSGAN-LA and ADA result in augmentation leaking. However, no evidence, such as citations or empirical results, is provided. I think this claim should be justified since it is the main motivation of the paper. - The proposed method has limited novelty in terms of technical details. It can be seen as a variant of SSGAN-LA, with the main difference being that the discriminator predicts continuous augmentation parameters instead of discrete ones. - The paper lacks a comparison with some previous methods, such as ContraD and SSGAN-LA. These methods are also designed to train GANs with data augmentation. It would be interesting (indeed necessary) to see how the proposed method compares to them. - The authors argue that the representation learning capability of the discriminator could be improved by the proposed method. However, there are not enough experiments to support this claim. The experiments are conducted on CIFAR10/100 only (Fig. 2), and there are no comparisons with the methods that claim similar advantages (i.e., ContraD).

Questions

Please see the above weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes, the authors have discussed the limitations of the proposed method, as well as the potential negative impact of this work.

Reviewer nM6o5/10 · confidence 4/52023-07-07

Summary

This paper proposes using a discriminator to predict the augmentation applied to the following sample, in addition to the standard real/fake discrimination objective. The authors demonstrate that using this objective training the GAN leads to minimizing to the Arithmetic-Harmonic Mean divergence objective. Further, the method shows improved results through FID in various data efficient setups of CIFAR-10, CIFAR-100, Few-Shot datasets etc.

Strengths

1) The proposed objective of predicting the augmentation through discriminator is novel to my knowledge. 2) The paper is well written and clear. 3) The technique of self-supervision of predicting the augmentation type seems to show significant improvement across baselines.

Weaknesses

1) Reasoning for Effectiveness of Proposed Method: The authors argue that discriminator being invariant to augmentations is wrong. However, that seems intuitively sensible however the theory presented does not properly justify the reasoning for it’s effectiveness. A possible direction for this to think upon could be how the discriminator distribution expands to augmentation. 2) Convergence Plot: As the method seems to improve performance, an analysis of convergence of FID could be provided for clearing whether the proposed method delays collapse or prevents it all together. In L136 the authors mentioned about better convergence but do not provide FID plot. Minor: 1) Fig. 2 is not described in text. 2) L136 misses citation. 3) Rephrase L49-51, as it's unclear.

Questions

1) Missing Details: The major missing detail is about how to determine the value of w for various augmentations introduced in the paper. What is the general principle which is followed to determine this? This is also one of the major weaknesses, if there is no general way. 2) Missing Evaluation using CLIP FID. Recently it has been argued in multiple papers [R1,R2] that CLIP based FID is a better metric for evaluating GANS. Hence, I request authors to provide the same in rebuttal. [R1]: The Role of ImageNet Classes in Fr\'echet Inception Distance [R2]: NoisyTwins: Class-consistent and Diverse Image Generation through StyleGANs Post Rebuttal Update: The authors responses were helpful and some of my major concerns have been resolved. I will maintain my borderline accept rating.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have discussed limitations

Reviewer b4Yw5/10 · confidence 5/52023-07-08

Summary

This work proposed an augmentation aware training framework to improve the synthesis of GANs under the low data regimes. The discriminator is also required to regress the augmentation parameters, preventing undesired invariance introduced by augmentations. Theoretical analysis also shows the connection between the given objective and the arithmetic-harmonic mean divergence. Experimental results alos demonstrate the effectiveness of the proposed approach under various benchmarks.

Strengths

- This paper is well-organized and easy to follow. - Making discriminator aware of the appiled augmentation is inspiring since some undesired invariance is always introduced more or less. - - Experiments are quite solid, including results on multiple benchmarks and insightful analysis (Sec. 6.2). - Section 5 also presents the theoretical analysis, showing the connection with arithmeticharmonic mean divergence, which may gives more deep understanding for audience.

Weaknesses

- There might lack the comparisons against prior SOTA. For instance, InsGEN and Vision-aided GAN achieves appealing performances for FFHQ and while Table.2 did not presents these. Obviously, the consistent gains could be obtained compared to the baseline StyleGANs. However, SOTA performances could make this work stronger. - Experiments on AFHQ are missing. I believe AFHQ is also a good benchmark to evaluate the approaches for limited data synthesis. - Performances on the sufficient data for FFHQ and LSUN-Cat would further give a whole picture of how well the proposed method performs. - Could this augmentation aware scheme be combined with more complex augmentations like ADA?

Questions

Please see Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Please see Weaknesses

Reviewer CpCq5/10 · confidence 2/52023-07-08

Summary

This paper addresses the challenge of training GANs with limited data, where the discriminator is prone to overfitting. Previous approaches, such as differentiable augmentation, have improved data efficiency but introduce undesired invariance to augmentation for the discriminator. To overcome this limitation while retaining the benefits of data augmentation, the authors propose an augmentation-aware self-supervised discriminator. This discriminator predicts the augmentation parameter of the augmented data, distinguishing between the prediction targets of real and generated data during training. The generator is encouraged to learn from the self-supervised discriminator by generating augmentation-predictable real data. This formulation connects the learning objective of the generator and the arithmetic-harmonic mean divergence.

Strengths

This paper proposes an augmentation-aware self-supervised discriminator. The authors include both experimental results and theoretical analysis to show the effectiveness of the proposed method. According to the results, AugSelf can improve the performance than SOTA methods.

Weaknesses

The major concern is the limited novelty. For example, SSGAN-LA works on augmenting the GAN labels (real or fake) via self-supervision of data transformation. The only difference is the form of self-supervised signals. Though the authors try to highlight the differences between these two works, these two methods still share many similarities. In the experiments, the paper also doesn't include a direct comparison with the closely related work SSGAN-LA. Furthermore, the method relies on the used augmentation method. How to select the appropriate augmentation is a problem. The paper doesn't include guidelines of how to select these augmentations.

Questions

1. Is there a way to determine which augmentations are beneficial to the training? 2. What are the major differences between this work and SSGAN-LA ? 3. SSGAN-LA can addresses catastrophic forgetting problem. Is AugSelf also able to tackle this challenge?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The limitations are elaborated in the prior sections. In summary, the limited novelty is the major concern.

Reviewer xecU5/10 · confidence 3/52023-07-26

Summary

This paper deals with the challenge of data-efficient GAN training. It is built upon DiffAugment and attempts to mitigate its issue of undesired augmentation invariance. The proposed method introduces an additional linear layer that predicts the augmentation parameter from the features extracted by the backbone of the discriminator.

Strengths

- The paper’s target task is clearly stated and holds significance. - The proposed method is simple and exhibits consistent effectiveness across various scenarios.

Weaknesses

- The motivation of the proposed method is unclear and lacks of evidence. The authors discuss the issue of DiffAugment in lines 36-42, citing [23] to argue the importance of augmentation sensitivity. However, it is questionable whether the motivations discussed in [23] are directly applicable to this work since the former aims at enhancing the transferability of the pretrained features. It is comprehensible that augmentation invariance can be harmful for the transferability of representation learning, and Figure 4 and 5 in [23] demonstrate intuitively that augmentation awareness helps the model learn position- and color-sensitive representations. However, in the case of image generation, it is hard for me to understand why the discriminator should be transformation sensitive. - One of the major benchmark dataset, ImageNet, is not considered in this work, which is a common choice in GAN literature [1, 2, 3]. Given that the proposed method is mainly an improvement of DiffAugment, it would be vital to show the comparison on the primary dataset used in the original paper. - The paper could benefit from a  qualitative comparison with the baselines by making it easier to intuitively grasp the effectiveness of the proposed method. [1] Brock et al. Large scale GAN training for high fidelity natural image synthesis. 2019. [2] Lucic et al. High-Fidelity Image Generation With Fewer Labels. 2019. [3] Zhao et al. Differentiable augmentation for data-efficient gan training. 2020.

Questions

Can you provide more explanation about how the theoretical analysis that “reveals a connection between the optimization objective of the generator and the arithmetic harmonic mean divergence” is relevant to the proposed method?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed the limitations of their work. Please see the Weaknesses stated above to see other limitations I find.

Reviewer nM6o2023-08-13

Thanks for the Rebuttal

Dear authors, I have read the rebuttal and reviews from the other reviewers. I also find that current work closely relates to SSGAN-LA, which predicts discrete labels. I checked the response. However, I am not very convinced why the continuous task should provide improved results over discrete task. Also, can you provide concrete examples of how the $w$ parameter is defined for various augmentations? Thanks

Authorsrebuttal2023-08-14

Thanks for the Reply (1/2)

Dear Reviewer nM6o, Thank you very much for your reply. We would like to elaborate further to answer your question. --- - **The current work closely relates to SSGAN-LA, which predicts discrete labels. Why the continuous task should provide improved results over discrete task.** We want to first emphasize that AugSelf-GAN and SSGAN-LA have substantially distinct contributions despite sharing similar methodological ideas. Firstly, SSGAN-LA aims to address the biased learning objective issue in existing discrete transformation-based self-supervised GANs [1] by improving the self-supervised loss function [2]; whereas the goal of our AugSelf-GAN is to enhance GAN training with limited data by proposing a seamless-adaptive continuous adversarial self-supervised task based on DiffAugment. In terms of methodology, SSGAN-LA employs a discrete adversarial self-supervised task (classification), while AugSelf-GAN utilizes a continuous adversarial self-supervised task (regression). It should be noted that the transition from discrete to continuous self-supervised adversarial task is not straightforward. Specifically, the adversarial idea requires us to differentiate self-supervised labels between real and fake data. In the discrete case, the classification task makes it natural for SSGAN-LA to use label augmentation to differentiate self-supervised labels between real and fake data. However, continuous labels are generally predicted through regression which cannot utilize the label augmentation to differentiate self-supervised labels. AugSelf-GAN needs to design a novel continuous adversarial task that fits continuous labels. However, there are many ways to differentiate continuous labels, such as adding extra dimensions, adding offsets to the labels, etc., but not all of them can fit the requirement of generating modeling (fitting real data distribution). We innovatively choose the origin-symmetry scheme to differentiate self-supervised labels between real and fake data for AugSelf-GAN and provide theoretical explanations to show that the origin-symmetry scheme can fit the requirements of generating modeling under certain conditions. Moreover, with regard to the specific implementation of the discriminator, SSGAN-LA does not accept the raw data as input, and thus requires data augmentation to be reversible, otherwise it cannot learn the real data distribution, while AugSelf-GAN accepts both raw data and augmented data as input and is not subject to this requirement. The most important thing is that our theoretical analysis reveals **why the continuous task should provide improved results over discrete task in data-limited GAN training**. We link a relationship between AugSelf-GAN and the arithmetic-harmonic mean (AHM) divergence. The AHM divergence is smoother than the reverse KL divergence optimized by SSGAN-LA (Fig.4), which theoretically makes the training of AugSelf-GAN more stable in the limited data scenario, as it receives more robust feedback [3]. The essential reason behind this is: there is no size relationship between discrete labels, while continuous labels have a size relationship, so that it can control the strength of the discriminator to distinguish between real and fake ($c$ in Eq.10), and alleviate its overfitting. The Table below compares AugSelf-GAN and SSGAN-LA on the CIFAR-10 and CIFAR-100 with different percentages of training data. According to the FID scores, AugSelf-GAN outperforms SSGAN-LA in most settings. We will add this comparative experiment in the revised version of our paper. | FID | CIFAR-10 | CIFAR-10-0.2 | CIFAR-10-0.1 | CIFAR-100 | CIFAR-100-0.2 | CIFAR-100-0.1 | | --- | --- | --- | --- | --- | --- | --- | | SSGAN-LA | 8.05 | 11.16 | 15.08 | **9.77** | 15.91 | 23.17 | | AugSelf-GAN | 7.68 | 10.97 | 15.68 | 9.88 | 16.11 | 21.30 | | AugSelf-GAN+ | **7.54** | **9.95** | **12.76** | 10.09 | **15.33** | **18.64** | | IS | CIFAR-10 | CIFAR-10-0.2 | CIFAR-10-0.1 | CIFAR-100 | CIFAR-100-0.2 | CIFAR-100-0.1 | | --- | --- | --- | --- | --- | --- | --- | | SSGAN-LA | 9.30 | 8.84 | 8.30 | 11.13 | **10.52** | 9.81 | | AugSelf-GAN | **9.43** | 8.98 | 8.76 | **11.19** | 10.25 | 9.78 | | AugSelf-GAN+ | 9.27 | **9.08** | **8.79** | 11.12 | 10.14 | **9.93** | We believe that the above significant differences demonstrate the novelty and contribution of this work compared to SSGAN-LA. --- [1] Tran, Ngoc-Trung, et al. Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Game. NeurIPS 2019. [2] Chen, Ting, et al. Self-Supervised GANs via Auxiliary Rotation Loss. CVPR 2019. [3] Tseng, Hung-Yu, et al. Regularizing Generative Adversarial Networks under Limited Data. CVPR 2021

Authorsrebuttal2023-08-14

Thanks for the Reply (2/2)

- **Concrete examples of how the parameter $\omega$ is defined for various augmentations?** As our AugSelf-GAN relies entirely on data augmentation used in DiffAugment, we would like to use the three data augmentation employed by DiffAugment as an example to answer your question. DiffAugment uses color, translation, and cutout as data augmentations. For color, there are three parameters: $\lambda_\text{brightness}$, $\lambda_\text{saturation}$, and $\lambda_\text{contrast}$, which are randomly sampled from the range $[0,1]$. Therefore, we directly combine them as the parameter $\omega_\text{color}\in [0,1]^3$. Translation includes two parameters, $x_\text{translation}$ and $y_\text{translation}$, which represent how many pixels the image is translated on the x and y axis, respectively. In the DiffAugment code, they are randomly sampled from the integer range {$-X_\text{translation},\dots,X_\text{translation}$} and {$-Y_\text{translation},\dots,Y_\text{translation}$}, respectively. We transform them into the range of $[0,1]^2$ through a linear transformation ($(x_\text{translation} / X_\text{translation} + 1) / 2$, $(y_\text{translation} / Y_\text{translation} + 1) / 2$) as the parameter $\omega_\text{translation}$. Cutout includes two parameters, $x_\text{offset}$ and $x_\text{offset}$, which represent the center coordinate of the cutout.They are randomly sampled from the integer range {$\{0,\dots,X_\text{offset}\}$} and {$\{0,\dots,Y_\text{offset}\}$}, respectively. Similar to translation, we transform them into the range of $[0,1]^2$ through a linear transformation ($x_\text{offset} / X_\text{offset}$, $y_\text{offset} / Y_\text{offset}$) as the parameter $\omega_\text{cutout}$. For the specific implementation of the above scheme, please refer to the `DiffAugment_pytorch.py` file in the anonymous code repository provided in our paper. Thank you for your time and effort in reviewing our manuscript. If you have any further questions or suggestions, we would be happy to hear them. Best regards, Authors of Submission 5698

Reviewer nM6o2023-08-20

Thanks for the response. However, there are still two concerns that I have: 1) Is it optimal to combine different augmentations with the same value of $w$? If the value of $w = 0.5$ for both the translation and cutout, are they equivalent augmentations? If not, then why should the discriminator predict the same for both? I don't understand the reasoning behind the same $w$ parameter for all augmentations. Hence, I am unsure of the correctness of the above procedure. 2) As the AHM divergence is shown to be the major reason for performance improvement. Can the authors give some concrete evidence regarding experiments to verify the claim regarding AHM divergence?

Authorsrebuttal2023-08-21

Response by Authors

Dear Reviewer nM6o, Thank you for your feedback on our paper. We sincerely apologize for any confusion or misunderstanding caused by our insufficient description of our method. We appreciate the opportunity to clarify the details of our approach. **The self-supervised labels $\mathbf{\omega}$ in different augmentations** Regarding the use of color, translation, and cutout as self-supervised signals, we would like to clarify that we employ separate self-supervised discriminators for each task, and they are not shared. In other words, our method includes four losses for the discriminator component: one for the original discriminator $D$ and one each for the color, translation, and cutout self-supervised discriminators ($\hat{D}\_\text{color},\hat{D}\_\text{translation},\hat{D}\_\text{cutout}$). These four networks share a common backbone $\phi$ and differ only in the final layer ($\psi,\varphi\_\text{color},\varphi\_\text{translation},\varphi\_\text{cutout}$), respectively. Therefore, different augmentations do not share the same $\mathbf{\omega}$, as can be seen from lines 122-124, where the dimensions of $\mathbf{\omega}\_\text{color}\in\mathbb{R}^3$ and $\mathbf{\omega}\_\text{translation}\in\mathbb{R}^2$ are not the same. We will rewrite Eqs. 7 and 8 as the following to make the details of our method more clear. $$ \min\_{D,\hat{D}\_\text{color},\hat{D}\_\text{translation},\hat{D}\_\text{cutout}} \mathcal{L}\_D^\text{da}+\lambda\_d\cdot\left(\mathcal{L}\_{\hat{D}\_\text{color}}^\text{color}+\mathcal{L}\_{\hat{D}\_\text{translation}}^\text{translation}+\mathcal{L}\_{\hat{D}\_\text{cutout}}^\text{cutout}\right) $$ $$ \min\_{G} \mathcal{L}\_G^\text{da}+\lambda\_g\cdot\left(\mathcal{L}\_{G}^\text{color}+\mathcal{L}\_{G}^\text{translation}+\mathcal{L}\_{G}^\text{cutout}\right) $$ where $\mathcal{L}^{\text{color}}\_{\hat{D}\_\text{color}}$ and $\mathcal{L}^{\text{color}}\_G$ are defined as follows: $$ \mathcal{L}^{\text{color}}\_{\hat{D}\_\text{color}} = \mathbb{E}\_{\mathbf{x},\boldsymbol{\omega}}\left[\|\hat{D}\_\text{color}(T(\mathbf{x};\boldsymbol{\omega}),\mathbf{x})-\boldsymbol{\omega}\_\text{color}^+\|\_2^2\right] + \mathbb{E}\_{\mathbf{z},\boldsymbol{\omega}}\left[\|\hat{D}\_\text{color}(T(G(\mathbf{z});\boldsymbol{\omega}),G(\mathbf{z}))-\boldsymbol{\omega}\_\text{color}^-\|\_2^2\right] $$ $$ \mathcal{L}^{\text{color}}\_G = \mathbb{E}\_{\mathbf{z},\boldsymbol{\omega}}\left[\|\hat{D}\_\text{color}(T(G(\mathbf{z});\boldsymbol{\omega}),G(\mathbf{z}))-\boldsymbol{\omega}\_\text{color}^+\|\_2^2\right] - \mathbb{E}\_{\mathbf{z},\boldsymbol{\omega}}\left[\|\hat{D}\_\text{color}(T(G(\mathbf{z});\boldsymbol{\omega}),G(\mathbf{z}))-\boldsymbol{\omega}\_\text{color}^-\|\_2^2\right] $$ $\mathcal{L}^{\text{translation}}\_{\hat{D}\_\text{translation}}, \mathcal{L}^{\text{cutout}}\_{\hat{D}\_\text{cutout}}, \mathcal{L}^{\text{translation}}\_G, \mathcal{L}^{\text{cutout}}\_G$ follow the similar definition. And $\mathbf{\omega}$ represents all augmentations, i.e., data $\mathbf{x}$ is sequentially transformed using color, translation, and cutout augmentations. **Concret experimental evidence on AHM divergence** We are very grateful for your constructive questions. However, it is difficult to directly verify the feedback robustness of AHM through experiments, as we cannot truly construct and obtain the results of the f function. Nevertheless, based on current experiments, we can assist in verifying the advantages of our method from the loss curve of the discriminator in data-limited scenarios. Specifically, the lower the loss in the discriminator, the more obvious the overfitting phenomenon, and the **higher** it indicates that **overfitting can be alleviated**. Unfortunately, we are no longer able to upload the loss curve through a PDF. According to the existing experimental records, as shown in the table below, the average value of the training loss on the DiffAugment discriminator is significantly smaller than that on our AugSelf-GAN, indicating that AugSelf-GAN can effectively further alleviate the overfitting of DiffAugment. | D_loss | CIFAR-10 | CIFAR-10 20% | CIFAR-10 10% | CIFAR-100 | CIFAR-100 20% | CIFAR-100 10% | | --- | --- | --- | --- | --- | --- | --- | | DiffAugment | 1.0902 | 0.7157 | 0.6964 | 1.0474 | 0.6404 | 0.8974 | | AugSelf-GAN | **1.4564** | **1.2364** | **1.1331** | **1.3818** | **1.1073** | **1.1719** | Thank you again for your coment. We sincerely hope that our response can address your question.

Reviewer xecU2023-08-14

Thanks for your response

Thank you for your effort to respond. I think my concerns are well addressed except that the lack of ImageNet experiments could be a major flaw. I have raised my score to borderline accept.

Reviewer NBv72023-08-19

Thanks for the response and additional experiments. - Comparisons with SSGAN-LA We recognize the difference between SSGAN-LA and the proposed method. BTW, why "the transition from discrete to continuous self-supervised adversarial tasks" is necessarily more critical? Any implications? Besides, SSGAN-LA occasionally performs better than the proposed AugSelf-GAN or AugSelf-GAN++. Does the authors have any explanations or reasoning for performances? - Comparison with ContraD We gracefully disagree that ContraD does not target a limited data setting. It developed a data augmentation technique for GAN, thus it is naturally effective in a limited data scenario. The paper also stated that “We evaluate our training method on the Animal Faces-HQ (AFHQ) dataset (Choi et al., 2020), which consists of ∼15,000 samples of animal-face images at 512×512 resolution, to further verify the effectiveness of ContraD on higher-resolution, yet limited-sized datasets.” (Besides, if the proposed method specifically targeted for a limited data setting does not outperform a general recipe, it is rather discouraging news. ) Regarding the limited time issue, the data this paper dealing with is "CIFAR10/CIFAR100/Tiny-imagenet". It is not that heavy to conduct the GAN training on these datasets, given the rebuttal period. Overall, we also read other reviewers' feedback and the authors' responses. Despite the difference of the proposed method from previous arts, we are still not convinced by the advantages given by the current experimental results.

Authorsrebuttal2023-08-21

Response by Authors

Dear Reviewer NBv7, Thank you for recognize the difference between SSGAN-LA and our proposed method. **Why "the transition from discrete to continuous self-supervised adversarial tasks" is necessarily** Our Theorem 1 links a relationship between AugSelf-GAN and the arithmetic-harmonic mean (AHM) divergence. The AHM divergence is smoother than the reverse KL divergence optimized by SSGAN-LA (Fig.4), which theoretically makes the training of AugSelf-GAN more stable in the limited data scenario, as it receives more robust feedback for the generator [1] for extremely large inputs ($p(x)/q(x)=D(x)/(1-D(x))$), which is estimated by discriminators and common in data-limited regimes where the discriminator's estimation of the proportion may be inaccurate. The essential reason behind this is: there is no size relationship between discrete labels, while continuous labels have a size relationship, so that it can control the strength of the discriminator to distinguish between real and fake ( in Eq.10), and alleviate its overfitting. **SSGAN-LA occasionally performs better than AugSelf-GAN(+)** We understand that SSGAN-LA outperforms AugSelf-GAN(+) on CIFAR-100 in terms of FID and on CIFAR-100 20% in terms of IS. For the first case, CIFAR-100 is a relatively data-rich dataset regarding the resolution, which may not play to the strengths of AugSelf-GAN(+). For the second case, it is being speculated that IS may not be an accurate metric for measuring the quality of the generated images, due to its inability to measure inter-class diversity [2]. This also contradicts the results of FID, where AugSelf GAN+ is superior to SSGAN-LA in this data setting. **Compare with ContraD on AFHQ** We appreciate the reviewer's suggestion to compare with ContraD on the AFHQ dataset and acknowledge that it would be a suitable dataset for evaluating our method. We are currently conducting experiments on the AFHQ dataset, but training on a single GPU for 512x512 resolution data takes at least 21 days and 22 hours to finish. We have obtained some promising results on AFHQ cat so far and will report the complete results in our updated version. | | AFHQ cat | | --- | --- | | DiffAugment | 3.49 | | ContraD | 3.82 | | AugSelf-GAN | **3.37** | Regarding the claim that training on Tiny-ImageNet is not heavy, we respectfully disagree. Training on the entire Tiny-ImageNet dataset requires at least 5 days and 12 hours on a single GPU to complete 1000 epochs, and conducting multiple experiments is a challenge for our computing resources. Thank you again for your valuable feedback. We sincerely hope that our response can address your concerns. [1] Tseng, Hung-Yu, et al. Regularizing Generative Adversarial Networks under Limited Data. CVPR 2021 [2] Heusel, Martin, et al. GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium. NIPS 2017

Reviewer NBv72023-08-21

Thanks for the additional experiments and explanations. After reading all your supporting arguments and materials, I can now accept the advantage of the proposed method, the robust scheme given by the large inputs. We strongly suggest adding empirical evidence on various datasets in your final version because the usefulness of the training strategy indeed needs empirical verification. Considering all, as the authors properly answer my questions and concerns, I will increase my rating, to borderline accept.

Reviewer CpCq2023-08-21

I thank the authors for providing the response and adding additional experiments. I have checked the author response and the reviews from other reviewers. I would maintain my ratings to accept the paper.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC