Toward Understanding Generative Data Augmentation

Summary

The paper presents a theoretical study of the generalization properties of training when the training set is augmented with artificially generated data. The main result of the paper is a theorem which bounds the generalization error by two terms – one representing the divergence between the original training distribution and the augmented distribution, and one representing the generalization error of the mixed distribution. The authors presents two empirical contributions. Firstly they study a gaussian mixture model with synthetic data and find that their theoretical predictions match the measurements. Secondly they consider Resnets trained on cifar and find that generative models are useful when augmentations are not used, and that diffusion models but not GANs are useful with augmentations. EDIT: increased confidence from 2 to 3 after clarifications from the authors.

Strengths

* The topic is topical and potentially impactful. * The writing is very good. * The results of the theorem are natural and intuitive.

Weaknesses

* It seems like a major assumption is that “because the distribution learned by the generative model is dependent on the sampled train set”. In practice, I don’t think this is necessarily true. If this assumption is removed, the theoretical results might be much easier to derive. * Gaussian mixture models are not really used in practice, so it’s not a very interesting experiment. * It doesn’t seem like the theoretical results will apply to deep neural networks.

Questions

1/ Can you comment on the assumption that “ the distribution learned by the generative model is dependent on the sampled train set”. Is it realistic? What happens if it is removed? 2. How does the experiments on cifar10 related to your theoretical results? 3. How well does your theoretical results apply to deep neural networks?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

n/a

Summary

The paper provides a theoretical analysis of the stability bound for generative data augmentation. The authors provide empirical evidence to validate the proposed theory on bGMM and GNAs.

Strengths

Data augmentation plays an important role in deep learning. The paper provides a theoretical analysis of using generative models for data augmentation. Understanding generative data augmentation can potentially benefit machine learning tasks in low-data conditions.

Weaknesses

- In Section 4.2, the authors use standard augmentation to approximate CIFAR-10 with larger $m_S$. The comparison of GDA on CIFAR-10 and augmented CIFAR-10 is unfair as the standard augmentations induce effective inductive bias in the dataset, e.g., flipping. A more proper way to approximate CIFAR-10 with different $m_S$ is to sample multiple subsets of CIFAR-10 data with different sizes. - Although the paper proposes a general stability bound on GDA, it is unclear how the stability bound can be utilized to advance existing baselines. The finding that GDA can improve test generalizations on small datasets, where awful overfitting occurs, is somewhat expected. It is still unclear how to set the hyperparameters, like the number of augmented samples given any new dataset.

Questions

- How to use the stability bound to advance existing generative data augmentation methods in practice?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

I find no negative societal impact in this work.

Summary

This paper studies generative data augmentation (GDA), in which the samples from trained generative models are added to the training dataset for training discriminative models. There have been several empirical research reports on GDA, and it is known that GDA is unlikely to be effective when real learning data is abundant. However, there has been no theoretical analysis provided so far to explain this phenomenon. This study assumes a realistic setting in which the distribution imitated by the generative model and the real dataset distribution are different, and analyzes the generalization error bounds of GDA in the non-iid setting in three cases: (i) general case based on the existing algorithmic stability framework, (ii) a binary Gaussian mixture model (bGMM) and linear classifier, and (iii) a deep generative adversarial nets and deep neural classifier. The main findings from these theorems are (a) the divergence between the distribution imitated by the generated model and the true distribution is important for the generalization error, (b) increasing the number of generated samples does not lead to a faster learning rate, and (c) GDA does not achieve faster learning rate in situations affected by the curse of dimensionality. The paper provides simple experiments on bGMM and CIFAR-10 to test the theory, showing the similarity between the upper bound of the generalization error given by the theorem and the measured trends of the error, and the effectiveness of GDA in situations where the curse of dimensionality occurs. While this generalization error analysis is not perfect, as the paper states in the Limitation, this paper will have a significant impact on the research field of GDA, where no theoretical discussion existed.

Strengths

+ The paper is well organized and clearly states its arguments. Also, the paper is very readable, with careful notation and explanation of existing frameworks to explain the theory. + The paper provides generalized error-bound analysis in three settings ranging from general to realistic settings, establishing a first step toward learning guarantees in GDA. + The paper confirms the implications of the theorem through several experiments.

Weaknesses

- **W1** Some parts of the explanation are difficult to interpret. In Theorem 3.2 and 3.3, the paper explains "constant-level improvement" by GDA, but it is not clear in which equation "constant" appears, making the argument difficult to understand. Further, in Eq. (2), most of the theorems depend on the explanation in the Appendix, and the paper is not self-contained in this respect. - **W2** The paper does not mention several important previous studies. For example, Shmelkov et al. [a] were the first to report that GDA with GANs degrades accuracy even in small training dataset settings. Subsequently, Tran et al. [b] and Yamaguchi et al. proposed methods to improve GDA by the principles of Bayesian neural networks and multi-task learning. Although these works use somewhat older generative models and there may be facts that partially contradict the claims of the paper, they are considered important milestones in explaining the motivation for your work. I recommend that the paper cites them appropriately. **Reference** - [a] Shmelkov, Konstantin, Cordelia Schmid, and Karteek Alahari. "How good is my GAN?." Proceedings of the European conference on computer vision (ECCV). 2018. - [b] Tran, Toan, et al. "A bayesian data augmentation approach for learning deep models." Advances in neural information processing systems 30 (2017). - [c] Yamaguchi, Shin'ya, Sekitoshi Kanai, and Takeharu Eda. "Effective data augmentation with multi-domain learning gans." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.

Questions

- **Q1** Can the theorems provided by the paper discuss GDA in the situation of transfer learning of generative models? We often apply transfer learning such as fine-tuning to compensate for the distribution approximation performance of the generative model in GDA. I understood that minimizing distribution's divergence by transfer learning is not matter because Theorem 3.1 makes no assumptions about training methods, is this correct? - **Q2** Why is GAN set as the generative model in Theorem 3.3? Is there any advantage in proving this theorem? - **Q3** This is mostly a comment, but I think that the experiments in Section 4.2 should be included in the main paper since the evaluation of GDA with diffusion models as well as GANs is a high-impact result.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

This paper adequately discusses the limitation of the tightness of the theoretical guarantee.

Summary

Generative data augmentation (GDA) aims to improve model performance by generating artificial labeled samples to enlarge the limited training dataset, but is also highly influenced by the size of training dataset, choices of augmentation methods and the number of augmented data. The paper seeks to develop a theoretical understanding to GDA by reproducing classical results built upon i.i.d. assumptions to the generatively augmented dataset, and establishes a general relationship between the stability bound and the learned distribution divergence besides the augmentation size. To further investigate and verify the proposed upper bound of generalization error, the paper also particularizes specific cases of bGMM and GAN, and provides insights to understand the theoretical results. Finally, empirical experiments on synthetic dataset and CIFAR10 are conducted to study the effect of different factors and validate discovered theoretical findings.

Strengths

1） Mathematical notations and theoretical assumptions are clearly exhibited and explained in the paper. Details required to understand the problem background and existing results are also provided. 2） The theoretical results are novel and the related analysis seems to be solid and intuitive. The proposed theorems also build connection to previous results under i.i.d. assumption and empirical findings. 3） The idea of extending classical generalization error theories to the generative data augmentation problem is novel and well-motivated.

Weaknesses

1） Despite theoretical results and corresponding conclusions of this paper are easy to understand and conformable to our common intuition, too many separated compositions of remarks make the paper annoyingly disorganized, forcing the readers to jump around while reading. The paper has to be re-organized to make readers easier to access the content. 2） Some figures (e.g. Figure 1a) are hard to recognize, and some are unnecessarily separated into parallel parts (e.g. Figure 1b and Figure 1c). 3） My greatest concern about this work is that when talking about data augmentation, we are most interested in the difference of model performance trained with and without data augmentation. However, the paper only considers the case where the optimal augmentation number $m_G^*$ is adopted. 4） According to Theorem 3.1, the generalization bound for GDA is composed of the distribution divergence and the generalization error with respect to the mixed distribution, where the former is determined by the model itself and the latter is controlled by the number of generative data. For a given model with fixed generative ability, what we can only do is to find a near optimal number of generative samples. However, the choice of this number is not inspiring in this paper in my opinion. 5） Experiments on real-world dataset seem incomplete and not convincing enough. First, only a single dataset (CIFAR-10) is adopted which fails to evade possible influence by the nature of dataset. Second, the experimental results show limited information and connection to the proposed theories, since the generalization error is inestimable in this case. Moreover, it is unfair to take generative models with totally different architectures for comparison, but fair comparison can be made between the same generative model with different training degrees. 6） The practical contribution of the paper may be limited since many conclusions are intuitive and show little help to practical application.

Questions

1) In the paper the mixed distribution with augmentation is simply defined as a weighted combination of the training set distribution and the generative model distribution, is it a proper definition particularly when these two distributions are mutually dependent? 2) Could the authors clarify the purpose of choosing DCGAN, StyleGAN and EDM for experiments since there are plenty of alternative choices such as classical variational autoencoder (VAE)? This part is unclear for me from the paper. 3) The theoretical results and conclusions given by the paper are solid. But could the authors illuminate what can be inspired from these conclusions especially for utilizing generative data augmentation in practical applications?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

see above

Summary

In this work the authors present new theoretical results for generative data augmentation. In particular, the authors introduce a new result that gives a bound on the generalization error of a model trained with data augmentation provided by a generative model. The authors use this bound to illustrate when GDA may or may not help in generalization performance. The authors then provide specific examples of applications of this bound to common models for generative augmentation; a binary Gaussian mixture and a deep generative model (GAN). For each the authors derive a more specialized bound and perform empirical experiments to validation the theory.

Strengths

- This work provides useful theoretical insight into the strengths and limitations of generative data augmentation. The authors present novel results that, to my knowledge, are the first results bounding the generalization error for models trained with the aid of generative data augmentation. - I have not checked proofs in the appendix, but the theory as presented in the main text seems sound. - The applications to both the binary Gaussian mixture and GAN setting are helpful for illustrating and expanding upon the main theoretical results. - The experimental results do support the theoretical results presented and the results on DGM showing the re-affirming the promise of diffusion models for this purpose are interesting. - The results presented could be a useful foundation for future work.

Weaknesses

- The deep generative model results presented are narrow in scope, applying to GANs trained in a class-separated manner and to fully-supervised learning, rather than potentially semi-supervised learning or transfer learning via generative models. - As mentioned by the authors, they do not investigate the tightness of the proposed bounds. - I'm not sure this type of data augmentation approach is widely used enough for this analysis to be immediately impactful, though it does appear to be gaining traction. - The clarity of the writing could be improved in places.

Questions

- I was very confused by the usage of the term "learning rate" in the context here. I think I now understand it to refer to rate at which the generalization bound shrinks as a function of the amount of data, but given that the paper also discusses SGD, where the term has a different meaning it's hard to follow. Or is there some connection that I'm not appreciating? - I'm similarly a bit confused by what is meant by "augmentation consumption" on line 166. Can you elaborate on this? - I found the overloading of $d$ as both the data dimension and as a way to denote a divergence confusing, particularly for theorem 3.2. - Can you offer further any insight into when and why GDA does help in practice?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss limitations.

Thanks for your reply

Thanks for your reply. I'm happy for the clarifications regarding how the theory applies to ANNs. I will keep my score but will increase my confidence.

Reply to authors

Thank you for your thoughtful responses! Reading through this and the other responses has convinced me to bump up my score. I agree that this seems to be an interesting step towards analyzing the performance of generative data augmentation. My confidence is still low as I am pretty unfamiliar with related work on generalization bounds.