Regularizing Neural Networks with Meta-Learning Generative Models

Summary

This paper proposed a regularization method 'Meta generative regularization' based on the bi-level optimization frame addressed for the generative data augmentation. The MGR is consisited of two terms: pseudo consistency regularization (PCR) and meta pseudo sampling (MPS). The training using MGR is formalized as alternating optimization of a main classification model and finder network for searching latent vectors of generative model(eg, StyleGAN).To maximize the gain from synthetic samples, MGR regularizes a feature extractor part of the classification model using PCR by effectively sampling useful samples for the generalization from GAN using MPS.

Strengths

- Use pseudo consistency regularization to address the distortion of decision boundary. - Introduce a subnetwork called a finder to improve the training of classifier, and address the unstable training of generator.

Weaknesses

- The data-driven data augmentation is not novel for the community, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. The proposed method is expensive in computation, and only achieve comparable or even worse performance than hand-designed data augmentation methods, e.g., SnapMix [4]. The advantage of proposed method is not clear to readers. - The used meta-learning technique is similar to Generative Teaching Networks [5]. - The experimental results can not support the effectiveness of proposed method. The compared method should contain other hand-designed data augmentation methods, e.g., mixup, cutmix, ect; and the data-driven data augmentation methods, e.g., AutoAugment [1], Population Based Augmentation [2], Fast AutoAugment [3], ect. Meanwhile, the hand-designed data augmentation methods SnapMix [4] can achieve a significant improvement on CUB, Cars, Aircraft datasets, while does not introduce additional expensive computation. - What if the classfication model totally training from scratch on the synthetic samples generated by the trained finder network for the StyleGAN? - Compared with existing data-driven data augmentation methods, proposed method is limited in transferability for other classification tasks. [1] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning Augmentation Policies from Data. In CVPR, 2018. [2] D. Ho, E. Liang, I. Stoica, P. Abbeel, and X. Chen. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. In ICML, 2019. [3] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast AutoAugment. In NIPS, 2019. [4] Huang S, Wang X, Tao D. Snapmix: Semantically proportional mixing for augmenting fine-grained data. In AAAI, 2021, 35(2): 1628-1636. [5] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, Jeff Clune. Generative Teaching Networks. ICML 2020

Questions

See Weaknesses. - Eq.(9) is wrong, since the numerator does not contain $\epsilon$.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The studied problem of this paper is somewhat out-of-date. The effectiveness is limited among existing researches. Especically, the proposed method does not show the effectiveness in some problems with sparse data, e.g., medical imaging.

Summary

The paper proposed meta-generative regularization (MGR) for improving generative data augmentation. MGR is optimized by alternative training between the main and finder network. To train the main network, contrastive learning is used. To train the finder network, the authors propose the bilevel optimization and approximate the solution.

Strengths

1. The proposed method does not update the generative model, but rather updates the finder network. Thus, the foundation generative model can be used. 2. I think that the proposed method does not depend on the type of generative models such as normalizing flow, auto-regressive, gan, and score-based generative models. 3. FDM to reduce the computation complexity. 4. The effectiveness of the reduced datasets

Weaknesses

1. CutMix [A] and MixUp [B] rather than AugMix are the strong baselines. These methods are simple, easily implemented, no training, and effective. A comparison is needed. 2. In Sec. 4.4, I am not sure about the interpretation of UMAP. From the visualization, the diversity of the proposed method is reduced. Thus, the loss term of contrastive learning, L_PCR, might be important. How much the performance is improved when the main model is trained on cross entropy and contrastive loss using only real data? 3. The result of diffusion model is in the supplementary. However, the experiment of recent generation models, including text-to-image generator, makes the paper more convincing because it shows promising results [C, D, E, F]. 4. How does this method affect the evaluation in terms of robustness, generalizability, or bias? These is possibility to generate the biased samples. [A] CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, ICCV 2019 [B] mixup: Beyond Empirical Risk Minimization, ICLR 2018 [C] IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION?, ICLR 2023 [D] Fake it till you make it: Learning transferable representations from synthetic ImageNet clones, 2023 [E] Synthetic Data from Diffusion Models Improves ImageNet Classification, 2023 [F] Generative models improve fairness of medical classifiers under distribution shifts, 2023

Questions

Please refer to the weaknesses. === I update my rate from 4 to 5 because my concerns are addressed and the authors will reflect the discussion below to the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is described in Sec. 6 in the main paper.

Summary

The authors propose a method for using synthetic images from GANs to augment training image classifiers. The naive approach to this problem is to generate samples for each class and treat these as supervised examples, but this can degrade performance due to image artifacts. Instead, the authors propose to use the generated data only for consistency regularization of the featurizer (”PCR”). Further, they meta-learn the codes to use for generation (”MPS”). On 6 datasets, the authors see performance improvements from using synthetic data this way, rather than performance drops.

Strengths

- The motivation is clear, the method is clean, and the approach makes intuitive sense. The overall paper presentation is very good. - The authors select a nice set of baselines and show their method outperforms them. In particular, GDA + MH was an important comparison to have. - The paper contains a solid set of ablations, including: the importance of PCR v. MPS (either can improve performance, both are best), the effect of MPS on sample quality as measured by FID, experiments with multiple generative models for one dataset, and experiments showing that gains are additive with standard data augmentations.

Weaknesses

- A missing baseline is consistency regularization without generative data augmentation, i.e. equation 5 applied to real examples $\mathcal D$ rather than $\mathcal D_p$. How does the method compare to training with this objective, and are gains additive? - Standard training with TrivialAugment / RandAugment outperform this method on Cars, presumably with much lower computational cost — though the authors do show these can be combined with the method to obtain further gains. I’d like to check if this trend holds for other datasets as well. - Because of the metalearning loop, the method is too slow to apply with diffusion models, as noted in the Limitations and Appendix B.3. All experiments are done with GANs trained from scratch on the target datasets. - Experiments are done on relatively small datasets. It would be interesting to compare this method on ImageNet.

Questions

A point of confusion for me is why experiments avoided using pretrained generative models / finetuning from them. As noted in the related work, massive pretraining may help produce higher quality samples. The authors attribute degradations after training on generated data to *class leakage*, i.e. samples where artifacts of multiple classes are combined into the same image, such as the tennis ball with a dog face generated by BigGAN. This seems like the kind of artifact that improved generative models can remove. Does generative data augmentation with large pretrained models result in such performance drops? Have the authors tried their method with finetuned GANs?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors note that the meta-learning loop renders the method unusable for computationally intensive generative models, e.g. diffusion models.

Summary

This paper leverages synthetic images from generative models to train classifier models, effectively using them as an augmentation tool. However, instead of incorporating these images in a simplistic fashion, the synthetic images are utilized as a regularizer. The paper posits that synthetic samples may not always perfectly mirror class categories found in real data distribution. As a response, a meta-learning framework is employed to dynamically ascertain which synthetic samples should be used to minimize validation losses. The paper introduces a feature-based consistency regularization loss for the generated images, a method similar to self-supervised techniques. The unique contribution of this paper is the proposition of a finder network that is trained within a meta-learning context, this network is designed to select images that will enhance the performance of the classification model. Through extensive experimentation, the paper illustrates that the proposed methodology can effectively circumvent the performance deterioration often linked with naive generative data augmentation while concurrently improving the baselines.

Strengths

* The paper is well-presented and easy to follow. * The paper presents a novel sampling network that is trained in a meta-learning setting to select synthetic images that will enhance the performance of the classification model. * Through a series of experiments, the paper successfully illustrates the effectiveness of the overall approach, as well as the contributions of different aspects of the methodology, i.e., consistency regularization loss and Meta Pseudo Sampling. * The proposed generative model augmentation method complements conventional data augmentation methods and can be employed together to enhance performance further.

Weaknesses

* The paper showcases its experiments predominantly on simple, fine-grained classification datasets. To bolster the credibility of their methodology, the authors would benefit from demonstrating how their approach enhances classification performance on more complex, large-scale datasets, such as ImageNet. * In Table 1, I suggest including an additional baseline: the base model + SSL. This means applying a consistency regularization loss on the real training images without utilizing any synthetic images. This will highlight the degree of improvement achieved using the generative model and the proposed model. The author acknowledges that training the sampling network in a meta-learning setting is computationally expensive, so this comparison would clarify whether the benefits of this approach justify its cost, particularly if the gains are minimal. * For the consistency loss, the paper solely employs augmentation in the image space. However, one might ask why not also utilize augmentation in the generative model's latent space by introducing small noise to the latent vector z. Given that the latent space of the generators implicitly represents the image manifold, performing augmentation in this latent space could lead to meaningful augmentation in the image space. Such augmentation could potentially be more challenging to achieve using conventional image augmentation methods. * Figure 5 could be improved further by incorporating a visualization of different class features along the decision boundary of the classifier. This enhancement would provide further insight into how these synthetic images influence the decision boundary under different cases. * (Just a suggestion) In lines 45-50, I would like to introduce an additional argument. Despite generative models being trained to model p(x), they frequently fail to capture the entire distribution of the training set, focusing mainly on the high-density region of the distribution. Therefore, the information extracted from these generative models is often less comprehensive than the information found in the actual dataset distribution. Furthermore, these synthetic images frequently exhibit disfigured object shapes; for instance, a majority of human shapes in StyleGAN-XL images appear distorted.

Questions

*The major novelty of the paper resides in the proposition of a sampling network, trained via a meta-learning method, to generate synthetic samples aimed at enhancing the classification model's performance. The authors further highlight that a superior FID score from the generative model correlates to improved performance in classification models (Table 2), and the FID score derived from their sampling network significantly surpasses that of uniform sampling (Fig 6b). In the generative models' literature, there exist works that advocate for effective sampling strategies from pre-trained GAN models, utilizing the multimodal truncation method, such as Makady et al. 2022's 'Self-Distilled StyleGAN: Towards Generation from Internet Photos.' How would their sampling approach fare when combined with the consistency regularization loss? It could significantly bolster the paper's case if authors demonstrated that the proposed sampling network exceeds the performance of other sampling methods, like the multimodal truncation method.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author themselves pointed out that the limitation of the method is the requirement of bilevel optimization of classifier and finder networks. This optimization is computationally expensive and may be less practical when used in generative models that require multiple inference steps, such as diffusion models.

The concerns are not perfectly addressed

Thanks for your detailed response, most of them has addressed the concerns. However, the major concern is that proposed method need to employ real samples and synthetic samples to train the classifier. However, existing data augmentation methods are easy and lightweight to improve the training. While the proposed method is expensive and cumbersome. Compared with data-driven data augmentation methods, proposed method can only be used for studied task at hand, and can not be transferred to new tasks. In the transfer stage, data-driven data augmentation methods is efficient and lightweight. Another concern is the proposed method use the data-augmentaion consistency regularization trick, which is a strong improvement strategy. It is unfair that compared methods do not use such trick to further improve performance. Besides, for data-scarce tasks, e.g., few/zero-shot learning, generated images have obtained SOTA performance, e.g., [1] etc. [1] He R, Sun S, Yu X, et al. Is synthetic data from generative models ready for image recognition?In ICLR, 2023.

Reviewer gssr

Dear Reviewer, The author has posted their rebuttal, but you have not yet posted your response. Please post your thoughts after reading the rebuttal and other reviews as soon as possible. All reviewers are requested to post this after-rebuttal-response.

Thank you for the detailed rebuttal

My apologies in not responding earlier. Thank you very much for taking the time and answering my questios. I have also gone through the other reviewers comments and I am now willing to change my score and recommend acceptance of the paper.