Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation

Summary

The paper proposes a generative data augmentation method, ALIA, which utilizes large image captioning and language models to summarize the domain description and employs language-guided image editing methods to create augmented training data. Experiment results show that ALIA outperforms recent data augmentation methods on fine-grained classification tasks.

Strengths

- S1. The paper is well-written and easy to follow. - S2. The idea of using prompts to extract domain information and class-agnostic descriptions is novel. - S3. The empirical result of the iWildCam experiment is promising. It is impressive that the task model trained on additional ALIA-generated images outperforms that uses real samples.

Weaknesses

- W1. Limited technical contribution. Although the idea of using prompts to extract domain information and class-agnostic descriptions is novel, most building parts of ALIA are off-the-shelf methods, for example, BLIP, GPT-4, Img2Img and Instruct Pix2Pix. - W2. The proposed method relies on pre-trained models heavily. As mentioned in the limitation section, ALIA is likely not as effective for unseen test domains. If ALIA performs poorly on a target dataset, no feedback can be sent to fine-tune ALIA on the dataset. - W3. ALIA is only tested on three datasets, where the dataset classes are mostly similar (see Q1).

Questions

- Q1. While being a generic data augmentation method, it seems that each tested dataset is comprised of similar objects, e.g., CUB contains birds, FGVC-Aircrafts contains aircraft and iWildCam contains animals. Does ALIA assume the dataset classes to be similar? On a more diverse dataset like ImageNet, can the large language model summarize good domain descriptions and create valid edits? - Q2. In line 118. The authors constrain the number of domain descriptions to be less than 10. Can the authors explain more about the choice of the number of domain descriptions? How does it affect the quality of the augmented samples? - Q3. In lines 191-192. The authors mention generating twice as much data as the original data to ensure “enough” data is available after filtering. How do we know whether the amount of filtered data is “enough” or not? Should we generate more data if the original dataset size is smaller?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I find no negative societal impact in this work.

Summary

The paper introduces ALIA to generate domain descriptions of the dataset from captions of each image. These captions are then used to generate more data using Stable Diffusion. There is also a filtering process to remove corrupted images or those with minimal edits. ALIA improves performance over the baselines on several datasets.

Strengths

- The paper was well written and easy to follow. - Fine-grained classification is an interesting and difficult problem. It is also nice to see a different setting compared to existing papers trying to improve robustness on IN-Sketch/R etc. Furthermore, the method performs well compared to the baselines. - I like the discussion on filtering failed edits. It is important but usually not given alot of attention.

Weaknesses

- Data and computational efficiency - How many new images were generated for the baselines and ALIA? It would be useful to have a plot of how the performance of the methods changes with generated data and compute time. Augmentation methods like CutMix, RandAug have the advantage of being relatively efficient and more scalable than generation methods. This may matter if the difference in accuracy is not as large e.g. between RandAug and ALIA in Fig 5. - Background domain from test images - ALIA seems to be informed of the test domain, at least for iWildCam, while the other baselines are not. Domain adaptation methods that makes use of unlabelled target images, or test time adaptation methods may be more relevant baselines.

Questions

Other than the questions above, - There are existing datasets with contextual bias e.g. Waterbirds, CelebA. I was wondering why the authors chose to create one with the Aircraft benchmark. - Is it interesting that img2img does not work well for the Aircraft dataset, do the authors have any intuition why?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper proposes a method to augment an existing vision dataset with samples that are likely reflective of task-relevant variations that are potentially missing in the same. To do this, the authors propose a method, ALIA (Automated Language-guided Image Augmentation), which utilizes off-the-shelf image-generation and language model pipelines to first generate diverse domain descriptions using a captioning + LLM pipeline which can then be used to edit an existing image to generate diverse variations of the same. Specifically, ALIA consists of three steps – (1) prompting an LLM to generate domain descriptions (likely capturing visual variations) based on a collection of generated captions associated with the original dataset, (2) using an image editing pipeline (Img2Img / Instruct Pix2Pix) to generate edits and (3) automated filtration steps to ensure the generated images and visually consistent and can reliably augment the existing dataset. From experiments conducted across three benchmarks – domain generalization (iWildCam), fine-grained classification (CUB) and classification in the presence of contextual bias (custom split of FGVC-Aircraft) – the authors show that ALIA guided expansion of the data distribution is most-effective in improving performance over a vanilla baseline and other prior augmentation strategies considered. Additional ablations outline the extent to which prompting, filtration and the editing mechanism impact performance.

Strengths

The following points outline the strengths associated with the submission. - The paper is generally well-written and easy to follow. The authors do a good job of motivating the base observations – (1) circumventing additional data curation by using generated images and (2) adopting conditional text-guided edits as a more structured way to generate samples to augment an existing dataset. The introductory section does a good job of outlining the necessity of individual components in ALIA – the necessity of invoking task-agnostic language descriptions to guide edits, avoiding fine-tuning of individual large-scale models and including an automated filtration step. - In my opinion, compared to prior work (as noted by the authors as well), the novelty of the proposed approach lies in automating the pipeline to generate augmented samples with minimal interventions. While it remains to be seen the extent to which “avoiding fine-tuning” will translate to more complex settings (images with multiple objects) and tasks (structured prediction), ALIA seems like a novel step in this direction of expanding data distributions by ensuring broad coverage of variations likely to be seen in test-time settings (modulo the obvious caveats acknowledged by the authors in Section 6). - The proposed method seems to work and leads to improvements over a vanilla baseline, other diffusion-guided (conditional / unconditional) editing schemes and prior augmentation strategies, and is often competitive with an oracle setting where one has access to data from the test distribution. With the exception of the points raised under weaknesses, the significance of ALIA lies in the fact that it is a straightforward and timely combination of existing techniques in language modeling and image generation / editing / personalization, and works fairly well off-the-shelf across multiple settings. Additionally, I particularly like the filtration process – designed by first identifying potential failure modes and subsequent methods to counter those. - The ablations (coupled with underlying hypotheses wherever applicable) are useful and provide actionable insights about the extent to which components in ALIA are sensitive to the data and the model at hand – for instance, while InstructPix2Pix is more useful compared to Img2Img, in the contextual bias settings, it leads to artifacts for iWildCam edits. These observations are likely going to be useful for future work attempting to build on top of ALIA.

Weaknesses

The following points outline the weaknesses associated with the submission. Most of these points are either associated with the significance and completeness towards the intended goal. - Given that the description of a “domain” is left somewhat open-ended in the current version (L83-85), the paper would benefit from including the discussion (supported perhaps by quantitative results) surrounding – (1) what kind of semantic / stylistic variations are missing in the base-dataset and (2) whether ALIA explicitly counters those scarcities by introducing relevant edits. While the examples provided for the contextual bias settings in Figure 7 help highlight how over-represented settings are pruned (and under-represented ones are highlighted), a discussion surrounding all different settings would significantly strengthen the submission. - Following up on the previous point, in addition to above, including an analysis informing the extent to which one needs to augment an existing dataset would be useful as well – is it necessary to consider an expansion of 20-100% in most settings? It may not matter as much for small-scale settings, but for someone intending to build on top of ALIA, it may be useful to know if one runs into a “diminishing return” scenario for different datasets at any point – i.e., if gains obtained by adding more diverse samples become increasingly marginal beyond a point. - L215-216 states that domain descriptions for the background images from the test-set were used to generate augmented samples in iWildCam. Coincidentally, iWildCam is the only setting where +ALIA > +Real. Since assuming access to the specific “kind” of target domain variations during training is not entirely fair (one does not know the test-time variation apriori), the paper would benefit from relaxing claims surrounding “beating real data” for this specific setting. - For the contextual bias experiments, it might be beneficial to consider settings where spurious correlations have been studied heavily (for instance, the NICO [A] benchmark). Not only would that involve a more diverse “base” dataset, but also help compare with more sophisticated algorithms (a subset of which are designed to explicitly counter spurious correlations). This is motivated from the fact that supplementing with “targeted” data may not always be the best solution in contextual bias settings. [A] – Towards Non-I.I.D. Image Classification: A Dataset and Baselines

Questions

The points outlined under strengths and weaknesses influence my rating. Regarding weaknesses, my suggestions are intended more towards improving completeness and significance of the results presented in the current submission. Among these, I think (1), (3) and (4) are crucial weaknesses which are perhaps central to the claims of the paper. Addressing these would definitely help me in improving my rating of the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The only potential negative societal impact that I foresee is that models trained on ALIA augmented datasets can “unintentionally” inherit biases present in the base models (LLM, Diffusion) in the ALIA pipeline. While this may not matter as much for the datasets being considered for experiments in the submission, it is perhaps crucial for other work attempting to build on top of ALIA for sensitive situations. Adding a discussion (with obvious disclaimers) would improve the current submission.

Summary

This paper explores improving robustness to variations in domains with the use of modern large vision and language models. They propose ALIA (Automated Language-guided Image Augmentation) which generates automated descriptors of data domains and augments training data with language-guided image editing. They use a model trained on the original dataset as a quality filter to ensure class-relevant information is maintained. They show this approach leads to nice gains on fine-grained datasets or data with significant clutter, for both classification and detection. After reading the authors rebuttal and discussing with them, I will maintain my score of 7.

Strengths

It’s clever to combine recent large models in this way, using image captioning to generate appropriate background or context caption templates across based on what is seen in the dataset, and allowing language-guided editing to provide diversity in a more realistic manner than randaug or cutmix based on those templates. In particular, the use of semantic features and visual features (via classifier confidence) to filter failures both where the image is only minimally changed or where the image is changed in a way that corrupts class information is quite nice. The experiments, across several datasets and dimensions of challenges including domain shift and contextual bias, were nicely done and visualizations were clear and informative.

Weaknesses

Since the domain descriptions are build with the existing dataset, it seems that this method may not work as well for cases where the dataset in question is not representative of all potential domains seen in practice, or where the dataset has significant bias or gaps. Additionally, since the diverse augmented data is conditioned on training examples and the category in question often is minimally changed, there is still limited diversity for rare species with few examples during training, particularly rare pose diversity. It would be nice to explore this more deeply on a large-scale fine grained dataset with significant imbalance, such as iNaturalist. The biggest weakness of this paper is lack of clarity and transparency about some of the data choices made. In particular, it wasn’t clear what protocols were used when selecting subsets of iWildCam for train and test.

Questions

Could performance be further improved by including domain descriptors for test data (assuming access to test data at inference time, but not test labels) for cases with domain shift between train and test? In some cases, context is highly important and valuable species identification information for human experts, and certain species would not ever been seen, eg, perched on a birdfeeder, out at night, in the water. Have you considered learning class-appropriate subsets of language templates per-species, to reduce potentially confusing and unrealistic generated context? I noticed that you considered only a subset of iWildCam classes and camera locations, what was the reasoning behind the choice of subsampling? Did you artificially balance the dataset at all (ie removing rare categories or capping common ones) or did you keep a realistic shift in subpopulation distribution across different domains? How were the real test-domain images sampled for comparison, uniformly across the test camera sites and categories of interest or?, Did you make sure that the test data was identical when comparing (ie remove that added test data from evaluation for all models)? It would be good to be very explicit about all of these choices whenever you use a non-standard split of a dataset Does performance ranking across baselines change if we consider different metrics? For example, if we consider top-1 accuracy or break down performance per class on iWildCam instead of looking at the macro f1? Does this method improve more for rare classes or common ones?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitations section is well done, and points out several clear potential failure modes of this method. More detailed analysis of failure cases would be appreciated. For which classes does this method help, and for which does it hurt? Are there any consistent patterns in data that the model finds difficult to predict accurately, even after augmentation? It would help the reader build additional intuition for where gaps remain and in which cases this method is ready for deployment.

Summary

This paper presents a data augmentation methods. It first leverages the advanced image captioning and large language models to generate prompts. Next, it uses diffusion-based models to generate more images. In addition, it trained a model to filter failure or similar cases during image generation. Extensive experiments over 3 specialised tasks demonstrate the effectiveness of the proposed method.

Strengths

1. This paper is general clear and easy to follow 2. The proposed method is simple and technically sounds 3. The proposed method effective improves the classification performances over the reported three tasks

Weaknesses

This paper studied the generated images through diffusion models on "specialised" classification tasks. However, generated images through diffusion models have been proved to be effective on improving "general" classification tasks [8, 35]. Therefore, the novelty and contribution of the proposed method seems to be incremental.

Questions

1. It have been studied in [8, 35] that images generated from diffusion models could help to improve the image classification performances. The author claims that "we use diffusion models to do image editing with text rather than generating images from text alone, resulting in augmentations that closely resemble the training data without finetuning." However, I'm thinking stable diffusion models (text2image models) could generate more diverse data as compared to text editing diffusion models (e.g. InstructPix2Pix). Then why text editing methods are better? Experimental results to compare the augmented data through text2image and text editing methods are expected. 2. This paper aims to improve the performances on specialised tasks that have very few training data. However, some of these tasks may contain out-of-domain data that may be difficult to be generated through diffusion models. Discussion regarding to these cases are expected. 3. It is mentioned "We fine-tune a ResNet50 for the CUB [40] and iWildCam [15] datasets and a ResNet18 for the Planes [21] dataset". Why use different networks for different datasets? 4. In Appendix, the generated images through img2img of Airbus and Boeing look very strange. Discussion regarding to these results are expected.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Thanks for the response!

Thanks for responding to my concerns and providing additional experimental results. My primary concerns surrounding the shifts ALIA tackles and the impact of the amount of generated data were adequately addressed by the rebuttal. Most concerns from other reviews seem to have been addressed (with supporting discussion as well) adequately as well. I would encourage the authors to include (additional) discussions from the reviews in the revised version.

Thank you for the clarifications

Thank you to the authors for the clarifications on data subset construction. This line of research on improving specialized OOD performance with augmentation from generated images is distinct from the more common in-distribution challenge presented by datasets like ImageNet. This work demonstrates a nice step forward in exploring what works well for OOD diversification, despite restricting the realism and possibly usefulness of the test case by focusing on common categories that are well represented in Stable Diffusion (I would still be very interested in seeing iNaturalist results in the camera ready). Because of this, I will keep my score of 7 for this work.

general vs specialized

Reviewer u22M, I'm curious how you define the difference between "general" classification tasks and "specialized" classification tasks?

Thanks for the clarifications!

Thanks for the additional results on waterbirds, runtime and clarifications. It is nice to see a paper focusing on non ImageNet tasks. I also appreciate the insights on how edits can affect datasets differently and the discussion on filtering failed edits. I have read the other reviews and responses too and have decided to increase my score to 7.