StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners

Summary

The paper investigates the potential of using synthetic images generated by text-to-image models to train self-supervised image embedding models. Methodologically, the authors propose StableRep, a multi-positive contrastive learning method that treats multiple images generated from the same text prompt as positive examples for each other. The paper demonstrates two key findings. First, when the generative model is appropriately configured, self-supervised methods trained on synthetic images can achieve comparable or superior performance to real image counterparts. Secondly, when language supervision is incorporated, synthetic data become more efficient.

Strengths

1. **Interesting topic:** The paper investigates the potential of using synthetic images generated by text-to-image models as strong visual representation learners. This exploration addresses an important and timely question. 2. **New contrastive learning method:** The proposed StableRep is new and tailored for synthetic data. 3. **Comprehensive evaluation:** The paper evaluates the representations learned by StableRep on large-scale datasets and compares them with strong baselines.

Weaknesses

1. **Potentially entangled comparison in terms of the amount of data:** The only concern I have for this paper is the claim of data amount. When claiming "synthetic and real images of the same amount", the authors refer to the amount of data for representation learning. However, Stable Diffusion is trained on LAION-2B. Therefore, 50M synthetic images may contain information from 2B images, making this comparison less meaningful. I'd like to see more discussion on this issue.

Questions

See Weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations are included in Section 7.

Summary

In this paper, the authors investigate the potential of learning visual representations using synthetic data generated by text-to-image models. The authors choose Stable Diffusion for exploration and extensive experiments demonstrate that self-supervised models trained on synthetic data can perform better or at par with training on real data. The authors also propose a multi-positive contrastive variant called StableRep to allow multiple images generated from the same text prompt as positives for each other.

Strengths

- The paper is well motivated. Given the recent progress of large-scale text-to-image generative models like Stable Diffusion, it is timing to investigate the effectiveness of these models in generating high-quality images to assist discriminative tasks. - The paper is generally well-written and easy to follow. - The experiments are extensive and the results seem promising.

Weaknesses

- The authors use large-scale image-text datasets like CC3M, CC12M, RedCaps for study. These datasets have well-collected image captions. Although the authors claim that the proposed method can reduce the reliance on collecting large-scale real images for learning representations, the cost of collecting these image captions is also somewhat expensive and should not be ignored. Thus, a more cost-effective and interesting setting should be to generate a synthetic ImageNet dataset using category labels as text prompts (e.g., a photo of [category]) since it does not require any human effects of collecting captions. Then self-supervised models could be pre-trained on the real ImageNet dataset as well as the synthetic ImageNet dataset for comparisons. I am curious about whether the self-supervised methods trained on synthetic ImageNet still have advantages over training on real ImageNet. - It seems that the synthetic data only have performance advantages for linear probing experiments. For few-shot experiments, self-supervised methods trained on synthetic data still have a large performance gap with training on real data, which limits the practical applications of using synthetic data to some extent. - The authors only evaluate the proposed method on classification downstream tasks. What about dense prediction downstream tasks like object detection or semantic segmentation? Will the synthetic data still have the advantages on these kinds of downstream tasks? - According to ablations in Sec. 4.2, generating multiple images per caption yields better performance. However, this may be due to simply increasing the number of views. To eliminate the interference of the number of views, the authors should also conduct an ablation by replacing multiple generated images per caption with the same number of cropped views per real image.

Questions

I am concerned about the questions mentioned above. Given the current status of the paper, I am leaning towards borderline accept and hope the authors could address my concerns.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed the limitations and broader impacts in Sec. 7 (main text) and Sec. D (supplementary material) in detail, which look good to me.

Summary

The authors presents a novel method for learning visual representations using synthetic data. The authors leverage text-to-image generative models (Stable Diffusion) to synthesize images from textual prompts, which are then used to train a self-supervised visual representation model. The synthetic data generation process is guided by a set of diverse and non-repetitive textual prompts, which helps in creating a wide variety of images. The authors demonstrate that their approach outperforms traditional self-supervised learning methods that rely on real images, especially when the amount of real data is limited. They also propose a multiple positive samples based contrastive learning approach called StableRep to utilize different synthetic images generated from a source text prompt, which outperforms training with simply real or synthetic images on a variety of downstream tasks.

Strengths

1. **Important Problem:** The authors study a very important problem, how to utilize synthetic images for training vision and vision-language models. They also show success at this central problem (with caveats, see weaknesses). They are one of the first studies showing successful use of synthetic data from generative models on standard computer vision benchmarks. 2. **Novel approach:** The multi-positive StableRep approach proposed by the authors for multi-caption invariance learning in self-supervised models is a novel idea and seems to work very well in practice leading to significant improvements over simply using synthetic images for training. 3. **Ablations:** The authors perform extensive ablations on the role of guidance scale, and also include a study on how additional language supervision could be used to increase caption efficiency while training with multiple positive synthetic images. The results indicate that proposed approaches (StableRep and StableRep+) indeed are more efficient versus training on real images.

Weaknesses

1. **Limited motivation behind why synthetic images:** It is unclear to me as a reader why Synthetic Images are being used in Section 2.2 as training data for pre-training. It seems that there is no real benefit from using them on state-of-the-art contrastive models like BYOL and MoCo-V3 in terms of linear accuracy. Plus, the key issue for training is the lack of real world labelled data as mentioned by the authors' central question "how can we collect such large amounts of varied data to train AI models?" (L21). There is plenty of real-world unlabelled data that is available and can be used to train self-supervised models. The real benefit that synthetic images present is unlimited labelled data. However no fine-tuning comparisons or supervised learning comparisons are provided except for fine-tuning MAE, which could very well be explained by the stochasticity in model training (82.9% vs 82.6%). Section 2.2 does not do much to justify the usage of synthetic images for training in a manner that addresses the authors' central question, and follow up sections justify it for efficiency but not performance. 2. **Limits of generative model:** While the authors include a limitation section, and mention briefly the issues that affect generative models like Stable Diffusion, there needs to be more rigorous evaluation of these issues. Conceptually, these methods suffer from issues like limited compositional understanding. There have been countless studies on the social biases reflected in these models. Very lately (after the NeurIPS deadline), there was an important work showing that models trained on data from generative models can suffer from model collapse, where tails of the original data distribution disappear from the subsequent trained model [1]. All these issues are quite major and should be part of this study, since it is one of the first to utilize synthetic data and show some improvements in results and could potentially effect future research in this area. In particular, the authors should include results on compositional benchmarks (like ARO, CREPE, Winoground, SugarCREPE etc) for their CLIP models trained with StableRep, discuss the fairness of the self-supervised models such as worst-class accuracy and geographic bias etc. ### References 1. Shumailov I, Shumaylov Z, Zhao Y, Gal Y, Papernot N, Anderson R. The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv preprint arXiv:2305.17493. 2023 May 27.

Questions

My main concern is with the motivation behind using synthetic images as well as the limited commentary and no evaluation of the downstream effects of using data from generative models for training. I would be willing to improve my rating if the authors address these issues (as discussed in weaknesses).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors have addressed the limitations of their work in Section 7, but it could use further commentary.

Summary

This paper investigates how synthetic data generated with the text-to-image diffusion model Stable Diffusion can be leveraged for representation learning. To this end, the paper analyzes established representation learning approaches such as SimCLR and CLIP, but trained on the synthetically generated data. Further, it introduces a novel method, dubbed "StableRep", specifically designed for representation learning from a generative model, to generate multiple "positive" image instances for downstream representation learning. Experiments demonstrate the potential of synthetic data for representation learning and mostly perform on-par with or better than when trained on real data only.

Strengths

This is a well-written paper, which approaches a timely research question: To what extent can large-scale, generative models which have been trained on internet-scale data be leveraged for other tasks, by augmenting or replacing real data. The experiments are well-designed and hint at the potential of using synthetic data. StableRep is a nice method specificially designed for generative models that can generate multiple positive examples for a given prompt. The experiments are encouraging for future research in this area.

Weaknesses

While experiments demonstrate potential, there remain a few unaddressed points: How does the choice of generative model influence the results for representation learning? In particular, Stable Diffusion is conditioned on CLIP text features, which themselves are pretrained through CLIP's contrastive objective. There exist, however, other generative text-to-image variants that use non-contrastive features for conditioning, such as T5-representations (e.g., Imagen [https://github.com/deep-floyd/IF], IF [https://github.com/deep-floyd/IF]). Further, the paper does not discuss how the size of the pretraining dataset for both CLIP (for conditioning SD) and the training data for SD change performance of StableRep (the pretraining data is much larger than CC3M/12M). In addition, the possibility of data poisoning w.r.t to the results presented in Tab. 2 and Tab. 3 is not discussed (for example, SD's training data might have contained examples from the linear probing datasets). The performance gap between CIFAR-10/100 and the other datasets remains unexplained. l. 1014-107 need more detail on hyperparameter settings like number of steps, learning rate used, convergence of the model. For further comments, please refer to the "Questions" section.

Questions

- Can we mix synthetic and real data? How does this affect the outcome of training approaches like CLIP and SimCLR or StableRep? - Other sampling hyperparameters than guidance scale w are not discussed. How does the choice of solver change the results? Same question for the number of denoising steps / function evaluations... - small note: In Eq. 1, t could be misinterpreted as timestep variable, as often used in the context of diffusion models. - Sec. 4, l. 229-233: Do the instabilities for larger models only occur for synthetic data? - Sec. 5, l. 240-245: This section speculates about the inferior results when training CLIP on synthetic data only (vs real data only). Why not plot the imagenet zero-shot accuracy vs clip-image similarity for different guidance scales? Using a better model than SD 1.5 might also help clarify this point.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

yes.

Summary

This paper studies pre-training by using generated image from diffusion models. It presents StableRep that generate different images with the same caption by using stable diffusion models. The model is hence pre-trained by using the generated samples and contrastive loss. Extensive experiments demonstrate the effectiveness of the proposed method by using synthetic images only.

Strengths

1. This paper studies the pre-training by using synthetic data only. Experimental results show that pre-training by using synthetic data only outperforms the pre-training over real images. 2. This paper is generally clear and well-written.

Weaknesses

The pre-training over generated images by using diffusion models have been studied in [24]. In [24], it seems direct pre-training on synthetic images could helps to improve the classification results without the proposed pre-training pipeline. As the comparison between the proposed method and the reported results in [24] is missing, it is difficult the determine the effectiveness of the proposed pre-training pipeline.

Questions

1. For general vision-language pre-training methods, image labels are usually very difficult to obtain. This paper studies the use of synthetic data, where the labels of synthetic images can be obtained during image synthesis. As such, we can simply pre-train the model by generating images together with the labels in the task-specific label spaces, which could improve the model performances [24]. As such, it is not clear why the contrastive pre-training is necessary or better when using synthetic data for pre-training. More discussion and experiments are expected. 2. According to the experimental results, pre-training on synthetic data helps to improve the performances on most of the datasets while may introduce degradation some of the datasets (e.g. CIFAR-10). How about combining the synthetic and real datasets together for pre-training?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Fix the cifar-10/100 performance caused by resolution gap

Dear Reviewers and AC, We have fixed the performance gap on CIFAR-10/100 caused by resolution issue, and now present the updated linear-probing / few-shot results (corresponding to Table 2 & 3 results in paper) here: - Linear probing: | method | cifar-10 | cifar-100 | aircraft | cars| dtd| flowers| pets| sun397| caltech101 | food101 | voc2007| Average | |--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | real SimCLR | 88.3 | 70.3 | 47.1 | 45.5 | 76.2 | 92.5 | 70.1 | 65.4 | 83.8 | 75.0 | 81.2 | 72.3 | | real CLIP | 94.0 | 79.0 | 53.2| 75.8 | 75.7 | 96.0 | **86.7** | 72.5 | 92.7 | 81.6 | 86.1 | 81.2 | | Stablerep | **96.4** | **83.8** | **56.8** | **80.5** | **79.4** | **97.2** | **86.8** | **72.9** | **94.3** | **83.3** | **87.1** | **83.5** - few-shot evaluation: | method | cifar-10 | cifar-100 | aircraft | cars| dtd| flowers| pets| sun397| caltech101 | food101 | Average | |--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | real SimCLR | 64.0 | 70.4 | 40.7 | 50.9 | 82.2 | 92.1 | 74.4 | 94.0 | 90.4 | 70.4 | 73.0 | real CLIP | 77.5 | 82.1 | **62.0** | 90.9 | 83.3 | 97.6 | 91.1 | **97.2** | 98.2 | **87.0** | 86.7 | | Stablerep | **92.7** | **92.0** | 60.4 | **91.8** |**86.4** | **98.2** | **91.9** | **97.3** | **98.8** | **87.1** | **89.7** What we tried is, during StableRep training, we randomly downsampled the image to a low resolution (64 or 128) and upsampled it to 256x256 (mimicking if we run into low resolution images). We prepended this augmentation to normal augmentation with a probability of $p=0.05$. After this augmentation, the performance on CIFAR-10/100 gets improved a lot, while the accuracy on the other datasets or ImageNet remains similar as before.

Response to rebuttal

Thank you for taking the time to write a thorough rebuttal, and also running the additional experiments based on my suggestions. The new results indeed solidify the utility of StableRep, and I am happy to say my concerns have largely been addressed. I have increased my rating from a Weak Accept to an Accept.

Response to rebuttal

Thanks for the authors' detailed response. Most of my concerns have been well addressed. I have one additional question. For the Syn+Real experiments provided in the rebuttal (reviewer HH2w, AnD7), could the authors explain why SimCLR Syn+Real performs worse than SimCLR Real on **few-shot avg**?

Summary of additional (interesting) results

Dear reviewers, We thank you again for providing lots of insightful and constructive feedbacks! Because of that, we are able to update many additional (and perhaps interesting) results during the rebuttal time. We present the **key** summaries here, eliminating the need for each reviewer to constantly reference other responses. **1. We identified the performance issue on cifar-10/100, and proposed a simple fix. Now StableRep consistently outperforms others** Please refer to [here](https://openreview.net/forum?id=xpjsOQtKqx&noteId=it4GmuE96a). **2. Our method significantly improves the fairness of the representations.** Thanks to reviewer qM14's suggestion, we evaluate our method on the FairFace benchmark. The comparison is: ||pre-train data|mean accuracy|best-class accuracy|worst-class accuracy| |--|:--:|:--:|:--:|:--:| |CLIP|Real|28.2|60.2|0.3| ||Syn|30.4|64.0|3.1| |Ours |Syn|**37.2**|**74.9**|**10.0** | CLIP w/ real data only achieved 0.3% accuracy with "southeast asian male" class, and CLIP w/ synthetic data improves this class to 3.1%, while our method improves this class to 27.2%. The worst class for ours is "middleastern male", which CLIP w/ real and w/ synthetic struggle too (they achieved 6.9% and 6.2%). This suggests a geographic bias and our method generally improves the fairness. **3. Our method also improves the understanding of compositionality.** We evaluate on the ARO (Attribute, Relation, and Order) benchmark: ||pre-train data| Relation accuracy | |--|:--:|:--:| | CLIP| Real|46.4| || Syn|**50.0**| |Ours| Syn |47.3| While our method slightly improves over CLIP w/ Real, CLIP w/ Syn surprisingly outperforms CLIP w/ Real by 3.6, highlighting the potential of using synthetic data. **4. Our method performs well on dense prediction task, and even outperforms the official MAE checkpoint** Following suggestion from reviewer DyTC, we evaluated representations on ADE20k segmentation dataset. We follow the original MAE paper to use a UperNet, and used the mmsegmentation library following their optimized parameters. We trained for the full-schedule (160k steps), and considered two setup: (1) frozen the pre-trained part; and (2) fine-tune the whole network. We present the results as below: Frozen: |method|pre-train data|mean IoU (%)|pixel acc. (%)| |--|:--:|:--:|:--:| |MAE|ImageNet, real|36.4|77.7| |CLIP|cc12m, real|37.9|78.5| |SimCLR|cc12m, real|39.5|79.5| |SimCLR|cc12m, syn |37.2|78.2| |StableRep|cc12m, syn|**42.3**|**80.4**| Fine-tune: |method|pre-train data|mean IoU (%)|pixel acc. (%)| |--|:--:|:--:|:--:| |Supervised|ImageNet, real|47.4|-| |MoCo v3|ImageNet, real|47.3|-| |MAE|ImageNet, real|48.1|82.9| |CLIP|cc12m, real|45.4|81.7| |SimCLR|cc12m, real|45.9|79.5| |SimCLR|cc12m, syn |44.7|81.9| |StableRep|cc12m, syn|**49.2**|**83.5**| We observe that: (1) StableRep consistently outperforms all other methods trained with real/synthetic data from cc12m/ImageNet dataset. (2) In the fully fine-tuning setup, StableRep even outperforms MAE trained with real ImageNet dataset. This is interesting and non-trivial results, because: - previously contrastive learning methods **without masking** strategy were consistently underperforming MAE in dense prediction tasks under the fine-tuning setup. In contrast, our method outperforms MAE - our method is trained purely with **synthetic** images **5. StableRep generalizes well to ImageNet** We fine-tune various pre-trained models on ImageNet, hinted by qM14. The results are presented as follow: ImageNet fine-tuning results: |method|pre-train data|Top-1 Acc. (%)| |--|:--:|:--:| |DINO|ImageNet, real|82.8| |MoCo v3|ImageNet, real|83.2| |StableRep|cc12m, syn|**83.5**| We note that StableRep performs better than other contrastive/Joint-embedding approaches (DINO, MoCo v3) pre-trained directly on ImageNet, despite the domain gap between cc12m and ImageNet. This shows that our StableRep possesses strong generalization ability. **6. Synthetic data can be unified with real data to further improve the performance** Results w/o resolution fix yet: |method|pre-train data|ImageNet LP|Downstream avg|few-shot avg| |-----------|:--------------:|:------:|:------:|:------------:| |SimCLR|Real|61.5|72.3|**73.0**| |SimCLR|Syn|63.7|72.7|70.8| |SimCLR|Syn+Real|**64.1**|**74.9**|71.9| |CLIP|Real|70.3|81.2|86.7| |CLIP|Syn|67.8|79.1|83.7| |CLIP|Syn+Real|**73.3**|**83.0**|**87.3**| |StableRep|Syn|67.9|78.0|81.4| |StableRep|Syn+Real|**69.1**|**79.3**|**84.2**| To sum up, our method has significantly improved the fairness of the learned representations and the understanding of compositionality. Besides, our method shows strong generalization ability in fine-tuning setup (including dense prediction), which is surprising since it may potentially change the previously widely-accepted story that contrastive methods do not do well in fine-tuning. Lastly, these results are achieved with **only synthetic** images.