Return of Unconditional Generation: A Self-supervised Representation Generation Method

Summary

The paper proposes a framework, coined Representation-Conditioned Generation (RCG), that aims to bring the advantages of Conditional Generation techniques to Unconditional Generation settings. To do so, they use the output of a representation generator network instead of class labels. This representation generator is trained in a prior stage to approximate the distribution of image features extracted by a pre-trained self-supervised network (MoCo-v3). They test this technique with a set of very different generative models, including a latent diffusion model, a diffusion transformer, and a masked generative transformer, showing improvement accross the board on the ImageNet Unconditional Generation benchmark.

Strengths

a. The proposed method is sound, the technical details are delivered clearly and thoroughly. b. Experiments and ablations are extensive, and results look robust, being confirmed for very different classes of generative models. c. The paper provides new empirical evidence and numbers on very relevant points: - Diffusion models can generate self-supervised representations convincingly. - Those generated features can serve as good conditioning signals to help train image generation models.

Weaknesses

a. Technical novelty is limited: [48] also generates images by first generating pretrained features to condition the image generator, and works such as [5,A] manage to generate high-quality images using representations. The remaining differentiator to existing work in terms of technical contribution is then in the choice of the pre-trained feature extractor and image generator. b. In that light, while thorough ablation results are presented, I find the interpretation of the results lacking. If those choices are important, and it seems to be the case considering the range of FID scores in the ablations, it would be very useful to provide intuitions about how to make those choices and why. -- [A] SODA: Bottleneck Diffusion Models for Representation Learning. Hudson et al. CVPR 2024

Questions

- In Table 8, it is difficult to have an interpretation of the FD as we don't have a reference point. Maybe showing FD between training and validation set could be an option, provided the number of samples for the estimates is handled carefully. Also, FD is evaluated on the training set. It would be good to have validation scores as well. - In a similar vein, I'd like to see the performance (FID) of RCG on features sampled from training images (similar to Figure 6). This would give another indication of how good the representation generation is. This is important since the representation generation is a major, and arguably the most important, contribution of the paper. - The authors mention in Appendix B that the FID in the corresponding section are computed on the validation set. Is that also the case for the other values in the main paper? - For the point raised in weakness b., a precise question I'd like to ask is why, as seen in Table 7a, MoCo v3 features are so much better for this task than DINO or iBOT features that are supposed to be better in other downstream tasks. - Also linked to weakness b., there are strong connections between generative models and self-supervised learning. In fact, works such as [61, A] or [41] on which the current paper builds a lot of their results are already generative models that explicitly double as representation learners, or even have representation learning as their main objective. Those ties should be discussed in the paper. Moreover, if a generative model can be a representation learner, and if using a representation can help achieve a better generative model, how to explain the good results in the submission becomes even less clear. I'd be happy to hear the authors' thoughts on that point. In any case, I think the submission is already sound and of interest. It shows very good and solid results on a relevant topic, and opens up interesting research directions. But also, it doesn't provide much in terms of technical novelty or in terms of analysis. The paper touches upon, but barely scratches the surface of very fundamental questions related to self-supervised methods and the training of generative models. I would definitely consider raising my rating to 7 or 8 if it would have a slightly more complete evaluation of the feature generation part and if it provided a deeper analysis of the results.

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

2

Limitations

The authors make an attempt to address limitations by showing failed generation. They could discuss the scope in terms of data type: are there results likely to transfer to data? what hurdles to expect in doing so? They also try to adress societal impacts by discussing generative model biases and hypothesize that unsupervised models should significantly mitigate the influence of human bias. I'd appreciate if they would either expand on that point or alternatively refrain from making such an assumption without a stronger backing. ImageNet photos certainly have been produced by humans and are not bias-free. Since they are willing to discuss the limitations and impacts of generative models in general, they could also mention potential misuses such as deep fakes and misinformation.

Summary

This paper proposes generative models conditioned on representation obtained from a pre-trained self-supervised encoder to achieve high-quality diverse generation.

Strengths

1. The writing is clear. 2. The experimental results demonstrate significant improvement over unconditional generation.

Weaknesses

1. The idea of generation conditioned on representation is not new. As the authors mentioned, conditioning on instance representations has been well studied in the community. Though the authors argue that they require ground-truth images to provide representations during generation, the issue can be addressed by adopting the idea from VAE to align those instance representations with the prior noise or by incorporating the representation generator proposed in this paper. Therefore, I would say the main contribution of this work is to improves those representation generation works by introducing a generative modeling for the representation, which however is limited. 2. The deep clustering community has also adopted the technique of self-supervised learning for a more clustering friendly representation and has achieved significant improvement in complex datasets, like ImageNet. Therefore, generation conditioned on clustering structures also has potential to bring gaps between unconditional generation and conditional generation. I would say the studied problem has been well addressed in the community. Comparisons with conditioning on clustering should be included.

Questions

1. When extending to class-conditional generation, is it required to fine-tune the model?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

1

Limitations

The limitations should include the discussions on the necessity of a self-supervised encoder and how to obtain such an encoder for datasets, especially for other modalities such as text, speech.

Summary

This paper proposes RCG, a novel framework to enhance unconditional image generation by leveraging self-supervised representations. The main idea is first to generate self-supervised representations unconditionally using a pre-trained encoder and then condition the image generation on these representations. This process does not require human-annotated labels and involves training a lightweight diffusion model to generate representations efficiently. The authors demonstrate the effectiveness of RCG by significantly improving the quality of unconditional generation across various architectures, closing the performance gap between unconditional and conditional generation methods.

Strengths

**[S1]** The paper is well-motivated, addressing the importance of unconditional image generation for utilizing abundant data, which has remained stagnant compared to conditional generation. **[S2]** The idea of utilizing self-supervised learning for image generation makes sense and is a novel idea. **[S3]** The paper shows significant performance improvements.

Weaknesses

I found no weaknesses.

Questions

**[Q]** Is there any intuition as to why MoCo-v3 is the best representation for RCG?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

They addressed the limitations.

Summary

This paper proposes a very simple unsupervised image generation framework that does not rely on human labeled annotation without compromising generation quality. This framework has two stages: i) representation generator learning and ii) image generator learning. The representation generator is trained in the form of diffusion model training to take a noisy latent image as input and output the corresponding representation encoded by a self-supervised image representation model. The image generator is trained to generate the image corresponding the given representation (encoded by the self-supervised representation model). This paper demonstrates that this simple framework is effective in achieving generation quality comparable to the counterpart supervised learning methods, regardless of architecture type.

Strengths

1. Simple but effective framework for unsupervised image generation. The proposed method is designed to solve difficult problems very simply and intuitively, so it will be able to provide inspiration not only in this field but also in a variety of other fields. 2. Great presentation. It was very helpful in understanding this paper since it explained the information covered in each section in a very informative but concise manner. 3. A variety of experiments can support the authors' claim and the effectiveness of the proposed method.

Weaknesses

I do not have a major concern for this work. I only have two questions in the method. 1. When training the image generator, why was the representation of the SSL model used instead of the output of the representation generator trained in the previous step? Although the representation generator is trained to distill the generation capability of the SSL model, isn't it more appropriate to train under the same conditions as in the inference in which the output (representation) of the representation generator is used? 2. Is there any way for the two generators to be trained together in an end-to-end manner?

Questions

Please respond the two questions I raised in the weakness section.

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

This paper properly deals with the limitation and potential societal impact in the supplementary material.

Thanks for the responses.

I thank the authors point out their ablation study about conditioning on cluster labels. So, I raised my score. However, I still feel that the technical contribution of this paper is marginal as I demonstrated in Weakness 1. Moreover, as shown in their results, the performance of conditioning on cluster labels is relatively good, though a little worse than the proposed method. That proves that generation conditioned on clustering structures has great potential to bring gaps between unconditional generation and conditional generation. Image generation without labels does not lag so behind its conditional counterpart with labels.

Latent variables with non-collapsing posteriors

Just think of the representation space of the MoCo encoder as a latent space and the diffusion model trained over this space as a prior over the latents. You can induce a reasonable posterior distribution over a finite sample of these latents using the softmax formed by scaled cos sim between the MoCo representation for an input image X and a finite sample of representations generated by the diffusion-based prior over MoCo representations. You can sample a generated representation based on that softmax and use it to condition generation of X. You could also play around with, eg, varying the temperature of the softmax in order to limit the information capacity of the latent variable (higher softmax entropy means less info about X). We can think of the softmax in contrastive learning, when evaluated only over negative samples, as estimating a conditional distribution over a non-parametric dictionary of representations that assigns higher probability to representations which are most similar to the representation of the "positive sample". This conditional should work reasonably well for sampling "semantic" information about X to use in conditioning. This also avoids a train/test mismatch in the representations used for conditioning the decoder, since they're always samples from the prior. Approaches like the one described above decouple decisions about what information to cram in a latent variable and how to decode that information to produce a generated image. In the olden days, with deep VAEs and such, there were often issues with "posterior collapse" when adding a more powerful decoder p(x|z). Basically, z would be ignored since the model for p(x|z) was powerful enough to just act like p(x) without much impact on the training likelihoods. In the "representation conditioned" setting described above, we can decide what information should be in the latent variable (eg, whatever MoCo happens to capture), and how much information to condition on when generating X. In the setup described above, the amount of info about X is limited to the log of the number of prior samples in the contrastive softmax minus the entropy of the contrastive softmax. This lets us define a nice latent space and non-collapsing posteriors over that latent space which represent strictly controlled amounts of "semantic" information which can guide data generation. I also think this setup would work fine without the diffusion model over MoCo representations. You could probably get away with just defining the prior as uniform over the appropriate hypersphere. This would make training and sampling a bit quicker. It's also straightforward to extend this approach to hierarchical latents, latents with variable bandwidth, etc.