Text-to-Image Diffusion Models are Zero-Shot Classifiers

Summary

This paper uses the text-to-image diffusion models as zero-shot classifiers. It proposes to compute a subset of the full scores matrix to be more efficient. It proves Imagen and Stable Diffusion have good zero-shot performance and are robust to misleading textural cues.

Strengths

1. It is novel to use pre-trained diffusion models as zero-shot classifiers. 2. Building a scores matrix with each label prompt is reasonable for me. 3. The efficiency is very important in this setting. The author well discussed this point in section 3.1. 4. It is interesting to see the generalization ability of large pretrained model leads to robustness to shape-texture conflicting cues. 5. Empirical results are sufficient to support the conclusion.

Weaknesses

1. It lacks some theoretical discussion about why diffusion models can perform well in classification. 2. Classification is a simple task. I am not sure whether it is too easy for a large model.

Questions

1. Is it possible to extent this work to various downstream tasks such as detection or segmentation? 2. In large datasets like ImageNet, there are many kinds of sub-classes such as many types of foxes. In this case, is it reasonable to use the diffusion model since they are very similar in noisy images?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to "Questions".

Summary

The paper presents a new method that utilizes generative models as image classifiers and initiates explorations of notable, open-source models using this approach. Beginning with recognized image datasets, the researchers examine the models and evaluate the scores they have achieved. Several experiments are conducted to profile the behavior of these models, such as determining their optimal operating resolutions and dataset types. The paper assesses these models' capabilities by evaluating dataset competencies, such as pointing out the prominence of the Imagen model with the MNIST dataset due to its strong text generation skills. Further characterizations are made on the models' ability to handle shape-texture conflicting cues using the Cue-Conflict dataset. The paper suggests that generative models, utilized through their proposed method for recognition tasks, exhibit more robustness compared to traditional ConvNets. Lastly, the paper explores how these models perform in attribute binding tasks. The researchers report that while the CLIP model's performance is near random, both the Stable Diffusion (SD) and Imagen models show promising potential in these tasks.

Strengths

The strengths of this paper shine through in several ways. Firstly, the paper explains the research and results in a way that's easy to understand. This clear writing helps to make complex ideas accessible to a wider audience. The paper's thorough exploration of generative models also sets this work apart. They dive deep into understanding how these models behave. This includes looking at how well the models can handle images where shapes and textures don't match, as well as how they can connect attributes together. These insights are not just interesting, but they also add to our understanding of how these models work. Another key strength of this paper is the smart improvements the paper makes. For instance, they use a technique called timestep weighting to improve how the models classify images. This technique is a smart way to reduce the impact of noise, making the models more reliable as they handle larger timesteps if they are sampled. Such improvements amplify the significance of the paper.

Weaknesses

# Pre-rebuttal While the paper continually emphasizes the intention to introduce a method for using generative models as classifiers, there's a significant overlap with the methodology presented in "Your Diffusion Model is Secretly a Zero-Shot Classifier", which also uses the Stable Diffusion generative model as a classifier. Despite the similar scoring mechanisms, they attribute their improved results to the application of timestep weighting and a more efficient class pruning method. This does not, however, fully offset the lack of originality due to the similarity of their method with the cited work. Furthermore, by choosing not to pursue enhancements such as prompt engineering, the authors have seemingly missed an opportunity to establish a stronger benchmark for future studies. This decision, coupled with the questions regarding the paper's originality, could limit the overall significance and potential impact of their work. Certain parameters used in the paper, such as the cutoff_pval, are not adequately examined for their effects, making them appear as arbitrary choices or 'magic numbers'. Similarly, the weighting function used remains largely unexplored and unexplained, leaving a gap in understanding its impact on the results. The paper provides comparisons for their efficiency improvements, including shared noise and class pruning methods. However, the paper falls short in determining the peak accuracy reach for the vanilla method, leaving an element of uncertainty and a lack of thorough comparison between the methods employed.

Questions

1. L271: few-shot classification? 2. Why not use more commonly used zero-shot (or few-shot) learning datasets such as CUB, AwA or miniImageNet? 3. Why not compare against strong (even if not sota) ZSL baselines with various benchmarks? 4. How do you tune the hyper-parameters? Setting any HP wrt. test performance is a violation of ZSL protocol. Can you describe a reproducible HP tuning procedure? 5. Regarding the compositional generalization experiments: why not use existing compositional zsl benchmarks and compare against existing methods?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Most limitations are listed above. No additional (especially societal) limitation to be reported here.

Summary

This study explores the potential of text-to-image diffusion models as zero-shot classifiers. The models show competitive performance with CLIP on zero-shot image classification datasets and excel in shape/texture bias tests and attribute binding. The findings suggest that generative pre-training should be considered as a compelling alternative for vision-language tasks.

Strengths

1. The authors thoroughly investigate the zero-shot classification capabilities of text-to-image diffusion models through extensive experiments, covering both standard and challenging benchmarks. 2. Additionally, the authors compare the classification performance of Stable Diffusion (SD) and Imagen, revealing intriguing results that suggest diffusion models trained on original images exhibit superior generalization compared to those trained on the latent space of VAE. 3. The authors offer a range of practical and effective techniques to accelerate the classification process, providing valuable insights for improved efficiency.

Weaknesses

1. While the authors present various strategies to speed up the classification process, it remains significantly slower compared to traditional classification models. 2. It would be beneficial to include a comparison with other multimodal models like BLIP. 3. There appears to be a disconnect between lines 192 and 193, as the introduction of Imagen is only mentioned in that paragraph. 4. Considering that CLIP is trained with 400M text-image pairs and SD is trained on Laion-5B, a direct comparison may not be entirely fair. It would be interesting to see the performance gap between SD and CLIP when both models are trained on the same amount of data.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper inverts pre-trained text-to-image diffusion models by using bayes rule, and evaluates them over a variety of benchmarks. For the evaluation they use two diffusion models: Stable Diffusion and Imagen. They compare these models against CLIP-L/14. They show a variety of benchmarks where the diffusion model does better at classification than a standard SOTA discriminative model such as CLIP-L/14.

Strengths

i) The paper does a very dense evaluation of their proposed method ii) They give good analysis on why/where generative classifiers would be useful over their discriminative counterpart. iii) The paper proposes weighted timesteps and sampling methods that improve the accuracy and speed of the classifier.

Weaknesses

i) the analysis become a bit weak , as none of the models considered are trained on the same dataset. ii) Ablations are in the supplementary and results are ablated only on one dataset, would be good to have atleast 2-3 diverse datasets (small/high resolution) I have few qs that i have listed below. minor - Line 271 shouldn't it be zero-shot instead of few-shot?

Questions

i) In Table 4 weighting seems to help significantly help Imagen however not so much for SD? why is that? what if u learned weighting for SD seperately and tried to generalize it to different datasets? ii) Are any of the models trained on the same/similarish datasets (ViT22B vs Imagen)? It would be good to clarify this in the paper. iii) The paper raises resolution mismatch of the generative model (specifically Imagen), a big concern. Do the authors expect future generative models that are explicitly trained for classification to resolve this issue? If so how? as SD does train on higher resolution but doesn't get better performance than CLIP

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes