Are Diffusion Models Vision-And-Language Reasoners?

Summary

The paper introduces Diffusion-ITM, a new method that directly adapts diffusion-based models to image-text matching tasks without retraining. Additionally, the authors collected a new benchmark called Generative-Discriminative Evaluation Benchmark (GDBench), which includes seven complex vision-and-language tasks and bias evaluation. The results show that Diffusion-ITM performs competitively on multiple tasks and surpasses CLIP. This paper underscores the significance of jointly considering discriminative and generative models and provides a new benchmark for future work.

Strengths

* The paper presents a new method to adapt diffusion-based models for image-text matching tasks without retraining. This innovation has practical value to apply. The finding of the relative difference between the with and without text conditions (Fig. 3) is pretty interesting. * The authors thoroughly evaluate the proposed method on various vision-and-language tasks, offering insights into its performance and potential biases. * The introduction of new benchmark for image generation models over the discriminative tasks provides a useful tool for the research community and enables comparative analysis. * This paper is well written and organized with clear logic to read.

Weaknesses

Weaknesses: * The Stable Diffusion includes a pre-trained CLIP text encoder and has itself been pre-trained on numerous image-text pairs. Therefore, Stable Diffusion has sufficiently learned the joint distribution of images and texts. The new finding is interesting but not surprising. * The performance of the proposed method on more challenging benchmarks, like Winogroud, is relatively poor, suggesting room for potential improvement. * The paper lacks theoretical analysis—since diffusion models are generative, applying them to a discriminative task would benefit from more theoretical insight. * This paper ignores time cost/efficiency analysis. The Stable Diffusion inference is slow. According to Eq. 5, how to assign the t in your method? How many steps are needed and why? * Minor issues, such as missing punctuation after equations, formatting errors, and a redundant space at the start of line 141, need to be addressed. Question: * Can the Stable Diffusion method be extended to more challenging image-text tasks like Visual Question Answering (VQA)?

Questions

Please see weakness

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please see weakness

Summary

Recently diffusion-based text-to-image generation models have evolved rapidly, but it’s still challenging to evaluate them quantitatively in an efficient way. This paper smartly converts evaluations of Stable Diffusion based image generation tasks as simpler image-text matching tasks (e.g. image-text retrieval), and proposes an image-text benchmark (GDBench) including 8 carefully-chosen tasks to measure generative model’s fine-grained properties (e.g. compositionality) and fairness. CLIP and Stable Diffusion models are evaluated on this new benchmark. Overall, this paper bridges discriminative and generative model evaluations, which could be inspiring to the image generation community and improve the quantitative evaluations of generative models.

Strengths

Tackling a challenging problem, i.e., efficient quantitative evaluation of image generation models, with a simple yet effective method (predicting the noise of the diffusion for image-text alignment). Proposing an image-text benchmark (GDBench), covering diverse dimensions for measuring image generation (e.g. semantics, compositionality, fairness), which is beneficial for the image generation community. Well written and easy to read.

Weaknesses

The biggest concern on my side is the lack of enough evidence to show such a ITM eval aligns well with SD model quality, which IMO can’t be strongly supported by the evals to compare CLIP and SD models on retrieval numbers. If possible, I would suggest at least to prepare a few SD models with different capability (e.g. trained with different numbers of steps), and show they have different numbers on GDBench; bonus: give some qualitative examples for these models conditioned on the same text.

Questions

As in “Weaknesses”: how to prove the proposed method can measure the capability of SD models? To measure discriminative vision-language models, both classification and retrieval tasks are usually used. Besides “pets”, could we also add some other common 0-shot classification tasks as well? E.g., INET (most common) or ObjectNet (to measure robustness). minor question/suggestion: this paper’s citation seems to use a different format (e.g. “[name year]”) than others (e.g. “[number]”)? The latter is more convenient to read, especially for prints.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

4 excellent

Limitations

This paper has addressed model bias in one of GDBench tasks.

Summary

This paper studies how to use a pre-trained text-to-image generative diffusion model to do discriminative tasks like image and text matching. Building upon previous works [Li et al. 2023, Clark and Jaini 2023], this paper introduces two technical contributions: 1) Unconditional normalization largely improves image retrieval; 2) Tuning the diffusion model with hard negatives on better-quality data like MSCOCO. The paper also introduces a benchmark to evaluate different diffusion models by image and text matching performance, and specifically studies the biases of two versions of Stable Diffusion model.

Strengths

1. The two technical contributions (unconditional normalization & tuning on MS COCO) are effective and well ablated. 2. The introduced benchmark could be useful for the community to evaluate the prompt-following ability and biases new diffusion models. 3. The paper studied bias in state-of-the-models, which should draw more attention to the community.

Weaknesses

1. The generative-for-discriminative methods typically take many feedforwards of the diffusion model to evaluate a single sample. Although the paper briefly mentions its slow runtime at the end of the Appendix, I think the computational cost is an important factor of the proposed algorithm and should be rigorously presented and studied in the main text, especially when the paper proposes a benchmark - A clear study on the computational cost would be important to others who plan to evaluate on this benchmark. 2. The GDBench covers several hard image-and-text matching datasets. However, I think results on other standard discriminative datasets would be helpful for the audience to understand the advantage of the proposed algorithm and compare to previous works. For example, how does HardNeg-DiffusionITM work on ImageNet compare to Diffusion Classifier [Li et al. 2023] / DiffusionITM (Ours) in Table 2(b)? I understand ImageNet is a 1000-way classification and would be a long time to run for this kind of methods (therefore studying runtime is important as mentioned above), but Diffusion Classifier [Li et al. 2023] has already reported their performance on ImageNet and an apple-to-apple comparison would be very helpful. Same applied to the CLEVR dataset. Another question is how tuning on MSCOCO works for image-text matching on MSCOCO? 3. I am not sure what is the use case to introduce MSCOCO tuning into Diffusion Classifier. It indeed improves the performance of image and text matching compare to the plain one, but all these methods are too slow (or not good enough) to be practically applied. On the other hand, tuning the weights of diffusion models prevents it from serving as a tool for evaluating diffusion models. Then in what case do we need MSCOCO tuning?

Questions

1. I am not sure how much MSCOCO HardNeg is clearly better than MSCOCO NoNeg - To me, MSCOCO NoNeg seems to be even better than HardNeg quantitatively, or at least on par with in Table 2. How do the authors land with HardNeg as the default setting? 2. Why is Winground's random accuracy (25%) higher than the algorithms' accuracy in Table 1(a)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper studies the discriminative capabilities of diffusion models measured by image-text matching. A new matching score computation enables text-based image retrieval beyond simply text retrieval in existing works. A new benchmark, augmented from existing image-text benchmarks, is proposed for researchers to evaluate current diffusion models.

Strengths

1. DiffusionITM enables image retrieval by making simple and minimal changes to existing text retrieval methods. The method generates good results on simpler datasets and performs better than random on challenging ones. 2. The proposed benchmark covers diverse aspects but is also optimized for lightweight testing. Differences in a few methods have been demonstrated with the benchmark.

Weaknesses

1. Evaluation on Flickr with CLIP retrieved negatives. This process might introduce bias towards or against CLIP based image diffusion models. For example, in Table 1a, CLIPRN50x64 obtains an accuracy of 71.9. According to line 545, does it mean that 28.1% of the cases CLIP does not select the correct positive? In order to fix this, maybe an evaluation against the original model-free Flickr retrieval metric is needed, e.g. measuring if CLIPRN50x64 or CLIP based method has an advantage/disadvantage compared with other methods, measured on the new simplified set vs. the original set. Similarly, the CLIP-based HardNeg training might be exploiting the bias, besides learning useful discrimination skills. 2. It is preferred to compare different kinds of diffusion model variants in the proposed benchmark and show some insights there. For example, one might see an advantage of T5 based text encoder in grammar/text related metrics, or one could study pixel based diffusion vs. latent diffusion etc. This comparison is challenging in obtaining the models but can provide more value to the community. 3. The current method makes an assumption that DiffusionITM, based on eq. 2 and 5, is a great way of ITM and will be the case for a while. However, given the field advancing so rapidly, researchers might discover a better zero-shot matching method that could infer quickly and outperform CLIP consistently for example. Then, at that time, why would researchers not adopt the original retrieval tasks, e.g. on Flickr with all images, but adopt GDBench with 10 images? 4. It is claimed that the proposed tuning strategy keeps the generation capability. Examples are shown in Figure 5 and positive human evaluation is shown, but is it possible to evaluate existing standard metrics and see the difference? FID might not apply as the tuning happens on COCO. One might even see better generation quality with discriminative tuning applied. 5. How is GDBench different from an ensemble of image-text matching evaluation metrics, except that it is made light-weight, e.g. measuring 10 samples. It seems DiffusionITM is a general enough method that can be evaluated on theoretically any image-text matching tasks. From another perspective, is it possible to show correlation of the GDBench metrics with real compositional generation quality, e.g. measured by human evaluation. 6. Given the limited samples in some of the evaluation datasets, what are the standard deviation for different runs? From Figure 9, it looks noisy.

Questions

Table 1 typo: Difussion. And see the weaknesses section for questions.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

This paper proposes novel techniques for improving the performance of text-to-image diffusion models on zero-shot image-text matching tasks. They first propose subtracting the unconditional denoising error $\|\epsilon - \epsilon_\theta(x_t, t)\|_2^2$, which reduces the problem in image retrieval where one image is a priori far more likely under all captions. The also propose using a hard negative finetuning loss to increase the denoising error on hard negatives, which improves its discriminative (and generative) capabilities. Finally, they assemble GDBench, a benchmark of 7 existing datasets that they use to quantify different aspects of text-image alignment performance. They find that their proposed model significantly improves on prior work that uses text-to-image diffusion models (Diffusion Classifier), and is competitive with CLIP ResNet50x64. They also benchmark model bias and find that Stable Diffusion 2.1 is less biased than Stable Diffusion 1.5 and CLIP RN50x64.

Strengths

- The paper proposes 2 novel modifications that significantly improve performance over Diffusion Classifier (a prior work on using diffusion models for text-image matching tasks). - The hard negative finetuning method in particular is intuitive yet highly useful. It is also evaluated extremely thoroughly in its effect on both discriminative and generative behavior. - Authors also gather a set of 8 image-text-matching tasks into a proposed GDBench benchmark for easily measuring and comparing the performance of different ITM methods. - Authors do a thorough analysis of the bias of Diffusion ITM, across versions of Stable Diffusion, as well as in comparison to a few CLIP models. - Extensive ablations and experiments on variants of HardNeg, comparison between CLIP, BLIP, and Diffusion ITM, and the relationship between the number of timesteps and the performance of the method.

Weaknesses

- Computational: The proposed method runs in $O(n)$, where $n$ is the number of candidate images or text captions to choose from. This becomes highly impractical for real retrieval applications, compared to approaches like CLIP where fast approximate nearest neighbor solutions work extremely efficiently. - Baselines: while the method does outperform previous work on using diffusion models (e.g., Diffusion Classifier), it doesn't compare against the strongest discriminative baselines, namely OpenCLIP Vit-H/14. This is fair since Stable Diffusion 2 actually uses this model as its text encoder. The paper currently only compares against CLIP ResNet50x64, which is quite weak in comparison.

Questions

**Main questions/issues**: - How well does OpenCLIP ViT-H/14 do on these ITM tasks? This is the most relevant discriminative baseline since SD 2 uses its text encoder. - The Flickr30K benchmark is biased against the CLIP model used to produce it. The CLIP model is used to produce the top 10 candidates, and *the correct candidate is added if it's not among the top 10*. If the model being evaluated is CLIP, it has to choose between 9-10 hard negatives, whereas other models will receive a boost from a weak ensembling effect with CLIP. I'd suggest another way of creating the 10 candidates without correlation to any model output. - Equation 6: is there a major typo here? This appears to still minimize $\mathcal L_{neg}$. - L 246: Hard negative finetuning uses $\lambda=-1$? Why is it negative when it's only used for $clip(\mathcal L_{neg}, |\lambda \mathcal L_{pos}|)$? Does this mean that $\mathcal L_{neg}$ is clipped to the range $[-\mathcal L_{pos}, \mathcal L_{pos}]$? Something is wrong here. **Minor questions/comments**: - Table 1b: Where does the 37.5 score come from with Diffusion Classifier/DiffusionITM on Winoground text? The Diffusion Classifier paper reports 34.0. - Page 6, footnote 2: This is not too important, since the prompt is fixed across methods, but Pets prompt template used in the CLIP paper is "a photo of a {}, a type of pet." This probably increases scores across the board for all methods. - L240-241: "we drop the complicated procedure that iteratively prunes classes after a number of noise-timestep samples" -- Diffusion Classifier doesn't use its adaptive strategy for its ITM experiments either, so its evaluation strategy matches DIffusionITM here. - Appendix G (runtime) mentions that "batch size has diminishing returns in terms of runtime after batchsize=4." However, based on the Diffusion Classifier github, using FlashAttention + FP16 + `torch.inference_mode()` + batchsize=32 achieves about 80 evals/s on an A6000. I'd suggest double checking your implementation. Computation speed should be an issue overall with this method, but not to this level.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes, authors address the main limitations of their paper and method.

Added Citations

[1]: Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models (Grover et al., AAAI, 2018) [2] Pixel recurrent neural networks (Van Den Oord et al., ICML 2016) [3]: [Diffusion Models as Masked Autoencoders](https://arxiv.org/abs/2304.03283) (Wei et al, Arxiv 2023) [4] On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. (Ng and Jordan, NIPS 2001) [5] On the generative-discriminative tradeoff approach: Interpretation, asymptotic efficiency and classification performance. (Xue and Titterington, Computational Statistics & Data Analysis, 2010) [6] Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification. (Ardizzone et al., NeurIPS 2020)