Information Theoretic Text-to-Image Alignment

We thank you for the comments. We reply to each point below. 1. *Note that the paper mainly focuses on SD-based (SD 2.1, SDXL) models. These models are mostly the same styles, e.g., similar network structures and traditional denoising training strategies. Is there any possibility that the MI tuning incorporated with flow-based models like DiT-based models (SD3, Pixart series or so). And it is interesting to see if the proposed MI tuning behaves different with different types of models.* In the first comment, the reviewer is raising an interesting point regarding two complementary methodological aspects with respect to what presented in the submission, namely DiT models and flow-based models. 1. Considering DiT models, if they rely on DDPM (as Pixart), MI-TUNE can be seamlessly applied to them. To show this, we applied MI-TUNE to Pixart and we are ready to include additional results in the camera ready version of our paper. In the meanwhile, we will update our answer before the end of the discussion period, as soon as experiments terminate to share the new results. 2. Considering flow-based approaches, to fully exploit the self-contained nature of MI-TUNE (self-supervised fine-tuning, with no need for auxiliary models), it is important for the generative model to be amenable to estimating mutual information. While MI estimation with normalizing flows has recently appeared in the literature [1], the generative quality of such models is currently not on par with diffusion-based or rectified-flow based methods. MI estimation with rectified-flows (used by SD3 and Flux) is an open problem that, while very interesting, falls outside the scope of this work. Indeed, adapting MI estimation would require re-deriving the time evolution of the mutual information along the generative dynamics of the deterministic rectified flows, and combine together the value of the learnt parametric drifts with score functions. In principle, such extension is within methodological reach, and its actual implementation will be an interesting avenue for future works. To conclude, given a rectified-flow MI estimator, we believe that the MITUNE method we propose in this work could be seamlessly applied to such generative models, and contribute to improved text-to-image alignment. [1] Butakov et al, “Mutual Information Estimation via Normalizing Flows”, NeurIPS 2024

Thanks for the detailed responses from the authors. Part of my concerns are solved. However, some of the points regarding the experiments on rectified-flow-based models remains unsolved, which makes me feel that the strategy strongly depends on the DDPM-style learning (unknown performance to flow-based model) and may weaken the contribution the paper should have made. Overall, it is still good to know that using mutual information to improve the DDPM-based model learning on text-to-image alignment for image generation and I will keep my score after considering the detailed response and other comments.

We thank you for the comments. We reply to each point below 1. *MI scores are missing from comparison tables and images despite being central to the method.* The first comment regards the lack of using MI as an evaluation metric in the benchmark. **We do not report MI scores as an evaluation metric in the comparison tables, because using MI for both fine-tuning set selection and evaluation would represent an unfair advantage for MI-TUNE.** This would be the same scenario GORS authors (Huang et al., 2023) call “biased evaluation”, i.e., the metric used for evaluation is also used during fine-tuning. To be even more precise, GORS results in Table 1 represent a “biased evaluation”, as this mechanism is an intrinsic aspect of the GORS method itself. That said, even in this condition, GORS presents inconsistent performance and often does not even result to be the best alternative fine-tuning method. **For additional information about MI scores computed on generated images in our qualitative analysis (see Fig. 1), we added a new appendix J** with a discussion of distribution of such scores. 2. *Figure 1 only demonstrates MI effectiveness on simple category prompts (color, texture, shape), lacking validation on more challenging cases like spatial relationships or complex compositions* The second comment regards missing categories from Fig.1. The reviewer is right in pointing out that Fig.1 could have been exhaustive and include all categories. However, this was a mere editorial choice, dictated by lack of space, and to underline that Fig.1 was reporting only a **qualitative analysis**. In other words, Fig.1 motivates our intuition that MI is a meaningful signal for alignment. Sec 3.1. further corroborates this, with more data analysis and points to a user study detailed in Appendix B.1. That said, **ultimately, the final assessment of MI is carried over in Table 1 with the actual comparison against all alternative methods (including HPS and Human evaluation too), with additional results about SDXL (Table4) and DiffusionDB (Table5).** For completeness, in **Appendix I of the revised manuscript we added example figures for all categories** complementing the one reported in Fig.1. The new figures show the same trend as the one already provided in the original Fig.1. 3. *How were the 700 prompts selected for the MI quantitative analysis?* The third comment relates to the train/test composition of the datasets we used. Based on the reviewer’s question, we realize that some clarifications are in order, thank you! Our evaluation considers two datasets, namely T2I-CompBench and DiffusionDB. T2ICompBench pre-defines 700/300 train/test prompts for each category. So, to answer the reviewer’s question, **the 700 prompts we use are those that are predefined by T2I-CompBench.** In our evaluation, we use the pre-defined splits (to foster replicability and comparison against other studies using this same dataset). Quoting T2I-CompBench authors [...]*we introduce three categories and six sub-categories of compositionality, attribute binding (including three sub- categories: color, shape, and texture), object relationships (including two sub-categories: spatial relationship and non-spatial relationship), and complex compositions. We generate 1,000 text prompts (700 for training and 300 for testing) for each sub-category, resulting in 6,000 compositional textprompts in total. We take the balance between seen v.s. unseen compositions in the test set, prompts with fixed sentence template v.s. natural prompts, and simple v.s. complex prompts into consideration when constructing the benchmark. The text prompts are generated with either predefined rules or ChatGPT [45], so it is easy to scale up*[...] Conversely, for DiffusionDB, we randomly sampled 5,000/1,250 (train/test) prompt-image from the dataset. We’ll make available our selection as part of the code artifacts, in an effort to foster reproducibility of our results. In the revised manuscript we clarified this in section 3.1.

We thank you for the comments. We reply to each point below. * *More detailed ablations are needed. The authors employ MI as the metric to select fine-tuning samples, which eliminates the extra usage of other models. However, what if we use SOTA VQA models as the metric? Intuitively, SOTA VQA models are more precise than the MI metric.* 1. The first comment regards the evaluation protocol used. We agree with the reviewer that the fine-tuning set composition and the evaluation metric are important. In this regard, we highlight the following points: 1. First of all, using BLIP-VQA for fine-tuning set composition AND final evaluation leads to biased results. This is demonstrated by GORS authors (Huang et al., 2023) as we mentioned in Sec.4.4 of the original submission. Note also that **the results for GORS reported in Table1 of our paper already reflect the experiment suggested by the reviewer (BLIP-VQA is used both during fine-tuning of GORS and its evaluation)**, as this an integral mechanism of GORS. Yet, GORS performance is inconsistent across categories and it often does not even result as the best method within the fine-tuning family. 2. **We also consider alternative approaches to the use of MI for composing the fine-tuning set. Table 3** presents an ablation considering HPS (as a possible alternative alignment metric) as well as a mix of generated and real pictures. 3. Finally, **Table 1 also includes a Human study** where real people were asked to provide alignment feedback (details of the survey campaigns are available in Appendix B). While the surveys were done on a relatively small set of users, they are still relevant to investigate the meaningfulness of MI to steer model alignment. * *An inherent drawback of MI is that it can measure how much help comes from the prompt but cannot guide in the right direction. For example, in cases of color misalignment, how should we deal with this issue?* 2. The second comment is focusing on the role/importance of MI. We agree with the reviewer in saying that MI can measure “how much help” comes from the prompt. Indeed, it helps the model to better denoise the generated images, during fine-tuning. However, “guiding the model in the right direction” is what inference-time methods such as A&E do, while MI-TUNE is a fine-tuning method. Abstracting from our evaluation, we are comparing the two alternatives: i) an implicit guidance through fine-tuning vs. ii) an explicit guidance at inference time by acting (for instance) on cross-attention maps. Taking color misalignment as a case study, as suggested by the reviewer, we show below that our method yields the best results (taken from Table 1 in the paper). | Method | BLIP-VQA | |---|---| | SD2.1-base | 49.65 | | A&E | 61.43 | | MI-TUNE | 65.04 (+5.88% wrt A&E) |

We thank you for the comments. We reply to each point below. *1. Comparison methods, "Attend and Excite (A&E) (Chefer et al., 2023b), Structured Diffusion Guidance(SDG) (Feng et al., 2023b) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al., 2024)" mentioned in line 329, is implemented on SD1.4, but the proposed work is implemented on SD 2.1, which is powerful than SD1.4 and may introducing evaluation bias.* The first comment raises attention to model version and fairness of the comparison. We acknowledge the need for a fair comparison and believe the comment is due to insufficient clarity in the description of the setup used in our evaluation. To clarify, for results of the 6 alternative methods reported in Table 1, **we did NOT report results from the related papers. Rather, we ran those methods ex-novo on the benchmark, testing them all on SD 2.1** – this was the only way to guarantee a fair comparison. We clarified this in Sec. 4.3 by adding *We underline that since results reported in the literature for both families do not necessarily refer to the same base models, to guarantee a fair comparison, we adapted and evaluated all methods on SD-2.1-base.*

Please find the following additional results: 1. The Mutual Information scores of baseline (SD2.1-base) against our proposed method MI-TUNE, in the answer to Reviewer LaKZ. These results clearly indicate a substantial improvement for the Mutual Information as a result of our MI-TUNE strategy, when compared to the baseline. 2. The additional result of MI-TUNE on the DiT-based PixArt-$\Sigma$, in the answer to Reviewer 8y4V. These results corroborate our claim that MI-TUNE does not depend on specific model architectures, and that it can be used to improve alignment of a variety of base models (all in all, we have now results on SD2.1-base, SDXL, PixArt-$\Sigma$). Thanks to the reviewers for helping us further strengthen our work!