Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesis

Summary

The authors tackle semantic binding, where T2I models often fail to correctly reflect the relations between objects (object binding) or objects and their attributes (attribute binding). To address this, they introduce Token Merging (ToMe), a method that aggregates related tokens to a single composite token, which ensures they share the same cross-attention map. To do that, two training-free optimizations are introduced. The first, Semantic Binding loss, which makes sure the composite token leads to noise prediction that is consistent with the full phrase the token is based on. The second, Entropy loss, which helps the tokens focus exclusively on their designated regions.

Strengths

* Creating a composite token and then optimizing it is a creative and elegant approach to the problem. I'm curious to see what future work can be built on top of this contribution. * The focus on object binding is indeed missing from the literature, a topic that is well-addressed in this work. * The idea of end token substitution to remove attribute information is simple.

Weaknesses

* The related work underplays the role of Attend-and-Excite and Linguistic Binding in Diffusion Models (SynGen) have on this paper. The former introduced the idea of semantic guidance, while the latter used dependency graphs to extract syntactically related tokens from prompts and presented a loss that encourages cross-attention maps of syntactically related tokens to agree. Both of which are ideas this paper meaningfully builds on. * You do not describe how you obtain the syntactic information: entities and their relevant objects nor if it is automatic or an input constraint. This is particularly confusing because in section 3.2 you begin describing exactly that, but do not delve into the specifics. Furthermore, in section 4.1 (line 248), you mention spaCy and that you "identify each object and its corresponding attributes for token merging", but do not provide actual details about your method. Does it mean it can accurately capture *all* syntactic structures? It is possible that I misunderstand, but it feels like the syntactic analysis portion of this work borrows from the SynGen paper. If this is the case, give appropriate credit. If it is not, then add details, as this is part of the method and is quite unclear. * If I understand correctly, there are no human evaluation experiments, a user study would provide merit to ToMe’s superior performance. As a side note, in table 1, ‘Human-preference’ is a slightly misleading title. If I understand correctly, it is the output of the Image Reward human-preference model, but it sounds as if it is a human-evaluation result. * I find that despite the claim that ToMe works well on highly complex prompts, there are no such examples or experiments given. For example, Figure 5 depicts prompts that are supposed to be complex, but they are no different from the simplistic prompts in Attend-and-Excite (“a {color_1} {subject_1} and {a color_2} {subject_2}”), only that they address object binding.

Questions

1. What input text is used in the global encoding part? 2. Do you expect your method to extend to SD-3? 3. I worry that ToMe suffers from too many gradient updates in the event of highly complex prompts (like all training-free approaches), which would push the latent out of distribution. 4. Does ToMe work on long captions, with multiple sentences? How? 5. You say in the appendix that SynGen was modified to work with SDXL. Can you elaborate on the modifications? What would be needed to reproduce this experiment too?

Rating

7

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

See questions 3 and 4.

Summary

The authors propose a method to mitigate semantic binding, a common phenomenon in text-to-image models. While previous methods explicitly control the attention maps so that nouns and attributes attend to the same regions, the authors propose combining the nouns and attributes into a single token. This approach enforces the attention maps to concentrate in the same area. Additionally, they introduce an inference-time optimization loss to ensure the attention maps are focused, and that each composed token retains the same semantic meaning as all the different tokens that comprise it. The authors also analyze the information encoded in different tokens at the output of the text encoder.

Strengths

1. The authors present an interesting analysis of the information encoded tokens, specifically they show that much of the semantic information is also encoded in the EOT tokens. 2. The authors provide a method to mitigate semantic binding in prompts of the form: “A <noun_A> <Attribute_A> and <noun_B> <Attribute_B>”, that outperforms or performs comparably to existing methods across various benchmarking subsets.

Weaknesses

1. It is not clear how the model performs on more complicated prompts where both the noun and its sub-objects have attributes, such as: “A blue cat wearing sunglasses and a yellow dog wearing a pink hat.” 2. The method is on the slower side compared to other existing methods, due to inference time optimization. 3. This work is missing a user study, as the proposed automatic metrics can be inaccurate, and human evaluation is common in previous works.

Questions

1. How does the method perform on more complex prompts? For example, for a prompt like “A blue cat wearing sunglasses and a yellow dog wearing a pink hat”. From my understanding, the method should not succeed in separating “pink hat” and “yellow dog” as the whole expression will be combined into a single token. 2. Is it not clear to me what prevents semantic binding across nouns, e.g. both the dog* and the cat* tokens to have the same appearance, (e.g. a cat-dog hybrid), which is a common problem in text-to-image models. What prevents the dog* and cat* attention maps from attending to the same regions?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors propose limitations relating to the underlying models, but I would be interested to learn in which areas of semantic binding the paper still struggles, such as with more complex prompts or a larger number of nouns, etc..

Summary

This paper aims to solve the semantic binding problem in T2I models. The authors introduce a semantic binding method by merging tokens of entities and related attributes. Besides, several other tricks like semantic binding loss and entropy loss are introduced to improve the performance of semantic binding.

Strengths

- The idea is straightforward and it is also reasonable that it will work with some other tricks. - The paper is well-written.

Weaknesses

- The importance of a few tricks like entropy loss is not well-emphasized. - The motivation for some tricks like substituting the EOT token is not well-explained.

Questions

- 1. From Table 2, we know that entropy loss + token merging bring the biggest impact. This is reasonable because token merging actually changes nothing, all information still exists in the composition token, and semantic leakage will also happen. However, the reviewer does not fully understand what happens when doing token merging + entropy loss. Why do they work so well? Actually, we don't change the information and the way the information interacts. - 1.1. The reviewer thinks Figure 7 is quite clear for the motivation, considering entities and their related attributes as one object. The reviewer is confused about what function entropy loss works. - 2. What is the impact of substituting the EOT token? Is it important in the whole process? - 3. How about the efficiency compared to the baseline in Table 1?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper focuses on the problem of lack of semantic binding in text-to-image generation models, and specifically on the misalignment between objects and their sub-objects. The paper introduces a training-free T2I method named ToMe after analyzing the properties of CLIP text embeddings and diffusion models. Utilizing the composition recognition ability of diffusion models, ToMe binds the embeddings of objects with their associated sub-objects together, and retains only the semantic information in the [EOT] tokens. ToMe is shown to be simple yet effective from the experimental results.

Strengths

1. The paper is well-written and easy to follow. 2. The authors present detailed analysis of the CLIP text embeddings including their coupled and entangled properties and the semantically additive properties of text embeddings shown in diffusion models. 3. The structure of the proposed ToMe model is training-free, simple and mostly clear. 4. The authors provide extensive experimental results including quantitative, qualitative and ablation studies on the T2I-CompBench and GPT-4o Benchmark (introduced in this paper).

Weaknesses

1. The Iterative composite Token Update (Sec. 3.2.2) is not clear enough to me. The authors introduce two losses: semantic binding loss and entropy loss. And since ToMe is training-free, it is not clear in the paper how these losses help the update the tokens at test time.

Questions

1. Can the proposed method generalize to more objects, or background with various properties? It would be great if the authors can explain how generating more objects will change the pipeline, or if this is not feasible, please give an explanation why. 2. Please demonstrate how the two losses are integrated in the T2I pipeline, and how they are used to update the tokens.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors presented the inherent limitation of SDXL that ToMe is based on like producing artifacts in generated images, and unable to generate images with complex layouts. Also the limitation of CLIP to produce text embeddings could restrict the performance of ToMe.

Thanks for your comments and look forward to further discussion

Dear Reviewer 8gpY: Thank you for your valuable feedback on our paper. We deeply appreciate the time and effort you’ve put into reviewing our work, and your constructive comments are invaluable to us. Regarding the weaknesses you mentioned, we include the corresponding experiments and discussion as follows: 1. Expanded discussion on related work, particularly Attend-and-Excite and SynGen. 2. Added detailed explanations on syntactic information extraction using SpaCy. 3. Conducted a user study with 20 participants to validate ToMe’s performance. 4. Provided additional examples of complex prompts generation. 5. Clarified the input text used in global encoding. 6. Discussed the generalizability of our method to other T2I models. 7. Discussed the potential latent distribution drifts. 8. Detailed the modifications made to adapt SynGen for SDXL. Your feedback is incredibly important to us, and we sincerely thank you for considering our rebuttal. Please let us know if there are any further concerns or questions—we are more than happy to discuss them. Thank you again for your time, and we look forward to your response. Best regards, Authors of submission 1763

Thank you

Thank you for taking the time to write back and answering some of my questions. **W2** Note that the current 4.1 section does not mention the SynGen paper, but I trust that this will be clarified in a revised version. **Q1** Please add it to the paper. **Q2** Great. For your consideration: it would be interesting to see a FLUX or SD-3 implementation in a revised manuscript. **Q3** Thanks, please add that to the paper, as these optimization methods are indeed quite sensitive, and understanding the tradeoffs is important. **Q5** Please specify the hyperparameters that were used and any other detail that can be used to reproduce your calibration of this baseline. There are lots of details that were not provided in the discussion, but I trust that you will incorporate everything in great detail in a revised manuscript. I'm raising my score to reflect this.