Summary
The authors propose _Magnet_ to solve the attribute binding problem. (1) Initially, they specifically analyze how the improper binding problem occurs in text embeddings. By comparing embeddings for each token, they demonstrate the attribute bias phenomenon where attributes do not bind well to the object token $c_{object}$ in rare concepts like "blue apple." Additionally, through cosine similarity analysis of padding embeddings, they hypothesize that the latter padding tokens, e.g., $pad_{73}$, tend to represent prior bindings (like red apple) rather than binding the target concept (blue) with the object (apple). (2) Magnet suggests a method to slightly edit the embedding of the object to align with the desired concept, based on these observations. They demonstrate their method's efficacy through ARC-6k and CC-500 datasets.
Strengths
- Writing quality
The writing quality is good for understanding the core motivation and method. Analyzing embeddings using cosine similarity is intuitive and effectively demonstrates how attribute bias negatively impacts attribute binding. The flow from the initial analysis to the method is also easy to follow.
- Novelty
Resolving attribute binding in the text space, as done by Magnet, seems quite plausible and novel. Identifying the limitations of the T2I diffusion model from the VLM model has been necessary but has not been extensively conducted until now.
Weaknesses
Overall, I think it's a good paper. It would be better if the evaluation were a bit more solid.
- Evaluation
I agree that a user study w.r.t performance comparison as shown in Table 1 is necessary, but for this task, it is necessary to evaluate whether each method achieved binding rather than comparing different methods. Since we don't know the precise performance of each technique, it's difficult to understand the actual performance of the Magnet and how much it has improved.
The results in Table 2 are weak. It's disappointing that the automatic results are quantitatively worse compared to attend-and-excite. While I agree that manual inspection is the most accurate, trying out a recent VLM like GPT-4o, which can discern characteristics that image encoder-based evaluations fail to capture, might be worthwhile.
Questions
Q1: How is the analysis in lines 64-78 and Figure 2-(a) connected to the method?
Q2: Is there no need to modify the padding? If the padding tokens positioned at the end contain prior knowledge, it seems they might also interfere with binding.
Q3: How did you perform detection using GrondingDINO in Table 2? I am curious whether you detected "blue apple" or just "apple." It might be somewhat out-of-scope, but I am also interested in how different the results are between detecting "blue apple" and "apple."
Q4: As mentioned in the limitations (Figure 20), Magnet seems relatively weak at handling positional information. I suspect this issue arises not so much from a problem with Magnet itself but rather because CLIP's embedding does not effectively bind positional information to the desired token. Is there an analysis similar to Figure 1 for positional information? It seems only color is present in the appendix. Showing this could further demonstrate, as per the title of your paper, an understanding of VLM’s function, providing clarity on what can and cannot be bound.
---
- Relevant references
Before this paper was submitted, methods for editing text tokens were proposed in papers like [1, 2]. Although these papers also deal with a different task, and attribute binding, and were not considered in the evaluation due to being on Arxiv, I recommend citing them in the camera-ready version.
[1]: Uncovering the Text Embedding in Text-to-Image Diffusion Models, https://arxiv.org/abs/2404.01154
[2]: TexSliders: Diffusion-Based Texture Editing in CLIP Space, https://arxiv.org/abs/2405.00672
Limitations
The authors describe the failure cases of Magnet in the appendix. They have adequately addressed the limitations.