Summary
This work advances the understanding of token-level directions within the commonly used CLIP text embeddings, demonstrating their efficacy for fine-grained, subject-specific control in text-to-image (T2I) models. The authors observed that diffusion models provide a smooth interpretation of CLIP text embedding vectors, allowing for the control of target attributes in generated images by identifying token-level semantic vectors within the CLIP space. Two primary methods are proposed. First, an optimization-free approach is introduced that contrasts embeddings from prompts that describe specific attributes (e.g., "a man" vs. "an old man") to determine the direction vector, with more accurate vectors obtained by averaging the differences across multiple prompt pairs. Second, a learning-based method is employed to identify attribute directions in the noise prediction space of the diffusion model. In summary, this work elucidates the effective application of token-level directions within CLIP embeddings, thereby enabling precise control over multiple attributes of individual subjects in image generation.
Strengths
1. The author identified a token-level attribute vector using a simple yet effective method, which allows control over the representation of specific attributes in the target generated by a diffusion model.
2. Unlike other methods, this approach achieves fine-grained control over specific attributes of the target in the generated images by adding the vector to a specific word in the prompt.
3. Experimental results demonstrate the effectiveness of the proposed method and illustrate its transferability.
Weaknesses
1. The author derives attribute vectors through contrasting descriptions (e.g., "a man" vs. "an old man"). A more intuitive method would involve directly adding the embedding of "old" to "man," yet the author does not compare this simpler approach. It is advisable for the authors to compare their proposed method with this more intuitive approach.
2. While the authors mentioned that direct subtraction of CLIP vectors is only applicable to prefix attributes, the paper does not provide a clear explanation of how the optimization-based method addresses this limitation. Furthermore, the experimental results never examine related content, such as suffix attributes.
3. Will directly adding a vector to the corresponding word completely avoid affecting other targets? During image generation in diffusion models, text tokens interact with pixels through cross-attention layers, and in this process, an attention map is generated, which can reflect the degree to which image pixels focus on a particular word. It is possible that a pixel might exhibit high attention to multiple tokens, which may lead to overlapping influences on the image regions associated with different words. The authors are encouraged to consider such scenario and discuss it.
4. The paper mentions that experiments were conducted using SDXL, which has two text encoders; however, the author does not clearly specify how the attribute vectors are incorporated. Please provide a specific description of how the attribute vectors is incorporated in SDXL's dual text encoders. Additionally, since optimization involved training with images generated by the diffusion model, the model's performance could significantly affect the results. It is recommended that the authors investigate the performance vary across different diffusion models.
5. Based on the results presented in Table 1, the proposed method does not demonstrate exceptional performance across various metrics. Moreover, integrating other methods even leads to a decrease in performance on the subject-specificity metric. The authors are encouraged to discuss the potential reasons for such decreases in performance.
6. The results presented in the paper are based on a few simple attributes and prompts. To enhance the persuasiveness of the method, more complex scenarios should be demonstrated. For example, attributes could involve verbs like " jump" or cases where the target includes multiple words, such as "a red sports car". Additionally, visual comparisons with the baselines should be provided to offer clearer insights into the method’s effectiveness relative to other approaches.