Summary
This paper presents an approach called “SynGen” to improve attribute binding between nouns and their modifiers for text-to-image latent diffusion models, specifically Stable Diffusion. They propose a two-part loss on attention maps that (1) encourages attention maps of nouns and their modifiers to be similar, and (2) encourages attention maps between the noun/modifier to be different from those of other tokens in the prompt.
Datasets and baselines: They run experiments on three datasets: the attribute binding contrast set (ABC-6K) from the Structured Diffusion (baseline 1) paper, data from the Attend-and-Excite (baseline 2), and a newly proposed challenge set called Diverse Visual Modifier Prompts.
Evaluation:
(1) Human evals for concept separation and visual appeal. They find that SynGen outperforms baselines.
(2) Qualitative analysis with sample prompts and generations for Syngen and the two baselines, demonstrating that SynGen overcomes failure cases of the other two models.
(3) Ablations on the loss components and loss weighting.
Strengths
1. This paper tackles an important and challenging drawback of text-to-image diffusion models struggling to faithfully generate objects with the right attributes.
2. This paper proposes an interesting two-part loss that imposes localized pairwise constraints on the attention maps between image patches and token embeddings. The first constraint enforces that the same image patches attend to both the noun and its modifiers. The second constraint enforces that image patches attending to these nouns/modifiers do not attend to any other tokens in the prompt.
3. This paper attempts to evaluate on a large set of prompts including those from prior work as well as proposing their own challenge set.
4. This paper attempts to better understand the effects and side-effects of the proposed losses via ablations.
Weaknesses
1. The proposed method appears to be strongly grounded in the framework of the Attend-and-Excite paper. The writing doesn’t highlight this connection while introducing the approach in Section 2, beyond a minor footnote. Section 2 would probably need some revision to better draw this connection.
2. Considering that the paper is tackling a very specific problem, the general “content separation” rating collected in the human eval seems pretty weak. Why not collect finer-grained scores for: number of objects in the prompt, number of objects from prompt generated, number of objects from prompt generated with the correct attributes? Wouldn’t this give a much better sense of task success and could potentially allow computing recall?
3. The loss function ablation mentions that sometimes objects are omitted from the generated image. Is combining the proposed losses with global constraint defined in the Attend-and-Excite paper a viable option? Since the framework is identical, wouldn’t combining the losses have mitigated this issue? The Attend-and-Excite paper talks about attribute binding as well, so this appears to be a variant to try.
Nits:
1. Maybe more explicitly state that this is an inference-only guidance procedure?
2. There’s a typo on line 112 for the equation: $\\nabla_{z_t}\\mathcal{L}$ not $\\nabla z_t\\mathcal{L}$. Also $z’_t$ not $z’t$.
Questions
1. Was there a reason to not include numerical modifiers in the study?
2. Are there unexpected side-effects of such constraints on model generalization?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
One limitation that was perhaps overlooked is the lack of standardized evaluation. Having to rely on different crowdsourcing tasks and qualitative comparisons makes it much harder to track progress.