Diffusion Self-Guidance for Controllable Image Generation

Summary

The paper introduces a new method for detailed and versatile image editing and controlled image synthesis with large-scale text-to-image diffusion models. In particular, the authors propose self-guidance: Self-guidance uses the internal representations of the denoiser neural network of diffusion models for guidance. The work finds that the attention and feature maps of the denoiser network encode properties like objects' size, shape, position and appearance in an interpretable way. This is, some heuristics on the internal attention/feature maps can be defined that represent these properties well and then we can do guidance, leveraging these internal representations (note that this works only for objects and words that were mentioned in the text prompts driving generation). Hence, given an image (either generated by the diffusion model itself or a real one, reconstructed by the diffusion model), the novel self-guidance technique enables editing operations such as freely modifying objects' size, shape, position and appearance, without introducing noticeable artifacts. Importantly, this does not require any additional training or use of auxiliary models, and no additional paired data for training, which makes this a very powerful technique. The authors convincingly validate their proposed editing method mostly qualitatively by showing lots of interesting editing examples.

Strengths

Generally, the main strength of the paper is the introduction of the novel self-guidance technique, which it then uses for detailed and advanced image editing and controllable image synthesis. To the best of my knowledge no previous image editing methods with text-to-image diffusion models reach this level of fine control with respect to objects' shape, size, position, appearance, layout, and composition, while introducing almost no artifacts. **Clarity:** The paper is very well written and easy to follow and understand. It is very clear. **Originality:** The main idea of self-guidance leveraging the model's internal representations is novel and original. That said, it is closely related to, and probably inspired by, existing technique such as Prompt-to-prompt editing (Hertz et al. "Prompt-to-Prompt Image Editing with Cross Attention Control") and paint-with-words (Balaji et al. "eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers"). **Significance:** Controllable image generation and image editing with large-scale text-to-image diffusion is an important direction with many relevant applications. The paper proposes a very powerful and versatile method with little requirements, which makes this a significant contribution. **Quality:** The overall quality of the paper is high. It is clear, well written, all claims are supported, limitations and societal impacts are appropriately discussed, background is provided, and related work is thoroughly discussed.

Weaknesses

While the paper has few major issues, there are some concerns: - Editing and control is limited with respect to objects (and words more generally) that appear in the text prompt. This also means that given real images require an appropriate caption to be editable. Note that this limitation is acknowledged by the authors. - The method comes with a ton of hyperparameters: The guidance weights for the different edits need to be chosen manually and it seems like some trial and error can be necessary. Also, more detailed ablations on this would be helpful, like showing different guidance weights for same image/seed/prompt/edit, etc. - When performing a simple edit like moving a single object, we actually require a lot more guidance terms for keeping all the other objects in place and preserving their appearance. This means that even edits that may appear simple are actually quite complicated and require guidance weights for all different terms. - The way the different edits and controls are defined is largely heuristic (how size, shape, appearance, etc. is calculated from the attention/feature maps). It seems to work well, but in theory it would be great if there was a more principled approach to this. - The way the cross-attention maps are leveraged in self-guidance is related to the "paint-with-words" technique introduced in eDiff-I (Balaji et al. "eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers"). I believe this should be acknowledged and cited. - The approach is validated only in pixel-space text-to-image diffusion models. Can the same method also work in latent diffusion models? The most popular, and publicly available, large-scale text-to-image model is Stable Diffusion. It would be great to see the method also applied there, if possible. In conclusion, there are concerns, weaknesses and routes for further improving the paper. However, the strengths outweigh these weaknesses. Consequently, I am leaning towards suggesting acceptance of the paper. I am looking forward to the rebuttal by the authors.

Questions

I also have a variety of questions that did not significantly impact the rating of the paper: - I believe the paper intrinsically relies on a typical U-Net architecture for the denoiser network that gives rise to the useful attention and feature maps. There are various recent works that use pure vision transformer denoiser networks or advanced architectures like recurrent interface networks. Can the approach be extended to these architectures, too? I would be interested in the authors' thoughts on this. - In a similar spirit, do the authors believe that similar techniques can also be used in diffusion models that model video, 3D, audio, graph/molecule data, etc.? - While centroid, size and shape leverage all cross attention maps, the appearance term uses only the activations of the penultimate layer in the decoder as well as the final cross-attention operation. Why is that? Can the authors provide more intuitions for the qualitative differences in how the attention/activations are used in the different operations? - The paper entirely leverages DDPM-style sampling. Does the method also work with deterministic probability flow/DDIM sampling? Moreover, when using given real images, could there be an alternative way of editing when probability flow ODE/DDIM is used to deterministically encode a given image and then re-generate again while introducing the desired edits? I would be curious about the authors' perspective on this. Very Minor nitpick: - Please properly introduce the data $x$ as well as the noise schedule $\alpha_t$, $\sigma_t$ around equation 1. - I would suggest to properly introduce and define the indices in Line 155 and below, this is, what exactly is $i$, etc. to avoid any potential confusion.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations and potential negative societal impact have been appropriately discussed and are well addressed. There are no concerns in that regard.

Summary

Large-scale text-to-image generative methods have demonstrated impressive performance given a detailed text prompt. However, many aspects of an image are hard or impossible to specify with texts. The authors proposed self-guidance to guide the generation of diffusion models. Experimental results demonstrated that object size, location, and appearance of objects can be controlled from normalized attention matrices and feature activations.

Strengths

1. The motivation and background is explained clearly. Arguments and claims are supported by references or experimental results. 2. The proposed approach is simple and effective. The presented empirical results on multiple tasks demonstrated the effectiveness of the approach. The authors claimed that all results are non-cherry-picked, which is quite impressive. 3. Adding geometrical/compositional control to the image generative models is an interesting and important research topic. The authors proposed a simple and smart method to control existing generative models with diffusion guidance. 4. Multiple tasks were considered, including adjusting individual properties, compositional generation, and real image editing. This proposed approach showed potentials to a range of real-world applications.

Weaknesses

1. From the abstract, the authors aimed to add more control over the image generation beyond text prompts. However, the proposed approach is not a good solution in this regard – it is strongly based on the tokens in the text prompt. There are two limitations: * Can we control fine-grained objects/parts not naturally present in the text prompt? For example, control the dog mouth when generating “a photo of a dog wearing a baseball cap”? All current results are on the object-level with no finer control. * The proposed approach is limited to objects/parts easily describable by texts. This is a disadvantage compared to the keypoint inputs from DragGAN. For instance, annotating a keypoint would be more straightforward than inputting “left-front leg of the dog”, for both the user and the model. 2. Based on the framework, designing more control seems difficult, e.g., 3D viewpoint of objects. Compared to parallel works, ControlNet [2] finetuned the LDM model with 3D representations and DragGAN [1] used finer keypoint inputs to accomplish this. 3. The proposed method is built on existing large-scale image diffusion models. It heavily relies on a good representation learned by these trained models. The aforementioned limitations are mostly due to the latent representations having limited feature disentanglement or 3D awareness. The proposed approach doesn’t seem to have any easy fixes to address these limitations. References: 1. X. Pan et al. Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. 2. L. Zhang et al. Adding conditional control to text-to-image diffusion models.

Questions

1. To see how entangled the representations are, can the authors generate a video or a dense sampling of intermediate images to interpolate the “control space”? For instance, changing the layout of the objects in Figure 5 over 16 frames, or changing the styles in Figure 4 over 16 frames? In this way we could see: (i) if a continuous control corresponds to smooth and continuous changes in the image space, and (ii) how the other objects are influenced by the controlled changes made to the target object.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

1. The input space of the proposed control is still limited by texts: see weakness 1. 2. The proposed approach is haunted by entangled representation: see weakness 2 and 3.

Summary

This paper introduces diffusion self-guidance, an inference-time technique for controllable image generation using pre-trained text-to-image diffusion models. The key finding is that internal representations of the denoiser network carry meaningful information about the scene, and one can build custom energy functions around these representations to align image generation with user-defined scene properties, including object location, size, shape and appearance. Further, an appealing property of self-guidance is that the energy functions can be flexibly composed to support the simultaneous manipulation of multiple image attributes. Extensive experiments are conducted to demonstrate the effectiveness of self-guidance across a broad spectrum of image manipulation tasks.

Strengths

- The proposed method enables controllable image generation / manipulation using existing text-to-image diffusion model checkpoints without costly fine-tuning. As a training-free method, it achieves controllability by steering the generation towards matching the desired internal representations at inference time. This guidance-based design is conceptually simple, easy to implement, computationally efficient and highly flexible. - The method represents object properties using aggregated statistics of denoiser representations. It thus reveals the internal working of the generative process. To this end, the impact of the paper goes beyond controllable image generation / manipulation. - Self-guidance enables several edits that are not possible with concurrent methods. These include change in object location, size and modification of scene layout. In the meantime, self-guidance allows the composition of multiple edits in a single generation pass. This is another unique property that is rarely seen in previous and concurrent works. - Finally, self-guidance offers compelling editing capabilities and supports a wide range of image manipulation needs. Among them are the mixing of objects and layout from multiple source images, transfer of attributes defined by non-object words, and the editing of real images.

Weaknesses

I do not have major concerns about the proposed method. Some minor concerns are as follows. - The main text presents self-guidance in its generic form and provides the ingredients for defining a specific energy function. However, all concrete examples are left to the supplement. I understand that space is limited and the authors want to make room for the results (which are indeed compelling). However, it is awkward to alternate between the main text and the supplement in order to decipher how those edits are actually achieved. One possibility is to organize all instantiations of self-guidance in a big table and put them in the main text for the ease of understanding. - I did not find ablation experiments on hyper-parameters such as the number of inference steps, guidance strength and the range of steps to apply guidance. In practice, the editing quality is often quite sensitive to these hyper-parameters. While their choices are briefly discussed in the supplement, some visual examples will help visualize their effects. - The experiments are conducted using Imagen, whose design is substantially different from the publicly available Stable Diffusion (cascaded vs. latent diffusion). It is a well-known fact that an editing method may not work equally well on different diffusion models. Given that most players in the space of controllable image generation rely on Stable Diffusion, I would encourage the authors to showcase a few editing examples using Stable Diffusion in order to calibrate the performance of self-guidance.

Questions

Please find my comments in the section above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

This paper includes a discussion on the limitations and potential societal impact of the proposed method.

Summary

This paper proposes a method for image editing using pretrained text-to-image diffusion models. The method guides the sampling process with energy functions that are added similarly to classifier guidance. These energy functions are computed as the difference between some object's property and the target state we want to change it to. A property is either an object's location, size, shape, or appearance. These properties are represented by the cross-attention maps between the object's word token and the image, which has previously been shown to produce meaningful masks. Multiple energy functions can be composed to achieve composite edits. Experiments qualitatively demonstrate the appeal of the method.

Strengths

The method is practical since it does not require retraining while still achieving impressive results (as demonstrated qualitatively with some samples). The paper proposes a simple method with a clear and concise presentation. The addressed problem (image editing using pretrained diffusion models) is of interest to both academics and practitioners.

Weaknesses

Quantitative evaluations of the method (for example with user studies that rank/compare different editing results) would make this work more insightful. Currently, it is unclear whether the shown examples are representative of what to expect when using this method and where the limitations lie. A potential limitation that could be exposed through quantitative studies is the number of energy functions that can be composed at once. Editing real images is possible as shown in Figure 7, but it isn't clear how well it works when compared to generated images. Furthermore, from some examples (e.g., Figure 8 a-b) it looks like the non-edited parts of the image also change and do not remain consistent with the original image, as is usually desirable in image editing settings. There are missing details that could help in reproducing and better understanding the results. See the questions part.

Questions

What pretrained diffusion model is used in the experiments? Does this method perform differently for different diffusion models (Stable Diffusion vs. eDiff-I vs. Imagen vs. etc.)? I'm asking this because different diffusion models incorporate text conditioning differently (both architecturally and during training). How sensitive is the method w.r.t. hyperparameters like the attention layer, time step, and guidance strength? Is it necessary to tune the hyperparameters for every image individually? Maybe even for every energy function individually?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Many limitations regarding the method and potential negative societal impacts are discussed. Some that might be reasonable to also mention are listed above.

Thank you for rebuttal

I would like to thank the authors for their rebuttal and for replying in detail to all my questions. I do not have any further questions. It's great to see that the method also seems to work in Stable Diffusion. The additional results in the pdf are also helpful and make me a bit less concerned about hyperparameters (although that's still a weakness). Overall, I believe this paper should be accepted. I raised my score by one point.