Summary
This paper introduces a method for controllable generation using diffusion models. The approach is designed as a training free technique for 1) structure/layout controlled generation (like e.g. controlNet) and 2) appearance transfer. The approach leverages manipulation of attention mechanisms and information transfer from reference images to the generative image. Authors evaluate their methods in multiple settings, and compare to state of the art related work, showing that their approach can achieve similar performance to more expensive alternatives.
Strengths
The paper is, for the most part, well written and well grounded in the related literature. Relevant related works are well references, and works from which ideas are borrowed are acknowledged.
The main benefit of the work is its fully training-free nature, which can offer more flexibility when generating images with different types of controls. The proposed methodology is relatively simple, leaving room for future improvements. Another benefit is the fact that the method can do both appearance and structure control, while related works often focus on a single one.
Experiments show promising results, with a performance similar to FreeControl, but without inference time optimization steps. Some ablations experiments are provided, in an effort to analyse the different components of the model.
Weaknesses
The main limitation of the work is the limited methodological novelty. The tools employed are not very novel: the structure transfer simply uses the method proposed in [34], while attention map manipulation is very commonly used for structure control (e.g. for editing methods such as prompt-to-prompt).
Another noticeable limitation, highlighted in Figure 9a, is the lack of flexibility with regards to using appearance OR structure control. Results show that appearance control is required when performing structure control, requiring to generate a separate appearance image. This can increase generation cost, and reduces control over the content of the image, as the appearance image is simply controlled by a prompt.
While the training free nature of the approach allows to use different types of structure images, the approach seems limited to control types with exact edge definition. Higher-level constraints like pose or bounding boxes do not appear to be an option with this type of method.
While the experiments on video generation are interesting, I would recommend that the authors focus more space on structure-only generation (as in the appendix) and expand the ablation and limitation experiments. For example, studying the impact on the quality of generated appearance image for structure-only generation would be beneficial.
Questions
-Can the methodology handle pose or bounding boxes types of controls? If not, are there modifications that can be done to achieve this?
-Can the method handle images with more than one subject? All object centric experiments (structure + appearance) show generation with a single object, often at the center of the image. How does the approach perform with more complex images ?
-In certain settings, conditions appear to be too strong and can affect image quality (e.g. figure 11, dog image). The benefit of guidance/inference is that one can control the influence of a structure image, offering more flexibility. Is there a way to adjust the influence of a structure image in this approach?
Limitations
One experiment in figure 9 shows one limitation of the proposed approach. However, as pointed out in above sections, there are several additional points that should be discussed (flexibility, single subject, loose controls, etc). For some of these, additional experiments could have allowed to understand the behaviour of the methodology more clearly.
Broader impact is adequately discussed.