Summary
The paper proposes a unified diffusion framework, UnitControl for a more fine-grained controllable generation based on input visual conditions, utilizing two modules mixture-of-experts adaptor that extract features from different visual conditions, and a task-aware HyperNet that extracts language-based task embedding to control visual generation.
Strengths
1. The paper conducts a relatively comprehensive evaluation of the proposed method, including human evaluations.
2. The framework can be useful as it unified different modalities for visual controllable generation using diffusion models.
3. The authors promise to open-source 20M triple training datasets that can benefit the whole community.
Weaknesses
1. **The writing is not so well written and missing details that can hinder understanding of readers**. For example, Figure 2 should also mention parameters of encoder and decoder since I initially thought the introduced parameters are only from MOE adaptors and Hypernet. The authors should also highlight the differences and **similarities** (e.g., encoder and decoder are copied) between their work and ControlNet. In Section 3.3, how is hybrid task generalization done? Since your model only conditions on one visual modality when computing the score in one forward pass, do you compute scores individually for each condition for generation? If so, what sampling technique did you use? One sampling technique I can think of is the conjunction operator from Composable Diffusion [1] or do you use some other existing techniques? For zero-shot task generalization, I also find it confusing and don't quite understand how it is done. Mentioning it in the supplementary is not sufficient since it is meant to provide details that don't affect your understanding overall.
2. **The claims are quite shaky.** The authors claim that the model need to tackle the misalignment of low-level features from different tasks. However, based on Figure 4, I don't see too much of misalignment from baselines, i.e., ControlNet. Besides the visual quality, I don't see much of advantages over ControlNet. Even sometimes single-task ControlNet is better in some aspects. For example, for the object bounding box to image, control net seems to understand there should be two separate benches instead of one long bench generated by the proposed model. In addition, since one unified model is trained for longer time, model could be more robust, directly improving overall visual quality.
3. **The zero-shot generalization ability could be misleading.** Suppose in training, there doesn't exist any instructions that perform colorization. I don't expect the language itself can bridge the gap between visual signals and language to perform such unseen tasks. For example, in Figure 5, such zero-shot results doesn’t necessarily reflect the true reality that the model understands given instructions. One way to check this is to reverse the whole process - for example, can you do image decolorization using your prompt or blur the image instead. Since training images are mostly high resolution, and colored images, the model could automatically utilize learned prior to make images higher quality, which doesn’t necessarily mean they are doing what they are told to do. If so, then the model just does general reconstruction when prompted something unknown.
4. **Missing quantitative evaluations.** As stated, one contribution is to show that the visual quality of such controllable generation can be improved, then it is needed to include quantitative comparisons for image fidelity using metrics such as FID and KID, though I don't think they are good metrics but it provides enough insights.
5. **Lacking relevant baselines.** There are also other existing adaptor methods for such generation, for example, T2I-adaptor [2] and GLIGEN [3] seem to be highly relevant, and both of them are not used as a comparison in the paper.
5. **Related work.** If one of the contributions is to combine different modalities for compositional generation, then it would be important to add existing works related to compositional generation.
[1] Liu et al., Compositional Visual Generation with Composable Diffusion Models (ECCV 2022) \
[2] Mou et al., T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. \
[3] Li et al., GLIGEN: Open-Set Grounded Text-to-Image Generation (CVPR 2023)
Questions
1. How are both hybrid tasks generalization and zero-shot new task generalization done? For example, a detailed procedure (i.e., sampling technique) will be useful for better understanding.
2. There are 6 tasks overlapped with ControlNets, each of which is trained for 100K iterations. However, there are 8 tasks in your multi-task model, but it is trained for 900K, which in fact is not a fair comparison. If I misunderstood, feel free to point that out.
3. The method mainly uses modalities that are spatial, so is it capable of using global context, e.g., color, texture, to guide image generation?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have included limitations and broader impact such as training data bias.