Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling

Summary

Diffusion models generate high-quality images from text but often lack diversity, especially with high classifier-free guidance. Kaleido addresses this by using autoregressive latent priors, which generate diverse latent variables from captions. It enriches input conditions, resulting in more diverse outputs while maintaining quality. Experiments confirm Kaleido's effectiveness in increasing image diversity and adherence to guidance.

Strengths

1. Due to the tendency of samples to converge towards the direction indicated by the condition under high CFG settings, the approach of generating abstractions first to expand the diversity of conditions and then proceeding with generation seems a very reasonable approach. Moreover, these abstractions are controllable, enhancing interpretability and customization possibilities, which is also favorable. 2. The qualitative results also appear promising. 3. The approach of constructing fine-grained prior via autoregression is novel to my knowledge

Weaknesses

1. The formulation in Section 3.1 seems somewhat unintuitive. From my perspective, both text descriptions and the additional autoregressive priors you construct are forms of condition signals for diffusion models. Therefore, I do not see a compelling reason to modify or complicate the original classifier-free guidance formulation. Why not simply regard your method as an extension of the condition signal for classifier-free guidance, while still adhering to the existing CFG formulation? 2. The quantitative results are somewhat limited, with MDM being the only baseline. More comparisons with state-of-the-art diffusion models are encouraged.

Questions

1. What is the motivation behind introducing visual tokens in the autoregressive prior? From the overall narrative of your paper, it seems that the autoregressive prior is intended to provide more customized control for users. However, visual tokens are essentially uninterpretable, making them unusable for user control. 2. The generation process involves an autoregressive process at the beginning. How does this process affect the efficiency of your method?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

More results on the efficiency and quantitative performance of your method are encouraged.

Summary

This paper introduced Kaleido Diffusion which leverages an autoregressive model to first model the latent mode and then generate latents based on the sampled mode. The proposed method is reasonable. The authors explain the insight from a classifier-free guidance perspective. Several experimental results can support the authors' claim. In addition, the paper with the same contents is published as a workshop paper in ICML24.

Strengths

- This paper investigates how to improve the diversity of generated images with additional mode controls, which is interesting. - The writing is clear and the mathematical explanation seems reasonable.

Weaknesses

- The quantitative results are very limited (only Figure 5). - What is the context extractor (MLLM) used in the experiments? Although the authors claimed the mode selection is the major contribution of this work, the additionally enrolled pseudo labels perhaps also contribute to the performance improvements. - I am not convinced that the proposed approach is closely connected to the CFG explanation. Several previous studies, eg, ControlNet (ICCV23) and MaskComp (ICML24) have proven that dense controls will improve image quality. The proposed method is more like distilling knowledge from the pre-trained MLLM to obtain more detailed semantic information about the original dataset. - It would be better to discuss more implementation details to enhance reproducibility.

Questions

- The dataset section claimed the usage of ImageNet and CC12M. However, the results of ImageNet are missing in the entire paper. - Which dataset was used for the results reported in Fig 5? - Can you quantitatively measure the diversity?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper improves the diversity of diffusion generation by incorporating autoregressive latent priors. It leverages the autoregressive model to generate specific discrete latent features, and then concat them with the original extracted text features to serve as the condition of diffusion model. Experiments show that the method can achieve high quality and diversity even with high CFG.

Strengths

The investigated problem is interesting and critical. By incorporating the intermediate "mode" representation, the method can apply CFG after the "mode" is sampled, which alleviates the "mode collapse" phenomenon. The author introduces four specific modes which can also support fine-grained editing.

Weaknesses

(1) The writing lacks training and inference details. During training, whether all the modes are used in each step or they are randomly selected? For the visual tokens, whether the latent features from SEED are directly used or the sequence IDs are re-embedded? During inference, the autoregressive model is responsible for generating the latent modes if I understand correctly. Then how different latent modes are generated in control especially the combination of them? (2) Evaluation is poor. In the experiment section, only a figure is provided without providing numerical numbers. The main experiment setting is confused (class conditioned or text conditioned). The construction of the toy example is also not clearly stated. (3) Comparison with related works is not sufficient, making the position of this work unclear. There are related works also trying to improve the generation diversity of diffusion models such as [a]. [a] CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.

Questions

I hope the authors can answer my questions in Weakness section point by point. I am willing to raise my score if my concerns are well addressed.

Rating

6

Confidence

4

Soundness

2

Presentation

4

Contribution

3

Limitations

See Weakness.

Summary

In this paper, the authors propose a principled pipeline called Kaleido Diffusion for text-to-image generation with better mode coverage and diversity. The main intuition is that, conventional DM requires large CFG to make samples locate at high-likelihood modes, which would somehow constrain the diversity by only generating those limited modes. To solve this problem, the authors propose to first use AR to capture the distribution of such latent tokens, then use diffusion model to take them as extra conditions for generation. Experiments on both quantitative numeraical results and qualitative visualizations are used to evaluate the effectiveness of this work.

Strengths

1) The intuition of this work makes sense to me. The excessively large CFG would limit generated samples to certain modes, where using an extra model to capture such text-irrelevant information is a good practice. 2) The visual quality and especially the teaser of this work can well demonstrate the effectiveness of this paper. 3) The experiments are thorough to me.

Weaknesses

1) I still don't well comprehend why choosing AR as the first-stage distribution learner. One of the reason I guess here is that, the CFG in AR model is only modulating the predicted logits, so that we can still sample across multiple reasonable modes for the next-stage DM through temperature parameter or top-k/top-p sampling. However, I'm still a little bit confused by this part: what if we use a diffusion model w/o CFG sampling to learn such an intermediate distribution? Is the model architecture design choice mainly for the fact that the authors use detailed textual descriptions as the latent token, so that it's natural to choose AR? If so, then this is a little bit ad-hoc to me. 2) Choosing these four specific types of latent tokens is also kind of ad-hoc to me. For example, somebody may also say the excessive CFG could result in similar artistic styles in generation. In this way, we have to additionally extract the style token as ground truth, then laboriously train the whole pipeline again to enable it. Another drawback of such design is that: the training cost is large and it's hard to scale up, every time we want to solve the mode collapse of new categories, we have to first extract/tokenize that new aspect, then jointly train the diffusion model and AR part, which is too ad-hoc and hard to scale up to me. 3) I hope the authors could also discuss about the diffusion decoder idea and the difference between that and this work. Specifically, AR + diffusion decoder would similarly take the latent tokens predicted by AR and feed into DM decoder as conditions. The major difference is that their setting doesn't need the ground truth for those latent tokens, which means the latent tokens are implicitly learned via end2end training. My question is what's the benefit of Kaleido compared to their pipeline, and won't their pipeline better at scaling up?

Questions

They are mainly elaborated in the weakness section. Overall I really like the intuition of this work. I'd like to see the authors' responses on my questions.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are discussed in the appendix.