Text-to-image generation has recently witnessed remarkable achievements. We introduce a text-conditional image diffusion model, termed RAPHAEL, to generate highly artistic images, which accurately portray the text prompts, encompassing multiple nouns, adjectives, and verbs. This is achieved by stacking tens of mixture-of-experts (MoEs) layers, i.e., space-MoE and time-MoE layers, enabling billions of diffusion paths (routes) from the network input to the output. Each path intuitively functions as a "painter" for depicting a particular textual concept onto a specified image region at a diffusion timestep. Comprehensive experiments reveal that RAPHAEL outperforms recent cutting-edge models, such as Stable Diffusion, ERNIE-ViLG 2.0, DeepFloyd, and DALL-E 2, in terms of both image quality and aesthetic appeal. Firstly, RAPHAEL exhibits superior performance in switching images across diverse styles, such as Japanese comics, realism, cyberpunk, and ink illustration. Secondly, a single model with three billion parameters, trained on 1,000 A100 GPUs for two months, achieves a state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. Furthermore, RAPHAEL significantly surpasses its counterparts in human evaluation on the ViLG-300 benchmark. We believe that RAPHAEL holds the potential to propel the frontiers of image generation research in both academia and industry, paving the way for future breakthroughs in this rapidly evolving field. More details can be found on a webpage: https://raphael-painter.github.io/.
Paper
Similar papers
Peer review
Summary
This paper presents a new large model RAPHEL (short for distinct image regions align with different text phases in attention learning ) for text-to-image generation. Technically, RAPHEL builds upon the LDM pipeline, with VAEs as image encoder-decoder, and then incorporates the MOE layers for spatial and temporal (in terms of diffusion steps) refinement in the diffusion generation process to improve the text and image fidelity. For the experiments and evaluations, the model is trained with LAION-5B and some confidential internal data, comparisons w/ other large models show better performance using the zero-shot FID on the COCO.
Strengths
- The paper is well written with a clear structure and easy to follow. - The model achieves sota FID and qualitative results compared to other strong and powerful similar scale models like Stable Diffusion, DeepFloyd and DALLE2. - The high-level idea to incorporate MoEs to refine the spatial and temporal details in DPMs for text2image synthesis is intuitive and reasonable, with effectiveness proved in ablation studies.
Weaknesses
- This is another large-scale model work that requires 1000 A100 GPUs with 2-month training on LAION-5B plus internal dataset, while the reviewer acknowledges the popularity of the topic and its superiority in performance, it is another work that can be hardly reproduced by most researchers in the field, especially w/ internal training data inaccessible for the community. - In terms of the methodology design, while the idea to refine the generation process w/ MoEs is intuitive, the technical novelties are rather limited, and are largely limited to this specific text2img task, as there are several existing works w/ similar ideas [4,5]. - Some technical details remain rather coarse and unclear. See details in my questions.
Questions
I have several questions regarding several technical aspects listed below: - I am still confused on the working mechanism of time-MoE after reading the paper and appendix. For the time-MoE, it is a Time Gate Network at each diffusion step, located between the cross-attention and space MoE. The output of time-MoE is fed into the space-MoE, then what does this info depict for space-MoE at different diffusion steps? - How does this work in inference, if the info from time-MoE does convey critical information in terms of steps, then how can the info be used in inference especially with the skipping sampling steps? - What is the inference time cost using the proposed RAPHEL compared to other popular models? - How does the internal data impact the final performance? Does the performance change evidently w/o the inaccessible internal dataset?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The paper discusses the limitations and potential negative impact on the risk of generating images with misleading and false information.
Summary
The paper trains a large-scale latent diffusion model for image synthesis. It trains on a mix of LAION-5B (post-processed + filtered by aesthetic score) and in-house data. It proposes two novel technical contributions: 1) using a "spatial" mixture-of-experts (MoE) where an expert is predicted from each text token and processes the attended features; and 2) supervising attention weights via edge maps. Interestingly (and differently from most of the prior works), it also uses multi-scale training. Qualitatively, the generated images look substantially better than the ones from the baselines. It achieves SOTA FID on COCO, which is the main metric/benchmark for text-to-image generators.
Strengths
- The method achieves the very best known results for large-scale text-to-image synthesis (among those models which could be rigorously benchmarked against). - There are two novel ideas explored in the work: using spatial MoEs (for attended features) and using edge supervision for attention weights. Both ideas make sense and should be easily extendable to other setups. For spatial MoEs, there are also test-time visualizations provided which helps in understanding their influence. - The comparison to other methods is careful and thorough: a lot of non-cherry-picked qualitative results provided; human studies are performed, quantitative metrics are reported.
Weaknesses
I have two big concerns: 1) spatial MoE and edges supervision are not properly ablated; and 2) the paper does not contain enough technical details to be reproduced. I will elaborate on them below: 1. Improper ablations. After spending ~5 hours on reading the paper, it's still not clear to me where exactly the SotA FID score on COCO is coming from — architecture, data, or optimization, since they are all intermixed in the final model. Fig 6 denotes several ablation experiments, but it's not specified anywhere how were they trained. Does each run in Figure 6 was trained for 2 months on 1,000 GPUs as well? On the same dataset? Such lack of details about ablations makes them impossible to understand and analyze. For Figure 6c, what the FID scores for the line "Computational complexity" denote? This is confusing. 2. Lack of details. The current manuscript is something between an academic paper and a technical report. Here are some (of many) missing details: - How exactly does your U-net and VAE look like? Are they equivalent to LDM ones, but with larger channel sizes? Or there are other modifications (apart from spatial/time MoEs)? - How many images are in your in-house dataset, what are their resolutions and how it was collected? - How exactly was multi-scale training implemented? What are the resolution distributions in your final dataset? Do I get it right that different batches on different GPUs have different amount of images in them (since the resolutions are different)? Do you allocate the same amount of GPUs per bucket? Do I get it right that your VAE is multi-scale, while the diffusion model is not? Or vice-versa? - How random noise \epsilon is sampled for expert routing (L122)? - What is the motivation of using focal loss for edge prediction instead of other loss types? Using focal loss here is quite non-intuitive to me. Did you try ablating it? There are also several smaller (but still reasonable) concerns: - It's not clear whether spatial experts reflect any text semantics. Judging by Figure 4, the expert assignments are completely random. I would expect to see that visually similar concepts ("tiger"/"cat"/"leopard"; "dog"/"wolf"/"fox"; "tv"/"monitor") would get assigned to the same expert. Could you please provide any support or refutation to such intuition (e.g., buy checking the clusters)? - The writing quality could be improved. There are many variables introduced, and it would ease reading if they would be described in text, i.e., instead of writing "we set \lambda to X", one should write "we set learning rate \lambda to X". Otherwise, a reader needs to jump back and forth trying to recall the variable meaning. - There is a quite confusing notation clash: - \epsilon denotes random noise in diffusion and random noise (L74) in experts routing (L122) - \alpha denotes variance schedule (L74), focal loss hyperparameter (Figure 6a), routing multiplier (L129) - Limitations are not properly discussed (see the "Limitations" form below) - GigaGAN is a missed reference since they also use MoEs, routed by text tokens. - It's not clear why SR-GAN is included into the exposition since there is nothing special about it, and SR-GAN can be combined with any other image generator. Does it work better for RAPHAEL than for other image generators? If so, then it's interesting. But if not — then it's not clear why claiming that RAPHAEL can generate 4096 x 6144 images when combined with it. Following such arguments, what prevents one saying that StyleGAN can generate 100,000 x 100,000 images when combined with tailor-mode bilinear interpolation? Typos and minor comments: - L114: "mean of all experts" => "weighted average of all experts"? I look forward to discussing my concerns with the authors and fellow reviewers and improving my rating.
Questions
I raised several concerns in the "Weaknesses" section and would be grateful to hearing the author's opinion on them. My main concerns are the lack of details about the method and experiments.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
There is a brief limitations discussion on potential negative societal impact. In this regard, I wouldn't demand more discussion from the authors since a potential misuse of powerful image generators is a well-known issue to the community and should be discussed at a "higher" level, rather than in this particular work. However, it would be good to see the discussion of other potential limitations of the work, for example, what are the disadvantages of binarizing the attention maps? whether it is possible to ablate the model properly — at least via convergence plots for partial runs (I understand that training a full model for each ablation is infeasible)? what could be a problem with edge maps supervision of the attention maps (i.e., I guess there should be failure cases in edge map detection)? And so on.
Summary
The paper proposed RAPHAEL, a text-to-image diffusion model. The model adopt MoE layers, including space-MoE and time-MoE layers. In addition, edge-supervised learning is proposed to enhance performance. RAPHAEL establishes a new state-of-the-art with a zero-shot FID-30k score of 6.61 on the COCO dataset, and surpasses its counterparts in human evaluation on the ViLG-300 benchmark.
Strengths
1. The authors promised to release a programming API for RAPHAEL to the public. 2. Explore spatial- and time-moe, and perform some abaltion study. 3. visualize spatial- and time-moe in the appendix. 4. achieved sota zero-shot FID score on coco.
Weaknesses
As a paper focusing on pretraining, more clarification of experiment details is needed. Including: a. data. The paper mentioned that "The training dataset consists of LAION-5B and a few internal data.". How many internal data is used? Its category distribution and collecting sources? b. model structure and hyperparameters. including VAE, and each stage of diffusion model.
Questions
1. Is the space-moe performed only on text-image cross attention? How about applying moe to self-attention within image? 2. In abstract the authors mentioned that "RAPHAEL exhibits superior performance in switching images across diverse styles" (L13). Is it benefit from spatial-moe? if so, is there any pattern in moe paths regarding different styles?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
see weakness and questions.
Summary
This paper proposes RAPHAEL, a new text-to-image generative model, based on the latent diffusion model framework. The main methodological contribution is the use of space-mixture-of-experts (space-MoE) layers. These are layers that focus on different concepts from the text prompt in different spatial areas of the synthesized image. Different space-MoEs are automatically chosen for the different text tokens, and they are assigned to the relevant regions in the image, which can be found through the cross-attention maps. A similar time-MoE is also incorporated, although that is less novel. The model is validated on standard benchmarks and achieves state-of-the-art performance. Ablations over all relevant new components and hyperparameters are performed. The paper shows various qualitative results and model samples, which are quite impressive. The authors also demonstrate that RAPHAEL can be easily combined with ControlNet, LoRA fine-tuning, and super resolution GANs to enhance the image resolution.
Strengths
The main strengths of the paper are: - State-of-the-art text-to-image generation performance, including visually impressive results. - Extensive ablation studies on all relevant components. - The space-MoE idea is well motivated, novel, and boosts performance non-negligibly. - I like in particular the gating mechanisms in both the space- and the time-MoE, which, for instance in the time-MoE case, automatically learn to assign different timesteps to various time experts. Previous work manually assigned experts to different diffusion time intervals (for instance, eDiff-I). **Clarity:** The paper is well written and easy to read and follow. There are no major concerns regarding clarity. However, some details seem to be missing (see below). **Originality:** In general, the mixture-of-experts idea is not new and has existed in language models and was also used in the text-to-image literature, for instance in the related eDiff-I (only time experts). However, the space-MoE technique specifically in the context of text-to-image generation is new, to the best of my knowledge, and it is well-motivated and seems to meaningfully boost performance. Hence, while the paper's originality is not groundbreaking, the main technical contribution is well-motivated and novel. **Significance:** Text-to-image generation is a highly relevant and impactful topic, and RAPHAEL achieves state-of-the-art performance in this competitive area. Its mixture-of-experts approach is well-motivated and may find wider adoption. Apart from the quantitative evaluations, its visual results are stunning. Hence, I think the paper is impactful and significant. **Quality:** The overall quality of the paper is high. The paper is easy to read, appropriately discusses the related literature, provides a background section, runs extensive ablation studies for all new relevant parameters, and supports its claims by appropriate experiments. The qualitative and quantitative results are strong. There are only relatively minor concerns with respect to missing details (see below).
Weaknesses
The paper does not have any major weaknesses. However, I have some minor concerns: 1. Many details are missing: - The paper only mentions on the side (Line 192) that it is using a latent diffusion model framework and does not operate in pixel space. That is fine; however, this requires more details. For instance: - Was the autoencoder regularized? The LDM paper uses either KL-based or VQ-based regularization. - Was the autoencoder trained only with a reconstruction loss? Or also with a (patch-wise) discriminator? - What was the downsampling ratio? - The work should explain the multi-scale training in more detail. How exactly is the model trained at all these different resolutions and aspect ratios at once? - The gating mechanisms incorporate an $argmax$ function. The $argmax$ is usually not differentiable. How did the authors deal with that, to enable regular backpropagation for training? I think it would be helpful to discuss this in a bit more detail. - The paper says that RAPHAEL is trained on LAION "and a few internal data". What is this internal data? Even if this data is internal and not released, the authors should describe it and what value it brings on top of LAION. E.g. what is the size of that internal data? Does it have any special characteristics? Only high-quality, for instance? Etc. 2. A discussion on limitations is missing. This would further strengthen the paper. In conclusion, the paper's main weaknesses are all related to missing details and I believe these issues can be addressed easily. I do not see any other major concerns. Considering the paper's strengths discussed above, I am consequently suggesting acceptance of the paper.
Questions
I have only one minor question (just curiosity, not impacting the paper rating): Figure 4 shows the diffusion paths for different simple concepts. Do more related or similar concepts also have more similar diffusion paths? For instance, would "strawberry" and "raspberry" share much of their paths, while "strawberry" and "car" would not?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Potential negative societal impacts have been briefly, but sufficiently addressed. Limitations have not been discussed. What are RAPHAEL's limitations? I would like to encourage the authors to add a critical discussion on this.
Follow-up questions
Thank you for your response, it helped me to understand your work much better and resolved several concerns. I would like to clarify a couple of more questions if that's possible: 1. Do I get it right that for your multi-scale training, the diffusion UNet model also operates on inputs/outputs of different resolutions? E.g. an image of shape [3, 448, 832] is encoded into the latent code of shape [12, 56, 104]? Does this also mean that there is an attention layer somewhere inside the UNet running on a 7x13 resolution? Or you use different downsampling factors inside the UNet? 2. What does "Use checkpoint" mean in Table 2 in the appendix? 3. How many overall steps/epochs do you do over the course of your 2 months training? For my previously raised concerns: - About Q1 (ablations). Honestly, the provided ablations are quite unusual, since the performance might drop after removing the components/objectives simply because it's a too drastic change in the architecture (even with the presence of residual connections). For Figure 6a, do I get it right that you trained the model with $\alpha=0.2$, $T_c=500$, and then, after 2 months of training, fine-tuned for other hyperparameter values? If so, how did you know that $\alpha=0.2$, $T_c=500$ were optimal? - About Q7. Is it possible to attach any plots to the rebuttal or update the previously attached PDF? I would be curious to see the diffusion paths for those kinds of prompts.
Response to follow-up questions
Thanks for your follow-up questions. I'm delighted to address your concerns. **Q1: Multi-scale Training** Yes, the diffusion UNet model also operates on inputs/outputs of different resolutions. Yes, there is an attention layer somewhere inside the UNet running on a 7x13 resolution. Consequently, this necessitates the scales of buckets to be divisible by 64. Notably, my experience has shown that cropping images to a fixed scale (e.g. 640) can **destroy** the performances of text-to-image diffusion models. Therefore, the importance of adopting a multi-scale training approach becomes more pronounced. Based on my experience, fine-tuning a model trained at a fixed scale (e.g. 640 x 640) to a multi-scale version takes 5 days. **Q2: Use of Checkpointing** The term "use_checkpoint" originates from the configurations within the stable diffusion (unet_config). Pytorch Checkpointing is employed to conserve memory and mitigate CUDA out-of-memory errors. **Q3: Iterations** Our model has undergone training for approximately 1.02 million iterations, equivalent to roughly 2.9 epochs. **Q4: Ablation Study** **Experimental Setup** Firstly, the objective of the ablation study is to validate the efficacy of each module outlined in our paper. The deletion operation works very well in our experiments. Fig. 6c provides further affirmation – **by resuming training from the same initial point and progressively increasing the number of experts, a consistent reduction in FID is observed. This unequivocally confirms the efficacy of both space-moe and time-moe.** The deletion of the edge-supervised learning module doesn't significantly alter the architecture, and this module exhibits relatively rapid convergence. Secondly, we believe this to be the **optimal** approach for ablating RAPHAEL. This is particularly because our model is designed for a **text-to-image** and **large-scale** image generator to measure the **zero-shot** FID. Running these partial settings of ablation study has also entailed a substantial investment of 1.5 months and millions of dollars. **Tuning $\alpha$ and $T_c$** Regarding these two hyperparameters, we adopt a proxy-based strategy to identify the optimal values before commencing full-scale training. As both space-moe and edge-supervised learning align with stable diffusion, we incorporate them into stable diffusion v1.4. This involves training each configuration over a 14-day period, and subsequently searching the hyperparameters based on the evaluation outcomes. It's important to note that this isn't an ablation pertaining to RAPHAEL but rather an investigation based on stable diffusion. As such, the inclusion of this aspect in our paper is unfeasible. However, if necessary, we can upload this part to the appendix for the camera-ready version if the paper will be accepted since we can't edit it now. This practice is widely adopted within the foundation model community prior to embarking on full-scale training. **Q5: Diffusion Path** I have lost the permissions to edit any contents of my paper, PDF files, and even the preceding rebuttal during this author-reviewer discussion phase. I will definitely add these visualization results to our camera-ready version to facilitate the community and enhance the accessibility of our work if the paper will be accepted. Please let me know if you still have more concerns, and I'm very happy to discuss them with you.
Thank you for additional clarifications
I am grateful for receiving so many clarifications. It greatly resolved my concern about the paper missing important details. Also, they have > we believe this to be the optimal approach for ablating RAPHAEL I do not agree with this statement. You could have just trained a smaller model on something like MS-COCO for a 1-2 weeks on a couple of nodes. If the proposed technique is general enough, then it would lead to a superior quality in this setup as well. Or you could have trained a much smaller model on your internal dataset. > by resuming training from the same initial point and progressively increasing the number of experts, a consistent reduction in FID is observed That's a strong and valuable observation, I apologize for missing this out. > As both space-moe and edge-supervised learning align with stable diffusion, we incorporate them into stable diffusion v1.4. This involves training each configuration over a 14-day period, and subsequently searching the hyperparameters based on the evaluation outcomes. It's important to note that this isn't an ablation pertaining to RAPHAEL but rather an investigation based on stable diffusion. As such, the inclusion of this aspect in our paper is unfeasible. I do not see why the inclusion of this is infeasible. Showing that the proposed technique works for Stable Diffusion would be valuable for a reader. I think that the paper would benefit from incorporating such exploration on top of SD into the appendix, since it would show that space MoE works in a slightly more general setting, rather than for RAPHAEL only. Btw, is it possible to re-export the plots as PDF rather than PNG in the paper? The text on it (e.g., on Figure 5) is not selectable/searchable. -------------------------------------------------------------------------- Overall, I believe that if the final version will incorporate all the details uncovered in the rebuttal and also additional visualizations for diffusion paths (as discussed in the previous response), it will be an interesting read for the community. Especially given that it even outperforms Midjourney (though it was not confirmed with human evaluation, only with qualitative evaluation on the selected prompts?). This is why I decided to increase my rating to "Accept".
Thanks for your comments and suggestions
We sincerely thank the reviewer for the constructive feedback and the kind support of this work! I will incorporate these details into the final version if the paper is accepted.
I would like to first thank the authors for the rebuttal and for carefully responding to my questions and concerns. I have read the rebuttal and (have always) acknowledged that the RAPHAEL is a technically solid work with its valuable contributions to the community. However, I also do not think the fact that other similar works, such as Imagen, have been previously awarded with the prize, or other large generative models have also used internal datasets is the expected answer to my raised concern (to be honest, it is never a question or concern specifically against RAPHAEL but a general research question/concern), in terms of reproducibility, model accessibility or performance. Personally, I would be more appreciative of works that actually improve our understanding of the model itself and can be beneficial and insightful for other practical and widely accessible applications. Despite the above, I think my questions regarding the technical details have been answered, and thus raised my score after rebuttal.
Thanks for your feedback
We deeply appreciate your valuable feedback. And we will incorporate the technical details uncovered during the rebuttal into the final version if the paper is accepted
Thank you for the rebuttal
I would like to thank the authors for their rebuttal and for providing extensive details to answer my questions. It would be great to incorporate all these explanations into the final version of the paper. I do not have any further questions or concerns. I have been already positive about the paper and still suggest acceptance.
Thanks for your comments
We sincerely thank the reviewer for the kind support of our work! We will incorporate the details into our final version.
Thanks for your comments
We deeply thank you for the kind support of our work!
Decision
Accept (poster)