MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation

Summary

This paper presents MultiFusion, a multilingual multimodal image generation model, which can be effectively trained by fusing existing pre-trained visual models, language models, and stable diffusion models. Using a multilingual autoregressive language model as a bridge, MultiFusion follows MAGMA to enable multimodality by learning adapters. Before connecting the language model with the stable diffusion module, it learns semantic embeddings with a contrastive learning objective in a parameter-efficient setup. Finally, it connects the language model with the diffusion model with monomodal data. i.e., an image or caption. Experimental results demonstrate that the trained MultiFusion model can generate high-quality images with multimodal interleaved prompts. Besides, with modular design and the fusion of pre-trained models, the training can be quite effective compared to training from scratch. Several analyses such as attention manipulation also provide insights into the multimodal language models.

Strengths

- The paper makes a clever combination of pre-trained models and adapter learning techniques, including 1) MAGMA for cross-modal adaptation/fusing, 2) contrastive learning before fusing LM with the diffusion module, 3) cross-attention learning of SD to align the conditioning with the new embedding space. These operations delicately combine the pre-trained modules into an end-to-end multimodal-text-to-image model. - Experimental results show that MultiFusion can produce high-quality images conditioned on multimodal and multilingual inputs, with a wide range of applications and use cases. - The analysis on attention manipulation is quite interesting.

Weaknesses

- It would be great to improve the presentation of the paper, especially methods and implementation details. Although Figure 1 presents an overview of the architecture, I have to guess some of the implementation details and carefully find clues from a large amount of text. Some suggestions: (1) you could provide some figures to show the details of how the adapters connect the pre-trained models and how you learn them; (2) you could also clarify the training tasks, and data in tables. - Existing works have explored how to learn adapters to connect pre-trained modules. For example, Flamingo learns gated adapter modules to connect language models with visual models, and generate text conditioned on multimodal inputs. The paper provides a full solution to the problem with careful design but it is kind of an integration of existing adapter methods.

Questions

- Are all the outputs of the language model connected to the diffusion model or only the last token? E.g., Input: [img_tok1] [img_tok2] [text_tok1] [text_tok2] [text_tok3] -> output vectors: [h1] [h2] [h3] [h4] [h5]. Are all five vectors passed to the diffusion model or only the last one? - Why is an autoregressive model used as the encoder? I understand the CLIP model has disjoint text and image encodings, but why not use bidirectional models as the multimodal encoder? Any explanations?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

As mentioned in the paper, the model always produces variations of input images, which limits its applications in image editing. I think it is worth mentioning this limitation, which provides further understanding of MultiFusion.

Summary

In this paper, the authors present a novel approach to expressing complex concepts with arbitrarily interleaved multimodal and multilingual input. Their approach leverages pre-trained models and allows an efficient fusion of different component without training a model from scratch.

Strengths

1. The paper is well-written and easy to follow 2. The experiments are well-designed and allow one to use existing pre-trained models while reducing the demand to train a system from scratch. 3. The result on various benchmarks are promising and would invite more discussion in this line of work.

Weaknesses

The motivation to attempt such a problem is rather weak. Under what circumstances, would one want to have interleaved multimodal input to generate images? Is it because we want to control the input? If so, why not compare the proposed approach with similar models such as ControlNet and DreamBooth?

Questions

See my comments in Weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Summary

In this paper, the authors introduce MultiFusion, a novel approach that enables the expression of complex and nuanced concepts in text-to-image diffusion models (DM) through arbitrarily interleaved inputs of multiple modalities and languages. The “fusion” concept is at the core of the whole work: to fuse modalities together, pre-trained models (a LLM and a stable diffusion backbone) are fused together. Experimental results highlight the efficient transfer of capabilities from individual modules to the downstream image generation module. Notably, MultiFusion empowers the image generation module to effectively utilize multilingual, interleaved multimodal inputs, even when trained solely on monomodal data in a single language. The contributions of this work include the fusion of modalities for image generation, experimental evaluations, and the introduction of a benchmark dataset for further analysis and comparison regarding the multimodal compositionality of the models.

Strengths

1. **Innovative model fusion approach**: The paper introduces an innovative approach by combining a partially frozen multilingual Language Model (LLM) with a stable diffusion backbone. This fusion results in an interesting multilingual and multimodal encoder capable of seamlessly interleaving between input items, treating them as a modality-agnostic sequence. 2. **Multilingual alignment investigation**: The authors conduct an investigation into the model's multilingual capabilities by translating the prompts from the DrawBench dataset. This exploration demonstrates an understanding of the importance of multilingual alignment. While there is a question regarding the accuracy of the translations, the authors acknowledge the potential benefit of utilizing literal translations in training the multilingual encoder, even though nuances in meaning may not be fully captured. This highlights the authors' attention to addressing the challenges and complexities of multilingual representation. 3. **Contribution of benchmark dataset**: The authors contribute to advancing research in multimodal compositionality by producing and sharing the MCC-250 dataset. This benchmark dataset, described in detail in the supplementary material, serves as a valuable resource for assessing the compositionality of multimodal inputs, specifically comprising English text and images. The production and release of this dataset demonstrate the authors' dedication to promoting reproducibility, comparison, and further progress in the field of multimodal compositionality.

Weaknesses

1. **Lack of clear architectural design and novelty**: The paper suffers from a lack of clarity in explaining and justifying its design choices. While references are provided, the underlying motivations and problem-solving aspects of these choices are not adequately explained. While Figure 1 attempts to illustrate the model structure, it is not accompanied by a clear rationale and explanation for the chosen modules and their interactions in the text. Enhancing the clarity of the architectural design would elevate the novelty and originality of the proposed approach. It is suggested to provide a high-level description that guides the reader in understanding the motivations behind specific choices. By focusing on the "why" rather than the low-level details, readers can, for example, grasp the purpose of unlocking only the biases in the LLM. Currently (line 130-131), it is unclear if this is a crucial step to obtain good results while keeping a parameter-efficient regime, or if it is marginal in that regard. The supplementary material can be utilized to provide additional low-level details for interested readers. 2. **Lack of clarity in Figure 4 and semantic search paragraph**: While Figure 4a demonstrates higher similarities of translated prompts in the authors' method compared to competitors, it does not provide insights into the similarities between the reference and other negative samples. This additional information is crucial to establish the range of similarities that can be considered as genuinely low. Furthermore, Figure 4b indicates that the AltDiff competitor generates potentially more consistent images in each language, suggesting that the embedding similarity between references and translations may not be entirely representative. Clarifying these aspects would enhance the understanding of the results and provide a more comprehensive evaluation of the proposed method's performance. 3. **Missing standard deviation in tables**: Including standard deviation in the results would provide important information about the variability and statistical robustness of the findings. By incorporating this measure, the paper would strengthen the reliability and credibility of the reported results. 4. **Performance comparison and insights**: In Figure 4b, the AltDiff method demonstrates better performance, raising questions about the potential benefits of adding more languages to the MultiFusion method. While the authors suggest that alignment remains similar despite MultiFusion being fine-tuned using only English data, additional experiments are needed to provide substantial evidence that adding more languages to MultiFusion indeed yields improved results. Further investigation and insights in this area would enhance the value and understanding of the proposed method. In offering these critical observations, I would like to emphasize that my intention is not to be harsh, but rather to provide constructive feedback. I acknowledge that explaining such a complex pipeline can be challenging and that significant effort has been invested in this work. However, I strongly believe that there is room for improvement in describing the architectural choices and highlighting the strengths of the paper, and I’ve tried my best to give possible suggestions in this regard. I’m convinced that addressing these aspects would significantly improve the quality and impact of the paper, but I don’t think this is something that could be fixed within the rebuttal period. In any case, I remain open to reconsidering my recommendation if any relevant insights emerge during the discussion.

Questions

**Lack of Chinese column in Figures 4a and 4b**: It is unclear why this column is missing or was included solely for the competitors if the proposed method was not tested on this language. There's a typo on line 139: extracted

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

1 poor

Contribution

2 fair

Limitations

The authors adequately addressed the limitations.

Summary

This work proposed a novel method to build multilingual multimodal generation models that supports prompts composed of interleaved text and image. It combines a strong pre-trained multilingual language model with the image generation model from Stable Diffusion (SD) and achieves alleviated capabilities such as prompting with text and image combined. It also shows that the new model does better in composition generations, as one can provide reference images as part of the prompt. In this work, a multilingual language model is trained in the first place (13B encoder-decoder structure model trained on 400B tokens), which itself is a strong multilingual model. It then adds an adapter module to the LLM model to support input in image format, following methods proposed in MAGMA. Finally, it aligns the trained encoder with the diffusion model taken from Stable Diffusion, with 15M text-image pairs. As a result, it can support multilingual, multimodal prompt for image generation, without training on massive text-image pairs dataset. In experimentation, it showed that using both text and image as prompts can be beneficial, especially in composition generations. It also shows superior performance to existing multilingual text-image generation model AltDiffusion, possibly due to better alignment in multilingual embeddings. Further, the support of taking image as prompts can enable varies applications such as negative prompting with image, image composition, image variation and style modification.

Strengths

* Novel and efficient method: fusing different pre-trained models works very well which can bootstrapping existing models such as stable diffusion to achieve different input format, avoiding the heavy cost of training model from scratch. * Enables prompting using both image and text and generates better images both in terms of metrics such as FID and human evaluation, comparing to baselines that only takes text prompt. * Better results on composition generations from considering reference images in prompts. * Well written overall and addressed limitations of the work very well.

Weaknesses

* Some of the details such as model parameter size, training data source and size are not presented in the main paper (included in appendix), which can be less clear when interpreting results presented in experimental section. It would be better to point those factors out when comparing with baselines in the main paper.

Questions

1. As you mentioned in the limitation section, the model cannot produce images that’s exact or close to the image in prompt. I wonder if there can be simple modifications made that can achieve this? 2. How is the interleaved data being used? Does it effect the results if you change the order of the interleaved data? 3. Have you tried other methods in addition to adapter to fuse image and text modality? 4. Have you done any ablation studies on the semantic embeddings? 5. On the MSCOCO dataset for generation, where is the image prompt come from (referring results in table 1)? 6. Have you compared the multilingual LM trained to other ones (mT5 etc.) in multilingual benchmarks?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The authors addressed the limitations relatively well in the paper: 1. The generated image cannot do copy of exact prompt images; 2. Sometimes the image prompts need to be carefully chosen and do not always work; 3. It suffers the same shortcomings (such as inappropriate content) as other generation models trained on very large-scale crawled dataset (LAION).

Summary

This paper presents an approach for creating a model that can take interleaved sequences of images and multi-lingual text as input, and generate novel images as output, by fusing together pre-trained models: (1) a ResNet image encoder from CLIP (2) an encoder-decoder text Transformer LM and (3) a Stable Diffusion (SD) image decoder. Most of the weights of the models are frozen, with some fine-tuning of adapter layers, the biases of the LM, and the cross-attention layers of the SD U-Net. Multimodal training is done on a combination of large scale image captioning, and VQA datasets, using the standard text-conditioned diffusion objective. The encoder-decoder text model was pre-trained on multi-lingual data, making the resulting image generation model also multi-lingual.

Strengths

S1) The aim of this work, enabling an image generation model to take both images and text (in multiple languages) as input, is exciting and well-motivated by some of the qualitative examples in the paper: multi-modal inputs give complementary info, and multi-lingual text capabilities should broaden model accessibility. S2) I found the experiments on compositional robustness (MCC-250), with the improved results from the combination of this text encoder and the image inputs, interesting and think it has the potential to be a timely addition to the ongoing conversation about the role of the pre-trained text-encoder in compositional robustness of image generation (but see suggestions on baselines below). Doing a human evaluation user study was also a real strength of these experiments. S3) The qualitative results were compelling, particularly Figure 5 in the main text and Figure 4 in the appendix.

Weaknesses

W1) The experimentation was a bit thin. - Although there is definitely a shortage of current benchmarks for the new capabilities presented by this model, the contribution of the paper would be stronger if it were able to reappropriate existing benchmarks or create new ones to evaluate some of these capabilities (e.g. negative prompting with images, multimodal image composition). - The method has a few steps (e.g. contrastive fine-tuning on a natural language inference dataset; training on a large number of multimodal datasets, both VQA and captions; and using attention manipulation), but I couldn't find any ablations on these components. This, in combination with the lack of details on the [apologies, the rest of this sentence was missing earlier] datasets, makes me worried about whether the overall approach will benefit future work. - The quantitative results that are presented here would be more convincing with a few (hopefully) easy-to-run variants of the current settings (another classifier-free-guidance weight; ablating image inputs in the MCC-250 experiment); see questions below. The compositional robustness results are interesting, but giving an image as input is a pretty strong (and potentially unrealistic) source of supervision. [update after response] : I still feel that point a) above, about capability evaluation, is a weakness, but the author response definitely helped address the other points. Thank you! W2) The method relies on proprietary datasets and models for the language model (and possibly also for the image datasets, see questions below). I don't think this would be a crucial weakness except that almost no information is given about these datasets and models, even in the appendix. Given that the LM is frozen when doing the multimodal training, and that the capabilities of the fused system (with respect to multi-linguality, and the compositional robustness experiments) seem very likely to me to depend on the properties of this LM, more openness (ideally, using a publicly-released multimodal encoder-decoder transformer, like mT5-XXL, which also has 13B parameters) would really enhance the scientific value of this paper. - The encoder is described as a "13B transformer encoder-decoder similar to GPT-3", but GPT-3 is a decoder-only model, trained with a language modeling objective. - The LM dataset is described only as "400B tokens of English, German, French, Italian, and Spanish", and it's unclear whether the multimodal training data includes datasets other than the ones listed in lines 17-18 of the appendix. - The German-English versions of SNLI and MNLI used for the semantic embedding objective also seem to be proprietary. W3) The writing was somewhat unclear. In particular, a lot of details about the model (the pre-trained models used, the training data for the full approach) were unspecified in the main text, although outlined in the appendix (Section A). Some details about the experiments were also unclear, see questions. [update after response]. The response effectively addressed both W2 and W3 -- thanks!

Questions

Q1) It would be helpful to give any of the following results that are available: - Scores for Table 1 with guidance scale 1.0, as SD seems to surpass MF as guidance scale decreases. - Results for Table 2 that also remove the image input (i.e. use just the text), to see if there's a benefit in compositional robustness from using the pre-trained encoder-decoder model (this would be very cool if so!). - Ablations of any components of the approach, e.g. the semantic embedding fine-tuning, the attention manipulation, or some of the datasets used in multimodal training (e.g., how much value do the VQA datasets add). Q2) Could more information be given about the encoder-decoder LM, in particular what objective was it pre-trained with (e.g. a denoising objective? prefix-LM)? What is in the training data (e.g. web pages? books? is there paired data across languages, or all monolingual corpora)? Q3) What data is used to train the multimodal adapters (to input images) and the SD U-net? It wasn't totally clear to me from the appendix. In particular, are other multimodal datasets used beyond the ones listed in lines 17-18 of the appendix? "such-as" and "like" make it seem like there could be others -- can you say anything about them, if so? Q4) What image is being provided as input in the Table 1 results? Is it the ground-truth image? If so, could you give some intuition for this? The results are much better with multimodal and image, but ground-truth image would be really strong supervision (and probably also explain why multimodal is worse). Not crucial for the author response, but it would also really improve the paper to clarify: - How are the translations of SNLI/MNLI (line 24 of the appendix) generated? - how is the contrastive training on the entailment dataset done? - give details of the multimodal training (e.g. learning rates, dataset size, GPU hours) for both the adapters and the SD U-net. - I didn't understand Fig 4a, as it seems that similarity needs a translation in two language pairs (e.g. en--de) but the categories here are single language.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

I felt that the limitations section was pretty solid in qualitatively outlining weaknesses of the approach, although I'd appreciate experiments to quantitatively support the claim that attention manipulation can help prevent image context overriding text context.