Generating Images with Multimodal Language Models

Summary

The paper introduces a new method called GILL that effectively integrates frozen LLMs with pre-trained image encoder and decoder models to create coherent image and text outputs. The authors show GILL's superior performance over baseline generation models in tasks involving longer and more complex text and its capability for image retrieval and generation at inference. GILL extends the capabilities of pre-trained LLMs to multimodal by mapping the embedding spaces of the LLMs to visual models with a mapping network. The proposed framework is inspiring and interesting. It may have a large impact for unified multimodal pre-training.

Strengths

* The proposed approach is innovative and inspiring, combining frozen LLMs with visual models to generate coherent image and text outputs. This approach is efficient, as it doesn't require training the image generation model from scratch. * The paper is well-written, logically structured, and accessible. * Experimental results show that GILL is effective in processing long-form text and generating images that are more closely matched to the text than baselines. It can process arbitrarily interleaved image-text inputs and generate retrieved images, novel images, and text, which expands its capabilities compared to previous models. * The paper’s approach to aligning visual tokens with LLMs is promising, and the proposed solutions are insightful.

Weaknesses

* The paper lacks clear implementation details. Despite Eq. 5 explaining the overall objective to optimize, the parameters to be updated are not clearly stated in the equations. This makes it unclear which parts of the model require training and which parts remain fixed. * While the GILL framework is novel, the GILLMapper idea is not new. Aligning a pre-trained/new encoder with the CLIPText encoder of the Stable Diffusion model has been previously explored [1,2], and this related literature is not discussed in the paper. [1] discussed altering the Language Encoder to extend the multilingual capacities for Stable Diffusion. [2] plug-ins the multimodal encoders to replace the CLIPText of the Stable Diffusion. The Eq. 3 of GILL paper is similar to the Eq. 3 of [2] even though GILL applies visual tokens here. [1] AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities. arXiv preprint arXiv:2211.06679 (2022). [2] GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation. arXiv preprint arXiv:2303.10056 (2023) * The paper lacks comparison with state-of-the-art multimodal methods such as BLIP. * The paper doesn’t discuss other relevant works, such as one [3] that unifies both text-to-image and image-to-text in one framework. [3] CoBIT: A Contrastive Bi-directional Image-Text Generation Model." arXiv preprint arXiv:2303.13455 (2023). * The paper lacks discussion of the inference speed, which is an especially essential factor for diffusion-based image generation task. * The paper does not address the possibility of the OPT model not perfectly aligning with CLIPText, which could create cross-model gaps that are challenging to bridge. It's unclear how the authors would tackle such challenges and whether these difficult-to-address domain gaps would cause a decrease in performance.

Questions

See weakness above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

See weakness above

Summary

The paper proposes a novel method to stitch together LLMs and text conditional image diffusion models, to process interleaved image-text and output interleaved image-text. The paper evaluates their methodological contributions on several tasks where outputting images is required.

Strengths

S1. The ability to output generated image or text OR retrieve images to satisfy a user’s query is not something I have seen in prior work and speaks to the generality of the formulation. S2. The paper explores how to stitch together large pre-trained models (i.e., LLMs and image diffusion models). It represents a creative exploration of stitching these models together using a combination of learnable modules, captioning losses, contrastive losses, and distillation losses, which is non-trivial exploration. S3. Generative applications of the model appear impressive in the demo figures. S4. Ablations related to image generation and retrieval are generally strong, thoughtful, and cover many different key design decisions. S5. Fine-tuning cost is cheap and reasonably accessible (~96 A6000 GPU hours).

Weaknesses

W1. Missing citation. The following paper conducts probes showing that pre-trained image and text representations can be aligned: [1] Ilharco et al. Probing Contextual Language Models for Common Ground with Visual Representations. 2021. W2. Missing citations. Since one potential use-case of the model is image captioning (i.e., the model can handle image input and textual output), I also suggest citing recent captioning work: [2] Yu et al. CoCa: Contrastive Captioners are Image-Text Foundation Models. 2022.; [3] Li et al. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. 2022.; [4] Li et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023; [5] Dai et al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. 2023. W3. Missing citation, unsupported claim. Recently, unified models take image and text inputs and generate image and text outputs for a variety of tasks. e.g., [6] Lu et al. Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks. 2022. Hence, lines 4-6 in the abstract seem a bit overstated: “Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs.” W4. Presentation. Consider adding percentage point improvements over baselines in the abstract and intro to give the reader an idea early on of what kinds of gains can be expected by using your model over competitors. W5. Presentation. Consider giving some more intuition about why you are making certain design decisions, instead of just saying what you are doing. For example, I found the motivation for your introduced IMG1, IMG2, …, IMGr tokens to be a bit confusing. It seems here, the supervision encourages learning when to produce IMG tokens, but not what content should be captured in the representation space, which would be different depending on the image. Why should this design decision will lead to learning good last layer hidden representations for arbitrary images. W6. Clarity. The Image retrieval section seems incomplete. An objective is provided, but it is not clear to me how the model is ultimately used to retrieve images at evaluation time. W7. Clarity. The work proposes to use a learned mapping module (GILLMapper) to go from text hidden states (of an LLM) to vision model embedding space (of a text-conditioned image diffusion model). However, it is not clear this module is needed. An alternative strategy might be to feed generated LLM text directly to the text-conditioned diffusion model, thereby bypassing the need to map representations. W8. Missing evaluation, unsupported claims. The paper claims that the model can generate image or text or retrive images. However, the experiments test only the abilities related to outputting images. How does the model perform on tasks that require outputting text (e.g., CoCo CIDEr and VQAv2 accuracy)? I consider such evals to be critical to verify claims that the model can generate sensible text outputs based on image inputs. Such evaluations also allow for comparison with Flamingo-like models. W9. Generality of the method. Deciding whether to retrieve or generate images depends on a liner classifier that is dataset specific. This means that a practitioner wanting to use this model out-of-the-box on data of their own may have to train their own classifier (annotate data etc.) to use the model. W10. Clarity. How is CC3M turned into an interleaved training dataset for model training? Are there any data sampling strategies that are important here? How many images and captions are sampled? Are non-interleaved sequences also trained on (i.e., image or text only)?

Questions

Here are my main questions distilled from the weaknesses above: Q1. Can some of the clarity related questions be addressed, specifically W5, W6, W7, W10? Q2. How does your model perform on CoCo captioning and VQAv2 (W8)? These also seem like useful evaluations to make sure that your model interprets the content in images. Q3. Can authors address the concerns related to the generality of the method for image generation vs. retrieval (W9)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No limitations or failure analysis is presented in the main paper. I suggest discussing conditions under which the model fails.

Summary

The authors train adapters to map embeddings of pre-trained image encoders and decoders to pre-trained LLMs. This allows them to input interleaved images with text into a pre-trained LLM and also make the LLM generate [IMG] tokens as required, which can be fed into a decoder to generate images or can be used to retrieve from a set of images based on cosine similarity.

Strengths

- Clean idea that introduces a fundamental novelty in the capability of text-to-image models - the ability to interleave image and text to generate images. - Adapting image to text space to use with pre-trained LLM's is neat since we don't need to train yet another large model. This idea has been shown to work well in contemporary models like LLaVA and miniGPT-4, but they show it works well for generation as well - a capability these other contemporary models lack. - The results look very promising. They show longer context helps, further justifying the need of such models that can handle long chains of interleaved image and text. - Such a line of work can have multiple interesting follow-up works on evaluating compositionality, etc.

Weaknesses

- One of the strengths of interleaving image and text is that one can expect to extract different concepts from different images to compose a new image. For instance: "a cup that looks like <image_of_come_cup>, but on a table that looks like <image_of_some_table>" Such an analysis could have been great similar to the goal of this paper: https://arxiv.org/pdf/2212.04488.pdf. - Some other recent methods aim to solve similar goals as advertised in the qualitative results. For instance, Figure 5 top row - papers like prompt-to-prompt and imagick aim to do this. Although their method cannot handle arbitrary interleaved images and text, a qualitative comparison of these applications that both methods can handle would make the paper very strong.

Questions

Is it necessary to train for both generation and retrieval? How does it work if we just train for generation and not include the retrieval loss? The ablation in the supplemental Table 1 is just during inference, correct? During training, the authors say they use CC 3M. This only has one image and text pair. Do they use any data that has multiple images and text? Are results on Visual Stories without training on it? What are the [IMG_r] tokens learning exactly and how does varying `r` (the number of image tokens) affect results? I am guessing each of the tokens are learning some image concepts present frequently among the images in data? An analysis on this would have been great!

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Authors discuss limitations in the supplemental.

Summary

This paper proposes to use LLM to do image generation. Their approach consists of two stages of training. In the first stage, they try to learn a linear layer to make VIT visual feature space is compatible with LLM space. In the second stage, they learn r new tokens representing image. They hided states of these 'image token' are used to do retrieval or image generation. For the image generation, they train a GILLMapper to transfer these hidden states into CLIP text space (which is the input to the unet of SD) so that the transferred feature can be directly used as input the SD. They also found that simple design of the GILLMapper such as linear layer does not work well, thus they train a encoder -decoder like model (like a DETR)

Strengths

The writing is clear and I like their idea and motivation in general.

Weaknesses

1, I feel maybe it is not appropriate to claim their method can 'generate image'. The generation part is still from SD; what they are doing is to use a multi modal language model to combine both image and text information into CLIP text space. This can be reflected from their training: Neither stage1 nor stage2 actually involve image generation. 2, for table 1, I think current 1 caption case does not make scene, and maybe they can just skip this, as it is not using any previous information at all. For 5 caption case, I don't know how exactly they feed them (maybe just stack them?). I think a better way is to ask a LLM such as GPT4 to combine all captions and generate a better prompt. 3, for 5 captions and 4 images case in the table 1, the SD can also do this. A simple approach is to use image captioning model such as BLIP2, LLAVA etc to convert images as caption, and prompt GPT4 to generate the final caption by giving 5 captions and 4 images (in the captioning format)

Questions

na

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

In my opinion, their comparison is not complete as they can have a simple extension (see weakness) so that it may potentially improve SD performance. I will raise my score if they could conduct this experiment. If they are going to do the experiment, I hope they carefully prompt a LLM and choose different captioning models to evaluate. ------------------------------------------------------------------------------------------------------------ The authors address my major concerns in their rebuttal and I recommend them to add these stronger baselines in their final version. W8 from reviewer QYwk seems a valid concern which I did not notice before. However, I am not very familiar with evaluation on text, thus I can not confidently evaluate their rebuttal in that question. Overall, I am lean towards the accept, but will not be supervised if got rejected due to other weakness raised by other reviewers

After Rebuttal

Thanks to the detailed explanation from authors. Most of my concerns are addressed and I'd like to raise the score to WA.