Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

Summary

Extending context length is a fundamental challenge for LLMs. Unlike previous approaches that focus on efficiently handling text tokens, this paper introduces a novel method: encoding lengthy text information into image renderings. These image tokens effectively increase the context length and enhance the performance of LLMs across various downstream tasks.

Strengths

The idea is simple yet surprisingly effective. Previous works like PIXEL and CLIPPO suggested treating text as images to eliminate the need for tokenizers and unify various languages into a single image format. In contrast, this paper uses image rendering to encode lengthy texts, employing a PIXEL-like method to enhance long-context understanding.

Weaknesses

1. Information loss from image rendering The current approach has two potential sources of information loss: 1) rendering long texts into an image, and 2) encoding an image into MLLM embeddings. This loss is not thoroughly investigated in the paper. For instance, Table 2 shows that using the original lengthy 426 tokens outperforms the proposed rendered image, albeit at higher computational costs. While it is acceptable to trade some performance for efficiency, the trade-off should be clearly demonstrated in the paper. --- 2. Optimal compression rate Due to the potential information loss, rendering very long texts into a single high-resolution image might not be optimal. For a context of, say, 2048 tokens, what is the best approach: a single image with 2048 words, or 32 images with 64 words each? The current ablation study only examines the rendering aspects like font size and font interval threshold. However, the trade-off between the number of images and the number of words per image is a crucial study that should be included in the paper. --- 3. Comparison with other long context methods A significant weakness of the paper is the lack of comparison with other long context methods. Specifically, the paper must compare the proposed compression-into-image approach with compression-into-text approaches, such as [1-3]. Indeed, text compression loses the original semantics, while image rendering retains all previous words. Therefore, combining both approaches—compressing the semantics first and then rendering them into an image—could potentially offer the best of both worlds. In addition to the text compression approach, it would be beneficial to discuss the pros and cons of this work in comparison with other long context methods. The current paper only discusses classic efficient self-attention models like Longformer. However, there are more recent and diverse approaches, such as handling long sequences by dividing them into multiple chunks [4-5] or using interpolation positional encodings [6-7], among others. [1] Mu et al. Learning to Compress Prompts with Gist Tokens. NeurIPS 2023.\ [2] Chevalier et al. Adapting Language Models to Compress Contexts. EMNLP 2023.\ [3] Ge et al. In-context Autoencoder for Context Compression in a Large Language Model. ICLR 2024.\ [4] Bertsch et al. Unlimiformer: Long-Range Transformers with Unlimited Length Input. NeurIPS 2023.\ [5] Song et al. Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs. ICLR 2024.\ [6] Chen et al. Extending Context Window of Large Language Models via Positional Interpolation. arXiv 2023.\ [7] Li et al. Functional Interpolation for Relative Positions improves Long Context Transformers. ICLR 2024.

Questions

This paper claims that long texts can be converted into images. If that's the case, do we need text tokens at all? Can we replace all the text with images, instead of just some prefixes as is currently done?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Discussed, but preliminary. The paper only addresses a minor technical limitation regarding static vs. dynamic tokenization of images. However, there are many more potential limitations to consider. For example, I wonder if the current approach would also be effective for larger models, such as Llama-3, which has a longer context length of 8192.

Summary

The paper proposes a method to increase the context size of multi-modal large language models. The goal is to increase the context with minimal GPU memory for both training and inference as well as floating pointing operations. Finally, the authors show that the method obtains good results on in-context learning.

Strengths

The paper tackles an important problem and can be of interest to the research community. It seems to achieve promising results and the idea seems interesting, but there are parts that are unclear or need better studying.

Weaknesses

I think that the paper is hard to follow and some parts hard to understand. Please see below for some questions My other concern is related to the experimental part, that I find a bit weak. The comparison is made against only two existing models and I don't clearly understand the numbers. Firstly, one concern is that there seems to be a degradation in performance with 0 shots in some cases. Also, probably I am missing something, but let's look at Tab. 1. Wouldn't an increase context size (ICL 256 vs 2048) would allow for more shots to be used? I don't understand what is different in terms of the theoretical context between 0 shot Open-Flamingo and VisInContext. Isn't the same information fed to the model? If yes, then I don't see how the experiment measures the impact of the context size. Or is it just way to feed more information from a particular document as opposed to sub-sampling it? In the later case, I would consider this to be a slightly unfair comparison and I don't really understand what are the benefits of the model if in the zero-shot setup where you have access to a larger portion from the document (as opposed to subsampling) the performance drops quite a lot in some cases (okvqa, vizwiz). This shows that the way that the context is processed is not ideal.

Questions

Questions related to lines 75-76. 1. Both images and text are concatenated to result in a sequence of 256 tokens? 2. Is m chosen so that the concatenation is 256 tokens? 3. What does the corresponding text represent? Based on Fig. 2, `I_i` seems to be an actual image and not an image representing the text, so I am confused. Or the bear image has nothing to do with `I_i`? If it doesn't where do the image tokens `I_i` are used in Fig 2? Questions lines 81-82. 4. Is this the same m as above? If yes, can you elaborate what the m represents? I assume no, since it's `M` vs `m`, but I think other letters can be used to make this more clear

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations are briefly discussed

Summary

This paper introduces a method called Visualized In-Context Text Processing (VisInContext) to address the challenge of processing long in-context texts in multimodal learning, which arises due to the high GPU memory and computational costs associated with lengthy textual content. VisInContext converts long textual content into images and uses a visual encoder to extract textual representations, thereby increasing the in-context text length that can be processed effectively. The method is based on a dual-stream encoder model that employs Token Masking and Text-Centric Contrastive Learning (TCCL) to improve the model's ability to learn from the rendered text images, and the paper demonstrates the effectiveness of VisInContext through experiments on various tasks, showing that it outperforms the baseline in terms of performance and inference cost, while also improving the optical character recognition (OCR) ability of the model.

Strengths

1. The paper is in well-written, which makes it easy to understand. 2. VisInContext can reduce GPU memory usage and FLOPs for both training and inference, allowing the model to handle much longer text contexts with lower computational cost. 3. The model trained with VisInContext delivers better performance on common downstream benchmarks for in-context few-shot evaluation and document understanding. 4. The method shows potential in document understanding tasks, as evidenced by the improvements on DocVQA and OCR VQA datasets, and the enhanced next-word prediction accuracy of the LLM on the Rendered Text dataset.

Weaknesses

1. The proposed method involves multiple steps, including text rendering, token masking, and contrastive learning, which might add complexity to the implementation. This could be a barrier to adoption for some practitioners. 2. From my perspective, the pipeline of "text -> text-rendering -> LLM" seems somewhat derivative from the initial purpose of LLMs (Large Language Models). That is, one may question whether we truly need such a complicated paradigm. If the answer is yes, then this approach appears more akin to an engineering exercise. Further, the performance may be affected by the selected font type, font size, and even the rendered canvas. 3. The process of rendering text into images and then processing these images through a vision encoder may introduce new bottlenecks, particularly as the text length increases. The paper could explore these potential limitations or trade-offs in greater detail.

Questions

refer to weaknesses

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

My primary concern is the necessity of the "text -> text-rendering" process. Perhaps it would be beneficial to explore a new perspective that focuses on aligning visual and textual information more effectively. However, as we know, image space typically contains much more redundancy than semantic space. For the current version, I am not fully convinced that projecting text into the image space is the optimal approach. Furthermore, rendering text may introduce several additional complications that warrant careful consideration.

Summary

The paper introduces Visualized In-Context Text Processing (VisInContext), a novel technique designed to enhance multi-modal learning models by efficiently expanding their in-context text length. This method transforms extensive text into visual tokens, substantially lowering GPU memory consumption and computational requirements during training and inference. Models utilizing VisInContext demonstrate superior performance on standard downstream benchmarks for in-context few-shot evaluation compared to conventional approaches. They also show improved document comprehension, particularly in document QA and sequential document retrieval tasks. An additional advantage of VisInContext is its compatibility with current context-extending techniques, allowing for potential combined applications.

Strengths

- The paper is well-written and easy to follow. - The suggested method presents an innovative and intriguing concept: converting text into visual representations to decrease computational expenses.

Weaknesses

- The motivation of “text-only in-context few-shots experiment” is not clear. These experiments appear tailored specifically to validate the proposed method rather than addressing practical applications. In particular, the use of text-only versions of visual question answering (VQA) or image captioning tasks for in-context learning seems questionable. The relevance and applicability of such text-only adaptations of inherently visual tasks in this context require further justification. - There are some unconvincing parts about token masking. In Line 87~88, the paper says the masking ratio of raw image tokens is 1.0. Then the model does not observe the raw image at all. Or, is the model initialized with OpenFlamingo Model, especially for cross-attention layer and resampler? If so, how does the token masking probability affect the model’s ability to learn text semantics from visual inputs? For my intuition, it seems there could be some trade-off of partly observing (masking) raw-image and learning text-image semantics from rendered text images at the same time. - Minor : - Reference, Figures needs compile check. There are some errors. (Line 159, 521, etc. ) - The formats of Table 1, 2 need to be more polished

Questions

- Table 1 exhibits an interesting trend where the baseline model occasionally shows improved performance with an increased number of shots in certain scenarios. This pattern might be attributed to specific characteristics of the datasets or peculiarities of the classification task at hand. What explanation do the authors propose for this counterintuitive observation?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The paper addresses thier limitation.

Rebuttal answer

Thank you for the rebuttal! I confirm I have read the rebuttal and I currently don't have other questions. The rebuttal brings more clarity and answers my questions, but I still think there might be a problem of clarity. Hence, I slightly raise my score.

Response to the Rebuttal

Thank you for the rebuttal. As other reviewers noted, this paper challenges the common belief that "image space typically contains more redundancy than semantic space" and argues that images can serve as better tokenizers than texts. Given the ambitious nature of this claim, it may be more challenging to convince people. However, the proposed concept of rendering images as text opens up several intriguing research directions. For example: 1) What is the optimal way to tokenize the combination of images and text? 2) How can we effectively train large models with these tokens, considering the challenges in next-token prediction? For these reasons, I believe this paper could make a valuable contribution to the conference, sparking new ideas among attendees, and I am inclined to maintain my original rating of acceptance.

Response to the rebuttal

Thanks for the response. I carefully read through the authors' responses and the discussions among the other reviewers. While most of my initial concerns have been addressed, I still have an unclear part, which is related to the token masking. In lines 87-88, the paper states that "which ensures that the model won’t simply be ignoring the text images during training." However, according to the rebuttal, "a 1.0 masking ratio" is needed for the situation when the model is not supposed to see the raw image tokens, such as text-only in-context few-shot, or zero-shot evaluation. The word "training" in line 88 confuses me. Could you elaborate on this more, please?