Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

Summary

This paper introduces a vision backbone pre-training method named Latent Compression Learning (LCL) to utilize interleaved image-text data. The proposed LCL approach maximizes mutual information between the inputs and outputs of a GPT-like model in autoregressive manner. The proposed method integrate both discriminative and generative objectives by contrasting preceding context and generate subsequent text based on visual representation. The extensive experiments demonstrate that LCL not only matches the performance of existing models like CLIP on paired datasets (e.g., LAION) but also effectively leverages interleaved pre-training data (e.g., MMC4) to learn robust visual representations from scratch.

Strengths

1. The paper is well written and easy to follow. 2. The paper introduces a new pre-training method, Latent Compression Learning (LCL), which utilizes interleaved image-text data for visual backbone pre-training for the first time. And this can effectively leveraging large-scale web-crawled data, which is easier to crawl compared to the image-text pairs. 3. Extensive experiments are conducted, demonstrating the effectiveness of the proposed method on both paired datasets (e.g., LAION) and interleaved datasets (e.g., MMC4).

Weaknesses

1. From Table 5, it appears that solely leveraging image-text pairs with LCL does not provide benefits over the CLIP baseline. However, when using the MMC4 dataset, which is manually composed of interleaved text, there is significant performance improvement on downstream tasks. I am curious whether this performance gain results from the increased number of training samples (i.e., the total number of images used during training). 2. According to Table 3, utilizing original interleaved datasets such as Obelics does not yield any performance gain. In comparison, the MMC4 dataset requires more computation for data filtering with the CLIP score and the use of image-text pairs to create interleaved data. It is unclear how to efficiently utilize the original interleaved data directly crawled from the web. Do you have any insights on the differences between these two types of interleaved datasets?

Questions

1. The scaling behavior is not demonstrated. While the author has shown the effectiveness of training on MMC4 and Laion-400M, it remains unclear how the model performs and correlates across different dataset scales. Understanding this could provide valuable insights into the feasibility and performance of scaling the proposed method to larger datasets, such as DataComp 12.8B and Laion 2B. 2. Can you show the seen samples of each model? It would be helpful for readers understanding the scale of model's training.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitation in their manuscript.

Summary

The paper tackles the problem of vision model pre-training. More exactly, it aims to exploit the interleaved image-text data that is very prevalent on the Internet. It proposes Latent Compression Learning that maximises the mutual information between the inputs and outputs of a causal attention model. When visual pre-training is applied to interleaved image-text data, visual latents are extracted using a visual encoding network and then combined with the text and fed to the causal model.

Strengths

The paper tackles an important task and proposes an interesting method that may be of interest to the research community.

Weaknesses

While the method seems interesting, my main concern is related to the experimental part that I find confusing. For example for BEiT3 the numbers reported are different from the ones reported in the paper. Also, I think that for Tab 6, more multi-modal LLMs need to be included. While there can be a debate on fair vs unfair comparison, I think that you present results on a dataset these need to be complete. So, they can be greyed out, put in a different section, etc and explained why the comparison is not fair, but I don't think it's suitable for models that perform better to not be included at all. So, missing comparisons: Fang, Yuxin, et al. "Eva: Exploring the limits of masked visual representation learning at scale." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. Zou, Xueyan, et al. "Generalized decoding for pixel, image, and language." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. or even some very recent ones for the sake of completeness: Sun, Quan, et al. "Generative multimodal models are in-context learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. Liu, Haotian, et al. "Improved baselines with visual instruction tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Questions

Why is OpenAI CLIP greyed out? Why are the numbers reported for BEiT3 different than what they report in the paper? For example for Flicker 3K, R@1, it's reported 73.2 while the paper reports 88.2 Why, for example the comparison with BEIT3 is only shown in Tab.1 when they report results on VQAv2? The same question for CoCA?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are barely discussed at the end of the conclusions. Some limitations can be inferred from the rest of the paper.

Summary

The paper pre-trains models with a combination of a contrastive image-text objective and a generative language objective. The authors provide many results on image classification and vision-language tasks suggesting the competitiveness of the method in controlled settings.

Strengths

S1. The paper is well framed and motivates nicely the need to pre-training on interleaved data. S2. The paper gives good intuition about what the various equations mean, making the manuscript more accessible. S3. Consideration of many pre-training datasets including LAION-400M, MMC4, and OBELICS. S4. Extensive objective ablations spanning both contrastive and generative losses.

Weaknesses

W1. [MAJOR] The paper presents the objective as novel (L44-54); however, it seems similar to CoCa (Yu et al., 2022.), which also employs a contrastive loss and a next token prediction loss. Can the authors clarify the differences and why the formulation is novel? W2. It seems equation 3 appears in prior work; however, when it is first presented it seems to be presented as a novel insight. I recommend making the attribution to prior work more clear before introducing the equation. W3. In the relation to previous pre-training tasks, it is important to also relate to CoCa. It seems the objective is pretty much the same suggesting that the objective is not actually a contribution of the work. Is there any reason CoCa is not mentioned here given the similarities? W4. Make sure it is clear that you train on sequences with more than one image per sample (I am assuming this is true because you train on MMC4, but when explaining the objectives you include only one sequence for simplicity). 3.3 is a nice place to add this information. Also any special tricks to get multi-image to work? If so, it could also be nice to mention this. W5. Why are the numbers for Flamingo in Tab 1 for IN-1k so low? Flamingo uses a pre-trained vision backbone, so I expect numbers to be good here. W6. Is the COCO CIDEr evaluation protocol zero-shot? If so the number in table 4(a) of 87.5 looks extremely high relative to open flamingo and Idefics. Please double check this number and if few-shot prompting is used here, please make this clear. Also why is Gen. only worse than Con. only for captioning. How is contrastive learner able to do captioning? W7. In the frozen transfer setting in Tab. 6 are all models fine-tuned on the same data? If so, what data? The specifics of the experiment are not clear to me, making it hard to interpret the results.

Questions

Please see the weaknesses section for specific questions and topics to address.

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors address limitations in a dedicated paragraph.

Summary

This paper aims to explore the use of weak supervision signals in multimodal interleaved image-text data to pretrain visual encoder, compressing the distribution of high-level features into the visual encoder. The paper employs contrastive loss and autoregressive loss to train the model. To prevent the collapse of visual representations, an entropy maximization constraint is applied. The paper derives the equivalence of maximizing the mutual information between the model's input and output as a latent compression and entropy constraint. The proposed pre-training method, called LCL, achieves performance comparable to CLIP on paired data while better utilizing the supervision information in interleaved data.

Strengths

This paper explores how to use weak supervision signals in more general interleaved image-text data to accomplish visual pre-training. Its advantages are as follows: 1. Unlike previous approaches that fine-tune pre-trained visual models to align visual representations with the text space (Flamingo, LLaVA), this paper explores how to train visual models from scratch using interleaved image-text data. This is a meaningful exploration. 2. To prevent the collapse of visual representations, where autoregressive generation relies solely on textual information, this paper imposes an entropy constraint and further derives it as optimizing mutual information. This approach aids in model training. 3. Extensive quantitative experiments have validated the effectiveness of the visual models trained using this approach.

Weaknesses

This paper has the following areas for improvement: 1. In some cases, the textual context may have little relevance to the image. It is worth investigating whether such data could harm the model's performance. 2. The paper lacks qualitative experiments to further demonstrate the effectiveness of the method. Designing reasonable visualization analyses would help to further elucidate the advantages of the approach. 3. Similar to CLIP, further demonstrating the model's transfer learning performance through domain adaptation tests and few-shot metrics would be beneficial.

Questions

see weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper's discussion on data bias and energy consumption limitations is commendable; however, it could further explore issues related to data privacy.

Rebuttal answer

Hi. Thank you for writing the rebuttal! I confirm I have read the rebuttal. I understand that the goal is to pre-train a vision encoder from scratch and the rebuttal sort of adds more clarity, but I think more details should be added in the paper to make things clearer. The authors have promised to add more details to the captions, so I don't have much to add here. However, when reporting results on a well defined benchmark, I personally believe all types of methods should be included, especially if they have a better performance. I understand that the space is limited and I do not propose to have a list of extensive non-related paper, but I consider including at least 1,2 especially can be very important for readers of the paper in order to make them aware of different methods and possibly increased performance on the benchmark. As I said in the original review, what is fair or not can be shortly explained and the methods clearly separated in the table.

General Response

We respond to the common concerns of the reviewers here. First, we claim that our proposed method is the first to use interleaved image-text data to pre-train vision encoder from scratch. Existing vision pre-training methods do not support interleaved data. Simple ways to use interleaved data, e.g., autoregressive text generation, do not perform well for vision pre-training due to the vision feature collapse. We address this by our novel training objective that leverages both contrastive and generation loss. ___ **Q1:** Comparison with CoCa's objective. **A1:** The major difference between our approach and CoCa's objective is that CoCa does not support interleaved data. For a detailed discussion, please refer to the response to Reviewer A7ig. In the revision, we will further elaborate on the differences between our method and CoCa, both in the figures and the textual descriptions. ___ **Q2:** Comparison with state-of-the-art methods. **A2:** We would like to emphasize that we propose a vision pre-training method with image-text data. Accordingly, it is appropriate to benchmark it against SoTA vision pre-training methods that use image-text data. We suppose that achieving SoTA performance at the system level of MLLM is not a requisite for a vision pre-training method. Below are two tables revised from the originally tables in our paper, which we will use to further explain our comparison with SoTA performance. 1. Existing vision pre-training methods only support paired data, and we compare with their SoTA performance on paired data. Due to the unavailability of some data, checkpoints, and source code, we have had to either reproduce existing methods or rely on available open-source implementations for a fair comparison (marked by * in Tab R1). The results in Tab. R1 show that our method is comparable to existing methods on paired data. 2. To show that using interleaved image-text data for vision pre-training is not trivial, we also consider existing ways to use interleaved data. Those methods are proposed as MLLM training methods, but we test them for vision pre-training. Tab R1 shows those methods do not work well for vision pre-training. 3. To further confirm our proposed method, we need to compare our pre-trained model with existing checkpoints. Tab. R1 shows that our pre-trained ViT-L/14 is on par or better than the OpenCLIP's checkpoint in a fair comparison. Other checkpoints listed in Tab. R1 could be the reference. They either use a larger vision encoder, more training data, or private data. 4. For multimodal benchmarks, as explained in the response to reviewer KZWZ, our goal is to fairly evaluate pre-trained vision encoders, not to compare different MLLMs at the system level. It is reasonable for us to adopt the open-source LLaVA-1.5. We also list some advanced MLLMs for reference in Tab. R2. Those MLLMs may outperform ours due to their use of stronger vision encoders, larger LLMs, or private data. We anticipate constructing a SoTA MLLM system in future work, but in the current stage, we focus on vision model pre-training due to resource constraints.