POINTS: Improving Your Vision-language Model with Affordable Strategies

Summary

The paper addresses the problem of model design choices and dataset selection in vision-language models learning. The authors propose a combination of existing methods to build a strong and sample-efficient baseline using a relatively small pretraining dataset of size ~1M. The main contribution is on the analysis of the dataset (individual select) and model selection (model soup) using and combining existing methods. The authors also propose a novel image splitting strategy to tackle the well known image distortion problem in models that use a dynamic high resolution image processing.

Strengths

* Well-written and easy to follow. * Comprehensive set of VL benchmarks including hallucination and OCR benchmarks * Table 2 shows the benefit of a perplexity-based selection of the base pretraining dataset, which is an interesting result in terms of compute efficiency. * The authors manage to get competing performance in a wide set of VL benchmarks with a much smaller pretraining dataset (~1M) than other methods. * All the datasets and pretrained backbones used by the authors are publicly available which make it easily reproducible.

Weaknesses

* Ablation study is thorough but the comparison with other baselines is not discussed. Specifically, POINTS is outperformed in many of the benchmarks the authors chose by similarly sized open-source models, but the differences are not discussed. * The authors also did not discuss the choice of the base LLM (l380 Yi-1.5-9B-Chat) compared to other baselines, which makes it difficult to disentangle the reasons for the difference in performance. * The authors introduce a QA dataset distilled form GPT-4O and include it in their base set. It is possible that this knowledge distillation alone explains why adding more data is not efficient and why a ~1M dataset is enough. The authors only justify this addition by the need to have prompt diversity but do not ablate it. **Minor Issues** * Small typo l480: refer to table 4 instead of table 5 * Some tables are missing the hyperlink ref (l460 and l480 for instance) * Typo l482 “increasing adding more” * The authors could rephrase l058 ” Unlike most existing works (Li et al., 2024; Chen et al., 2024c), we extensively ablate each newly introduced component in the strong baseline to verify their individual benefits” and not point out the lack of ablations in other works, especially when those cited work actually reference a whole blogpost about ablations in LLaVA-Next (see [1]) . [1] https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/

Questions

* Could the authors discuss in more detail the choice and comparisons of baselines ? I acknowledge that it is easy to get lost in the number of new VLMs coming out these days but since the authors propose an analysis of the training + modeling frameworks, i would expect a comparison table that could explain the choice (and exclusion) of VLMs instances based on a few criteria (datasets, size, LLM base, ViT base, presence of OCR data, dynamic high resolution processing etc..) * In the ablation study Table 1, what does it mean to not have IS? Does it mean training on all the potential candidates additional datasets? * Could the authors discuss and analyze the choice of the base LLM (Yi-1.5-9B-Chat l388) ? Specifically, why not choose a base LLM that is the same as the closest baseline (for instance InternVL 1.5 that also has a dynamic high resolution processing)? It would make it easier to disentangle the contributions that come from the dataset selection, the model soup and the base LLM compared to other baselines. Overall, the authors presented their method in a very clear and structured way. The authors also perform a thorough ablation study. However some key choices (such as GPT-4O distilled dataset, a different base LLM compared to the other baselines) are not discussed or ablated which makes it difficult to disentangle the benefit of the overall approach. The lack of discussion around the baselines and their performance makes it challenging to evaluate the novelty and impact of the proposed contribution.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Summary

The paper proposes to improve the baseline LLaVA model in terms of the following three aspects: (a) to refine the pre-training dataset quality, the paper introduces CapFusion utilizing an LLM to integrate world knowledge from existing captions while simplifying their structures, (b) the model architecture is improved through the introduction of dynamic high resolution component with consistent aspect ratio, and (c) the visual instruction tuning is enhanced through the usage of model soup that merges weights from models fine-tuned on different learning rates.

Strengths

- The proposed components are indeed effective and target the limitation of existing vision-language models in great detail. - Compared to finetuning dataset selection, the proposed model soup is a more sustainable design choice in terms of the return on performance with scale. - Perplexity-based dataset selection technique is simple and could have a potentially broader impact on data-efficient pretraining of large foundation models.

Weaknesses

- While the paper is overall well-written, it is hard to spot the motivation behind the introduced components now and then. For instance, the introduction states "... However, recent advancements have rendered its (LLaVa's) performance suboptimal. Thus, there is a need to establish a stronger baseline for further exploration..." Without proper references or an explicit description of how LLaVa's performance has gone suboptimal, the current motivation reads as "We know the model x and we want to improve upon it." While the latter motivation is indeed genuine, this however opens up a plethora of search space where the model x can be improved. As such, I am reluctant about why the paper would then improve model x in only a few chosen directions. - Continuing on the above note, a good part of the experiments are occupied with performance on OCR-related tasks. However, there is no such concrete discussion of why the paper chooses to highlight the OCR task, what limitations the current baselines have for the OCR task, and what exactly are the example scenarios where the proposed model performs better on OCR than the baseline LLAVA. - My major concern with the current experiments is a lack of discussion on the additional resource overhead brought about by the proposed components. For a practitioner to prefer the proposed POINTS over LLAVA, it would be helpful if the paper mentions: (a) the overhead in pretraining/finetuning in terms of computational resources such as GPU memory, training/inference times in terms of wall-clock/FLOPs as well as the data efficiency of the proposed method. On the lattermost note, I would like to see a discussion on how the POINT model performs compared to LLAVA if a downstream fine-tuning task has limited (few-shot) training examples. - How does the proposed CATTY method scale with the image size in terms of the model inference time? Minor suggestions: - Introduce the full name for OCR before the first usage of the acronym in the paper (line 057). - Lines 75-78: citation required for ".. Compared to conducting model soup on models fine-tuned with different hyper-parameters, e.g. learning rate, the improvement with model soup on models fine-tuned with different datasets is much more prominent."

Questions

See Weaknesses. Most of my current concerns with the paper are around a lack of proper motivation and analyses of the proposed components.

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Summary

The paper proposes affordable and efficient methods to improve vision-language models, focusing on three main parts: (1) building a robust baseline model with thorough validation, (2) using perplexity filtering to improve the quality of pre-training data, and (3) applying "model soup" to merge weights from different fine-tuned models. The final model, POINTS, achieves competitive performance with state-of-the-art models while remaining lightweight and accessible. The authors also introduce improvements like CATTY to reduce image distortion and enhance OCR capabilities, all while keeping computational costs manageable.

Strengths

- The use of perplexity-based filtering of pre-training data is inspired by its success in large language models. This approach is a creative application of an existing concept in a new context, addressing data quality for vision-language models. - The paper is well-structured, with clear divisions between each method's presentation and the experimental results validating those methods. - The paper's focus on affordable strategies is practical for the broader community, especially for those without extensive computational resources. Achieving competitive performance using lightweight techniques is useful.

Weaknesses

- Each of the proposed methods, such as perplexity filtering, CATTY, and model soup, are either existing methods or slight modifications of existing approaches. This limits the overall novelty of the contributions. More significant deviations or novel techniques could make the paper's contributions more impactful. - While the authors claimed model soup has introduced improvements, the paper does not thoroughly explore its potential limitations or generalizability across different tasks or datasets. There are only limited details on the possible negative effects of merging model weights, particularly if the datasets have significant divergence in distribution. Also, I wonder if there can be potential overfitting to benchmark from the Greedy Soup method and why "MathV360K" has no improvement while others have improved. - While perplexity filtering effectively reduces the dataset size while preserving model performance, there is a potential risk that the model's ability to generalize could regress due to a lack of exposure to a diverse range of data. The paper does not provide any experiments or discussion addressing this potential drawback. I think it is important to understand the gap in the understanding of the impact of data reduction on generalization performance.

Questions

- While perplexity filtering effectively reduces the dataset size while preserving model performance, there is a potential risk that the model's generalization ability could regress due to reduced exposure to diverse data. Could you provide more insights or experimental evidence on the impact of this data reduction on the model's generalization capabilities? - Regarding the Model Soup technique, is there a potential risk of overfitting to benchmarks due to the greedy soup method? Could you elaborate on why "MathV360K" showed no improvement while other datasets did? - Have you observed any cases where CATTY underperforms compared to DHR? It would be useful to understand any trade-offs involved, particularly in terms of applicability to different image types. - Typo in the table 4 "Greey Soup" -> "Greedy Soup"

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Ethics concerns

There are no ethical concerns in the paper.

Summary

This paper proposes a robust baseline vision-language model. The main contributions are threefold: (1) Framework improvement: the authors introduce Consistent Aspect Ratio Dynamic High Resolution for the vision encoder of the multimodal LLM; (2) Data filtering: the authors use perplexity to filter the pre-training dataset; (3) Model ensemble: this work employs model soup methods to merge models fine-tuned with different datasets to achieve better results.

Strengths

- Paper is clearly written. - Introduced approaches are technically sound.

Weaknesses

- Novelty concern: The proposed strong baseline model primarily integrates multiple existing advancements in vision-language models, with limited novel technical contributions or findings. Specifically, the proposed CATTY encoding combines dynamic high-resolution encoding and sliding window techniques, both of which have been explored in previous works [1-3]. Additionally, model soup or model ensemble techniques have been shown to effectively boost performance. The novelty of the introduced maximum, average, and greedy soup methods for merging model results is limited. - Incomplete experimental analysis: Experimental results comparing CATTY with DHR are shown in Table 1. However, the proposed CATTY encoding approach, which leverages a sliding window with overlapping, may also introduce extra computation costs, additional vision tokens, or repetitive information. This analysis is crucial for understanding the pros and cons of using CATTY for future work, yet these aspects are not examined. - Confusing table and figure references: In the experiments section, some table and figure references are confusing. For instance, in Line 480, should this refer to Table 3? And in Line 484, should this be Table 4? This paper may require more attention to detail in the writing. [1] Chen, Zhe, et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites." arXiv preprint arXiv:2404.16821 (2024). [2] Lin, Ziyi, et al. "Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models." arXiv preprint arXiv:2311.07575 (2023). [3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

Questions

- Is the baseline in Table 4 trained only on a base instruction tuning dataset, or is it trained on a combination of a base instruction tuning dataset and a series of visual instruction tuning datasets? How was the baseline model trained? - What specific off-the-shelf vision-language model was used to calculate perplexity in Table 2? In the experiments, did you observe any differences when using different VLMs?

Rating

3

Confidence

4

Soundness

3

Presentation

3

Contribution

2

General Response

We greatly appreciate the valuable suggestions from the reviewers. Here, we first address the common issues raised by the reviewers, and then we address each reviewer's individual concerns one by one. ### Novelty of this paper Regarding the novelty of our paper, we would like to further elaborate on our perspective: 1. As we mentioned at the beginning of the article, our primary goal is not to propose new methods, but to refine the existing model training procedures to build a multimodal model that can rival industry-leading models in three key aspects: i) constructing a stronger baseline for subsequent experiments, ii) processing pre-training data, and iii) enhancing the model's performance during the instruction fine-tuning phase by not only continuously introducing more datasets but also utilizing model soup. 2. It is well known that most of the current industry-leading models, such as Qwen2-VL and InternVL2, rarely explain the details of their model training to the community. This practice makes it difficult for others to train a model from scratch that can rival the state-of-the-art (SOTA), which is detrimental to the progress of the field. The most significant contribution of POINTS is that it provides a detailed explanation of every step in the model training process, from data collection and processing to model architecture and training configuration. We hope that this approach can offer valuable references to our peers in the community. 3. In the era of large models, one should not overly rely on the innovation of methods themselves to judge the contribution of a work. Instead, a more comprehensive and detailed examination of every step in the model construction process should be conducted to unleash the model's potential performance. Although POINTS utilizes many existing methods, compared to previous approaches such as LLaVA-Next, we have meticulously fine-tuned every step of the training process. ### Typos in this manuscrip ased on the reviewer's feedback, we have corrected the typos and updated the manuscript. Additionally, for the reviewer's convenience, we have included the appendix at the end of the main text (previously, the appendix was placed in a zip file).