Enhancing Large Vision Language Models with Self-Training on Image Comprehension

Summary

This paper addresses the problem of acquiring high-quality fine-tuning data for large vision language models (LVLMs) with minimum human effort. The paper presents STIC (Self-Training on Image Comprehension). STIC contains two stages: image comprehension self-training and description-infused fine-tuning. LVLM first generates descriptions based on images and corrupted images and regards each sample as preferred and dis-preferred responses, respectively. The base LVLM is trained on the generated preference data leveraging the directed preference optimization (DPO) framework. Next, the LVLM is fine-tuned on instruction following data. Experimental results show that STIC outperforms the compared methods on diverse benchmarks.

Strengths

- S1: Overall, the manuscript is well-written and easy to read. Preliminaries and figures help readers understand the paper. - S2: The paper presents diverse analysis and discussion in experiments.

Weaknesses

- W1: The manuscript has limited soundness for several reasons. For example, the authors did not validate the effectiveness of description-infused fine-tuning. In Table 2, much of the performance gain comes from the prompting method (DaR) which is outside STIC. - W2: While there is a rich literature on existing self-training algorithms [1,2,3,4,5], the paper only discusses recent self-improvement systems, especially in the context of LLMs. This limited investigation leads readers to question what the technical contribution of STIC is compared with the existing line of semi-supervised learning (or self-training) research. - W3: The paper experimented with a fairly small amount of unlabeled images (6k and 12k images). What and how much data should be used to maximize the performance has not been thoroughly investigated. **References** [1] Self-training with noisy student improves imagenet classification. Xie et al., CVPR 2020. [2] Rethinking pre-training and self-training. Zoph et al., NeurIPS 2020. [3] Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Sohn et al., NeurIPS 2020. [4] Revisiting self-training for neural sequence generation. He et al., ICLR 2020. [5] The dialog must go on: improving visual dialog via generative self-training. Kang et al., CVPR 2023.

Questions

- Q1: In Algorithm 2, why aren’t the model-generated description $\mathbf{y}_{\mathrm{des}}$ used for fine-tuning? - Q2: What is the motivation of the stage 2 (Description-infused fine-tuning)? It is not directly related to the motivation of the paper (difficulty of obtaining high-quality fine-tuning data). - Q3: How can we validate the effectiveness of description-infused fine-tuning? Did the authors check the performance of the method which did not use the model description? - Q4: Table 1 just shows the performance with and without STIC. How much does each stage contribute to performance improvement? - Q5: Table 2 shows that the describe-and-respond (DaR) prompting method improves overall performance. DaR was not mentioned until the section for experiments. Why did not the authors describe DaR in detail in the method section? - Q6: Is there any reason why STIC randomly selects unlabeled images? Some self-training algorithms present the methods for selecting unlabeled data to learn. - Q7: How can we guarantee that all samples generated by the LVLM inputting clean images are preferred responses? **Typos** Figure 1: SPIC to STIC

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations were adequately addressed.

Summary

Summary: The paper presents STIC, a method to enhance LVLM by reducing the need for labeled data. STIC generates image descriptions using unlabeled images and improves reasoning by reusing existing instruction-tuning data. It demonstrates performance gain across seven benchmarks, showing potential to effectively leverage vast quantities of unlabeled images for self-training.

Strengths

Strength: 1. The proposed method improves VLLM's performances with efficient cost during data collection. 2. The performance improvement is consistent on diverse benchmarks and achieves an average accuracy gain of 3.8%, which is quite significant.

Weaknesses

Weakness: 1. Interesting idea about using different prompts to generate both good and bad captions. However, I wonder if there is any method needed to ensure the correctness of the prompts generated by the "step-by-step" prompt strategies. Based on my own observations, VLLM is not good at following instructions, which means the description will be even worse if the prompts are complicated. Have the authors observed similar problems?

Questions

Have the authors changed the default latex template? There should be no anonymous submission ID. And it seems that the authors gain more space because they remove the anonymous authors part.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper proposes a two-stage method to enhance Large Vision Language Models (LVLMs) using unlabeled images. In the first stage, well-designed good and bad prompts are used to make the LVLM generate preferred and dis-preferred completions, respectively, conditioned on the unlabeled images (from COCO). Then, direct preference optimization (DPO) is used to fine-tune the LVLM using the generated preferred and dis-preferred completions. In the second stage, the fine-tuned model is used to generate descriptions for images in an instruction-tuning dataset (from LLaVA's data). The generated descriptions are inserted into the instruction-tuning data to fine-tune the model. The fine-tuned model shows significant improvement over the baseline (the LVLM before fine-tuning) on seven VLM benchmarks.

Strengths

- The biggest strength is the significant improvement over the baseline LVLM achieved by the proposed method. There is an average improvement of 4 points on the seven VLM benchmarks. - The proposed method mainly leverages unlabeled images for training, which gives the proposed method a great potential to use a vast amount of unlabeled images. The authors also show using more unlabeled images for training improves the performance.

Weaknesses

- The paper raises some questions unanswered. - The effect of the prompt set is less explored in the paper. It seems that the prompts play a crucial role in data construction. It is unclear how the authors designed the well-curated captioning prompt and the hallucination prompt set. Are there any principles behind the design? Especially for the well-curated captioning prompt that generates the preferred data, how do different design choices affect the final model performance in the evaluation? - Is the performance gain dependent on the MSCOCO data set? Do other image datasets (such as Flickr30k) work in stage 1? - It seems counterintuitive that fine-tuning an LVLM on MSCOCO will help improve its performance on science-related benchmarks like ScienceQA. It will help us better understand the mechanism by comparing the model generations in the benchmarks before and after using the proposed method. - The description of describe-and-respond (DaR) prompting is a bit unclear. I could not fully understand the setting.

Questions

- What is the difference between the image captioning prompt set $P$ and the well-curated captioning prompt (Algorithm 1)? - What is the prompt $x$ used in DPO in stage 1? - What is the model performance if only stage 1 is performed (i.e., without stage 2's SFT)?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

See Weaknesses.

Summary

This paper introduces Self-Training on Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, the model reuses a small portion of existing instruction-tuning data and appends its self-generated image descriptions to the prompts. Improvements in several benchmarks are reported.

Strengths

1. The challenges of self-training with VLM are discussed, which is appreciated. 2. The proposed STIC approach is claimed to be a novel two-stage self-training method that targets both image perception and reasoning over images and texts, which is intriguing. 3. STIC does not require pre-labeled information on the images, 4. The methodology of constructing dis-preferred data using bad prompting is pretty interesting.

Weaknesses

1. The experiments are conducted with 7B-level LLava 1.5 and 1.6. The method's scalability remains questionable.

Questions

What if we try the proposed method with smaller or bigger models or other ViT families (e.g., EVA-CLIP models)? With a higher representational capacity, will the model benefit more or less from self-training?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations.

Invitation for discussion

Dear reviewer qFQt, Thank you again for your strong support and constructive feedback. We hope that our responses have adequately addressed your questions. Specifically, we added the additional experiment on scaling up model sizes for STIC and provided discussion on further extending it to various models. We would like to inquire if you have further questions regarding our rebuttal. We are more than happy to discuss any remaining questions and provide additional details. Thank you once again for your time and efforts on this paper!

Invitation for discussion

Dear reviewer B7oX, Thank you again for your support and constructive comments. We hope that our responses and clarifications have adequately addressed your questions and concerns. Specifically, we provided detailed explanations toward the prompt design (Global A1) and DaR (Global A3). We further added an experiment on unlabeled images from a different distribution than MSCOCO, where we observed that a more diverse unlabeled image data can provide better improvement for STIC. Regarding W2 and Q3, we provided specific generation examples before and after STIC, as well as its stage-wise performance on ScienceQA. We would like to inquire if there are any questions about our rebuttal, for which we're happy to provide additional information and further clarifications. Thank you once again for your time and efforts on this paper!

Invitation for discussion

Dear reviewer s8oQ, Thank you again for your time and feedback to this paper. We hope that our clarifications have adequately addressed your questions. Specifically, we provided detailed explanations into the prompts of our method and further discussion in Global A1. We are following up to inquire if there are any remaining questions. We are more than happy to further discuss and provide clarifications. Thank you once again for your time and efforts!

Invitation for discussion

Dear reviewer uhYN, Thank you again for your detailed and valuable feedback to this paper. We hope that our responses and clarifications have adequately addressed your questions and concerns. Specifically, 1. We provided detailed clarifications on the results of Table 2 and explanations on DaR (Global A3). 2. We added a discussion paragraph on the similarities and differences with the mentioned related works, emphasizing our specific focus. These works will be incorporated into our revision. 3. We conducted additional experiments scaling up the data to 30k and explained the data efficiency of our method. 4. We provided clarifications and explanations for each of your specific questions. We hope these responses adequately address your concerns. If you have any further questions about our rebuttal, we'd be happy to provide additional information or clarification. Thank you once again for your time and efforts!