VisMin: Visual Minimal-Change Understanding

Summary

The paper introduces VisMin, a new benchmark for assessing fine-grained understanding in VLMs. VisMin evaluates the ability of models to distinguish between minimally different images given a caption, testing recognition of changes in objects, attributes, counts, and spatial relationships. The benchmark is created using an automated pipeline and human verification. Empirical results show that current VLMs have notable deficiencies in spatial relationships and counting. Fine-tuning models like CLIP and Idefics2 with a large-scale dataset generated by this pipeline leads to improvements in fine-grained understanding and image-text alignment.

Strengths

1.This paper introduces VisMin, a benchmark that challenges models to detect subtle semantic differences between similar images, revealing significant shortcomings in VLMs and MLLMs. 2.This paper uses minimal-change image-text data to finetune VLM and MLLM models, improving the fine-grained understanding capabilities. 3.Table 2 illustrates that current open-source MLLMs perform poorly in fine-grained image understanding. This paper suggests using minimal changes in images to enhance fine-grained image understanding, and I believe this idea is promising.

Weaknesses

1.After fine-tuning on VisMin data, MLLMs showed improved performance on fine-grained understanding benchmarks. However, their performance on POPE and MMMU decreased, which contradicts the authors' claim of improved text-image alignment in MLLMs. The authors should provide results on additional benchmarks such as TextVQA, MathVista, and MMBench. 2.The authors attribute the poor performance to the binary-choice task and limited computational resources preventing training of an 8B model. This explanation seems insufficient; given the computational constraints, the authors could reduce the model size, such as using Qwen-1.8B, to validate the method's effectiveness. 3.For VLMs, the authors should also provide some zero-shot image classification results, such as on ImageNet. I am curious to know if fine-tuning on VisMin data affects zero-shot classification performance.

Questions

Please refer to Weaknesses. There are also some typo like L76: can -> can, L169: he -> the, Duplicate citation [39], [40], Table 3 and 4: Vismin -> VisMin.

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations.

Summary

This paper studies the fine-grained understanding of objects, attributes, and relationships between objects for VLMs and MLLMs. Specifically, it focuses on the capability of VLMs and MLLMs to distinguish between two very similar images given a caption. Firstly, by leveraging an image generation diffusion model and an LLM, this paper introduces a new benchmark termed Visual Minimal-Change Understanding, which requires models to predict the correct image-caption match given two images and two captions. Secondly, it performs evaluations on multiple VLMs and MLLMs and analyzes the performance in terms of different types of changes: object, attribute, count, and spatial relation. Thirdly, it fine-tunes existing VLMs and MLLMs on the generated datasets and shows improvement in fine-grained understanding.

Strengths

- The motivation is clear, and the problem addressed in this paper is well-explained. - The proposed dataset is interesting and benefits the research of fine-grained understanding of foundational VLMs. - This paper is well-written and easy to follow. - The evaluation protocol and the methods for fine-tuning are technically sound. - The analysis of VLM and MLLM performance on the proposed benchmark is interesting.

Weaknesses

- Some notations are confusing, and implementation details for the evaluation are missing: how are two captions chosen in text scores for VLMs? It appears from line 247 that captions chosen from a single image are paired instead of being randomly sampled. What's the difference between C_0 and C_1 in lines 247 and 248? The notation here is a bit confusing. Do the notations C in line 247 and T in line 294 both refer to the captions for the images? What's the difference? - Implementation details for fine-tuning are missing: How do the authors construct the VisMin Instruct-IT dataset? In line 344, some rule-based approaches are applied, but the reviewer cannot find details of these rule-based methods. It would be great to show some examples of how VisMin is converted to VisMin Instruct-IT. It also appears that the authors fine-tune the model using QALora and instruction fine-tuning techniques. How many instruction-image pairs are in this dataset? - Insufficient dataset analysis: The authors mention in this paper that after human filtering of the proposed dataset, there is still some noise, such as deformation and mismatches. Conducting a human evaluation on the proposed dataset to determine the percentage of these noise data would be great. This would provide a valuable reference for future works using this dataset.

Questions

- What is the motivation behind designing text scores and image scores for MLLMs? In Table 2, there is a significant drop in performance from text scores to image scores, especially for MLLMs. This drop might be because most MLLMs do not undergo instruction fine-tuning with two images, potentially affecting their ability to perform this task correctly. Under such circumstances, if the performance on image scores is low (for example, in spatial relation tasks), it becomes unclear whether the issue is the model's inability to process information from two images or its inability to detect minor changes in spatial relationships. - Visual minimal changes occur frequently in videos. Do you have any experience in this direction, such as how this dataset and the fine-tuned model could benefit video understanding? (If it's discussed in the paper, could you point it out? I may have missed it.) - Will the model and dataset be released? Also, VisMin Instruct-IT dataset. - (Minor comment) GPT-4V has different versions; it would be great to specify which version was used in the paper.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

- The authors discuss limitations and potential solutions in Section 7, such as the possibility of noisy data being included in the proposed dataset. The reviewer suggests conducting human evaluations on some samples from the dataset to provide statistics on the noisy data and to offer visualizations of these noisy data samples. - The social impact should be discussed in this paper. This paper propose fine-tuned VLMs and MLLMs which are able to detect visual minimal-change and can have potential social impact.

Summary

This paper introduces a new benchmark, VisMin, which mainly challenges models to detect semantic differences between visually similar but semantically different images. It uses an automated data curation pipeline and human verification to create dataset. The authors benchmark the dataset with current VLMs and MLLMs, showing that the proposed dataset can help improve image-text alignment and overall performance.

Strengths

1. The motivation of the paper makes sense and the paper writing is clear. 2. The curated dataset contains hard negative samples that will be beneficial to improve fine-grained understanding of current models. The dataset curation pipeline is well designed and the data quality looks quite good. 3. It indicates huge performance on CLIP and Idefics2 after finetuning with the proposed dataset.

Weaknesses

1. The paper only uses T, I, G to evaluate the performance of different models on the proposed dataset, which is insufficient to show the ability of understanding attribute, count and spatial relation. I think some basic metrics like accuracy are also needed. 2. The models being benchmarked are not sufficient. It lacks some latest or popular models, like Flava, CogVLM, etc. 3. Another concern is that the finetuned CLIP/Idefics2 models do not demonstrate improvements in all aspects, indicating that the benchmark may introduce some side effects. I am curious about the underlying reasons for this and recommend that the authors propose a new baseline method alongside the benchmark.

Questions

1. I'd like to see evaluation results of more latest models, with more evaluation metrics. 2. I am also interested in seeing the authors provide justification for the finetuning results on the CLIP/Idefics2 models, since they are not consistently strong.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors use editing models to modify original images. I suggest that the authors carefully review the curated dataset to ensure there are no ethical issues.

Follow-Up on Our Response and Request for Score Reconsideration

Dear Reviewers, We hope this message finds you well. We are writing to follow up on the explanations we provided in response to your valuable feedback. We genuinely appreciate the time and effort you’ve invested in reviewing our work. If our clarifications have adequately addressed your concerns, we would kindly ask you to reconsider your score. We sincerely hope that our work, which introduces the VisMin benchmark to challenge models in detecting fine-grained differences, develops an automated pipeline for high-quality data creation, and enhances fine-grained understanding in VLMs through fine-tuning, will be recognized as a valuable contribution to the community. As August 13th marks the end of the discussion period, we would be grateful if you could share any additional questions or concerns you may have before then, so we can address them in a timely manner. Thank you once again for your thoughtful review. Best regards, Authors

Response to rebuttal and further questions

Thank you for the further clarifications and the updated results/data. I appreciate the additional details and examples provided in the rebuttal PDF. The response has addressed most of my concerns. However, I would like to clarify a question I raised in my initial review. When I referred to an insufficient dataset analysis, I was specifically referring to the training set. The reason for this concern is that noisy training data used for fine-tuning MLLMs can lead to hallucinations during testing. Therefore, it would be beneficial for future work utilizing this training data to include a human evaluation on (a small subset of) the training set to determine the percentage of noisy data present. How noisy is the training set? Have the authors observed any hallucinations in testing? It would be greatly appreciated if the authors could provide a discussion or share their thoughts on this aspect.

Has the rebuttal addressed your concerns?

Dear Reviewer dNLV, Thank you again for your time to review this paper. Could you please check if the authors' rebuttal has addressed your concerns at your earliest convenience? The deadline of the discussion period will end in about 24 hours. Thank you! Best regards, AC