Fine-Grained Visual Prompting

Summary

This paper proposes Fine-Grained Visual Prompting (FGVP) that incorporates Blur Reverse Mask to improve the semantic localization capability of VLMs, like CLIP. It provides a comparison to other possible methods for highlighting the different parts/objects in the image, based on SAM and other techniques. The resulting method improves the performance on RefCOCO* datasets.

Strengths

The paper is written in a clear way and describes the algorithm well. The evaluation shows clear improvements and the ablation study is convincing. The idea to blur and mask the background of different object/part proposals is both innovative and significant.

Weaknesses

The related work section missing two relevant works in the visual prompting domain: 1. Bhang et al., "Exploring Visual Prompts for Adapting Large-Scale Models", 2022 2. Bar et al., "Visual Prompting via Image Inpainting", NeurIPS, 2022 Moreover, while SAM is a powerful method for segmentation, it requires running the model with a relatively dense grid of keypoints. This runtime can be significant and should be discussed in the limitations section.

Questions

Can SAM be replaced with a simpler segmentation model, like color-based unsupervised segmentation methods? Any ablation of SAM will be useful here for understanding the scale/quality that is required from the segmentation network for achieving good results.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation section is present and addresses some unexplored directions. I suggest adding runtime estimates.

Summary

This paper proposes a new “visual prompting” method. Visual prompting refers to the idea of altering images to guide the “attention” of a vision-language model when the model is used to embed the image. For example, to obtain an embedding for an object in an image that contains many objects, the user could draw a red circle around the object of interest and then embed the image. Since the vision-language model might have seen images during training in which important objects are highlighted with red circles, the resulting embedding might focus on the circled object. This approach can be used to partially solve tasks such as referring expression comprehension: Given a set of bounding boxes and text descriptions, visual prompting can be used to find the best-fitting description for each box. A separate object detector is needed to obtain the bounding boxes first. The paper proposes a visual prompting method that consists of blurring everything in the image except for the object of interest. This is done by first using a pretrained segmentation model (Segment Anything) to obtain a mask for the object of interest, and then blur everything outside of the mask. The motivation for this approach is that it simulates the shallow depth of field seen in photographs taken with a large aperture, which are commonly found in VLM training data. The blur method is compared to the “red circle” method and several variants in object detection and referring expression tasks. The blur method consistently performs best. In addition, hyperparameter sweeps for the blur radius and other hyperparameters are shown.

Strengths

1. The proposed visual prompting method consistently improves over other methods. 2. The blurring approach is well motivated by the abundance of photographs with shallow depth of field. This is a clever way to exploit “natural supervision” present in large-scale web image training data.

Weaknesses

1. The proposed method is significantly more complex than the “red circle” method, since it relies on a large segmentation model. The inference cost of the proposed method is therefore much higher than the “red circle” method. This should be acknowledged in the discussion and/or limitation sections. 2. While the proposed method works well, it is an incremental improvement over the “red circle” idea (https://arxiv.org/pdf/2304.06712.pdf) and the evaluation is not as comprehensive as in the “red circle” paper. For example, no analysis of the relative biases of red circle vs blur methods are performed, and no failure cases are discussed. It is not clear if the contribution is substantial enough for NeurIPS.

Questions

Please use descriptive method names (keypoint/box/circle/…) instead of just A/B/C in Table2 and elsewhere, so that the reader doesn't have to refer to Figure 1 all the time.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The runtime cost of the proposed method needs to discussed further. The biases of the method compared to the "red circle" method need to be evaluated and discussed. A generic "broader impact" statement is given in the appendix.

Summary

This paper works on visual prompting. They proposed FGVP, together with Blur Reverse Mask, to improve the semantic localization ability of the vision-language model.

Strengths

Solid experiments showing the effectiveness of their method.

Weaknesses

In general, this is a good work. However, there’re several things concerning me: - The novelty of this work is not well established. Seems like an engineering combination of previous works. - The discussion of the upper bound is based on the assumption, while the legitimacy of the assumption is not well discussed.

Questions

No further questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No notable limitations.

Summary

This paper proposes a visual prompting method that exploits the segmentation masks of interested objects in images to generate more fine-grained visual prompts. Experiments show that the proposed methods achieve competitive results on zero-shot referring expressions comprehension and part detection.

Strengths

1. This paper is well-organized. The motivation and the framework is clearly presented. 2. The experiments confirm the effectiveness of the visual prompt design.

Weaknesses

1. The ablation studies of different VLMs are missing. Since visual prompting is a zero-shot framework, it is natural that the performance of the visual prompts differ on VLMs trained on different data. The paper adopts the CLIP as the VLM in all experiments. How do different visual prompts perform on other VLMs? 2. Figures 2 & 3 are very similar and thus redundant. The authors should consider merging them into one figure.

Questions

1. Basically, I like the section 3.3 of the discussion of the prior knowledge introduced by the large amount of photography images in the large-scale image-text dataset. Could the authors provide more discussions on the analysis of the training data of the VLMs, and the effect of better alignments between the visual prompt design and the VLM training data? 2. How good are the zero-shot results compared with the few-shot / many-shot methods? The few-shot / many-shot methods may be provided for general reference.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The proposed visual prompts can be further verified on more object-based tasks.

Summary

The paper proposes Fine-Grained Visual Prompting (FGVP), which uses precise semantic masks from SAM as visual prompts to improve spatial localization of vision-language models (VLMs) like CLIP for instance-level tasks. The key contributions are: - Systematically study different visual prompting techniques like cropping, boxes, circles, masks, etc. Show that blurring background outside target mask (Blur Reverse Mask) works best. - Achieve state-of-the-art results on referring expression comprehension benchmarks RefCOCO/RefCOCO+/RefCOCOg, outperforming prior works. - Demonstrate FGVP can enable zero-shot part detection on PACO dataset without needing any box proposals, again outperforming other prompting methods.

Strengths

Originality: The paper presents a novel idea that using SAM to generate semantic masks from detected bbox as better visual prompts of specific instance in the image, which is an original idea not explored in prior works. Quality: The overall approach is technically sound. The experiments follow standard protocols and are extensive in studying different visual prompting design. The results demonstrate benefits over existing methods. Clarity: The paper is well-written and easy to follow. The problem context, proposed method, experiments are clearly explained. Figures and tables aid understanding. Significance: FGVP pushes state-of-the-art in two important vision-language tasks - referring expression and part detection. The analysis may inspires more research on the properties of VLMs regarding spatial understanding.

Weaknesses

While the paper presents a novel fine-grained visual prompting technique and achieves state-of-the-art results, there are some aspects where the analysis could be strengthened: More Insights from Study: The paper performs an extensive set of experiments on different visual prompt designs. However, it could provide a more detailed analysis of the inferences and insights derived from this study. For instance, comparing the gap between using ground truth and predicted bboxs would give insights into the impact of mask quality. Explaining the differences in various design choices (VP, PP, proposals etc.) in Table 3 would be informative. Attention visualizations could help reveal why fine-grained prompting is more beneficial. Computational Overhead: The paper proposes generating semantic masks using SAM models. However, the computational overhead this introduces is not analyzed. Reporting inference times, scalability, etc. would provide a better understanding of its practical viability. Joint Language and Visual Prompting: The study is currently limited to exploring only visual prompts. Evaluating prompts on both visual and textual modalities could offer a more comprehensive understanding of VLMs. It is good that authors discussed this point in the limitation though. Granular and Failure Analysis: Providing per-class performance breakdowns and detailed failure analysis compared to other methods through examples would provide useful insights into where the improvements come from. In summary, while the core ideas are promising, performing a more thorough empirical analysis along the above dimensions would strengthen the paper and provide a better understanding of the factors behind the efficacy of fine-grained visual prompting.

Questions

See the main suggestions in weakness. - For the RegCOCO/RefCOCO+/RefCOCOg results in Table 4, which subsets are the numbers reported on? - Line 288-290 if there are better grid size / NMS thresholds, how do we still use the suboptimal default ones?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are discussed in the paper

Thanks for the rebuttal

Thank the authors for their response. Since the rebuttal addresses most of my concerns, I raise my score to "accept".

Reviewer comment after rebuttal

I really appreciate the author's rebuttal. Most of my concerns are addressed. Therefore, I raise my score by one. However, I do think the paper could go a bit deeper when it comes to the insights from the empirical studies on which prompting works best. Take for example the "middle zebra" and "second guy from right white shirt" mentioned in the rebuttal attachment. These seem to hinge on understanding spatial relationships. Perhaps they'd benefit more from a global context than just crop-based prompting. But with blurring, it seems we're still cutting down on that global context info, making things trickier. Also, I'm curious about the per-class performance analysis - why does FGVP fall short compared to RedCircle for things like the baseball glove and traffic light? Digging deeper into these questions and sharing insights would really help in understanding FGVP better. It would be great to get a clearer picture of what makes one visual prompting strategy stand out over another.