What Makes Good Examples for Visual In-Context Learning?

Summary

This paper studied in-context learning abilities of large vision models and finds downstream task performance to be highly sensitive to the choice of examples. They observe that the closer the in-context example is to the query, the better the performance. Since manually designing prompts would be time-intensive, they propose a (contrastive) supervised and an unsupervised version of a framework for prompt retrieval guided by a score function. They evaluate the proposed framework on the foreground segmentation, single object detection and image colorisation tasks.

Strengths

- In-context learning in vision is a new and active area of research and methods for selecting better examples for in-context learning will likely be of interest to the computer vision community. - The proposed method for prompt retrieval outperforms randomly selected examples on all tasks considered. - Distribution shift results are interesting, suggesting that the supervised prompt retrieval method acquires in-context knowledge that is robust to distribution shift. - The analysis and ablations are interesting and insightful.

Weaknesses

- The finding that examples in the prompt should be semantically close to the test example is known from language and not that surprising. - The proposed method is technically relatively straightforward compared to an average NeurIPS paper. - The analysis is limited to one model for in-context learning and may not transfer to other models.

Questions

1. It is surprising to me that the smallest gains are observed on the colorisation task and supervision is not helpful, despite the fact that both training and evaluation are done on ImageNet. Do you have any intuition for why this may be the case? 2. In Sec 3.3, you mention that 20% of the data is enough for good performance. Is this 20% of 50,000, i.e. 1000 images? 3. What is the intuition behind MAE in Table 5 performing worse than UnsupPR?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The paper identifies that visual prompting is sensitive to the choice of input-output example(s). To address this, the authors propose a retrieval framework to better select the examples. The authors propose supervised and unsupervised retrieval approaches which significantly improve the performance compared to random selection.

Strengths

- Through extensive empirical study, the authors demonstrate the role of input-output examples for visual prompting. - The authors present two different retrieval strategies to choose the best visual prompting example and find that both approaches are superior to random choice. - The authors show that choosing the right example can also improve performance under distribution shifts. - It is also nice that the retrieval similarity function is class-agnostic. But regardless, empirically it extends to three different tasks.

Weaknesses

Overall, the main two weaknesses in my opinion are: 1. The assumption that we have a set of tagged examples to retrieve from. 2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity. Minor: 3. I think there is a broader spectrum that can be analyzed for visual prompts. Please see the questions for extended comments/feedback.

Questions

1. The main weakness is the assumption that we have a set of tagged examples. In this case, why should we retrieve rather than use our tagged examples? e.g., train a new model overall tagged examples or use an ensemble? Perhaps the best way to address this would be to add another ensemble/supervised-trained baseline in Figure 5 (left). These baselines should utilize the same pool of examples. 2. The supervised/unsupervised similarity function is straightforward but currently very specific to mIOU similarity. Can we improve this by utilizing similarity in feature space? (e.g., using existing CLIP or DINO, etc?). To be fair, from the empirical results it seems like by optimizing mIOU there is somewhat improvement for colorization and single object detection as well. Minor: 3. I think there is a broader spectrum that can be analyzed for visual prompts. E.g., using synthetic task data as examples, using examples from different classes, then examples from within classes as analyzed in the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors clearly stated and discussed the limitations of this work.

Summary

This paper investigates in-context learning for large vision models. Specifically, the authors propose to automatically retrieve prompt for vision models in two methods: (1) nearest example search with off-the-shelf model, (2) supervised prompt retrieval method. Experimental results show that the proposed method bring non-trivial improvement, comparing to the random selection on foreground segmentation, single object detection and colorization.

Strengths

- the motivation of this paper is important: investigating how vision models could benefit from in-context learning. - the writing is clear and easy to follow. - The experiments show improvement comparing with random selection.

Weaknesses

1. Visual in-context learning emerged from large autoregressive language models. In-context learning was referred to use language models for a wide range of downstream tasks without updating the model itself. But this paper, though claims to study visual in-context learning, only studies a limited set of tasks (assign labels to pixels) with in-paining model. I wonder if the authors could provide thoughts on how to apply to a wider visual tasks, like classification, or standard segmentation tasks (semantic and instance). 2. The finding of "a good in-context example should be semantically similar to query and closer in context" seems to be a problem if the testing examples are unique, and there is no semantically similar {image, annotation} pair in the retrieval pool. This goes to the practicality of the method, and goes to the claim "potential of using prompt retrieval in vision applications".

Questions

please refer to the weakness section

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

This paper aims to address the problem of vision in-context-learning that the performance highly depends on the choice of visual in-context examples. In the paper, the authors propose automatically retrieving prompts in unsupervised and supervised ways without reaching internal weights of large vision models. Besides, this paper comprehensively studies how to select good examples for visual in-context learning and shares some valuable insights with the community on choosing good visual in-context examples.

Strengths

The paper discusses several aspects that influence visual in-context learning and propose methods to choose in-context learning samples automatically to optimize visual in-context learning with inpainting method. The proposed methods both outperform random selection baseline and the supervised version requires training performs better than the unsupervised method in several tasks. Also, this paper discusses other factors that might influence visual in-context-learning including the number of examples, order of examples, and size of retrieval set, which shares some useful practical experience to the community.

Weaknesses

1. This paper only discusses the previous paper of using inpainting as visual in-context learning as the visual icl frame and experiment on the dataset from that paper. It's hard to tell if experiments and conclusions conducted on a specific framework will generalize to more general "visual in-context learning" settings. Besides, although the supervised example retrieval method outperforms the unsupervised one, the additional model and training seem to be contradictory to the main advantages of ICL learning that requires no additional training.

Questions

1. The paper discusses the impact of the order and number of in-context examples. While more in-context examples generally lead to better performance, is there an upper limit to this? Will adding more examples become redundant or even detrimental at some points? 2. The paper compares the proposed methods with random selection. Are there other baselines that the authors considered for comparison?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Response to authors

Thank you for addressing my concerns, following the author's rebuttal I raise my score and I now support the acceptance of the paper.

Follow-up

Dear Reviewer, In our rebuttal: 1) Regarding concerns about the generalization of our method to broader "visual in-context learning" settings: - Our method has demonstrated easy generalization to visual-language tasks such as image captioning and VQA. 2) On the point of "additional training" potentially contradicting ICL advantages: - We claim that since the SupPR doesn't train the pre-trained model, it does not conflict with the benefits of ICL. Furthermore, we've incorporated experiments exploring: 1) The upper bound of in-context examples: - Our findings indicate an almost linear performance increase from 1-shot to 16-shots. However, the gains plateau from 16 to 32 shots. 2) Additional baseline comparisons: - We add a baseline about randomly selecting the in-context examples from the entire training dataset, which further strengthen the superiority of our proposed UnsupPR and SupPR. We would love your feedback on whether our answer has solved your concern or if you have further questions.

Follow-up

Dear Reviewer, In our rebuttal: 1) Regarding concerns about the limited set of tasks (assign labels to pixels) with an inpaining model: - Our method has demonstrated easy generalization to visual-language tasks such as image captioning and VQA. 2) On the point of "Seems to be a problem if the testing examples are unique": - We underscore that (1) Our study follows the in-context learning paradigm, where a given set of testing examples is assumed. (2) Even without perfect semantically similar pairs in the retrieval pool, our SupPR and UnsupPR methods consistently outperform random selection, underscoring our method's practicality. We would love your feedback on whether our answer has solved your concern or if you have further questions.

Follow-up

Dear Reviewer, In our rebuttal: 1) We highlight the insight found in our paper. We claim that most of our study presents findings that are significantly novel to the computer vision community. And we indicate that the Table.5 in our paper shows that, despite our method is straightforword, the superiority of our method becomes apparent only when we meticulously develop an appropriate learning pipeline. 2) We conduct more experiments showing that method has demonstrated easy generalization to visual-language tasks such as image captioning and VQA. 3) We clearly answer your question about the detail of our paper. We would love your feedback on whether our answer has solved your concern or if you have further questions.