Towards In-context Scene Understanding

Summary

The authors propose a self-supervised approach to adapt models learned via SSL to in-context learning via retrieval. To obtain the best performance, the authors propose to add attention pooling to encourage features attention (even though within an image) and supervised training for retrieval. Through an extensive number of ablations and empirical studies, the authors show the features produced via Hummingbird outperform other SSL approaches on segmentation and depth estimation.

Strengths

1. The paper is well written and the presentation is very good. 2. The idea is to perform in-context via retrieval is very simple 3. The empirical results are effective and impressive compared to the baseline

Weaknesses

Major: 1. There are a few potential limitations with the current approach to in-context learning. Since the main idea of in-context learning is the adaption to new tasks, it is important to explicitly mention what can and what cannot be done with Hummingbird. 2. With Hummingbird (no retrieval loss) there is a discrepancy between train time to test time Minor: 1. Some missing RW. 2. L109 - slight inconsistency in notation going from HxWxD then using a single index for spatial location Rating 3. What is the ViT-B patch size used? can you add it to Table 1/4?

Questions

Major: 1. There are a few potential limitations with the current approach to in-context learning. Since the main idea of in-context learning is the adaption to new tasks, it is important to explicitly mention what can and what cannot be done with Hummingbird. * The title implies that scene understanding, however, the retrieved features might actually be very local and specific. The features are retrieved independently of each other and there is no mechanism to consolidate the retrieved features into a whole scene output. * Correspondence - with the current approach, the authors assume there are spatial correspondences between features and their associated labels. This is the case for segmentation or depth estimation but this is not necessarily the case for every possible task. E.g, if the task is to translate, crop and resize, rotate, or flip the image, etc, this assumption breaks. * Retrieval approach might not result in smooth outputs. This is ok for segmentation but how would this work for style transfer? I hope that the authors can explicitly clarify these in the final manuscript. On the one hand, the author's main claim is on segmentation and depth estimation, but on the other hand, in-context learning is very broad and therefore I think it should be stated explicitly what can be achieved and what not under this framework. I see this as the most important point, and I'll be willing to increase my score if this is addressed. 2. Discrepancy between train time to test time (attending within the image to attention across examples). I assume that with an end-to-end objective (perhaps similar to [2]), the gap between Hummingbird to Hummingbird++ can be diminished. Minor: 1. Some missing RW. * Recently retrieval-based a retrieval-based approach was proposed for choosing visual prompts [1]. To be fair, this work might be parallel research. * Conceptually, the idea here is similar to a (now old) paper [2]. E.g., learning to attend to different features to colorize, which can later be used for segmentation. I think it is at least worth citing. 2. Why Hummingbird? Is there any connection from Hummingbird to the idea/method I'm missing? [1] Zhang et al. "What Makes Good Examples for Visual In-Context Learning?". 2023. [2] Vondrick et al. "Tracking Emerges by Colorizing Videos", ECCV 2018.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Please consider adding the following limitations (also stated above): * The title implies that scene understanding, however, the retrieved features might actually be very local and specific. The features are retrieved independently of each other and there is no mechanism to consolidate the retrieved features into a whole scene output. * Correspondence - with the current approach, the authors assume there are spatial correspondences between features and their associated labels. This is the case for segmentation or depth estimation but this is not necessarily the case for every possible task. E.g, if the task is to translate, crop and resize, rotate, or flip the image, etc, this assumption breaks. * Retrieval approach might not result in smooth outputs. This is ok for segmentation but how would this work for style transfer?

Summary

This paper investigates in-context learning in computer vision. It proposes the Hummingbird model, which uses a pretraining protocol and a nearest neighbor retrieval mechanism. This model avoids the need for task-specific parameters or finetuning, offering fast and data-efficient adaptation to various scene understanding tasks, including semantic segmentation and depth estimation (at the current stage). The pretraining protocol utilizes a unique combination of attention across and within images, providing robust image representations. Tested on common benchmarks of semantic segmentation and monocular depth estimation, the model's performance approaches the capabilities of specialist models finetuned for each task, suggesting potential for in-context scene understanding.

Strengths

I believe the studied direction, in-context scene understanding, is very important, and this work takes step towards it. The proposed nearest neighbor retrieval framework reminds me of nearest neighbor baselines in few-shot recognition [1,2]. Once the learned representation is powerful enough, a simple NN-baseline can have comparable or even higher performance than SOTA meta-learning algorithm. It is interesting to see such simple framework being re-proposed and shown strong performance in in-context scene understanding. All the used techniques (cross-attention, patch-level attention, self-contrastive objective) are sound and composed in a reasonable way. Their effectiveness are strongly proven in Table 5. Overall, the paper is easy to follow. [1] Large-Scale Few-Shot Learning: Knowledge Transfer With Class Hierarchy [2] SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning

Weaknesses

First, I am unsure if Figure 1 is the best design: (1) I cannot know the components used in Hummingbird; (2) does the nearest neighbors (on the top) refers to image-level or patch-level or pixel level? (3) does query image is people-horse the nearest neighbor also contains furniture images? L 142-144 said: "we train representations to locate the most distinctive part of an image". Does this claim has any supportive evidence? L 251-254 shows the author adopts a NN search way to let cross-attention to a small set. What's the relationship between the NN search time and memory length. What's the cost for the pre-training stage? Will this be a huge commitment? (1) If Hummingbird pre-training with semantic segmentation and fast adopt to monocular depth estimation, how will performance change? (2) If Hummingbird pre-training with monocular depth estimation and fast adopt to semantic segmentation, how will performance change? (3) if Hummingbird pre-training with both semantic segmentation and monocular depth estimation, will this model outperforms than the models pre-trained with one type of data?

Questions

Please address concerns raised above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The author has discussed broader impact and limitations in Section 7 (L 273~285). I personally feel one potential limitation was not discussed: training time and inference time. The training/inference time could be huge if adopting large memory. Also, what's the relationship between training/inference time and memory size?

Summary

This paper presents a novel in-context learning paradigm based on NN retrieval, which can be adapted for downstream dense vision tasks without requiring fine-tuning or specialized decoders.

Strengths

1. The proposed method addresses a promising regime for reducing task-specific training requirements with prompt images and labels. 2. The framework is conceptually simple yet powerful, providing a concise and reasonable solution. 3. The methodology is clearly presented, and the paper is well-organized and easy to follow.

Weaknesses

1. The current paradigm seems to only be applicable to pixel-wise prediction tasks. As it is unclear how the in-context learning paradigm could be extended to other vision tasks, its application may be limited. 2. The retrieval process is inefficient compared to previous paradigms during inference.

Questions

1. In Tab. 3, why does the result of MoCo-v3 significantly drop for E2E FT? 2. In Sec. 3.2, why contextual pretraining can learn meaningful representations while "enforcing its representation to be expressed as a combination of representations of nearby examples", even without supervision? I don't understand its intuition.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have discussed the limitations.

Summary

The paper presents a non-parametric nearest neighbor retrieval method designed to efficiently adapt a pretrained model for dense scene understanding tasks. The fundamental elements include a self-supervised pretraining method, which incorporates attention across images, and a spatial attention-pooling mechanism. The conducted experiments demonstrate improved data efficiency and training speed compared to fully-finetuned methods, while maintaining competitive performance.

Strengths

- The paper shows that non-parametric nearest neighbor retrieval method can be more effective than linear probing in existing self-supervised learning approaches. - The self-supervised pretraining does not necessitate either fine-tuning or specialized decoders. - The proposed method significantly outperforms existing self-supervised baselines in downstream tasks and demonstrates improved data efficiency.

Weaknesses

- The paper only consider two downstream tasks (semantic segmentation and depth estimation) in the experiments, despite aiming for general-purpose representations. - When dealing with a complex dataset like ADE20K, the training approach without any supervision (Hummingbird) exhibits lower performance than some baselines, particularly when the data is scarce (n=158, 316) or when fine-tuning is applied.

Questions

- Why does the proposed contrastive learning objective enable the encoded features to be directly applied to downstream tasks without any parameter adaptation? This seems to align well with semantic tasks like semantic segmentation, but can you explain its applicability to geometric tasks like depth estimation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and broader impact.

Summary

This submission proposed a new representation and pre-training method, named Hummingbird for its fast adaption ability. Hummingbird could perform inference simply by nearest neighbor retrieval from a prompt of annotated features. Unlike the previous pre-training and then fine-tuning/linear probing protocol, the new representation could predict dense outputs by computing the cross attention (dot product similarity with propagation) between the testing query image and the memory bank. The pre-training process could be fully self-supervised or mixed with class label supervision. The new representation of Hummingbird outperforms strong representations like DINO, MAE on nearest neighbor evaluation of semantic segmentation and depth estimation benchmarks by a large margin. Hummingbird also shows great data efficiency in low-data regime.

Strengths

1. I like the idea of nearest neighbor propagation very much and I am very delighted that authors make it works. With the nearest neighbor propagation protocol, a single model could adapt to any vision task without fine-tuning. It's also very interesting that Hummingbird uses patches as the memory bank instead of the whole image, which saves a lot of storage and computation. 2. The fast adaption ability of Hummingbird is very surprisingly good. I like Figure 3 very much, and it shows with just 10s on V100, Hummingbird could achieve very good results already, which is nearly real-time. 3. It's very interesting to see the proposed self-supervised objective works very well with retrieval-based supervised objective. The self-supervised Hummingbird already achieves amazing results and retrieval-based supervised objective could further boost it.

Weaknesses

1. I am a little bit concerned about the scalability of Hummingbird. I truly appreciate that authors provide ViT-L version of Hummingbird and Hummingbird++. But if we are going to scale Hummingbird to ViT-H level, would ImageNet-22k be sufficient enough? Is there any more gain from scaling up the model? Or if we follow the self-supervised objective only, how much gain could we expect if there is more data? 2. Depth results in "Fast adaptation to downstream tasks" and "Fully finetuned scene understanding" sections are missing. I would also be very interesting to see the adaption ability on tasks other than semantic segmentation.

Questions

1. I found that there is no softmax operation in contextual pretraining Eqn 2 but there is softmax in Eqn 3. Is there any explanation for this design? 2. Would the work be open-sourced? I think this work would be highly impactful for the community and it would be quite beneficial if authors would like to release the implementation and checkpoints. 3. Is Hummingbird ViT-L fully fine-tuning yields higher accuracy than ViT-B? 4. Since Hummingbird is very good at nearest neighbor, it would also be interest to see the linear probing results and comparison. 5. It would be interesting to see whether retrieval-based supervised objective works solely without self-supervised objective.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Response to authors

Thanks for posting a rebuttal. Given that you plan to add the paragraph above to your limitation section, I changed my rating and I now support the acceptance of the paper.

Reviewer response

I thank the rebuttal. I encourage the author to include all the details required at this review into their revision. Also, refer to [50] is not strong enough to support your claims. As you also adopted the contrastive objective, you should can prove via qualitative results if the claim holds.

Response to authors

Thanks for the rebuttal. I like it very much. All my concerns are resolved in the authors' rebuttal. I would keep my score as it is and look forward to your implementation's open source. I think it would inspire the community a lot.