Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection

Summary

This paper uses the vision-language pre-trained model CLIP for long-tailed object detection. The key idea is to consider not only image-level semantics but also region-level semantics, and fuse them under a soft-label scenario. The overall idea is deployed on different object detectors with different backbones. Experiments are conducted on three long-tailed object detection datasets with extra data. Besides, the experiments and discussion on ablation studies and foundational models are very extensive.

Strengths

+ To the best of the reviewer’s knowledge, it is the first work (excluding CVPR2023) to use vision-language pipelines for long-tailed object detection. The contribution is significant. + The ablation studies and other discussions are very extensive, in both the main submission and the + This paper is well-written and easy-to-follow.

Weaknesses

- This work, in its current form, lacks theoretical insight, especially on how the proposed method fits into the long-tailed feature distribution. As the proposed method is targeted at the context of long-tailed object detection, not the generic object detection, it is very necessary to extensively discuss how the framework and the specific module design can work well or fit into the long-tailed feature distribution. Unfortunately, as the reviewer goes through the entire manuscript: In Fig.1 motivation case and Fig.2 framework, there is no element for the long-tailed feature space or long-tailed elements. Besides, in the methodology section, from learning the image / object semantics to use soft labels, all these presentations are generic and are suitable for generic recognition, detection, segmentation and etc. More importantly, in the experimental section, there is no feature space visualization on how CLIP benefit the long-tailed feature distribution. - The state-of-the-art comparison is insufficient, and to some extent, unfair. Specially, It is really strange that in Table 1, the authors only compare some generic vision transformer based detectors such as DERT. Why the methods, using vision-language pre-trained model for generic object detection, such as [11,18,27,34,57], are not involved for comparison on the long-tailed object detection datasets? It is very likely that these methods [11,18,27,34,57] for generic vision-language object detection can work well on long-tailed detection datasets. - The performance of the proposed method seems to be mainly brought high by the use of Swin-Transformer. This remark is not saying that the state-of-the-art performance with Swin-Transformer backbone is not important, but the reviewer would like to raise the attention that, when use the old ResNet-50 backbone (Row 1 of Table 1), the proposed method, even with vision-language pre-trained model, does not lead a significant performance against some SOTA in 2021. Thus, the reviewer feels that in the future, why with Resnet-50 is not that effective for the proposed framework, worth to be further investigated. Other minor issues to improve: - Extensive feature visualization on the context of both long-tail and CLIP features, please. - Please use \mathcal{} to present the loss function, and use \mathbf{} to present the tensors and vectors. These notations can distinguish them from scalars.

Questions

The reviewer really appreciate that the work is the first to extend vision-language pre-trained model to the long-tailed object detection task. However, the weakness of this submission is also very obvious. In the rebuttal, please address the weakness part point-by-point, on lack insight for long-tailed context, lack state-of the-art comparison, and to-some-extent performance ineffectiveness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitation is not properly discussed

Summary

- This work deals with Long-tail object detection. Authors identify two problems with using additional data, namely Semantic ambiguity and Location sensitivity. - Authors identify that semantic ambiguity arises due to supervision with one-hot encoded labels from the image datasets and instead propose to use CLIP scores for supervision. - CLIP's ability to provide sufficient semantic information even with course locations is leveraged to tackle location sensitivity. - Authors propose RichSem, a simple yet effective method, that adds an additional semantic branch to the detector to learn rich semantics from images. - The semantic branch is only required during training and with extensive experiments is shown to achieve state-of-the-art results on LVIS dataset in overall and rare categories.

Strengths

- The paper is well written and the presentation makes it easy to follow. - The experiment section is exhaustive and supports all the claims made by the authors. Authors test their method with the transformer and R-CNN based family of detectors and the ablation experiments clearly explain the contribution of each component. - RichSem is a principled way to leverage additional data for long-tailed object detection.

Weaknesses

- Authors miss the comparison with [1], which also uses additional data. Please compare appropriately in Table-1. - The current work heavily relies on CLIP but it is widely known that CLIP has several limitations [2]. It would be interesting to address the robustness of the current method to the limitations of CLIP. [1] Bo Li, Yongqiang Yao, Jingru Tan, Xin Lu, Fengwei Yu, Ye Luo, and Jianwei Lu, Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learning. [2] https://stanislavfort.github.io/blog/OpenAI_CLIP_stickers_and_adversarial_examples/

Questions

- Authors mention constructing a mosaic with multiple images in L51, where in the whole method is this done? - What is $f_v^i$ in Eq. 4? Is that $s^t$ obtained in Eq. 2? - From table 2c, rows 4,5 the increase in improvement is because of image level labels. But how are the annotations used in the whole pipeline? The semantic branch only uses CLIP similarities for the KL divergence loss? Is a hard supervised loss also being applied for the image labels? Is there a localization term for the image level labels? - What is $f^t$ in the semantic branch in Fig. 2? If $t\in (\text{loc},\text{cls})$, then is the semantic soft loss applied to both $f^{\text{loc}}$ and $f^{\text{cls}}$?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

I do not foresee any potential negative societal impact of this work.

Summary

To address semantic ambiguity and location sensitivity, this paper introduces a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. And their RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors.

Strengths

1. The paper is clearly written. And main idea of the paper is easy to understand. 2. This paper introduces a novel semantics learning framework, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels. 3. Extensive method demonstrates significant results on long-tailed datasets

Weaknesses

1. In Long-tailed object detection, the scarcity of samples or natural constraints results in a limited number of instances in the tail classes. Can exploring extra data effectively address the issue of scarce tail classes in practical applications? 2. CLIP is a large-scale model designed for the joint processing of images and text. The lack of representative samples for tail classes may lead to a relatively weak understanding and recognition ability of CLIP for these classes. Can CLIP still provide stable semantic guidance?

Questions

Please check the paper weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please check the paper weaknesses.

Summary

This paper adopts the CLIP model to obtain a 'soft label' supervision to train the detector under the long-tail distribution dataset and derive rich semantics from the CLIP part to enhance the tail-categories representations, which can be removed during the inference. The authors claim that the CLIP model can well capture visual semantics conditioned on only coarse locations, the whole-bbox for extra data in this paper. They then elaborate soft-label supervision to train the detector so as to alleviate semantic ambiguity and location sensitivity issues. Various ablative studies have been conducted to validate the effectiveness of the proposed method.

Strengths

1. The novelty of this paper sounds technically reasonable. 2. The writing of this paper is easy to follow. 3. The final performances look competitive and the improvements are obvious.

Weaknesses

1. In Line 172, the subscript of f and o maybe should be superscript. 2. The proposed semantic branch training is parallel to the detection heads during the training to avoid the training conflicts for the CE loss and KL loss, but the detection heads are only trained on the tailed samples in LVIS without some kind of interactions on the semantic branch and how to ensure that the detection heads can well handle the rare categorical samples during inference? 3. ImageNet-21K is used as extra data in this paper, but the backbone is pre-trained in ImageNet too. So is there any dataset overlapping between the pertaining and fine-tuning stages?

Questions

Please regard the weakness parts.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please regard the weakness parts.

Response to author rebuttal by Reviewer vo8U

Thanks for the authors to provide such a detailed rebuttal. Indeed, my concerns on Question 1, 2, 3 and 4 have been well addressed, which I appreciate a lot. However, before the discussion end period, could the authors also provide how the proposed pipeline can improve the feature space of long-tailed tasks against the baseline? I feel this is necessary for the scope of NeurIPS, and it can convince me to raise my score above accept threshold. Many thanks, and look forward to the update !

Rebuttal Response

Thanks for your detailed explanation and more experiments. In sum, most of my concerns have already been addressed. I keep my initial rating 'borderline accept'. Moreover, I suggest the authors should carefully take into account about the further demonstrations in rebuttal phase into their revised version, especially 'q2: How can semantic learning on extra classification data boost rare categories detection?'