Open-Vocabulary Object Detection via Language Hierarchy

Summary

The paper proposes a novel method for leveraging image-level annotations for object detection pretraining, specifically for zero-shot and open vocabulary detection settings. The proposed method, DetLH, leverages self-training to generate pseudo-labelled object proposals, and then adjust the pseudo-label for each box according to the image's class and the classes in WordNet that are hierarchically related to it. Furthermore, the authors propose a method for prompt generation that does not require additional training. The proposed method is evaluated extensively and achieves state-of-the-art results in relevant zero-shot and open-vocabulary object detection benchmarks.

Strengths

1. The paper is well written and its key ideas clearly communicated. 2. The proposed method is well presented and novel, and constitutes an improved way to leverage labelled object centric images to facilitate object detector training. 3. The conducted experiments are extensive and demonstrate that the proposed method consistently achieves state-of-the-art performance in various settings and datasets.

Weaknesses

1. Unless I am mistaken, there is no ablation study related to the impact of the reliability score. I would be very interested to see how it affects performance. 2. While justifications are provided in the appendix, I believe more comparisons should have been included with relative methods (L661-671). More importantly, I believe that the ommision of evaluations with LVIS/COCO is a significant issue, and I am not convinced by the authors stated reason (L683-687) that they focused on better suited datasets. The paper has extensive experiments with many datasets, and it is confusing to me why they chose to leave out what is arguably the most common dataset for detection tasks.

Questions

My primary concern regarding the paper is related to the evaluations: a) Whereas the paper is a clear improvement over Detic, it avoids extensive comparisons with more recent methods (Detic being a ECCV 2022 paper) such as the ones included in the paper (L661-671), and b) evaluations with LVIS, a shared benchmark for most relative papers, are not conducted. Overall, however, I am very satisfied with the paper's contribution and I believe that the conducted experiments adequately demonstrate the proposed method's effectiveness. My main reason for not giving a higher rating is because LVIS/COCO results are not included, which would be informative and are significant to facilitate comparisons with existing and future relevant works.

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the relevant limitations.

Summary

This paper focuses on scaling the detectors’ vocabulary with image-level weak supervision. To better leverage the image-level labels in this task, this paper introduces a language hierarchical self-training (LHST) framework that incorporates language hierarchy (i.e., WordNet) with self-training. In addition, the authors introduce a language hierarchical prompt generation (LHPG) method enabling open-vocabulary object detection. The proposed method outperforms baselines across 14 commonly used benchmarks. Comprehensive ablation experiments validate the effectiveness and generalization of the proposed method.

Strengths

+ This paper is well organized and written. The overall paper is easy to follow and technically sound. + On extensive benchmarks, the proposed method achieves non-trivial performance improvement and shows strong generalization.

Weaknesses

- The necessity of the investigated problem (i.e., scaling the vocabulary of the detectors with image labels) should be further stated, as the recent MLLM-based detection/grounding methods inheritly have strong zero-shot capabilities. - The proposed method is somewhat trivial and heuristic, where only the usage of WordNet is new and interesting. Most techniques are from existing WSOD and semi-supervised object detection methods. It would be better to highlight the novelty. - Several semi-supervised WSOD methods trained with both instance- and image-level annotations are missing. It would be better to review them for completeness. [a] UniT: Unified Knowledge Transfer for Any-Shot Object Detection and Segmentation, CVPR’21 [b] Cyclic Self-Training with Proposal Weight Modulation for Cross-Supervised Object Detection, TIP’23 [c] H2FA R-CNN: Holistic and Hierarchical Feature Alignment for Cross-Domain Weakly Supervised Object Detection, CVPR’22 [d] DOCK: Detecting Objects by transferring Common-sense Knowledge, ECCV’18 - In Line 149, the authors claimed ‘a single-label annotation could be expanded into a multi-label annotation’. It would be better to give some visualization examples. In Line 164, authors claimed ‘the labels expanded by WordNet are not all reliable’. Giving some examples would be helpful to improve the readability. - Why only the AP results of oracles are reported on Object 365? - How about the efficiency of WordNet in training and inference?

Questions

Please refer weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations, and this works does not show potential negative societal impact.

Summary

* The authors propose to expand the training data for object detection using classification datasets through two main contributions: 1. Combine a language hierarchy with self-training to extend the training dataset, while minimizing label noise by re-weighting categories with reliability scores. 2. Bridge the vocabulary gaps between training and testing using a vocabulary proxy (WordNet) improving zero-shot learning. * Extensive experiments in multiple datasets are provided showing an improvement over the state-of-the-art.

Strengths

1.The authors clearly exposed their ideas and the paper is reasonably well written. 2.Even though using a language hierachy to improve object recognition is an old idea, [Marszalek, et al. Semantic Hierarchies for Visual Object Recognition] it's appealing to see it applied to a modern, large-scale, object detection setting. Moreover, the combination with self-training is original and well justified. 3.The results show a significant improvement over the state-of-the-art. 4.The evaluation is very extensive and performed over an impressive quantity of diverse datasets.

Weaknesses

1. The experimental section and ablation do not clearly show if the authors' assumptions are correct. * The idea of the paper heavily relies on the assumption that there are discrepancies on the taxonomy levels of the categories of image and detection datasets. This assumption may hold for some datasets but not for others and it should have a significant impact on the method's results. I miss a section presenting an analysis of the taxonomies of image vs box categories, and how it may impact peformance. 2. Discussion on the specific impact of each contribution to the final result is lacking * The ablation study (Section 4.2) does not seem to add much value to the experimental discussion. First, showing the performance increase over the baseline [68] seems misleading when the authors compare DetLH against only-box supervision, since the work of [68] already proposes to expand detector (boxed) training data with image-classification data. The work of [68] already demonstrated that adding image-based training data improves detection results, and it should not be the goal of the ablation section to show this. A far more interesting baseline could have been to train DetLH with uniform category weights to demonstrate how DetLH deals with noisy labels. * Another important ablation experiment would be to analize the impact of narrowing the gap between training and testing label distributions. The CLIP embeddings would already help in this regard, but how impactful is the use of a proxy vocabulary on top of it is not clearly quantified. 3. Wording is sometimes ambiguous or could lead to contribution misunderstandings: * For instance the authors say "These detectors usually yield constrained detection peformance when trained only with detection datasets... [68]", when the work of [68] is actually trained with both detection and classification data. If the authors want to point out that [68] already suggested this, it should be more explicitly stated. Even more so, if it's followed with a claim that the current method overcomes such limitations: "DetLH introduces large-scale image-level datasets to enlarge the data [...] in detector training.", which is misleading as this is not something the present work introduces, but the work of [68].

Questions

* Futher analysis and validation of main assumption. How do the category taxonomy levels differ across datasets? * Where does the performance gain come from? Robustness to label noise or vocabulary proxy? * In general in the experimental section I miss more insight and discussion on results, experimental setup and data setup. I feel that most discussion repeatedly states the obvious (performance increases due to leveraging larger amouts of data). Whenever more specific insight is provided, it is very speculative. For example, authors state that performance gains in Wildlife are due to the "introduction of the language hierarchy", but there are no specific experiments designed to proof such claim.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes, the limitations and potential negative impact are addressed.

Reply to authors

I thank the authors for addressing the issues raised by my review. I would like to also see what other reviewers think about the responses to the issues they raised, but I am satisfied by the authors’ response to my review and am inclined to recommend acceptance (i.e. raise my score to 7 after the discussion).

Reply to authors

Thanks to the authors for the thorough clarification of my cocerns. I'm happy to boost my score to "Weak Accept" as long as the updated ablation study, as well as further discussions regarding experiments (and taxonomy analysis) are included in the main paper. In my opinion those are important points that should not be left in the supplementary material.