Domain Adaptation for Large-Vocabulary Object Detectors

Summary

This paper addresses the domain generalization problem of large-vocabulary object detectors. Without requiring additional annotations, it uses the Knowledge Graph Distillation technique to transfer knowledge from CLIP, enhancing the detector's generalization capability on downstream datasets.

Strengths

1. Strong performance across 11 widely used dataset. 2. Detailed implementation details, which I really appreciate, are provided in the appendix. It includes thorough analysis and explanation from the motivation to the final results. 3. The proposed method is highly intuitive and easy to understand.

Weaknesses

1. Speed: Although the appendix provides training times, considering that Equation 6 requires cropping proposals from the detector and sending them into CLIP, this step is very time-consuming (as seen in VILD[1], which took almost a few days on the COCO dataset). Given that the training set includes much larger datasets like Object365, how was this training time calculated? 2. Performance: As shown in VILD and other open-vocabulary papers, adding CLIP scores improves detection performance on novel classes. Considering this paper also uses CLIP to assist in out-of-domain detection, and the compared methods do not use CLIP, directly embedding CLIP scores as a baseline to highlight the necessity of GCN is essential. [1] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

Questions

see weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

yes, the authors has adequately discussed the potential negative societal impact of their work and limitations.

Summary

This paper highlights the problem that in domain adaptation, detectors often correctly localize but misclassify. To solve this problem, this paper proposes the Knowledge Graph Distillation method, which uses the pre-trained knowledge of VLM to supplement the relational knowledge of various categories in vision and language required for object detection. Experiments over multiple benchmarks validate the effectiveness.

Strengths

1 The proposed method is easy to follow. 2 The proposed method exploits the prior knowledge of VLM to annotate unlabeled domains through knowledge graph distillation, which is a promising and efficient direction. 3 Experiments are conducted to verify the effectiveness of the proposed method.

Weaknesses

1 Computation overhead: Although the paper shows the training time and inference speed compared with other methods in Table 11, it does not compare memory usage and computational overhead. 2 Insufficient and unfair experimental comparison. The proposed KGD utilizes the strong generalization capability of the visual-language model (VLM) in the region (proposal). Therefore, it is unfair to compare directly with the traditional source-free domain adaptation method as they did not make any special design for VLM. The proposed KGD should be compared with methods based on visual-language models, such as RegionCLIP, which also uses the idea of ​​regional distillation. However, the comparison with methods such as RegionCLIP is only conducted on the Cityscapes dataset (Table 9), and the improvement is not particularly obvious (1.7%).

Questions

1 I would like to know, if traditional methods (such as MT, etc.) are applied to methods like RegionCLIP, what is the performance gap with the proposed KGD?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper addresses the challenges faced by Large-vocabulary object detectors (LVDs) in recognizing objects across diverse categories, particularly due to domain discrepancies in data distribution and object vocabulary. The paper proposes a novel approach named Knowledge Graph Distillation (KGD) that leverages the implicit knowledge graphs (KG) within the CLIP model to enhance the adaptability of LVDs to various downstream domains. The paper validated the effectiveness on a wide range of datasets.

Strengths

1. The task tackled in this paper is highly significant because successfully generalizing Large-vocabulary detectors (LVDs) to downstream tasks can substantially increase the practical utility of these models. 2. The paper presents a detailed description of the method and offers numerous formal justifications and clarifications.

Weaknesses

1. The paper has not clearly explained the motivation for using knowledge graphs as a tool. If the goal is to employ LVDs for various downstream tasks involving unlabeled data, one potential approach could be to use CLIP to obtain pseudo-labels and then incrementally update the LVDs through semi-supervised learning. In this scenario, it appears that a specifically designed knowledge graph may not be necessary. 2. The methods used for comparison in the experiments are outdated, with no works from 2023 or 2024 included. This makes it difficult to adequately demonstrate the effectiveness of the proposed method. 3. The knowledge graph distillation technique is likely based on prior research in the field. From a design perspective, it would be beneficial if the paper could emphasize the improvements the proposed method offers over previous approaches in this area. 4. The ablation experiments are not sufficiently comprehensive. It would be advantageous to investigate the performance of using only Detic+MT without incorporating the knowledge graph distillation technique. 5. Cropping the object regions and using CLIP for recognition could considerably slow down the training speed of the model. The paper should offer a detailed explanation regarding this aspect.

Questions

Please refer to weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

No.

Summary

The authors propose a method for domain adaptation for large-vocabulary object detectors. To perform the adaptation, the authors first construct a Language and a Vision graph from the set of classes of the target dataset. The language graph is built with nodes as the description of the target class and the hyponym set of the class as retrieved from WordNet, and each description is encoded with the text encoder of CLIP to define the nodes’ features. This graph is exploited to adapt and improve the predictions on an image with a graph convolutional network (GCN). The Vision graph is initialized with nodes for all classes of the target dataset, with nodes represented by the text embedding of each class which are then adapted using the visual embedding centroid of each category. The similarity between the bounding boxes’ features and these adapted visual nodes’ representation is exploited to improve the class predictions. Experiments conducted on multiple datasets with multiple baseline comparisons show state-of-the-art performance of the proposed method. Ablation studies also show the contribution of each component of the method.

Strengths

- The authors present a sound method for domain adaptation for large-vocabulary object detectors exploiting graphs built from the textual and visual modalities from the set of classes of the target dataset for adaptation - The proposed method achieves state-of-the-art performance across a wide range of object detection datasets - Detailed ablation studies show the contribution of each component

Weaknesses

- The main weakness of this paper may be in the terminology used by the authors. For example, the authors talk about Knowledge Graphs for things that are not aligned with the common meaning of these terms, as the edges in the LKG and VKG graphs do not convey semantic relationships but are simply affinity edges based on the distance between the nodes' representation. On the contrary, the authors could name more explicitly the underlying methods and principles adopted in their approach, e.g. the step in Eq. (7) seems to perform a semi-supervised label propagation with a graph convolutional network, and Eq. (13) is performing embedding propagation. The term “encapsulation” may not be well adapted either, and it also hides the main visual adaptation steps in Eq. (11) and (12). Overall, this hinders the readability of the paper and induces confusion. - Some elements of the method should be detailed (see questions) - Some experimental results could be presented more clearly (see questions)

Questions

- Figure 2, what do the dashed reddish lines between LKG and VKG represent? - In the WordNetRetrieve step, it is unclear if the hyponym set just contains the hyponym class names, their descriptions, or both. Similarly, is the initial class name used? - p6 Eq. (12), what is the value of $\lambda$? How is it selected? How sensitive is the method to this value? - p6, under Eq. (13), from the reference (54) it is not immediately clear what is the value of $\alpha$, how it selected, and how sensitive the method would be to this value. The authors should discuss this directly in the paper. - Ablation studies tables (Table 3, 5, 6, 7) are somewhat redundant and hard to read due to the layout. The authors could invert rows and columns and combine all these tables into one to improve readability. Interestingly, it seems the most simple “LKG Extraction with category name” and “Static VKG Extraction” already show significant improvement over the baseline 46.5 -> 51.9, while adding all the other elements of the method brings less than one more point of performance. Do the authors have the results of using these two jointly? How far would that be from the whole KGD (Ours) results? Ablation results are only reported on the Cityscapes dataset, have the authors conducted similar analyses on the other datasets? If so, is the behavior consistent across datasets? Typos etc: - p6-l195: this sentence is unclear “the update-to-date object features to update it using manifold smoothing” - p9-l306-311: use “smoothing” instead of “smooth”

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes, in A.4.8.