Learning Domain-Aware Detection Head with Prompt Tuning

Summary

1. Proposed a novel framework for domain-aware object detection with a) a vision-language model-based backbone to extract highly generalized features b) a domain-aware detection head by prompt tuning 2. Design the prompt includes domain-invariant tokens, specific tokens, the token for class, domain-related textual description 3. The domain-adaptive prompt tuning maintains a prompt buffer with an ensembled strategy, and the buffer is saved and used for inference.

Strengths

1. The idea of this work is evident, reasonable, and well-expressed. 2. The generalized semantic knowledge makes practical help with prompt tuning. 3. The proposed method is evaluated on several benchmarks and obtains significant improvements compared with related works.

Weaknesses

1. Some details about the methodology are not clear: - I see that the box head is frozen when tuning the prompts, so the box head is trained with the baseline detector only using source data, right? Thus, the domain-aware head is only trained on the classification branch. Won’t the box regression accuracy be influenced when changing the domain? If I understood correctly, there seems to be no adaptation process to deal with the regression. - The image regions rj is obtained from RPN with RoIAlign, so is each rj in fj=f(rj) already a fixed-sized feature patch? Since f is the frozen visual encoder, where are the features R={rj} extracted from? 2. How much extra inference time will be increased by introducing the textual encoder?

Questions

see the "Weaknesses"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a new domain adaptive object detection (DAOD) method named DA-Pro. Unlike previous methods, which ignore the domain bias in the detection head, DA-Pro applies the learnable domain-adaptive prompt to generate the dynamic detection head for each domain. To do so, the prompt is designed to be composed of domain-invariant tokens, domain-specific tokens, domain descriptions, and class labels. The proposed method is evaluated in three scenarios and shows favorable performance compared to existing methods (non vision-language-model (VLM) based) and a baseline method (VLM based).

Strengths

S1. The proposed method is reasonably designed. S2. The proposed method shows favorable performance compared to existing non VLM-based DAOD methods and a VLM-based baseline method. S3. The ablation study reveals the effectiveness of each proposed component. S4. The paper generally reads well and easy to follow.

Weaknesses

W1. The novelty of the paper is a little weak. I do acknowledge that the paper has certain novelty in a sense that it reasonably integrates [4] into the DAOD task and the authors showed not only the proposed method achieved better performance than existing non VLM-based DAOD methods and the VLM-based baseline (table 1), but also the newly introduced components contributed to the performance gain (table 2). However, the contribution is rather straightforward extension given the previous work of [4] and the DAOD task. W2. The proposed method is evaluated on rather limited datasets. All the datasets used are related to cityscape. In addition, K->C and S->C scenarios only deal with car class. The evaluation on wider variety of datasets such as PASCAL VOC, clipart, and Watercolor2k would make the paper more convincing. W3. The details of the baseline method, which is very important because the proposed method can be fairly compared only to this baseline method due to the usage of strong VLM, is a bit unclear. I suggest to show the architecture of the baseline method as in Fig 2 and explicitly show the difference with the proposed method. I believe this would more clearly reveal which component in the proposed method is important. I wonder if the pseudo labels are used in the baseline method. Minor: W4. It is nice to have the ablation on the hand-crafted prompt. The paper says “clear” and “foggy” is used for Cityscape and Foggy Cityscape, respectively. I wonder how the performance changes if alternative words are used. Which words are used for the other scenarios?

Questions

Please discuss the points that I raised in the weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is no discussion on the limitation of the work.

Summary

Most existing methods incorporate a visual encoder (detection backbone) to mitigate the shift across the domain. This paper leverages domain adaptive prompts comprises of domain invariant tokens, domain-specific tokens, and domain-related textual description with class label. These domain adaptive prompts introduce detection heads across the domain having a backbone of vision language models, exploring its generalizability for adaptation tasks. Experiments include three domain adaptation scenarios for evaluation.

Strengths

+ This paper is well-written and easy to follow + An interesting application in domain adaptation space, utilizing the vision language models and analyzing the importance of the detection head instead of the backbone (visual encoder) + Figures are helpful to understand the overall proposed method

Weaknesses

- Authors need to consider Pascal to Clipart, watercolor, and comic experiments. That allows the method to be evaluated in more challenging domain shifts and multi-class problem settings. - Impact of uncertainty on pseudo labels? It would be interesting to see that in this problem setting, how uncertainty is helpful for a selection of pseudo labels instead of probabilities are taken into account above certain thresholds. Also how accurate are the pseudo labels? - Evaluation set 1500 in Cityscapes to Foggy Cityscapes? Authors need to be consistent in the evaluation e.g. in the works [see SIGMA, TIA, etc], the evaluation set is 500 images with the highest level of density in Foggy Cityscapes. - It would be interesting to see error bar plots on multiple runs

Questions

Please see the weakness section for relevant questions

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are not mentioned explicitly. Authors are encouraged to add limitations

Summary

This paper designs a novel Domain-Aware Detection Head with Prompt Tuning (DA-Pro) framework for domain adaptive object detection. The motivation is learning the discriminative detector for each domain instead of reducing the domain bias as in the traditional DAOD methods. Specifically, the authors leverage the vision-language model (VLMs) to build domain-aware detection head. The domain adaptive prompt consists of the domain-invariant tokens, domain-specific tokens, and the domain-related textual description along with the class label. The experimental results seem to be effective on serval DAOD benchmarks.

Strengths

1. This paper addresses an important research problem and the prompt tuning in vision is a popular direction. 2. This paper is well-written and the proposed method is easy to comprehend. 3. The experimental results appear to be effective compared to previous approaches. 4. The motivation for designing a domain-aware detection head for domain adaptive object detection is reasonable and the technical implementation is easy to follow.

Weaknesses

1. One major concern of this paper is the differences between the proposed prompts and those in COOP and DetPro. They utilize similar learnable prompts for the text prompts. In line 145, this paper claim “both the proposed prompts in RegionCLIP [39] and DetPro [4] cannot model the domain-specific knowledge”. In my opinion, RegionCLIP just fills concepts into prompts so that it does not model domain-specific knowledge. However, DetPro learns the prompt in a certain domain and could indeed learn domain-specific knowledge. 2. The experimental results are not adequate. There are two benchmarks (KITTI to Cityscapes and SIM10K to Cityscapes) that are only evaluated on the ‘car’ category and do not adequately show the effectiveness of the proposed method. 3. Why the bounding box head is shared for both domains? The proposed domain-aware detection head may be further improved for the bounding box head. 4. The ClipRegion and DetPro have the ability to handle open-vocabulary object detection, this work degrades them to close-set object detection. Why not directly study the domain adaptive open vocabulary object detection that handles the domain shift and knowledge shift (recognizing new concepts) such as [A1]? [A1] Rethinking Open-World Object Detection in Autonomous Driving Scenarios.

Questions

1. In line 260, the baseline is RegionCLIP and DetPro with a domain classifier, so are the baseline results reported in the Tables based on RegionCLIP, DetPro, or both? It is recommended to clarify the setting of the baseline model. 2. What is the initialization for the detection model (e.g., the visual encoder and Bbox Head in Figure 2)? Which datasets are used to pre-trained the detection model? How about the backbone only being pre-trained in ImageNet as previous DAOD methods (e.g., DA-Faster RCNN, SIGMA, AT, etc.)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The ClipRegion and DetPro have the ability to handle open-vocabulary object detection, this work degrades them to close-set object detection and limits the real-world application of ClipRegion and DetPro.

Summary

This paper introduces VLM to domain adaptive object detection. To be specific, this paper uses highly generalized VLM as detection backbone, and adapts detection head instead. To learn the domain-invariant and domain-specific knowledge, this paper extends the prompt to domain-invariant and domain-specific ones, which are optimized with corresponding loss. Besides, to adapt domain-adaptive prompt for unsupervised object detection, this paper uses the CLIP to get pseudo labels and Prompt Ensemble to stabilise the training. Experiments on DAOD benchmark show better performance.

Strengths

- This paper shows that using highly generalized VLM seems as a promising direction with relatively high performance

Weaknesses

- The comparison method and evaluation could be improved. See questions for details.

Questions

- Source and Target classifier in Figure 2 means similarity calculation, and drawn entity is misleading - What is the motivation of prompt dl_d. I mean it seems to play the same role with domain specific prompts. - An important comparison method [1] is missing, which as far as I know, is the SoTA of DAOD, which pushes the K2C performance to a very high level. - CLIP initialized visual encoder is an image level extractor, I am afraid if it is suitable for detection. - I am curious if the learned prompts are independently semantic. For example, K2C and S2C both learned domain specific prompts of domain C, and what performance will get if exchanging the corresponding prompts.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n/a