CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection

Summary

The paper presents CoDet, a novel approach for region-word alignment in vision-language representations for open-vocabulary object detection. Unlike existing methods that rely on pre-trained or self-trained models, CoDet reformulates alignment as a co-occurring object discovery problem. By grouping images that mention the same concept, CoDet establishes correspondences between shared concepts and common objects through co-occurrence, enabling it to leverage region-region correspondences across images for object discovery and open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects, while also showcasing scalability with visual representations, indicating its potential for benefiting from advancements in visual foundation models.

Strengths

+ In terms of originality, the authors propose a novel approach, CoDet, which reformulates region-word alignment as a co-occurring object discovery problem, diverging from the reliance on pre-trained or self-trained vision-language models. This unique perspective brings a fresh perspective to the field. + The quality of the paper is commendable, as the authors provide a clear description of the proposed approach, detailing how CoDet groups images based on shared concepts, leverages co-occurrence for region-region correspondences, and utilizes open-vocabulary supervision. The experimental results consistently outperform state-of-the-art methods, validating the effectiveness of CoDet in detecting novel objects. + The clarity of the paper is also notable, as the authors provide a concise and coherent presentation of their work, making it easily understandable to readers. + The significance of the paper lies in its potential impact on the vision-language representation field. By addressing the limitations of existing methods, CoDet opens up new possibilities for reliable region-word alignment and object-level vision-language representations, which can benefit various applications such as open-vocabulary object detection.

Weaknesses

- The paper would benefit from a more explicit discussion of its differences and a comparative analysis with related work [a]. By highlighting the distinctions between the proposed CoDet approach and the existing method [a], the authors can provide a clearer understanding of the unique contributions and advantages of their approach. [a] Aligning Bag of Regions for Open-Vocabulary Object Detection. CVPR, 2023. - The absence of VLDet in Tables 2 and 3 raises concerns and it is important for the authors to provide an explanation for its omission. Including VLDet in the comparative analysis is crucial to assess its performance against the proposed CoDet approach and other existing methods.

Questions

Please refer to the weaknesses mentioned above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations

Summary

This paper proposes CoDet, a novel approach for open-vocabulary object detection that reformulates region-word alignment as a co-occurring object discovery problem. The approach groups images that mention the same concept in their captions, leveraging region-region correspondences to discover the common object and adopt the shared concept as category label for open-vocabulary supervision. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.

Strengths

1. Introducing a new method, CoDet, for solving the open-vocabulary object detection problem, which reformulates region-word alignment as a co-occurring object discovery problem, achieving more accurate localization and better generalization ability. 2. Experimental results demonstrate that CoDet outperforms existing state-of-the-art methods in detecting novel objects and exhibits strong scalability, benefiting from advancements in visual foundation models. 3. The authors provided open-source code, which ensures the reproducibility of experiments and promotes further research. 4. The paper is well-organized, with a clear structure and logical flow.

Weaknesses

1. The method is simple, and prototype-based solutions are common in other tasks. I am slightly concerned about the novelty of the approach. 2. There are fairness concerns in this paper. See Questions.

Questions

1. The fairness for comparison cannot be promised. For LVIS, CenterNet2 is used in this paper. But in previous methods, FasterRCNN is usually used. 2. Some implement details are confusing. Why using CenterNet2 [62] with ResNet50 as the backbone for LVIS?. But for OV-COCO, Faster R-CNN with a ResNet50-C4 backbone is adopted. Is this fair to compare with previous methods? The same backbone should be used for fair comparison. 3. Some OVD methods are not compared in this paper. e.g., "Open-vocabulary detr with conditional matching."

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author have addressed the limitations.

Summary

The paper proposes a novel perspective in discovering region-word correspondence from image-text pairs, which bypasses the dependence on a pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. An open-vocabulary object detection framework named CoDet is built and achieves state-of-the-art performance across multiple standard benchmarks. The proposed method also demonstrates the effectiveness of visual guidance in region-word alignment. Experimental results validate the effectiveness of the method.

Strengths

The paper is well-written and easy to understand. The idea of using co-occurrent regions as visual guidance is interesting. Experimental results are promising.

Weaknesses

1. During prototype construction, how to ensure that the prototype does not contain noisy information such as hard negative samples. 2. In line 16, the authors claim that the proposed method can benefit from advancements in visual foundation models. However, no further experiments about different visual foundation models are carried out.

Questions

Please see above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors addressed the limitations.

Summary

This paper argues that existing OVD work relies heavily on visual language pre-training tasks, and current methods cannot provide more fine-grained cross-modal alignment information for OVD, resulting in limited performance on open vocabulary detection tasks. To this end, the authors propose to adopt co-occurrence guided region-word alignment for the OVD task. This method is novel and interesting, and has a good reference value for fields such as OVD and open world segmentation. Overall, the writing of this article is clear and the content is comprehensive. But I think the author still needs a clearer motivation, formulation and explanation for how to discover the co-occurrence proposal in the image.

Strengths

As expressed in the summary, the ideas in this paper are novel and interesting, and the writing is clear and complete.

Weaknesses

In this paper, the motivation and solution of how to obtain the co-occurrence proposal corresponding to the text from the mini-group image is not clear. For example, - How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170. - In addition, the author's description of the change in the shape of the similarity matrix in figure2 is too late (in line 185), and it may actually be more appropriate to arrange it in line 166. This caused some difficulty in understanding.

Questions

1.How the author avoids the problem of large differences in the visual appearance of target objects in different images under the same concept mentioned in line 170. 2.In line 189, why should the co-occurrence region appear in the last dimension of S? 3.In line 177, the text feature should be an embedding of a sentence, why does the author say that the relative size of text features in different dimensions indicates their relative importance in concept classification?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors provide an objective discussion of the limitations of the article.

Summary

In this paper, they propose CoDet, a novel approach that overcomes the reliance on pre-aligned vision-language space by reformulating region-word alignment as a co-occurring object discovery problem. Specially, CoDet groups images that mention the same concept in their captions, which brings a natural correspondence between the shared concept and the common objects within the group through co-occurrence. Experimental results demonstrate that CoDet consistently outperforms state-of-the-art methods in detecting novel objects.

Strengths

1. This paper propose an interesting idea to reformulate region-word alignment as a co-occurring object discovery problem. 2. Experiments show its effectivenesses.

Weaknesses

1. Experimentation Missing, Need to compare with the fine-grained word-region alignment method proprosed in DetCLIPv2[1], another OVD method try to learn with image-text pair more efficiently with region-word loss. 2. Some more questions, see the questions. [1] DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment

Questions

1. The image-text pairs currently used are relatively clean. It is hypothetical that if data with larger noise, such as YFCC, are used, the results might differ. In their images, the content described in many texts does not align with the images. Would the algorithm proposed in the paper have limitations in such cases? Furthermore, how efficient is the proposed algorithm? For instance, when expanded to larger datasets like YFCC26M, how does it affect the increase in training time and the stability of performance improvement? 2. Suppose there are two noun that your models have never learned similar concepts (where text guidance is ineffective), and the objects corresponding to these two nouns always appear simultaneously, such as eyes and eyelashes. Would it be difficult for this framework to distinguish between them, or is all existing methods fundamentally incapable of solving such issues? 3. Regarding the technical details in lines 160-161, it appears that multiple prototypes and text losses are computed simultaneously in a single forward pass, followed by iterative selection of the query image, correct? 4. How sensitive is this framework to RPN? For example, in the case of Figure 2, if the RPN proposes a dog's head, is there a possibility that the dog's head might be recognized as a complete dog?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The limitations are addressed.

Experiment Results on Visual Foundation Models

Here we report the results of CoDet using EVA02-L as the backbone on OV-LVIS, which further verifies that CoDet can consistently benefit from stronger visual representations. In comparison with F-VLM [1] which uses the pre-trained CLIP visual encoder as backbone, CoDet achieves superior performances with fewer parameters. | Method | Backbone | Params. | mAPr | mAPc | mAPf | mAP | |--------|-------------|------------|----------|----------|----------|----------| | | R50 | 25M | 22.7 | 30.3 | 34.7 | 30.7 | | CoDet | Swin-B | 88M | 29.4 | 39.5 | 43.0 | 39.2 | | | **EVA02-L** | **304M** | **35.2** | **49.1** | **49.2** | **46.7** | | F-VLM | R50x64 | 420M | 32.8 | -- | -- | 34.9 |     [1] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

Response to the author

Thank you very much for your detailed explanation of my concerns. I keep my initial score.

Welcoming further discussions

Dear Reviewers, We sincerely thank you for your efforts in reviewing our paper and your suggestions for polishing the manuscript. As we are approaching the end of the discussion period, we would like to ask whether there are any remaining concerns regarding our paper or our response. We are happy to answer any further questions. If you find our responses have addressed all of your concerns, we would be much grateful if you could kindly consider raising the rating of our paper during your final evaluation.