Scaling Open-Vocabulary Object Detection

Summary

The paper extends previous OWL-ViT detectors with self-training an 10B Web image-text data. The authors design different self-training strategies, such as token dropping, instance selection and Mosaics augmentation. It is an achievement to beat all previous methods on open-vocabulary benchmarks.

Strengths

- The authors provide necessary implementation details about scaling up the open-vocabulary detector on massive web data with a self-training method. - The performance on the open-vocabulary LVIS benchmark is truly impressive. - The authors promise to open-source their code.

Weaknesses

- GLIP2 performs better than OWL-ST on the ODinW dataset, can the authors explain the possible reasons? - The authors may consider adding the ablation studies about w or w/o the mosaic augmentation. The additional gain from the mosaic augmentation is not clear.

Questions

- What's the meaning of the $AP^{mini}$ metric in Table 1? Does it mean the LVIS test set?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I like this paper which includes comprehensive experiments and excellent performance, so I think this paper deserves to be accepted. But to me, this paper is not pretty interesting since this paper is more like a technical report with lots of engineering efforts rather than a research paper.

Summary

The paper proposes a self-training recipe for open-vocabulary object detection that leverages weak supervision in the form of image-text pairs from the Web. The authors identify three key ingredients for optimizing the use of weak supervision for detection: choice of label space, filtering of pseudo-annotations, and training efficiency. They propose to use all possible N-grams of the image-associated text as detection prompts for that image and apply only weak confidence filtering to the resulting pseudo-labels. The self-training recipe is applied to the OWL-ViT detection architecture and is called OWL-ST. The authors introduce OWLv2, an optimized architecture with improved training efficiency. The proposed method surpasses prior state-of-the-art methods already at moderate amounts of self-training and achieves further large improvements when scaled to billions of examples. The authors also evaluate their models on a suite of "in the wild" datasets and study the trade-off between fine-tuned and open-vocabulary performance.

Strengths

This paper is well-written, easy to understand, and very rigorous. This paper thoroughly discusses the methods and challenges of extending OWL-VIT to the billion-level image scale. This paper achieves surprising performance.

Weaknesses

As the authors mentioned, the biggest issue with this paper is the significant amount of computation required, which makes it difficult for general research institutions to reproduce. Therefore, my question is whether the authors will make the research results available as a foundational model, similar to SAM, for public use. Additionally, can it create a wave of downstream tasks, like SAM did, by providing basic capabilities?

Questions

Since the authors have mentioned SAM, I am curious about how it compares to SAM. Although one is a segmentation model and the other is a detection model, SAM can also be combined with Grounded, such as in https://github.com/IDEA-Research/Grounded-Segment-Anything, to achieve detection. I'm wondering how the authors view the comparison between their model and SAM in terms of certain task capabilities. This is not a necessary question to answer, I just want to have a friendly discussion with the authors.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The main issue is still the enormous amount of computation required. However, if it is open-sourced as a foundational model for public use, it would still make a significant contribution to the community.

Summary

This paper presents OWLv2 model and OWL-ST self-training recipe for open-vocabulary object detection. The detection data is greatly enriched with the aid of self-training. Concretely, the authors use WebLI dataset as the source of weak supervision for self-training. The dataset consists of approximately 10B images and associated alt-text strings (noisy captions). Then OWL-ViT CLIP-L/14 is utilized to annotate all images with pseudo boxes. For self-training at scale, the authors present several techniques including Token dropping, Instance selection and Mosaics to make the training efficient. Experiments on LVIS show the effectiveness of the proposed approach.

Strengths

1. This paper attemps to scale open-vocabulary object detection with self-training on large-scale weakly labeled dataset. This is a good engineering work built upon the existing detectors and datasets. This work verifies that large model + large data is a good option to enable zero-shot open-vocabulary detection. 2. The experimental results look like promising. Result on LVIS dataset is good.

Weaknesses

1. Authors should carefully polish their paper and fixtypos such as: In L12, L/14->VIT/14; L109, consist->consists. 2. The authors are encouraged to discuss how to handle noise in self-training. What's effect without handling these noise. 3. What's the inference process? The authors are encouraged to describe this in their next version.

Questions

See weaknesses box.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitations are adequately discussed in Section 5.

Summary

This work adopts a self-training strategy to generate web scale object annotations for open-vocabulary object detection. It uses an existing open-vocabulary object detector (OWL-ViT) to annotate 10B image-text pairs, and uses the annotated dataset for self-training. A various architecture is also proposed to improve the training efficiency. Experimental results show a large performance gain on open-vocabulary detection.

Strengths

1. A self-training strategy to annotate the bounding boxes for the large-scale image-text pairs. This provides an open-vocabulary detection dataset for various location-specific tasks, including detection, segmentation, visual grounding, etc. Despite self-training being widely explored in many previous works (discussed in Sec 2.3), this work provided a detailed ablation in the filtering strategy in Sec. 4.4. 2. The performance is promising in rare classes, indicating the ability of zero-shot transfer with the self-training strategy.

Weaknesses

See questions.

Questions

1. This paper is a coherent work of OWL-ViT. It's not straight forward for me to drop the DETR decoder in OWL, but rather predict a bounding box with each patch token. Such a dense prediction results in a significant number of background proposals (Instance selection in Sec. 3.2), which leads to inefficient detection. 2. While N-grams tend to maintain a higher number of image-text alignments in comparison to rule-based methods (such as highest-scoring or threshold-based approaches), they also tend to generate more redundant alignments, particularly for obvious objects where the confidence scores are higher (e.g., both 'boy', 'a boy', 'a boy in red' might correspond to the same region). Does this bias the model towards learning obvious objects while potentially overlooking the hard examples?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The self-training strategy in the paper requires a large amount of computing resources and web-crawled data, which is not feasible for many colleges and research institutes.

Thanks for your reply

Most of my concerns are addressed. Despite this work being an extension of object detection self-training on a large dataset, it provides some insights into scaling up the detection dataset. Besides, the experimental results impressed me, thus my decision tends to accept.