The human visual perception system demonstrates exceptional capabilities in learning without explicit supervision and understanding the part-to-whole composition of objects. Drawing inspiration from these two abilities, we propose Hierarchical Adaptive Self-Supervised Object Detection (HASSOD), a novel approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. Furthermore, HASSOD identifies the hierarchical levels of objects in terms of composition, by analyzing coverage relations between masks and constructing tree structures. This additional self-supervised learning task leads to improved detection performance and enhanced interpretability. Lastly, we abandon the inefficient multi-round self-training process utilized in prior methods and instead adapt the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. Through extensive experiments on prevalent image datasets, we demonstrate the superiority of HASSOD over existing methods, thereby advancing the state of the art in self-supervised object detection. Notably, we improve Mask AR from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B. Project page: https://HASSOD-NeurIPS23.github.io.
Paper
Similar papers
Peer review
Summary
The paper proposes a self-supervised object detection method (in fact, it can also do instance segmentation). This is based on an idea of bottom-up merge by strong feature representation like from DINO feature. The image patch grouping is hierarchical because of setting different stopping thresholds. This makes the model interpretable. Overall, the idea is simple and straightforward. The model can be trained by teacher-student framework. Experiments show better results than recent work MaskCut.
Strengths
- the bottom-up merging method to solve the self-supervised object detection might be new. - Figure 1 and 2 are good to understand the method. - experiment results look good.
Weaknesses
- The writing is bad. Many critical details are omitted which makes it super hard to under the method. - Some inconsistencies in the paper, needs to clarify.
Questions
Some critical details are missing in the paper and they need to be clarified. 1) What is learned, the whole DINO backbone or just a head network? 2) For the Mean-Teacher, what are the losses used to supervise the student networks training? The grouping process in section 3 is iterative and seems to be too slow. What is the time complexity? Line 138, how to ensemble? Line 195, in section 3, looks like only the feature representation is used, why train a Cascade mask rcnn here? Obviously, Cascade mask rcnn is not self-supervised. Line 233 and 234 are inconsistent with section 4.2. What exactly Objects365, Lvis, SAM are for evaluation or training? Lvis should have a LVIS-rare benchmark. Whole, part, subpart should be given rigorous definitions. For the limitations 2) and 3) in the second paragraph, I don't understand how they are solved after reading the whole paper.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The discussion at the end of the paper is good.
Sorry for the delay. Thanks for the effort of the authors. Most of my concerns have been solved. I increased the score.
We are glad that our responses have addressed your concerns. We appreciate your helpful feedback and reconsideration of the rating.
Summary
The authors propose a self-supervised object detection approach by employing a self-supervised pre-trained model (i.e., DINO) to hierarchically and adaptively group regions into object masks using multiple pre-defined thresholds based on cosine similarity in the feature space. The authors then adapt the Mean Teacher framework to train a student object detector with the initial pseudo-labels from the clustering process as well as the progressively refined pseudo-labels from the teacher model. Extensive experiments demonstrate the superiority of the proposed approach over existing self-supervised object detection and instance segmentation methods.
Strengths
- The experiments are extensive and the results are promising. - The paper is generally well-written and easy to follow.
Weaknesses
- The authors use a set of pre-defined thresholds (i.e., {0.1, 0.2, 0.4}) to merge region features. However, considering different datasets may have different distributions, such COCO-tuned hyper-parameters may not be suitable when merging region features for other datasets. Thus, the authors have to select the optimal thresholds for each dataset when performing clustering, which limits the practical applications of the proposed method to some extent. - The authors use some post-processing techniques like CRF to refine the masks. I am curious about the mask quality without such post-processing. Will the performance be dropped significantly? - Although the proposed method can segment an image with hierarchy, it seems that the mask quality is somewhat inferior to CutLER according to the qualitative results as shown in the paper. - The authors choose DINO with ViT-B/8 model, i.e., each spatial element in the feature map corresponds to a 8x8 patch in the original image. What is the effect of the patch size? It seems that the patch size tends to affect the performance significantly and should also be ablated. - The authors use Cascade Mask R-CNN to verify the effeteness of the proposed method. What about other kinds of detectors? It would be better to provide results based on more detectors to verify the generality of the proposed method.
Questions
See questions mentioned above. I am currently leaning towards borderline reject and hope the authors could address my concerns during the rebuttal.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors have well addressed the limitations and the broader impacts in Sec. 5, which looks good to me.
Summary
This goal of this paper is to improve the performance of self-supervised object detection by enhancing the detection pseudolabels. In addition to semantic masks provided as detection labels to an object detector, the authors utilize a rule-based approach to automatically generate a hierarchical structure of objects and provide it as targets to the detector. By training on hierarchical pseudolabels, the detector learns about the compositional structure of objects and therefore performs better. Additionally, the authors use a teacher-student network to train the detector instead of multi-round self-training approach. The proposed approach outperforms two other self-supervised detection approaches.
Strengths
- The authors tackle an important problem in perception; the hierarchical structure of objects. By providing object labels at multiple levels in a tree structure, the detector can learn patterns in compositional structures of objects (subparts - parts - whole), which helps improve the predictions. For example, a model can learn to condition a whole object prediction on the existence of its parts and subparts. - This hierarchical structure is not only found in vision, but also other perceptual modalities and NLP. Therefore, understanding how to generate and utilize this structure can be helpful in other tasks and across other modalities. - This method is completely self-supervised, based on features extracted from DINO trained on ImageNet. The improvement in detection performance comes at no supervision cost, only additional computational cost to generate the pseudolabels. - The approach shows promising quantitative results, surpassing other self-supervised methods.
Weaknesses
- This approach, like many others, will always be upper bounded by the quality of the features and the robustness of the self-supervised representation learning approach used to generate them. The generation of labels depends on frozen features, so the detector will be sensitive to the quality of those features. - Approaches that use hierarchical clustering lose representation quality as they naively average high dimension feature representations to merge clusters. Therefore these methods cannot cluster the object representations to generate class labels for training. This is similar to approaches like TW-FINCH [1] that takes the same route for event segmentation. This being said, I'm interested in seeing whether high-level representations of objects generated by averaging low-level pixel/patch representation would provide good matching to true class labels. An alternative would be to add a (non-)linear on the average-based representations to classify the objects. - While there is no supervision cost to generate the extra hierarchical labels, there is computational cost in clustering and structure discovery. It is important to share the computational cost in order to determine whether this approach is practical and can be scaled up. - I am not convinced that simple thresholding of the similarity metric provides accurate separation between the hierarchy levels. Using an arbitrary threshold will guarantee a tree structure but not necessarily a part-whole structure. The goal is not to find any tree structure whith parent masks covering children masks. These rules do not guarantee a part-whole structure. Each entity in the tree should be a coherent object (part-whole tree). - One suggestion is to plot $\theta^{merge}$ against the number of objects in the image and detect regions with lowest slopes to be the thresholds. The intuition is that in these regions the model is less sensitive to changes in $\theta$, therefore there could be good separation between hierarchical levels at these thresholds. - How do the authors decide on $\theta^{merge}$? Is it using the ablations in Table 2? Table 2 is generated using groundtruth masks. So if the decision on $\theta^{merge}$ is based on evaluating the masks with ground truth masks, this would be a significant flaw in the experimental protocol. It would violate the self-supervision claim. Usually researchers decide based on a small validation set, and even then it is still arguably supervised. This is an important point. [1] Sarfraz, Saquib, et al. "Temporally-weighted hierarchical clustering for unsupervised action segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
Questions
- When using a single threshold to generate the masks, my understanding is that we should only get one level of objects. In other words, the output masks should not intersect with any other masks. How are the numbers in Table 2 generated when using a single $\theta^{merge}$ threshold? This approach should only be possible when having intersecting masks (i.e., ensemble).
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
- Limitations about not aligning hierarchical levels with human perception are addressed. But it's missing limitations concerning the inability of this model to generate representations useful to classification of the detected boxes/masks.
Summary
This paper proposes a hierarchical Adaptive Self-Supervised Object Detection (HASSOD), an approach that learns to detect objects and understand their compositions without human supervision. HASSOD employs a hierarchical adaptive clustering strategy to group regions into object masks based on self-supervised visual representations, adaptively determining the number of objects per image. HASSOD identifies the hierarchical levels of objects in terms of compositionality, by analyzing coverage relations between masks and constructing tree structures. The proposed method adapts the Mean Teacher framework from semi-supervised learning, which leads to a smoother and more efficient training process. The Mask AR is improved from 20.2 to 22.5 on LVIS, and from 17.0 to 26.0 on SA-1B.
Strengths
Good results of the process, exhibited in Fig. 3. Many quantitative comparisons in Tables 1-3. Nice to read the overall paper. Good to see the reference of the vital work of [7], even after more than a decade. Appendix Doc highlights the importance of a better metric than AR.
Weaknesses
Lot of experiments, results, but no analytics. Not even a single equation, cost function, hyper-parameter set mentioned. Adaptation of mean-teacher model, still makes it a self-supervised framework., The overall designed architecture is explained with examples of images, in a diagram in fig. 2. A flowchart would still be better. Which part requires CNN-based learning, which part requires shallow learning (if any) or heuristics is not so clear (except a bit in Sec. 4.1, a small 2nd para). Only in the Appendix - The platform/environment/resources used for implementation are specified. The computational time required for each sub-stage of the process is also not mentioned, including the inference time. This is very important for practitioners. Few failure cases should have been highlighted with causes/reasons; Appendix is not thorough on this too. Color labels used for inference vs bounding box, often causes confusion - the color overlap for anyone to clearly understand.
Questions
If this proposed work is inline with self-supervision, what are the equivalent/compatible stages of - pretext learning using base data with no labels; fine-tuning with support set; target set for inference on downstream task? A table specifying the dataset distributions (although names given) and stages exploiting the same for self-supervision should have been given. Any scope of few-shot learning or meta-learning framework in your proposed approach?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Lack of analytics ; NO ablation studies with cost functions; Lack of mapping to standard Self-supervision paradigm, is a concern. Object Detection in complex cases of overlap of objects, background clutter not provided. I think its high time that researchers also look at identification/detection in camouflage (a personal opinion).
Follow-up on Detection in Camouflage
Thank you for highlighting Camouflage Object Detection (COD) [Ref 1]. COD presents a unique set of challenges, given the intrinsic similarities between the target and its environment. Currently, the best approaches to COD [Ref 1, 2] are based on high-quality human annotations and supervised learning. Given this context, utilizing self-supervised learning may not be immediately suitable. Exploring the overlap of self-supervised learning and COD is indeed an interesting future avenue. [Ref1] Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, Ling Shao. Camouflaged Object Detection. In CVPR, 2020. [Ref2] Deng-Ping Fan, Ge-Peng Ji, Ming-Ming Cheng, Ling Shao. Concealed Object Detection. In TPAMI, 2022.
Hope this message finds you well. This is just a friendly reminder of our recent rebuttal and follow-up responses, in which we have addressed your valuable feedback on our work. Could you please take a moment to review our responses? If any issues remain unresolved or further clarification is needed, we are more than willing to continue the discussion. Thank you once again for your time and expertise. We look forward to hearing from you soon.
Comment on the Rebuttal
Thank you for the rebuttal. I have read your answers to my questions and the other reviews. I am still not convinced that the performance is not upper-bounded by the **quality** of the frozen features you use for hierarchy discovery. My understanding is that the better the frozen features, the better the separation between hierarchical levels (as you mentioned in answer 4 about DINO features coherence). A better hierarchy improves object detection performance, because this is the premise of the paper. Therefore the improvement in object detection performance is always bounded by the quality of frozen features. As I mentioned in the review, I understand this procedure is similar to other methods that use pretrained frozen features and will lead to object detection not improving beyond a certain level, because the features are frozen. Showing that object detection performance trained on the hierarchy results in higher performance than objects detected from frozen features seems irrelevant to the concern I raised. If you cannot enhance the frozen features quality, you cannot improve the hierarchy quality, therefore cannot improve the object detection performance. Is my understanding correct, or am I missing something? Thank you!
Follow-up Response
Thank you for your insightful comment. We would like to further clarify our approach in light of your remaining concern about the pretrained features in the following aspects: 1. Focus on Object Detection, Not Representation Learning We would like to emphasize that the core of our research is on self-supervised *object detection*, rather than on improving self-supervised *representations* themselves. As the reviewer mentioned, consistent with previous state-of-the-art work including CutLER, our approach leverages existing pre-trained features to perform the object detection task. We have demonstrated that our approach HASSOD, while using the same pre-trained features by DINO, leads to enhanced results compared to prior works (see Table 1), highlighting the effectiveness of our specific contributions to self-supervised object detection. 2. Two Avenues for Performance Improvement Enhancing the frozen feature quality is *not the only way* to improve the object detection performance. In fact, from the perspective of pre-trained features, we believe that there are two ways to improve self-supervised object detection performance: - Approach 1: As you have correctly suggested, and we fully agree, one possible way for performance enhancement is through *better pre-trained features*. Currently, DINO has proven to be the *most effective pre-trained features for object discovery* and has been utilized by state-of-the-art methods such as CutLER. Future advances in self-supervised representation learning could yield even better pre-trained features, and using such improved features would further improve detection performance. However, developing more advanced self-supervised representations is beyond the scope of this paper. - Approach 2: Contrary to relying solely on frozen features, our work showcases the ability to **refine these features through the object detection training process**. We find this a critical contribution to highlight. In both HASSOD and existing self-supervised object detection work, the initial pseudo-labels generated by the frozen features could be noisy and coarse. For example, when the boundary between a foreground object and its background is vague, the frozen features are less distinguishable between the foreground and background, thus leading to inaccurate mask edges or even missed objects. However, after observing more similar instances in the whole dataset, the detector can gradually learn to fix errors in pseudo-labels, and detect such objects accurately. During this procedure, the pre-trained features are also adapted in an end-to-end manner for detecting more challenging objects. Therefore, we can achieve higher performance by training a detector than directly deriving objects from frozen features – this result was pointed out in our previous response. We hope our explanation here also clarifies the reviewer’s question regarding how this result is relevant to addressing the reviewer’s concern. 3. Refinement Beyond Frozen Features It is important to clarify that while we start our process with pre-trained DINO features, these features are **not permanently frozen** in our approach. During the training of our object detection model, we do fine-tune the DINO backbone. This fine-tuning, coupled with self-correction in detector training and the Mean Teacher strategy, enables us to adapt and potentially improve the DINO representations specifically for the object detection task. We provide a table below to compare the quality of objects discovered directly using the *frozen DINO ResNet-50 backbone* versus those discovered using the *fine-tuned backbone in our trained detector*. We use our hierarchical adaptive clustering strategy and standard post-processing to discover objects. Please note that this table is a direct comparison on the feature quality between the two backbones, rather than the final detection results. This comparison explicitly illustrates our ability to *substantially enhance* the pre-trained representations for the task of object discovery. | Backbone | AR | AR$_S$ | AR$_M$ | AR$_L$ | | ------------------------------------------ | ---- | ---- | ---- | ---- | | Frozen DINO ResNet-50 | 1.8 | 2.1 | 1.3 | 2.0 | | Fine-tuned DINO ResNet-50 in HASSOD (Ours) | 3.4 | 3.9 | 3.0 | 2.2 | In summary, our work not only leverages pre-trained representations, but also refines and adapts them to the specific task of self-supervised object detection.
Follow up question
Thank you for the reply. I have a follow up question. Are the enhanced features by object detection used to generate a better hierarchy or is the hierarchy fixed based on the frozen DINO features? I understand that the method optimizes the DINO features during object detection training, but that happens during training of any task. If the enhanced features from object detection are not used to enhance the the hierarchy (perhaps in an iterative or a joint training algorithm), then your proposed idea of using a hierarchy to improve the object detection performance is still bounded by the quality of those features you used to create the initial fixed hierarchy, because this hierarchy is fixed and is not affected by any enhancement of the frozen features. Please clarify this point. Thank you.
Yes, the enhanced features are indeed used in improving the hierarchical pseudo-labels. The hierarchical pseudo-labels are not fixed during our training. In our work, the enhanced DINO features engage in **providing better pseudo-labels through the process of Mean Teacher**. As introduced in Section 3.3, the student detector is supervised first by (a) fixed initial pseudo-labels, and the learning target gradually moves to (b) improved pseudo-labels produced by the teacher detector. The teacher detector contains the enhanced DINO backbone and utilizes its fine-tuned features. Please note that we do not have to actually rerun the hierarchical adaptive clustering algorithm on the enhanced DINO to discover hierarchical objects. Instead, since we have tuned a detector based on the enhanced DINO, **we can directly read out the predictions from the detector as the refined hierarchical pseudo-labels**. Generating improved pseudo-labels directly from the **trained teacher detector** is preferred, because it is more efficient than clustering DINO backbone features, and leads to hierarchical pseudo-labels of higher quality. A similar design has been applied in prior work such as FreeSOLO and CutLER (but with a multi-round iterative self-training routine that is inferior to our Mean Teacher). However, we cannot do so in the beginning when only the pre-trained DINO is available. Our detector model receives supervision from the combination of the (a) fixed initial pseudo-labels and (b) hierarchical pseudo-labels of higher quality from the teacher detector. Our adaptive target strategy modulates the weights for (a) and (b) during the training process, so that the impact of noises in (a) can be gradually reduced, the benefit of consistently improved (b) can be gradually magnified, and in the end, we are **not bounded by the quality of (a)**.
Thank you for clarifying. I now better understand the proposed online process of refining the hierarchy.
We sincerely appreciate the constructive discussion we’ve had, and we are glad our clarification has helped you understand our online refinement process. Considering the progress of our discussion and your latest positive comment, we kindly request you to reconsider the rating of our submission, if possible. Thank you once again for your valuable feedback.
Follow-up on Use of Cascade Mask R-CNN
As clarified in our previous response, we chose Cascade Mask R-CNN mainly to ensure **a fair comparison with CutLER**. Meanwhile, we do acknowledge the importance of demonstrating the versatility of our method across different detectors. To this end, we have trained a **Mask R-CNN** model using our proposed method. The following table compares LVIS detection performance of Mask R-CNN trained by CutLER and HASSOD. The results demonstrate that our method is indeed generalizable and consistently outperforms CutLER with different detector architectures. We will include the results in the revised version as well. | | Box AR | Box AR$_S$ | Box AR$_M$ | Box AR$_L$ | Box AP | Mask AR | Mask AR$_S$ | Mask AR$_M$ | Mask AR$_L$ | Mask AP | | ------------- | ------ | ------- | ------- | ------- | ------ | ------- | -------- | -------- | -------- | ------- | | CutLER | 20.7 | 10.4 | 33.3 | 52.0 | 4.1 | 18.5 | 9.6 | 29.1 | 44.9 | 3.4 | | HASSOD (Ours) | 23.8 | 13.5 | 38.3 | 50.9 | 4.3 | 21.5 | 11.8 | 35.1 | 46.6 | 4.1 | In summary, while our primary focus was on Cascade Mask R-CNN for a consistent comparison with CutLER, our method is certainly not restricted to it and shows superior performance with other detectors as well.
Hope this message finds you well. This is just a friendly reminder of our recent rebuttal and follow-up responses, in which we have addressed your valuable feedback on our work. Could you please take a moment to review our responses? If any issues remain unresolved or further clarification is needed, we are more than willing to continue the discussion. Thank you once again for your time and expertise. We look forward to hearing from you soon.
Response to rebuttal
Thanks for the rebuttal. I have read the authors' response and the other reviewers' comments. Overall, I think this is a borderline paper given the current status of the paper. I would like to raise my score to borderline accept to reflect the efforts the authors made during the rebuttal. The paper could be made much stronger if the authors could revise their paper based on the discussion during the rebuttal.
Thank you very much for your time and thoughtful consideration of our work. We deeply appreciate you raising the score based on our rebuttal. Following your constructive feedback, we are committed to further revising and strengthening the paper, by incorporating all of our discussion during the rebuttal. We would also like to highlight the growing interest within the object recognition community in class-aganotic object detection and instance segmentation, as exemplified by the emergence of models like the Segment Anything Model (SAM). In contrast to the supervised nature of SAM which extensively relies on human supervision, **our work shows the promise of a fully self-supervised direction** with strong, generalizable performance. As you have commented in the initial review, “extensive experiments demonstrate the superiority of the proposed approach over existing self-supervised object detection and instance segmentation methods.” We hope our self-supervised approach can benefit the research community timely and profoundly.
Follow-up on LVIS Metrics
As clarified in our previous response, our main objective is to identify **all objects from all categories** present in scene images without distinguishing based on classes, so we did not evaluate LVIS-rare/common/frequent metrics, which are typically used in a different task of class-aware object detection. While we do believe that the full LVIS is more suitable as the evaluation benchmark for our class-agnostic detection task, per the reviewer’s request, we have evaluated our model with the LVIS-rare/common/frequent metrics, and compared it against the prior state-of-the-art method CutLER. The results are summarized in the following table. Again, we demonstrate improved performance in every metric, which is consistent with our previous results. In contrast to class-aware object detection, we do not observe a significant gap between the detection performance for rare/common/frequent objects, since our method is developed in a **self-supervised and class-agnostic** manner. | | AR$_r$ | AR$_c$ | AR$_f$ | AR | | ------------- | ---- | ---- | ---- | ---- | | CutLER | 17.9 | 22.2 | 20.2 | 20.2 | | HASSOD (Ours) | 20.8 | 23.9 | 22.6 | 22.5 |
Hope this message finds you well. This is just a friendly reminder of our recent rebuttal and follow-up responses, in which we have addressed your valuable feedback on our work. Could you please take a moment to review our responses? If any issues remain unresolved or further clarification is needed, we are more than willing to continue the discussion. Thank you once again for your time and expertise. We look forward to hearing from you soon.
Decision
Accept (poster)