Rank-DETR for High Quality Object Detection

Summary

This paper focuses on the ranking problem in object detection. Inspired by the misalignment between class and location scores in DTER models, the paper proposes to redesign the model architecture and the loss to modulate the rank information. Experiments show that , the proposed method can build upon the current strong DETR detectors such as DINO and H-DETR to further boost the performance by a notable margin (>1 mAP). The experiments also suggest the complementarity between the ranking model architecture and the ranking loss design.

Strengths

+ The paper tackles an important problem in DETRs for object detection: the misalignment of the class scores and the localization scores. + The paper is well written and easy to follow. The paper has provided informative statistical plots. For example, the density distribution for QRL and HWC, and the illustration of box scores with and without the proposed ranking mechanism. + The experiments are solid and sound. The paper uses detailed ablation studies to justify the design choices of the proposed components. The method seems effective according to the experimental results, i.e., around +1% AP metric for DINO and H-DETR.

Weaknesses

- The whole process seems a little engineering. E.g., the addition of learnable bias. - The main boost originates from the GIoU-aware loss, which has been investigated in Varifocal loss, Align-DETR and Stable-DETR (I am aware that the latter two are not formally published). This somehow hurts the novelty of the paper. A thorough discussion/comparison is expected. - The method seems complicated and includes many steps. For example the rank-driven model design seems not to invite as much performance gains as expected.

Questions

- The title is too broad and has little meaning. I recommend to change a name including the technical contribution. - The addition of the RCH module even hurts the performance, especially for AP_L, it degrades 0.6 points. Please explain this phenomenon. - The learnable logit bias vectors S and the randomly initialized content query C are confusing to me. Are they tricks or can the author give an explanation (or illustrations with figures) for their practical functions? I think further ablations on the two learnable bias are required. - The result for DINO, R50, 12epochs in Table 1 is different from that in Table 3. Please check.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The current solution is somewhat straightforward, and the topic can be deep investigated.

Summary

In this paper, the authors study the problem of object detection. To be specific, they introduce rank-awareness into transformer-based detectors both at the architecture-level and the loss/cost-level. After the rebuttal: The authors have addressed my concerns about the comparison with other ranking-based solutions. Therefore, I recommend the paper to be accepted.

Strengths

1. Ranking is an important problem in object detection. 2. The paper introduces and incorporates many interesting ways of integrating ranking into transformer-based detectors. 3. Strong results compared to existing methods.

Weaknesses

1. The paper is missing citations and comparison to significant ranking-based object detectors. To name a few, AP Loss, aLRP Loss, Rank & Sort Loss, Correlation Loss. 2. "Rank-adaptive Classification Head" => What does it have to do with ranking? This module just adds a learnable bias to classification scores, which do not depend on negatives or positives or localization qualities. 3. Some aspects require clarifications: 3.1. Eq 2: How does updating the positional encodings based on the sorted boxes disrupt positional information of the boxes? 3.2. Eq 2 & 3: Do you not need Sort() to be differentiable to pass gradients through? 3.3. "High-order Matching Cost" => Why high-order? What is high and what is order in this context? 4. Improvement over Varifocal loss (0.3% mAP) is insignificant. It is not clear what was missing in existing ranking-based loss functions. 5. Presumably there was not sufficient time to compare against [1, 21]. If you have performed a comparison in the meantime, can you please share the results? 6. It is a pity that there are no results on the LVIS dataset.

Questions

Please see above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None.

Summary

This paper proposes a DETR training method named Rank DETR that integrates multiple (four) rank-oriented designs, i.e., rank-adaptive classification head (RCH), query ranking layer (QRL), GIoU-aware classification loss (GCL), and high-order matching cost (HMC). Among these four components, the former two (RCH and QRL) are relatively novel, while the latter two (GCL and HMC) are close to already-existing IoU-based loss function and matching criterion (e.g., as in Stable DINO). Experimental results show that the proposed Rank DETR improves multiple DETR baselines. However, some recent methods (e.g., Stable DINO) with fewer components achieve comparable or even higher results.

Strengths

- Two (out of four) major components, i.e., rank-adaptive classification head (RCH), query ranking layer (QRL) are novel. - Ablation experiments show that most components bring considerable improvement (the improvement from RCH is relatively trivial) and integrating them brings further improvement.

Weaknesses

- Two (out of four) major components, i.e., GIoU-aware classification loss (GCL), and high-order matching cost (HMC), share close insight, motivation and mechanism with recent methods, e.g., Stable DINO and Aligned DETR (though the detailed implementation is different). Moreover, the improvement brought by these two components is smaller than the similar ones in the competing methods. - More importantly, though the proposed Rank DETR adds two more components (RCH and QRL) based on IoU-related loss and matching (GCL and HMC), the overall results of Rank DETR is still lower than the recent methods that only has IoU-related loss (and matching), i.e., Stable DINO. For example, with DINO baseline, the proposed Rank DETR achieves 49.6 AP (12 epochs), while Stable DINO achieves 50.4 AP (12 epochs). - The definition of ranking in DETR is not clear enough. What criterion is the ranking based on? The IoU or the predicted confidence. This should be clearly pointed out at the very beginning. - The relation between the proposed method and IoU-related methods is not clear enough. Section 2 should explain their differences against IoU-related methods, as well as connections, in more details.

Questions

- Eqn. 2 is confusing. What does Sort(A, B) perform? Sorting B and duplicate the sorting results onto A? - In Eqn. 2, positional embedding is irrelevant to the rank of a predicted results, but determined by the coordinates. Since the ranking (sorting) does not change the coordinates, how does the ranking impact the positional embedding? If there is no impact, why do you enforce ranking in Eqn. 2?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have discussed the limitations.

Summary

This paper proposes Rank-DETR for image object detection. The key contributions of Rank-DETR include (1) a rank-oriented architecture design, which comprises a rank-adaptive classification head and query rank layer to ensure lower FP and FN in predictions; (2) a rank-oriented loss and matching design, which introduces GIoU-aware classification loss and high-order matching cost to boost the AP under high IoU thresholds. Experiments on COCO demonstrate each component can contribute to the overall performance and align the design expedition, and RANK-DETR outperforms the current SOTA detector DINO.

Strengths

1. The paper is well written, neatly written, clean, and easy to follow. 2. The motivation of the paper is clear, 1. Rank-oriented Architecture Design: ensure lower FP and FN 2. Rank-oriented Matching Cost and Loss: boost the AP under high IoU thresholds. Experiments demonstrate the proposed component can align well with the design intent. 3. The experiments are very solid, it compares two SOTA query-based detectors H-DETR and DINO, and outperform them. The ablation study thoroughly analyzes each component's effect and performance contribution to demonstrate its effectiveness.

Weaknesses

1. Some module designs are not very novel. (1) Rank-adaptive Classification Head learns a class-aware and input-independent logits vector to model the class distribution and calibrate each query's score predictions. Such technique is well used in other fields like long-tailed classification/detection, and few-shot classification/detection. (2) GIoU-aware Classification Loss is very similar to Veri-Focal-Loss, except the modulating factor of positive classification loss is $t - \hat{p}[l]$ instead of $t$, and such a minor modification leads to a slight performance boost, $~0.3mAP$. 2. The Query Rank Layer design decouples content and positional query. The content query is constructed from MLP mapping of the fusing of a static query $C_l$ and output of the last decoder layer $O_l$. The positional query is constructed from PE encoding of ranked predicted boxes. This is a new design of query initialization, it will be interesting to compare it with other query initialization methods in DETR, Deformable-DETR, and DINO, but I can't find such comparisons. 3. There is no analysis or comparison of parameter FLOPs and computation cost (training & testing speed).

Questions

Please refer to the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Looking forward to hearing the response from all Reviewers

We sincerely value the reviewers for your thorough assessments and invaluable suggestions that have significantly enhanced our submission. We eagerly anticipate additional suggestions and improve our work from your further invaluable comments.

Re: Rebuttal by Authors

Thank you for the detailed rebuttal. I am happy with the responses and the new results. Therefore, I will keep my original recommendation as Accept.

Looking forward to hearing the feedback from JDo7

We sincerely value your dedicated guidance in helping us enhance our work. We are eager to ascertain whether our responses adequately address your primary concerns, particularly in relation to the comparisons with Stable DINO and Aligned DETR. We would be grateful for the opportunity to provide any needed further feedback.

Thanks for the authors' rebuttal

The authors have clarified most unclear statements and explained their differences and connections with recent IoU-based DETR methods. Though the proposed method shares similarities with some recent IoU-based DETR methods (e.g., Stable DINO and Align DETR), the reviewer is convinced that these works are actually concurrent, and thus has no more doubt on their contributions. Moreover, during rebuttal, the authors have showed that compared with Stable DINO, their method can actually achieve comparable results on the popular baseline DINO and even slightly higher accuracy on a more recent method (H-DETR).Therefore, I would like to change my rating to "weak accept".

Thanks the authors for the rebuttal

I have read the rebuttal and other reviews. The rebuttal has addressed my concerns sufficiently. I would like to keey my original rating as "borderline accept".