Cal-DETR: Calibrated Detection Transformer

Summary

The paper proposes a method to improve the calibration performance of transformer-based object detectors. In their approach, they first present a way to quantify the uncertainty of each logit using the variance of the outputs of different transformer decoder layers. Then, with the motivation that a higher uncertainty generally implies a poor calibration, they downscale the logits with high uncertainty. Second, they introduce a mixup based approach that is applied to the queries that match with the objects. Specifically, they create a prototype as the mean of the logits of the positive queries and mix it up with the logits of the foreground objects. They incorporate their method on Deformable DETR and UP-DETR, and observed notable improvements across different datasets.

Strengths

- The proposed method is a training time approach, hence does not require an extra hold-out validation set. - The improvement in the calibration performance is notable and outperforms existing approaches consistently in several datasets. - While the reilabiility of the estimated uncertainties are not investigated, the proposed approach to do so is intuitive and does not introduce an additional burden on the detection architecture. - The paper is written clearly and it is easy to follow.

Weaknesses

- I think the logit mixup strategy that the authors introduce is quite related to a cited work [36]. The main differences are that the authors apply mixup in the logit space (which is again special case of manifold mixup), they use a fixed mixing coefficient instead of sampling it and the method is applied for object detection. I haven't seen in the paper (especially in related work or in Section 3.3.2) such a clarification. Therefore, I'd see the contribution of the authors in logit mixup as the extension of the regularized mixup [36] to object detection. I think this is still an important contribution but I'd like to see an explicit discussion with the prior related work. - Recently [A] showed that the number of detections that are input to the calibration measure has a clear effect on the calibration performance. Specifically, such measures are easy to be misleading due to the maximum number of allowed detections, which is for example 100 for COCO dataset. For example, a detector that outputs 100 detection has more advantage to achieve lower calibration error compared to a detector that outputs less and does not fill the quota of 100. So, I wonder whether such a case exists here. Specifically for example for Table 1, how many detections per image on average do Baseline D-DETR, temp scaling, TCD (as the closest counterpart) and Cal-DETR output (which are then used as the inputs to estimate Detection ECE)? Or is there any thresholding of the top-100 detections while estimating the DECE? If so, how? Related to this, a minor suggestion (that I am not considering in my rating as a weakness as [A] came out in CVPR in June) can be to include a comparison in terms of Localisation-aware ECE in the way that [A] suggests to avoid such a doubt in the final version if the paper is accepted. - The authors claim that they propose a method to quantify the uncertainty for each logit. However, further insight on these uncertainties are not provided. For example, there is no evidence that the estimated uncertainties are reliable and can be used for different purposes. [A] Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration, CVPR 2023

Questions

- Specifically for example for Table 1, how many detections per image on average do Baseline D-DETR, temp scaling, TCD (as the closest counterpart) and Cal-DETR output (which are then used as the inputs to estimate Detection ECE)? Or is there any thresholding of the top-100 detections while estimating the DECE? - Can you please confirm that you obtain the prototypical representation query for each iteration during training? And I'd recommend making this explicit in the paper (L217-221). Also I'd recommend making it more explicit that you compute the mean of the logits across all positive queries (L217-218). - In constrast to several mixup strategies that sample mixing coefficient (\alpha in Eq.(4)) from a distribution, why did you choose a single \alpha value? - How do you obtain the labels after smoothing ( `c_i in L225). I think this should be explicitly defined.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Briefly mentioned

Summary

This paper proposing a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR and UP-DETR, which consists of quantifying uncertainty, an uncertainty-guided logit modulation and a logit mixing approach. Results show the method improves the baselines in calibrating both in-domain and out-domain detections while maintaining or even improving the detection performance.

Strengths

I'm familiar with object detection and DETR, but not Model calibration. It seems that the model calibration is a valuable problem, and the proposed method does improve it without performance drop.

Weaknesses

It seems the performance of selected baselines is relatively low, e.g. the performance of Deformable-DETR is only 44.0 in COCO as mentioned in Table 1. Will the method still be effective when using a strong or even a sota model, like [DINO](https://github.com/IDEA-Research/DINO)?

Questions

Will the method still be effective when using a strong or even a sota model?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors may try more baselines and report results.

Summary

This paper focuses on performing calibration for DETR, particularly for Deformable-DETR and UP-DETR. The authors first propose an approach for quantifying uncertainty in DETRs, which is built from the variation in the output of decoder layers. Then they develop an uncertainty-guided logit modulation mechanism and a logit mixing approach as regularizers. Experiments on three in-domain and four out-domain scenarios show the effectiveness of the proposed method in calibrating.

Strengths

The results of D-ECE are good in both in-domain and out-domain scenarios. The experiments are extensive across various datasets and settings.

Weaknesses

1. The proposed method has limited novelty. Although the authors claim that they do calibrate on object detection, the proposed method also performs the logits and shares large similarities with the methods in classification. 2. The way to quantify uncertainty in the DETR pipeline using the variation in the output of decoder layers is a straightforward idea, which may not be propriety to claim as a contribution. Moreover, the relationship between the proposed uncertainty-guided logit modulation mechanism and the logit mixing approach is not clear. It is more like a combination of two techniques, lacking deep insights. 3. When introducing D-ECE in Sec3.2, it would be better to cite [22] to indicate that the used measures follow previous works.

Questions

1. I am confused that what benefits can we get when the D-ECE is lower. It seems that the box AP does not show clear differences between Cal-DETR and the baseline model in both in-domain and out-domain scenarios. If the calibration can not bring benefits in detecting objects, then what benefits can we get by performing calibration? 2. It seems that the proposed uncertainty-guided logit modulation and the logit mixing approach only perform on the output logits of the last decoder layer. Do the authors try to implement them into all decoder layers' output logits? 3. In Table 1 and Table 4, it seems that the D-ECE of the out-domain is lower than the D-ECE of the in-domain scenarios. Why? Does this phenomenon have a further explanation?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See the above sections.

Summary

This paper proposes a calibrated detection transformer model, which equips the DETR variants with an uncertainty-guided logit modulator and a mixup augmentation for the classification branch of the detector. Specifically, the authors first quantify the uncertainty with the variance among the predicted logits from different decoder layers. Then logits with more uncertainty are suppressed. For mixup, it is similar to the common mixup used in detectors except the mix is between the average of the batch and each sample. Through experiments on various benchmarks, the authors demonstrate the effectiveness of the proposed method.

Strengths

1. This paper studies an important but underexplored problem, calibration in object detectors. 2. The proposed method is lightweight and does not require any network modifications, which makes it more likely to be generalized to more architecture and application scenarios. 3. The authors perform very comprehensive experiments and the results show the effectiveness of the proposed method.

Weaknesses

1. It is unclear why the variation in the output of decoder layers can measure the uncertainty of the class prediction. Is it based on intuition? And is there any experimental support? 2. As one of the two major contributions, logit mixing for confidence calibration is not technically novel enough. First, applying mixup for the classification branch of the object detector is considered a well-known technique [1][2]. Second, in [1], the mixup is considered a trick instead of a technical contribution. Finally, the difference in mixup between [1] and this paper is subtle. [1] Zhang, Zhi, et al. "Bag of freebies for training object detection neural networks." [2] Ge, Zheng, et al. "Yolox: Exceeding yolo series in 2021.

Questions

Please refer to the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A