Summary
This paper tackles the end-to-end multiple object tracking with transformers. It observes the issue of imbalanced distribution of the detection and tracking labels assignment. The majority of the queries are assigned to the track queries causing the detection queries are not be sufficiently supervised during training. The authors propose release-fetch supervision (RFS) to increase the proportion of detection queries. Besides RFS, the paper also uses the pseudo labels from other strong detectors to further boost the detection performance of the proposed tracker. The authors also explore augmentation such as adding noise to the reference point and creating track query groups. The experiments demonstrate slightly better performance than MOTRv2 without relying on external detectors on DanceTrack but get worse performance on MOT17 if compared to the original MOTRv2.
Strengths
1. The paper is well-written and clearly organized.
2. The observation regarding the disproportional assignment of track and detection queries is insightful, with a clear analysis that adds valuable understanding for the MOT community.
3. The proposed Release-Fetch Supervision (RFS) method is simple yet effective. Due to its simplicity, it could potentially benefit various Transformer-based MOT methods.
Weaknesses
1. Limited Scenarios for RFS Technique: The need for RFS is primarily in scenarios where there is more video data than image data for training, leading to a disproportionate assignment of track and detection queries. However, in most cases, such as in large-scale [1] and open-vocabulary MOT [2] tasks, the opposite is true, with more image detection data than video tracking data. In these cases, joint training with both image and tracking data is a common practice, providing sufficient supervision for detection queries, contrary to the paper’s analysis. It would be helpful if the authors could at least discuss these more common scenarios.
2. Besides the proposed RFS technique, other contributions, such as Pseudo Label Distillation (PLD) and Track Grouping Distillation (TGD), have been explored in prior works [3,4]. [3] also utilizes pseudo-detection labels to improve the detector training. [4] also, adds noise to the GT and learning the denoise process in the decoder. A short discussion of those similar ideas would be beneficial.
3. There is no comparison with simpler alternatives to RFS. A straightforward alternative would be to train detection queries jointly on image detection datasets along with video tracking datasets. For example, detection queries could be trained only on images when using image datasets. I would recommend an ablation study comparing RFS to this joint training approach to provide concrete evidence of RFS’s effectiveness relative to existing methods.
4. The paper uses the first five decoders to train with all queries, while the last one trains separately on detection and tracking queries. An ablation study could clarify whether alternative arrangements, such as using the first decoder for track queries and the remaining five for all queries, would impact performance. To deepen the analysis, I suggest testing multiple configurations (e.g., using the first 1, 3, or 5 decoders for mixed queries) and reporting on their effect on performance metrics. This would provide a more comprehensive understanding of the optimal configuration.
[1] Li, Siyuan, et al. "Tracking every thing in the wild." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
[2] Li, Siyuan, et al. "Ovtrack: Open-vocabulary multiple object tracking." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.
[3] Wu, Di, et al. "Spatial self-distillation for object detection with inaccurate bounding boxes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[4] Zhang, Hao, et al. "Dino: Detr with improved denoising anchor boxes for end-to-end object detection." arXiv preprint arXiv:2203.03605 (2022).