Type-to-Track: Retrieve Any Object via Prompt-based Tracking

Summary

This paper provides a new paradigm for MOT, using natural language descriptions to track objects of interest in videos. To accomplish it, the authors collected and annotated various object and scene descriptions. Meanwhile, the new evaluation protocols are also formulated following the evaluation of the original MOT. The authors also provide a new method, showing good performance in experiments.

Strengths

* This work has an intuitive and good motivation to solve MOT using natural language. * This paper is well organized, which contains collecting new benchmarks, designing effective methods, and conducting experiments. * More importantly, the authors provide many visualization results, which are interesting.

Weaknesses

* First of all, the main idea of ‘Type-to-Track’ has a similar idea to Referring Multi-Object Tracking [1] (released at Arxiv2023.03, accepted at CVPR2023), which both use language description to prompt multiple objects. It is important to discuss and compare the related work. * I’m curious about the annotation details of the descriptions. Are they manually annotated? Or they are annotated by the current techniques of GPT? * In Table 3, does the number of words represent the number of descriptions? Or the words are words of descriptions? * The evaluation of caption prompts confuses me. The crucial words of created captions have one-to-one object matching with the grounded objects. So each object is evaluated individually? * This paper is not easy to follow and sometimes overclaimed on some details, e.g., third-order tensor modeling (using three inputs), and correlation simplification (using separate attention). Besides, the authors define many new concepts but lack details and explanations, e.g., the single-stage end-to-end framework. How to deal with new entrance or exit objects? * How about the visualization performance of the proposed method? [1] Referring Multi-Object Tracking, CVPR2023.

Questions

See the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have already provided the limitations in the paper.

Summary

This paper systematically tackles the prompt-based multiple object tracking problem by (1) introducing a dataset for the Grounding Multiple Object Tracking Task (GroOT), (2) introducing a new tacking paradigm (type to track), and (3) developing a new language-based multiple object tracking model (MENDER).

Strengths

This paper systematically tackles the prompt-based multiple object tracking problem with a variety of relevant contributions. We list the strengths of the paper below, expanding each paragraph with comments on the relevance of each contribution. ## A new tracking paradigm (type to track) This paper introduces the type to track paradigm, which consists in prompting models with text descriptions and obtaining all the tracklets corresponding to it. This paradigm is of great practical importance, as it allows users to query videos with text prompts to retrieve the desired objects. ## A dataset for the Grounding Multiple Object Tracking Task (GroOT) This paper introduces GroOT, a dataset which extends MOT17 and TAO with a variety of text captions to learn models suitable for the type to track task. GroOT supports 5 settings, including tracking by : - category name - category synonyms - category definition - retrieval prompts: 3 retrieval prompts per video are provided. the prompts are sequential and each prompt could refer to multiple tracklets at the same time (one-to-many). - tracklet captions: one caption here corresponds to one tracklet, where the caption consistently describes the object throughout the tracklet. In the TAO subset, the caption describes the object appearance. In the MOT17 subset, 2 types of captions are provided for each object, one describing the action and the other the appearance. ## Class-agnostic evaluation metrics To evaluate the type to track performance on GroOT, this paper introduces CA-MOTA and CA-IDF1, two class-agnostic tracking metrics. ## A new language-based multiple object tracking model (MENDER) The authors also introduce MENDER, a proof-of-concept tracker capable of retrieving tracklets based on text prompts. MENDER is composed of an image encoder, a text encoder, and a transformer decoder. The **image encoder** encodes in parallel the image at time $t$ and the cropped bounding boxes for tracked instances at $t-1$. The **text encoder** is a RoBERTa encoder that encodes the text prompts. Cross-attention between image features, track embeddings and text embeddings produces the inputs to the transformer decoder, which is trained with a GIoU loss to detect the objects based on the track and text queries. The proposed model is simple enough and addresses the type to track task effectively.

Weaknesses

## Missing baselines The `category_name` setting could be tackled by any multiple object tracking algorithm. The current version of the paper lacks an absolute estimate of the tracking performance of MENDER in comparison to state-of-the-art tracking algorithms. It is possible that the current design produces suboptimal tracking performance, and only empirical results can show whether this is true or not. The paper would thus benefit from a comparison of MENDER to SOTA tracking-by-detection methods on the `category_name` setting. ## Class-agnostic evaluation metrics **Evaluation of retrieval performance.** The proposed metrics focus more on evaluating the tracking performance of retrieved tracklets rather than the retrieval performance itself. Although the limitations section already indicates this as a limitation, I find this to be a weakness since a benchmark for tracking retrieval should specifically provide suitable metrics for the evaluation of such task. **CA-HOTA.** Although MOTA and IDF1 are established tracking metrics, the community has reached an agreement that they cannot solely represent the performance of a tracker. The HOTA metrics (DetA, AssA, and HOTA) disentangle different components of the tracking pipeline and it would be beneficial for the proposed benchmark to also introduce a class-agnostic version of such metrics. ## Missing ablation on the tokenizer and the text encoder The authors adopt the RoBERTa tokenizer and text encoder. However, they do not provide an explanation for choosing it over other popular architectures, e.g. CLIP. The paper would benefit from a comparison to other text encoder architectures and initializations.

Questions

I find the paper well-written and relevant to the community. However, the weaknesses listed above somewhat hinder the contributions of this paper. In my opinion: 1. a comparison on the `category_name` setting to SOTA tracking-by-detection methods on GroOT is essential for this paper 2. retrieval-specific metrics would complement the contribution of the dataset, which as of now does not allow directly assessing the retrieval task performance 3. the paper would benefit from ablations on the tokenizer choice. I would consider increasing my rating and supporting this paper if my current concerns are addressed, in particular points 1 and 2.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately pointed out limitations of their work.

Summary

This paper delves into language-grounded multi-object tracking, a highly important task that has been underexplored due to the lack of suitable datasets. Firstly, the authors propose a novel dataset that extends existing Multiple Object Tracking (MOT) datasets with additional captions describing appearance or action of each tracklet, as well as retrieval prompts describing main content of a scene. To evaluate language-grounded multi-object tracking, the authors consider five scenarios with different types of text input (prompts) and suggest class-agnostic MOTA and IDF1 that decouple the classification accuracy. Also, the authors introduce MENDER, an efficient single-stage model that autoregressively tracks objects relevant to the prompt. The effectiveness of this model is validated through a comparison with a two-stage baseline model consisting of MDETR and TrackFormer.

Strengths

* This paper addresses a significant problem, aiming to retrieve multiple objects relevant to an input prompt from a video. This setting is novel compared to existing spatio-temporal video grounding methods (e.g., TubeDETR[1]), which are limited to retrieving only a single object. The proposed task can not only serve as a user-friendly video analysis application but also work as a video perception foundation model for fine-grained video understanding. * The proposed dataset is significant to further research on video-language model. * The proposed diverse scenarios for evaluating grounded MOT are interesting and valid. [1] A. Yang, et al., TubeDETR: Spatio-Temporal Video Grounding with Transformers

Weaknesses

## Inconsistent annotations * In the released dataset, some inconsistent annotations are observed. How quality assurance is achieved in the manual annotation process? Is there criteria to accept or reject the captions from the annotator? * In Sec 3.1, L116 states that the appearance-related caption and action-related caption are provided in order if the tracked object is person. However, in the annotations, the captions are randomly ordered. For example, in mot17_train file, object whose instance_id is 170 has captions of ["person walking on sidewalk", "person wearing black jacket"], where action-related caption is the first in the caption list. * Also, object whose instance_id is 9 has three captions: ["a person sitting on a bench", "child in stroller", "baby stroller"]. And, in TAO dataset, there are some instances with six captions. What is the reaons of such variance in caption annotations? * I would rather suggest to have separate field for appearance described caption and action described caption. ## Heavy dependence on TAO dataset * MOT Challenge datasets include more challenging tracking cases caused by complex motion and occlusion, compared to TAO dataset. However, its scale is very small (e.g., only 7 videos for training). * I think captioning other human-centric MOT datasets with complex motion would improve the importance of the dataset. For example, human-centric datasets with dense FPS video (e.g., DanceTrack[2] and HiEve[3]) are the possible candidates. ## Unclear evaluation metrics * In the paper, I cannot find the details of matching GT-prediction pairs which are necessary to compute FN and FP. * What is the reason of cls exists in Eq. 1 even though we are not considering class of tracklets? Isn't that the class of tracklet do not exist in the case of grounded MOT? ## Lack of experimental details * Unclear details about implementing the two-stage baseline model which is based on the MDETR and TrackFormer. TrackFormer jointly performs object detection and tracking, but how MDETR is integrated into TrackFormer to achieve grounded object detection? [2] P. Sun, et al., DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion, CVPR 2022 [3] W. Lin, et al., Human in events: A large-scale benchmark for humancentric video analysis in complex events, arxiv 2020.05

Questions

Major concerns are raised in the weakness section. Here are some minor questions: * Why only text prompt for every frame exists, but there is no prompt-tracklet paired annotations in the MOT files? * In Table 3, dataset statistics for MOT17 videos are not correct. As far as I know, there are 7 unique videos for the training and test subsets, respectively. Also, your released dataset seems using the GT tracklets from the unique videos. * How do you have the mot17_test annotation file? Is the tracklet obtained from GT or just prediction value? * TAO dataset is the most important one as the proposed dataset is built on top of this one, but why it is not included in Table 1? * Why do you design the MENDER as online tracking model that runs autoregressively, instead of predicting multiple spatio-temporal tubes at once? * In Table 4 and 5, is mAP the absolute value or percentage value? * In Table 4, some bolded values are not the best in the metric.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The limitations and future works are well addressed.

Summary

The paper introduces a novel task and dataset that involves tracking multiple objects in videos using language prompts. It proposes two evaluation protocols: retrieval prompts, where the prompts can change during the tracking process, and caption prompts, which remain consistent. The paper also presents class-agnostic evaluation metrics by modifying MOTA and IDF1. Additionally, a novel framework called MENDER is introduced to address the grounded MOT task.

Strengths

1. The significance of the proposed task and dataset is well-defined, as existing tracking datasets using language prompts have been focused on single-object tracking (SOT) tasks. 2. The evaluation protocol for the proposed dataset consists of five distinct protocols, each with clear explanations of their purpose. This dataset stands out from previous tracking datasets due to its diverse range of prompts. 3. The proposed MENDER framework demonstrates efficient and effective performance in addressing the grounded MOT task. 4. The paper is overall well-written and easy to follow.

Weaknesses

1. Referring video segmentation datasets like Ref-YTVOS and Ref-DAVIS are also related to this grounded MOT task since they also aim at tracking a referred target in a video. I think these datasets should also be mentioned in the paper as a comparison benchmark. 2. Moreover, I think Refer-KITTI [1] could also be discussed in the paper as it also utilizes referring expressions for multi-object tracking. 3. The experimental comparisons in the newly proposed dataset are limited. It would be beneficial to include additional methods such as TransRMOT [1] and popular MOT baselines like ByteTrack [2] for a more comprehensive evaluation of the dataset's performance. Adding these comparisons would provide stronger validation for the dataset. [1] Referring Multi-Object Tracking, https://arxiv.org/pdf/2303.03366.pdf [2] ByteTrack: Multi-object tracking by associating every detection box, ECCV 2022

Questions

As discussed in the weaknesses, I hope the authors would supplement some dataset comparisons and experimental validations.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have adequately addressed the limitations of their work.

Summary

The authors have created a new dataset by combining existing Tracking datasets and adding language prompt annotations. They also proposed a method by extending and modifying MDETR and Trackformer.

Strengths

1. The dataset quality looks sound, diverse, and sufficiently large enough to train Deep networks. 2. The dataset is well documented and the task is clearly defined. 3. The proposed solution is simple and should be easy to extend by new researchers.

Weaknesses

1. The authors claim that Type-to-Track is a novel paradigm. This is incorrect as there exists Datasets, e.g., Refer-Youtube-VOS, Ref-DAVIS, Ref-KITTI, having the same challenge. 2. The related work in incomplete. The authors fail to compare their proposed dataset with existing datasets from Video Instance Segmentation(VIS) literature. Note that, MOTS and VIS are the same task with different nomenclature. 3. The modeling part is unnecessarily written in a complicated way. It is a simple extension of DETR architecture to model the specific solution. Instead of complicating the method with notation, The authors should try to simplify the method section. 4. There are too many evaluation metrics. There are 5 different sub-categories of the same task with separate evaluations on the 3 base datasets. So there are in total, around 12*6=72 different evaluation metrics.

Questions

1. How is this dataset different from Refer-Youtube-VOS, Ref-DAVIS, and Ref-KITTI? 2. How did you evaluate MOT17 and MOT20 datasets? The test ground truth is not public for these datasets. What is MOT17 subset in Tab. 4 and 5? 3. Why is there no unifying evaluation metric. The authors report evaluation separately on all the different datasets they worked on. A new dataset needs to be either compatible with previous ones, or propose a new metric. Without a single metric, new line of researchers would find it difficult to adapt the dataset. 4. How many human hours was needed to annotate the dataset?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

yes

Poor dataset quality

I thank the authors for the rebuttal, and for addressing some of my concerns. I was originally positive towards this submission, I think that the community will greatly benefit from an MOT dataset with carefully-curated language captions and from a method to solve the type-to-track problem. After a more careful inspection of the dataset, I have however changed my mind. My main concerns are that: (i) the quality of the annotations is extremely poor; (ii) the annotation policy is inconsistent; (iii) the authors claim to have hired a team of ten professional annotators, but the quality of the captions seem to point out that a visual-language model was used for captioning; (iv) the proposed tracking algorithm trained on GroOT performs significantly worse than industry standards such as ByteTrack. - **Underwhelming performance of MENDER.** Regarding (iv), refer to the authors reply A26. While I appreciate that they disclosed the performance of MENDER, it also shows that the proposed algorithm is not particularly effective. - **Lack of retrieval metrics.** I believe that a paper introducing a new task (type-to-track) and a new dataset (GroOT) should provide metrics to evaluate the effectiveness of a given method at solving the type-to-track task. Since type-to-track is a retrieval task, I find it hard to accept a paper that does not introduce metrics to evaluate the retrieval performance. - **Poor quality of annotations.** I will here report some examples based on the supplementary video shared by the authors themselves. 1. _Timestamp 1:50._ inconsistent annotations in the same frame. 1.a) Three very similar bikes are described as "bicycle", "a black bike", "two black wheels on a bike". First of all, all bikes are black. I wonder how one could evaluate type-to-track performance (i.e. retrieval) if descriptions do not follow a common pattern. Second, "two black wheels on a bike" is an odd description and is the first example that reminds of captions from visual-language models. 1.b) "A man riding a bicycle", "a man wearing a helmet", "a person in a pink turban". They are all riding a bicycle, and all should be retrieved by the prompt a man riding a bicycle. Yet, proper retrieval metrics would penalize retrieval of "a man wearing a helmet" and "a person in a pink turban" when using such a prompt. Using well-defined patterns such as "a man riding a bicycle and wearing a helmet" and "a person riding a bicycle in a pink turban" would assist the evaluation of type-to-track methods. 2. _Timestamp 1:58._ Object ID 5274 was annotated as "the headlight of a car". I wonder how a human annotator could come up with this kind of description for what is seemingly a black car. 3. _Timestamp 2:13._ The authors say that in MOT they provide two distinct set of captions, i.e. appearance captions and action captions. However, the annotations shown here are really confusing and inconsistent. In the same frame, appearance captions sometime represent actions, e.g. some objects are labeled as "a man walking in a store", "woman walking on a sidewalk", "woman walking down the sidewalk" and others like "a man wearing a black shirt", "a little girl in a green shirt and white pants", "a man wearing a white hat". 4. _Timestamp 2:25._ The "man talking on a cell phone" is actually just touching his chin. Overall, my greatest concern is the **underwhelming quality of the annotations**. A by-product of the poor quality of the annotations is that it is really hard to evaluate the performance of the type-to-track task. In my opinion, a dataset designed for a task should provide all the tools to evaluate such task, while these annotations make it impossible. While there is no right way to annotate visual-language datasets, this dataset does not seem to be going in the right direction. In my opinion, the dataset would need significant further refinement before getting published. However, I am waiting to hear the opinion of other reviewers and keep my mind open to different opinions.

About the comparison with other approaches in catogory_name setting

I would like to express my appreciation to the authors for providing a rebuttal and for addressing some of my concerns. However, my concern about the comparison with other baselines is not fully resolved. As suggested in A26, the performance of MENDER on `category_name` setting is far behind the SOTA tracker TrackFormer and ByteTrack. It is understandable that ByteTrack's superior performance could be achieved from their task-specific design for associations. However, it is hard to find reasons why MENDER's performance falls short compared to TrackFormer, which is essentially a simple extension of Deformable DETR. Considering that MENDER benefits from additional training with more language descriptions, one would naturally expect it to gain advantages from the increased annotations. Rather, the result in A26 rather gives an impression that the proposed method suffers from generalizing to a more standard `category_name` setting.

Regarding the Response

## Overall response Thank you for clarifying my concerns. I agree that the proposed dataset is significant at this stage for researching multi-modal MOT and the proposed task is also interesting. However, the current version of the dataset (only v0.2 is available to the public on 2023-08-14) is not yet satisfying and I would like to keep my score. ## Remaining concern about annotations Please note that as the v0.2 dataset is only available at this moment, my review is based on that. ### About captions describing the human action The captions are restricted to describing the appearance of an object or person, instead of action. In the MOT17 train set, there are only 14 words related to verb - filtered by NLTK - ['carrying', 'crossing', 'eating', 'holding', 'is', 'parked', 'pushing', 'riding', 'sitting', 'standing', 'striped', 'talking', 'walking', 'wearing']. Among these words, only some words are related to describing the actions of a person. Based on this observation, I cannot agree with your contribution point (L8) "textual captions describing their **appearance and action** in detail" which was appealing to me. ### Limited diversity of captions In MOT17 train set, among 175540 captions, there are 294 unique captions and its distribution is severely long-tailed, where top-5 frequent captions occupy 22.6%. Even such top-5 frequent captions contain the same meaning (1, 2, 4) and their description is not informative as well. I guess this is caused by the characteristics of the MOT17 dataset which focuses on the scene including pedestrians. However, I believe this would be handled effectively by carefully annotating the captions (e.g., describing the fine-grained appearance) or separately annotating the captions about appearance and action (e.g., walking or standing). 1. 'a person walking on the sidewalk' 2. 'person walking on the sidewalk' 3. 'person standing on sidewalk' 4. 'person walking on sidewalk' 5. 'person wearing black shirt'

Official Review by Reviewer j89C

Overall, the rebuttal from the author side solves most of my concerns. So I decide to keep my original rating for it. I would encourage the authors to conduct a comparison&discussion with RMOT in the revised version, since RMOT (developed in the last year) should be much earlier than this work. Moreover, I notice that other reviewers mentioned about the quality of annotation. I would also suggest the authors to address since it is really a big bet for this work.