ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
Visual object tracking aims to locate a targeted object in a video sequence based on an initial bounding box. Recently, Vision-Language~(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this paper, we propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions and enhance tracking performance. To this end, we propose a novel reflection-based prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, a simple yet effective VL tracking framework is proposed and can be easily integrated as a plug-and-play module to boost the performance of both VL and visual trackers. Experimental results show that our proposed ChatTracker achieves a performance comparable to existing methods.
Paper
Similar papers
Peer review
Summary
This paper introduces a simple yet effective VL tracking framework based on a multimodal large language model, called ChatTracker. The main idea is to utilize the rich world knowledge in multimodal large language model to generate high quality language descriptions and improve tracking performance. Specifically, a reflection-based prompt optimization module is proposed to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. A large number of experiments demonstrate the effectiveness of the proposed method.
Strengths
1.the paper is well written and can be easily understood by the reader. 2.The proposed tracker can achieve SOTA performance and adequate ablation experiments are accomplished.
Weaknesses
- Line 127 mentions only the initial bounding box and search frame as inputs and does not mention template frames as inputs. However, the equation in line 128 writes template as input. - In section 4.1, MixFormer-L and ARTrack-256 as visual coders for ChatTracker-L and ChatTracker-B? I have a question, why not use mixformer-B/L or ARTrack-B/L uniformly as visual encoders. Could the authors please provide more results for mixformer-B and ARTrack-L for performance comparison. - In section 4.3 and Table 2, when applying the proposed module to other visual trackers, the paper does not describe how the linguistic description is fused with the visual features, could the authors please explain this process specifically. - The LaSOT_ext and OTB_Lang test datasets are also included in the VL tracking task, and the authors are asked to provide performance on these datasets to help enrich the experimental results in this paper.
Questions
- It is an interesting idea to apply large language models to the visual tracking field. The general practice would be to do fine-tuning operations on the downstream data so that the large language model is more adapted to the downstream task. I have a question, does the proposed approach require fine-tuning in the downstream tracking dataset?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The limitations of the method have been accounted for in this paper.
Thanks to Reviewer Bwy6
Dear Reviewer, Thanks a lot for your recognition of our work. Best wishes, The Authors
Summary
This paper proposes ChatTracker, a novel framework that leverages MLLMs for visual object tracking. The Reflection-based Prompt Optimization (RPO) module can narrow the knowledge gap between the VL tracker and the MLLM. ChatTracker also achieves SoTA performance on several tracking datasets.
Strengths
1. This paper designs a new framework that leverages MLLMs for visual object tracking. The entire framework's process is clear and easy to follow. 2. The approach performs favorably against the state-of-the-art methods on several datasets.
Weaknesses
1. SOTA trackers are missing in Table 1, such as OVLM (TMM23), MMTrack(TCSVT23), etc. There is a discrepancy between the data from UVLTrack in Table 1 and the data provided by the official source. 2. I want to know the model's performance on MGIT (NeurIPS '23) and OTB99_Lang (CVPR '17) to verify the generalizability of the method in different scenarios (short-term tracking and global instance tracking). 3. Why is it necessary to design ChatTracker-L and ChatTracker-B, but use different backbone networks? Also, I believe comparing the L model in Table 1 is unfair, as UVLTrack provides the L model, and annotations are needed for models of different scales. 4. For analysis of language descriptions generated by ChatTracker, if the natural language description includes background information, is it reasonable to only crop out the target for calculating image-text similarity? 5. I believe the phrase "via Chatting with Multimodal Large Language Model" in the title overstates the contribution of the entire framework.
Questions
Please refer to weaknesses.
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes
Response to Reviewer z5a5（part2）
Due to the word limit in the rebuttal, we provide the remaining citations here. [1]OVLM: Huanlong Zhang, Jingchao Wang, Jianwei Zhang, Tianzhu Zhang, Bineng Zhong "One-stream Vision-Language Memory Network for Object Tracking" TMM 2023 [2]MMTrack: Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, Xianxian Li"Towards Unified Token Learning for Vision-Language Tracking" TCSVT 2023 [3]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439–461, 2021. [4]Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763–13773, 2021. [5]Shiyu Hu, Dailing Zhang, Meiqi Wu, Xiaokun Feng, Xuchen Li, Xin Zhao, and Kaiqi Huang. 2023. "A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and Causal Relationship." Paper presented at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), Track on Datasets and Benchmarks. [6]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226. [7] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151–23160, 2023. [8]Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9697–9706, 2023. [9]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang. Unifying visual and vision-language tracking via contrastive learning, 2024
Official Comment by Reviewer z5a5
I read the author's experiments and replies to all reviewers. I am grateful for the author's efforts during the rebuttal, which largely answered my doubts. There are still some small details that need to be paid attention to. On the one hand, MGIT provides flexible and diverse semantic labels. This benchmark is somewhat closer to the "chat" feature than current VLT benchmarks. Therefore, I suggest that the author conduct further experiments on this benchmark to analyze the effects and limitations of the method under different granularities and evaluation mechanisms, and consider adding this part of the analysis to the article to highlight the motivation and theme of the article. In addition, please note that the author cited the wrong MGIT literature (Reference 5) in the second reply window for me. I am unsure what the relationship is between the work cited by the author and the rebuttal itself. Finally, NeurIPS officially recommends that the reply to each reviewer during the rebuttal should not exceed 6,000 characters (only one window). I noticed that the author's reply to 2 reviewers had exceeded the limit of one window. In a sense, this is unfair (for other authors of the same period who only use one window for rebuttal). I hope the authors can pay attention to these details. In summary, I would like to temporarily maintain my original score and listen to the opinions of other reviewers on the authors' rebuttal.
Thanks and Response to Review z5a5
Dear Reviewer: Thank you again for taking the time to review our work and for providing us with valuable feedback. We now address your additional comments. ***Q1. MGIT provides flexible and diverse semantic labels. This benchmark is somewhat closer to the "chat" feature than current VLT benchmarks. Therefore, I suggest that the author conduct further experiments on this benchmark to analyze the effects and limitations of the method under different granularities and evaluation mechanisms, and consider adding this part of the analysis to the article to highlight the motivation and theme of the article.*** We appreciate your valuable suggestion. MGIT is a high-quality vision-language benchmark. The MGIT dataset focuses on the impact of text annotations at different granularities on tracking performance. However, our ChatTracker only uses the initial bounding box as input and does not utilize additional text descriptions as input. Although we agree that further applying our proposed method to MGIT is very interesting, employing different text annotation granularities is beyond the scope of this paper. In the rebuttal, we have provided ChatTracker's results on the MGIT dataset with the initial bounding box and we will include these results in the revised manuscript. Thank you again for your valuable suggestion. We will further investigate these text inputs into our framework in future work. ***Q.2 In addition, please note that the author cited the wrong MGIT literature (Reference 5) in the second reply window for me. I am unsure what the relationship is between the work cited by the author and the rebuttal itself.*** Thank you for your careful reading. We have corrected Reference [5] in the comment. ***Q.3 Finally, NeurIPS officially recommends that the reply to each reviewer during the rebuttal should not exceed 6,000 characters (only one window). I noticed that the author's reply to 2 reviewers had exceeded the limit of one window. In a sense, this is unfair (for other authors of the same period who only use one window for rebuttal). I hope the authors can pay attention to these details.*** Thanks again for your advice. For additional comments, we follow the [NeurIPS 2024] Clarification on author rebuttal email: > - Comments to paper and reviews will be fine. Comments can be seen in time. Please set the readers correctly when you post them. Reviewers are not required to take comments into consideration. To better discuss the issues, we responded to the longest review (up to 11 questions) in one additional Comments and provided relevant references in another Comment. We strictly follow the policy and instructions given by the NeurIPS 2024 PCs email. This email is sent to all authors and we don't think there is any fairness issue. We hope our responses have fully addressed your concerns. If you have any other questions or need further clarification, please feel free to let us know. Thank you very much! Best wishes, Authors
Summary
The paper proposes a novel Multimodal Large Language Model framework to improve the vision-language visual tracking performance. By introducing the reflection-based prompt optimization module, the tracking prompt can be iteratively refined via tracking feedback. The proposed method shows state-of-the-art results compared to prior vision-language trackers and visual trackers.
Strengths
- The paper investigates the multimodal large language models for the visual tracking problem. - By showing the limitations of prompts created by MLLM and manual annotation for visual tracking, the paper introduces a novel iterative prompt generation via chatting and semantic prompt verification. - The paper shows adequate evaluation comparing the proposed method with both vision-language trackers and visual trackers.
Weaknesses
The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module and their performance effect. The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.
Questions
Please refer to the weakness.
Rating
6
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have mentioned the limitations.
Summary
The paper proposes a new Visual-Language (VL) tracking framework called ChatTracker, that integrates MLLMs into VL tracking through iterative refinement of text prompts for VL trackers. The text prompts optimized using the proposed Reflection-based Prompt Optimization (RPO) module are more accurate than manual text annotations in the datasets and improve the tracking performance by taking into account both foreground and background objects.
Strengths
- The paper identifies a clear gap for Visual-Language trackers and how manually annotated or naively generated text prompts can provide sub-optimal results for language-assisted visual tracking. - The proposed framework combines the power of a strong MLLM like GPT-4V and a GVLMs like Grounding DINO-T to formulate the iterative refinement procedure for generating better text prompts for tracking tasks. It uses a GVLM powered by the optimized prompts in conjunction with a visual tracker to get a set of proposals. It also incorporates GVLM generated region proposals to train its Foreground Verification module, which not only does foreground classification but scores candidate regions with how they overlap with background proposals. While different parts of the framework are pre-existing and straight-forward, the paper uses them in a new way to improve visual tracking performance in relevant benchmarks. - The method is plug-and-play, and any visual tracker (regardless of whether it is a VL tracker or not) can be used as the baseline tracker in the Semantic Tracking Module of the framework. - The experimental results and ablations support the claims in the paper. - The paper is well-written, well-organized and clear.
Weaknesses
- The proposed framework does not take into account the potential temporal changes in the video that might require a more optimal foreground/background text prompt to proceed. For example, in Fig. 3 we see the framework landing on ‘hands, bar’ as the positive prompt, which focuses on hands being attached to the bar, which may not be true for the rest of the video. The background text of the video is also never updated. In fact, neither the GVLM that the framework heavily depends on nor the Foreground Verification module have any temporal components. Therefore, the framework would have to heavily rely on the baseline tracker in this aspect. - The quality of the region proposals coming out of the Semantic Tracking Module is un-addressed. How often GVLM proposals are preferred to the visual tracker results (unless the prompt optimization fails as mentioned in Appendix A)? How dependent is the final performance on how well the Foreground Verification module is performing (since it’s only been trained on the LaSOT training set)?
Questions
- The potential limitations mentioned in the weaknesses should be addressed or discussed in the paper. Temporal changes to target and background are the main challenges in visual tracking and the proposed framework does not add any mechanisms to address these even though it might heavily depend on GVLM results for its answers. - Related to the above point: Have the authors done any experiments where foreground/background prompts were updated while the video is being processed? What were the insights? - As mentioned above, how often does the visual tracker proposal get picked over a GVLM proposal and vice versa when both exist in the pool going into the Foreground module? Or does the visual tracker proposal mostly get used when there are no GVLM proposals? - Related works section can be expanded a bit. An important motivation for this paper is the disadvantages brought by the manual textual annotations in the VOT benchmarks (that have them). For the broader audience, the authors could add a short section on related benchmarks and if they have manual text annotations, explain how those came about (why are they low quality?). Line [300-301] mentions the prompt “swing swinging above a man in black pants” - which almost sounds like a generated text prompt, having this context in the paper itself would be helpful. - Equation (6) - What’s the purpose of $T^i_{pos}$ in this equation? - Lines [152-153] - What happens if a word matches to no proposals? Can we add that to negative words? Why not? - Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix. - Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals? - Please use $P_{norm}$ or NP consistently across tables. Also, for a broader audience you should mention what the metrics are even if it's in a couple sentences. - Some tables report NP and P, others one or the other. It would be better to report consistently even if it’s just across smaller tables. Moreover, baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1. If I’m missing some details, please let me know. - Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Authors did address some limitations of their method in the Appendix. Please see my discussions about other potential limitations in the weaknesses and questions sections above.
Response to Reviewer gPPX（part2）
Due to the word limit in the rebuttal, we provide the remaining responses here. ***Q7 Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix.*** No, we do not sample positive samples from the visual tracker's results during training. We only randomly choose a target patch from other frames of the same video as the positive sample. In Line 202, we use the trained $ f(.) $ (which is a neural network) to calculate the foreground score of the visual tracker's result. We will clarify this in the revised manuscript. ***Q8 Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals?*** - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Is it generally lower with the proposals? For the IoU threshold for foreground(i.e. $\theta_1$ in equation (5) ), we set $\theta_1$ to 0.3. This is to ensure there are enough positive words $T_{pos}^i$ as feedback during the early iterations, helping the iterative optimization process converge quickly. For accepted IoUs (i.e., $\epsilon$ in line157) ，we choose the IoU threshold based on the experiments. We tested on the LaSOT dataset, and the results are shown in the table below. The results indicate that an IoU threshold (i.e., $\epsilon$ in line157) of 0.4 leads to the best results. **Table.2 The performance of chattracker on Lasot dataset with different IoU thresholds** | IoU threshold | 0.3 | 0.4 | 0.5 | 0.6 | | ------------- | ----- | ----- | ----- | ----- | | AUC | 69.87 | 71.68 | 71.54 | 70.92 | - Do you have any statistics for what the accepted IoUs are from the foreground verification? Yes, the average accepted IoU is 0.79. We calculated the distribution of accepted IoUs on the LaSOT dataset, and the results are shown in the table below. **Table.3 Data distribution of IoU values on the Lasot dataset.** | IoU | [0,0.4) | [0.4,0.6) | [0.6,0.8) | [0.8, 1.0]| | ------------------ | ------- | --------- | --------- | ---------- | | number.of.sequence | 20 | 14 | 52 | 194 | Thank you for your question. We will add the relevant results and analysis in the revised manuscript. ***Q9 Please use $P_{norm}$ or NP consistently across tables. Some tables report NP and P, others one or the other. It would be better to report consistently even if it’s just across smaller tables.*** Thanks for pointing this out. In the revised manuscript, we will use $P_{norm}$ or NP consistently and report metrics consistently. ***Q10 baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1.*** Thanks for pointing this out. In Table 1, the baseline trackers (i.e., JointNLT[3] and UVLTrack[4] ) are initialized using both natural language and the initial bounding box to compare with the best-performing model. In Table 3, the baseline trackers (i.e., JointNLT and UVLTrack) are initialized using only natural language. The purpose is to compare the effectiveness of ChatTracker-generated text versus manually annotated text for Vision-Language trackers without the influence of the initial bounding box. We will clarify this in the revised manuscript. ***Q11 Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.*** Thanks for pointing out this typo. CiteTracker[5] is not included in this comparision and we reuse the legend picture in this figure mistakenly. We revise the manuscript accordingly. For clarification, we have placed the revised figure in the .pdf file.
Response to Reviewer gPPX（part3）
Due to the word limits, we provide the remaining citations here. [1]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439–461, 2021. [2]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226. [3] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151–23160, 2023. [4]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang. Unifying visual and vision-language tracking via contrastive learning, 2024 [5]Xin Li, Yuqing Huang, Zhenyu He, Yaowei Wang, Huchuan Lu, and Ming-Hsuan Yang. Citetracker: Correlating image and text for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9974–9983, 2023.
I want to thank the authors for all their efforts in responding to my and other reviewers' questions and comments. The analyses the authors provided were really helpful for clarifying the mechanics of their method for me. After reading the other reviews and the authors' responses to them, I'm more comfortable with my assessment of the paper. Thanks to the authors for including all this new information in their revised manuscript. I do agree with reviewer z5a5's comment that it would very valuable to analyze the proposed method on MGIT's multi-granular annotations.
Thanks to Reviewer gPPX
Dear Reviewer, Thank you once again for your valuable comments. We also agree with reviewer z5a5's advice on this further research direction. It would be very interesting to extend our framework in MGIT's multi-granular annotations and dataset. Since analyzing the effect of multi-granular annotations is non-trivial and beyond the scope of this paper, we will investigate this direction in our future work. Thank you very much! Best wishes, The Authors
Comment
Thanks to the author's reply, my concerns were largely addressed. I will maintain my original rating score.
Thank the authors for their response. I am satisfied with the answer. After carefully reading other reviews, I will maintain the score.
Thanks to Reviewer aCWh
Dear Reviewer, Thank you very much for recognizing our work. Best wishes, The Authors
Decision
Accept (poster)