TAPTRv2: Attention-based Position Update Improves Tracking Any Point

In this paper, we present TAPTRv2, a Transformer-based approach built upon TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from DEtection TRansformer (DETR) and formulates each tracking point as a point query, making it possible to leverage well-studied operations in DETR-like algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its reliance on cost-volume,which contaminates the point query\'s content feature and negatively impacts both visibility prediction and cost-volume computation. In TAPTRv2, we propose a novel attention-based position update (APU) operation and use key-aware deformable attention to realize. For each query, this operation uses key-aware attention weights to combine their corresponding deformable sampling positions to predict a new query position. This design is based on the observation that local attention is essentially the same as cost-volume, both of which are computed by dot-production between a query and its surrounding features. By introducing this new operation, TAPTRv2 not only removes the extra burden of cost-volume computation, but also leads to a substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves state-of-the-art performance on many challenging datasets, demonstrating the superiority

Paper

Similar papers

Peer review

Reviewer zD8r5/10 · confidence 4/52024-07-12

Summary

This paper proposes TAPTRv2, an improved version of TAPTR, which addresses the Tracking Any Point (TAP) task. TAPTRv2 introduces a novel attention-based position update (APU) operation that leverages key-aware deformable attention to refine point query positions. This operation removes the need for cost-volume computation, which was previously used in TAPTR but had the drawback of contaminating point query content features.

Strengths

1. The introduction of the APU operation is a novel improvement over TAPTR. By directly using attention weights to update point query positions, TAPTRv2 avoids the contamination of content features and leads to more accurate tracking results. 2. This work simplifies the overall pipeline and makes it more conceptually straightforward. This will help scale up the training process and accelerate the development of more practical TAP algorithms. 3. The paper conducts extensive experiments on multiple datasets and provides ablation studies to analyze the effectiveness of each key design.

Weaknesses

1. Due to the main contribution of simplifying the algorithm structure, speed comparison should be very important, but the author overlooked this point 2. Enhance experiments by including more common datasets (such as PointOdyssey) to assess generalization, and incorporate visual comparisons for long video sequences to demonstrate stability and accuracy of TAPTRv2. 3. Figure 4 does not show the result of Cotracker, which is inconsistent with the line 295. And it is preferable for Figure 4 to provide more key points to demonstrate the advantages of proposed method.

Questions

See Weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

My main concern is the lack of speed comparison for the proposed TAPTRv2. It is necessary to provide detailed efficiency comparisons for each module, such as the Effective Utilization of Cost-Volume and the APU operation.

Reviewer wowB6/10 · confidence 4/52024-07-12

Summary

The paper proposes an improved version of TAPTR, a DETR-based approach for point-based tracking in videos. TAPTR-v2 removes the dependency of TAPTR on the cost-volume, using local window features to define the keys and values to be used within the attention blocks of the DETR's transformer decoder. This simplification offers better computational complexity and improves performance.

Strengths

The paper is well written and motivated, with a deep analysis on the pros and cons of TAPTR and how it can be improved. The analysis in l. 163 regarding the Cost-volume with an ablation study is of great interest. Based on that, I liked the idea of building an attention block that directly uses the features as keys and values rather than exploring the standard cross-attention method w.r.t. the cost-volume. The intuition behind the APU block is clear and well presented, with Figure 3 giving a reproducible description of the proposed unit.

Weaknesses

I am a bit confused about the part regarding the feature preparation, in particular with Eqn (1). How is this initialized in the first frame and how is this updated after each frame? It is my understanding that the content and feature parts are those corresponding to the target points from the previous frame (i.e. t-1), although I am not sure I understood that correctly. If this is the case, I am not sure what happens if only one point is being tracked. Similarly, I wonder if the authors have considered adding as queries all former embeddings for a target point within a feasible window. For example, if a single point x is being tracked, then f_e and l_e would be {f_1, f_2, ..., f_t-1} and so forth for l_e. In such case, how would this combine for a multi-point tracking approach? Please clarify in the rebuttal. I am a bit concerned about the results in Table 2. While it is to acknowledge the simplification that TAPTRv2 offers w.r.t. TAPTR, the results are pretty much on par with those of the latter. Such marginal differences call for a study on the statistical significance of the results. While I still believe the method deserves its merit, I would like the authors to further explore why the results do not improve over TAPTR. I miss a computational complexity analysis between TAPTRv2 and TAPTR, as well as the contribution to the complexity of the different alternatives to the attention blocks in both. Where is the gain coming from? What's the computational complexity/performance tradeoff? I would also like to see a real example of the offsets considered by the method for the APU unit. Some visual examples of which points are selected as neighbors in the example in Fig.4 would be very interesting. Also, how many neighboring points are considered? I understand that this is 4, but I could not find that in the paper (I wonder if I missed that part), and I wonder and whether the authors considered studying a broader set of offsets, or a local grid within the neighboring pixels. Some proof-reading is needed: l. 152: dot-production --> product l. 250: Table 3 --> Table 2?

Questions

All my questions are included above. Overall I find the paper interesting and the questions above would help me set a final rating for the paper.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Authorsrebuttal2024-08-07

Some others that the reviewer may need.

Limited by the space of the above rebuttal block, we put some other information here. We apologize for the inconvenience. ## Summary of review ratings. | Reviewer | yaYP | wowB | zD8r | zGVx | | :--- | :---: | :---: | :---: | :---: | | Rating | Weak Accept | Weak Accept | Borderline Accept | Borderline Reject | | Confidence | 4 | 4 | 4 | 3 | ## Some statements that need the reviewer's further clarity. Some of the reviewer’s statements are not very clear, so we have not responded to them currently. If the reviewer wants to discuss these points in the discussion phase, we'd appreciate if the review could further clarify the following questions. 1. What does the 'different alternatives to the attention blocks' mean in the third-to-last paragraph? 2. What does the 'gain' mean in the third-to-last paragraph?

Reviewer wowB2024-08-13

Answers

1. I mean the attention using the cost volumes vs the proposed method 2. I mean what's the gain in the computational complexity

Authorsrebuttal2024-08-07

The video for the visualization of offsets has been permitted by AC.

Dear Reviewer wowB, The anonymized video link for the offsets’ visualization has been permitted by the Area Chair. As required by NeurIPS 2024, the link is provided in a separate comment block titled “Anonymized Video Link.” We apologize for the inconvenience.

Authorsrebuttal2024-08-13

Discussion Stage: Author Response.

## Discussion Due to space limitation, our description of decoder in the main paper is relatively abbreviated. We apologize for this and will provide a more detailed description in the camera-ready version. As shown in Fig. 3, the decoder incorporates both 'self-attention' as well as 'temporal-attention' mechanisms. The 'self-attention' considers multiple points that belong to the same timestamp, while the 'temporal-attention' considers the N frames of the same point. This decomposition of 'temporal' and 'multiple-point' dimensions not only clarifies feature processing but also helps reduce computational cost and memory usage. We have to thank the reviewer's thorough review and valuable suggestions again. At the same time, we would like to express our gratitude for the reviewer’s recognition of our efforts.

Reviewer yaYP6/10 · confidence 4/52024-07-14

Summary

The paper introduces TAPTRv2, an enhancement of the TAPTR framework, which is akin to a DETR-based point tracking approach. It critically examines the reliance on cost-volume in traditional Tracking Any Point (TAP) challenges and questions its necessity, particularly how it affects the query's content feature across different domains. Leveraging insights into the role of cost-volume, the authors propose an Attention-based Position Update (APU) mechanism to supplant cost-volume. This innovation purportedly alleviates the computational burden associated with cost-volume calculations without compromising the state-of-the-art tracking performance.

Strengths

The paper establishes a straightforward yet potent baseline that significantly broadens the horizon for advancements in the field of point tracking. The research methodology and rationale are persuasive. By decomposing various structural components, the paper effectively isolates the function of cost-volume and offers a viable alternative, leading to enhanced performance.

Weaknesses

The paper successfully reduces the reliance on cost-volume but does not thoroughly elucidate the tangible benefits of this elimination. Specifically, it would be beneficial to understand the impact on key performance metrics such as Frames Per Second (FPS), Memory Access Cost (MAC), Floating Point Operations (FLOPs), and the number of Parameters. Furthermore, the introduction of an additional Multi-Layer Perceptron (MLP) warrants an analysis of its effect on inference speed. In Equation 5, the paper presents the Disentangler operation, which is intended to manipulate the attention map. However, for maintaining distribution consistency, it might be more appropriate to normalize the attention weights by dividing by the square root of their dimensionality before applying the MLP. The current approach could potentially disrupt the inherent normalization of the attention mechanism.

Questions

Please refer to the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes.

Authorsrebuttal2024-08-14

A kind reminder.

Dear reviewer-yaYP, Thanks for your recognition and constructive suggestions again. Considering that the deadline is approaching, we would like to kindly remind you to check out our responses and raise the questions that you may still have. This will help us to provide more detailed explanations. Thanks for your time and effort.

Reviewer zGVx5/10 · confidence 3/52024-07-14

Summary

The paper presents TAPTRv2, an improved Transformer-based approach for the Tracking Any Point (TAP) task. Building on TAPTR, which utilizes designs from DEtection TRansformer (DETR), TAPTRv2 addresses a critical issue related to the reliance on cost-volume. This reliance was found to contaminate the point query's content feature, impacting visibility prediction and cost-volume computation. TAPTRv2 introduces an attention-based position update (APU) operation and key-aware deformable attention to improve query position prediction. By eliminating the need for cost-volume aggregation, TAPTRv2 not only simplifies the tracking pipeline but also achieves state-of-the-art performance on several challenging datasets, demonstrating its effectiveness and superiority over previous methods.

Strengths

1. The paper identifies a practical challenge in tracking any point and proposes a novel and effective solution with the attention-based position update (APU) operation. 2. The paper is well-organized, with a clear description of the problem, the proposed solution, and the experimental setup. 3. The inclusion of both in-domain and out-of-domain datasets strengthens the validity of the claims.

Weaknesses

1. In Sec. 3.2, the authors suggest that the detrimental effect is due to the different distributions needed for content and position updates but do not provide a rigorous theoretical justification or detailed analysis to support this claim. 2. Lack of comparative experiments with alternative methods to achieve weight separation, such as other attention mechanisms or different initialization schemes. 3. A more detailed analysis of the computational efficiency and resource requirements of TAPTRv2 compared to other methods would be valuable.

Questions

Please refer to the weaknesses section for details. The motivations raised by the authors seems reasonable, but their approach to the problem needs more explanation and justifications for their design choices are needed.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Authors provide a limitations section at the end of their paper.

Authorsrebuttal2024-08-13

Discussion Stage: Author Response.

## Discussion ### Q4. Whether the performance advantage is achieved at the expense of increased computational cost. We apologize for misunderstanding your question (Q3) and conflating it with other reviewers’ questions. __A4.1__ Firstly, we have to thank the reviewer for the constructive suggestion. Indeed, comparing with more methods from the aspects of efficiency and speed can better reflect the advantage of TAPTRv2. Limited by the deadline of the discussion stage, as shown in the below tables, we compare the performance, computational cost, speed, and the number of parameters with two most widely recognized works in terms of performance and speed: CoTracker (from Meta, using its open-sourced implementation with about 2.2K stars) and PIPs (from CMU, using its open-sourced implementation with about 600 stars). The results show that, although these two methods have fewer parameters than ours, their computational cost is about three times larger than ours, resulting in much slower speeds. These substantial computational costs are due to their redundant designs in their multi-layer refinement process as we have discussed in Sec. 1. For example, they need to recalculate the correlation map between each tracking point and every image feature at the beginning of each layer in multi-layer refinement (6 layers in total). __A4.2__ Affected by such redundant designs, their computational costs increase more rapidly with the number of points to be tracked. As shown in the second table, when the number of tracking points reaches 5000, CoTracker encounters an Out-Of-Memory error (OOM). Although PIPs does not experience the OOM error, its computational cost is about five times larger. __A4.3__ Although these methods requires much more computational cost than TAPTRv2, TAPTRv2 still obtains the best performance. Especially, even when CoTracker is tested in its 'Single' mode, which deliberately tracks each single point at a time and will brings much more computational cost, CoTracker's performance is still inferior to ours. We believe these results can prove that the performance advantage is not achieved at the expense of increased computational cost. (Since other methods are implemented with JAX and have not open-sourced their evaluation code (TAP-Net and TAPIR) or implemented with the on-the-shelf RAFT with numerous numpy computations on CPU (MFT), we need to spend more time measuring their computational cost. However, we believe that the comparison with CoTracker and PIPs is a solid support to our conclusion.) __4.4__ At the same time, we hope that our insight in the analysis of cost-volume, the novel APU derived from the analysis, and the computational efficiency and performance superiority of TAPTRv2 over TAPTR can be considered as nontrivial contributions. | 800 Points | Performance $\uparrow$ | FPS $\uparrow$ | GFLOPS $\downarrow$ | #Params $\downarrow$ | | :--- | :---: | :---: | :---: | :---: | | PIPs | 42.0 | 32.8 | 359.0 | 27.3M | | CoTracker | 60.7 | 38.2 | 308.6 | 23.0M | | CoTracker-Single| 62.2 | 2.1 | 873.2 | 23.0M | | TAPTR | 63.0 | 65.9 | 147.2 | 39.2M | | TAPTRv2 | 63.5 | 69.1 | 143.4 | 38.2M | | 5000 Points | Performance $\uparrow$ | FPS $\uparrow$ | GFLOPS $\downarrow$ | #Params $\downarrow$ | | :--- | :---: | :---: | :---: | :---: | | PIPs | 42.0 | 6.6 | 1777.2 | 27.3M | | CoTracker | 60.7 | OOM | OOM | 23.0M | | CoTracker-Single| 62.2 | OOM | OOM | 23.0M | | TAPTR | 63.0 | 11.8 | 426.8 | 39.2M | | TAPTRv2 | 63.5 | 14.6 | 354.2 | 38.2M |

Authorsrebuttal2024-08-07

Request for permission to provide an anonymized video link.

Dear AC, considering that reviewer-wowB requires the visualizations of intermediate results from the model, we provide an anonymized video link to show the required visualizations. The anonymized link will be provided in another comment block that is visible only to you. After obtaining your permission we will make it visible to all reviewers.

Reviewer zGVx2024-08-12

Thank you for your response, which has addressed some of my concerns. However, I still have some questions. Regarding the comparison of computational cost and the number of parameters, you have only compared TAPTR. In your original Table 2, you compared several different methods. I believe that you should compare at least 3 to 4 models in terms of computational cost and parameters to determine whether the performance advantage is achieved at the expense of increased computational cost.

Authorsrebuttal2024-08-14

A kind reminder.

Dear reviewer-zGVx, Considering that the deadline is approaching, we would like to kindly remind the reviewer to raise the questions that the reviewer may still have. This will help us to provide more detailed explanations and clarify our contributions effectively. Thanks for your time and effort.

Reviewer zGVx2024-08-14

Thank you for the author's response. This has addressed most of my concerns. I will increase my score to borderline accept.

Reviewer wowB2024-08-13

Answer

I thank the authors for their effort in the rebuttal, which helps me consolidate my rating. While the response is satisfying, I wanted to clarify an aspect in my review that might have been misunderstood considering it hasn't been replied (it might have been myself who misunderstood the answer). In A1 my question was if the self-attention was considering the N frames of the same point, or the multiple-points at a time. My understanding is that it is the former, but in such case I wonder how does the method work at inference for multiple target points. Is inference carried for every target point? Thank you for your time.

Reviewer zD8r2024-08-13

Thanks for the response, especially for providing the speed comparison experiment. I believe there are no obvious flaws in this paper now. I keep my initial score unchanged.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC