Discussion Stage: Author Response.
## Discussion
### Q4. Whether the performance advantage is achieved at the expense of increased computational cost.
We apologize for misunderstanding your question (Q3) and conflating it with other reviewers’ questions.
__A4.1__ Firstly, we have to thank the reviewer for the constructive suggestion. Indeed, comparing with more methods from the aspects of efficiency and speed can better reflect the advantage of TAPTRv2. Limited by the deadline of the discussion stage, as shown in the below tables, we compare the performance, computational cost, speed, and the number of parameters with two most widely recognized works in terms of performance and speed: CoTracker (from Meta, using its open-sourced implementation with about 2.2K stars) and PIPs (from CMU, using its open-sourced implementation with about 600 stars). The results show that, although these two methods have fewer parameters than ours, their computational cost is about three times larger than ours, resulting in much slower speeds. These substantial computational costs are due to their redundant designs in their multi-layer refinement process as we have discussed in Sec. 1. For example, they need to recalculate the correlation map between each tracking point and every image feature at the beginning of each layer in multi-layer refinement (6 layers in total).
__A4.2__ Affected by such redundant designs, their computational costs increase more rapidly with the number of points to be tracked. As shown in the second table, when the number of tracking points reaches 5000, CoTracker encounters an Out-Of-Memory error (OOM). Although PIPs does not experience the OOM error, its computational cost is about five times larger.
__A4.3__ Although these methods requires much more computational cost than TAPTRv2, TAPTRv2 still obtains the best performance. Especially, even when CoTracker is tested in its 'Single' mode, which deliberately tracks each single point at a time and will brings much more computational cost, CoTracker's performance is still inferior to ours. We believe these results can prove that the performance advantage is not achieved at the expense of increased computational cost.
(Since other methods are implemented with JAX and have not open-sourced their evaluation code (TAP-Net and TAPIR) or implemented with the on-the-shelf RAFT with numerous numpy computations on CPU (MFT), we need to spend more time measuring their computational cost. However, we believe that the comparison with CoTracker and PIPs is a solid support to our conclusion.)
__4.4__ At the same time, we hope that our insight in the analysis of cost-volume, the novel APU derived from the analysis, and the computational efficiency and performance superiority of TAPTRv2 over TAPTR can be considered as nontrivial contributions.
| 800 Points | Performance $\uparrow$ | FPS $\uparrow$ | GFLOPS $\downarrow$ | #Params $\downarrow$ |
| :--- | :---: | :---: | :---: | :---: |
| PIPs | 42.0 | 32.8 | 359.0 | 27.3M |
| CoTracker | 60.7 | 38.2 | 308.6 | 23.0M |
| CoTracker-Single| 62.2 | 2.1 | 873.2 | 23.0M |
| TAPTR | 63.0 | 65.9 | 147.2 | 39.2M |
| TAPTRv2 | 63.5 | 69.1 | 143.4 | 38.2M |
| 5000 Points | Performance $\uparrow$ | FPS $\uparrow$ | GFLOPS $\downarrow$ | #Params $\downarrow$ |
| :--- | :---: | :---: | :---: | :---: |
| PIPs | 42.0 | 6.6 | 1777.2 | 27.3M |
| CoTracker | 60.7 | OOM | OOM | 23.0M |
| CoTracker-Single| 62.2 | OOM | OOM | 23.0M |
| TAPTR | 63.0 | 11.8 | 426.8 | 39.2M |
| TAPTRv2 | 63.5 | 14.6 | 354.2 | 38.2M |