Fast Encoder-Based 3D from Casual Videos via Point Track Processing

Summary

The paper presents TRACKSTO4D, a fast, encoder-based method for reconstructing 3D structures and camera positions from casual videos with dynamic content. It processes 2D point tracks using a single feed-forward pass, leveraging inherent symmetries and low-rank approximations. TRACKSTO4D is trained unsupervised, reducing runtime by up to 95% compared to state-of-the-art methods while maintaining accuracy. It generalizes well to unseen videos of different categories, offering an efficient solution for 3D reconstruction in applications like robot navigation and autonomous driving.

Strengths

Method: 1. The method considers the symmetry of tracked points and the temporal sequence of the video. By designing an equivariant layer based on transformers and positional encoding, it effectively takes advantage of these properties. 2. It also uses a low-rank movement assumption to decompose the motion of the 2D tracked points into the global camera motion and the 3D motion of objects in the scene. This converts an originally ill-posed problem into a solvable and intuitive problem. The method incorporates hard constraints into the design, resulting in more efficient training and more accurate and constrained results. Overall, I think the method is very solid. Results: 1. TRACKSTO4D significantly reduces the runtime for 3D reconstruction from casual videos by up to 95%, making it highly efficient and practical for real-time applications. 2. The method generalizes well to unseen videos and different semantic categories, demonstrating robustness and versatility. 3. The use of unsupervised training without the need for 3D supervision minimizes the dependency on annotated datasets, which are often expensive and time-consuming to create. The evaluation results are fairly comprehensive.

Weaknesses

1. The paper primarily focuses on runtime and accuracy compared to state-of-the-art methods, but it lacks a detailed evaluation on other important aspects like robustness to noise or handling of occlusions. There is also limited discussion on how the method scales with increasing video length, number of objects, or more complex scenes. Scalability is crucial for deploying the method in large-scale real-world environments. In particular, I'm very curious how the chosen $K$ would affect the performance since $K$ is like an indirect descriptor of how complicated the point cloud's components are in my opinion. 2. The method's performance is heavily dependent on the quality of 2D point tracks extracted from videos. Poor quality or incorrect point tracking can adversely affect the 3D reconstruction accuracy. I'm curious how robust the method is against failing 2D point tracks. 3. While the low-rank assumption helps in reducing complexity, it may not capture the full variability of highly dynamic or complex scenes, potentially limiting the method’s applicability in such scenarios.

Questions

1. How robust is TRACKSTO4D to variations in the quality of 2D point tracks? For instance, how does it perform when the point tracks contain noise, are partially occluded, or have tracking errors? Does it strictly follow the 2D point tracks, or does it try to rectify them to some extent? Understanding this robustness is crucial for assessing its applicability in real-world scenarios where perfect point tracking is often unattainable. 2. Can the method be extended to handle dense point clouds or full 3D reconstructions rather than sparse point clouds? If so, what modifications would be necessary, and what impact would this have on runtime and accuracy?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

1. The method assumes that the movement patterns of the dynamic 3D points can be effectively represented using a low-rank approximation. This assumption might not hold true for all types of motion, particularly those involving complex deformations or highly non-linear dynamics. 2. There is an assumption on the motion parallax, without which the method fails to generate accurate camera poses. 3. The accuracy and robustness of the method are more or less dependent on the quality of the 2D point tracks extracted from the videos. Noisy or incomplete point tracks could adversely impact the 3D reconstruction and camera pose estimation.

Summary

The paper introduces a method for fast 3D reconstruction of dynamic structures (or 4D reconstruction) from monocular video. The model is a transformer architecture that takes a set of 2D point tracks as input, and lifts them to 3D. It is learned with re-projection losses on 2d tracks without 3D ground-truth. The method is trained on COP3D (containing cat/dog videos) and evaluated on two test datasets: (1) 21 self-collected videos of dogs and cats and (2) NVIDIA dynamic scene dataset, using depth metrics and camera tracking metrics. It is shown to perform on par with an optimization-based (CasualSAM) but much faster.

Strengths

- The method tries to tackle a challenging and significant problem, and presents a viable approach that works decently well and very fast. - It does not need 3D ground-truth, or depth supervision, and shows good generalization ability to novel scenarios. - The authors also collected a new test set that has more accurate depth ground-truth than COP3D.

Weaknesses

- The presentation can be improved. Specifically, I had difficulty understanding the points made in Sec 2.1. - The permutation invariance of point sets has been discussed in early literature (e.g., pointnet), and it might be better to connect to that instead of starting from symmetry analysis. - I also don't really understand what is the relation of "linear equivariance" and cyclic group, and how that relates to the proposed architecture. Is there any cyclic operation that moves the end of a sequence to the beginning? - It seemed to me the final architecture is a standard transformer with some modifications. It would be good to highlight which part enforces translation equivariance. - Since the method does not require 3D ground-truth. In theory, it can be trained using any video. However, the model is only trained on cats and dogs in practice. I'd like to encourage the authors to train it on larger scale dataset and observe whether there is any performance gain on the held-out set. - It uses some hand-crafted shape and motion prior (low-rank). I'd be interested in seeing an analysis of how changing the flexibility of the model (by changing the number of basis) affects the results. - The evaluation can be further enhanced by reporting 3D end-point-errors, since depth only evaluates one aspect of 4D reconstruction. Tapvid3d [A] or synthetic datasets like Kubrics, PointOdyssey could provide such ground-truth data. [A] Koppula, Skanda, et al. "TAPVid-3D: A Benchmark for Tracking Any Point in 3D." arXiv preprint arXiv:2407.05921 (2024).

Questions

- Is there any mechanism to handle the errors in co-tracker's 2d trajectory? - Spatialtracker [B] is another relevant work, although then train on 3D ground-truth and rely on depth inputs. It would be helpful to discuss what are the pros and cons of using unlabelled/3D data to train the model. [B] Xiao, Yuxi, et al. "SpatialTracker: Tracking Any 2D Pixels in 3D Space." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

This paper proposes a feed-forward network that takes a set of 2D TAP curves as input, outputs the 3D curves as well as extracts the camera rigid SE(3) poses.  This paper exploits the permutation and time shift equivariance when designing the encoding network. To train the model, the main loss is the re-projection loss (similar to BA). Experiments show in-category improvement and novel category generalization.

Strengths

- The reviewer really likes the paper's style, exploiting the underlying structure of dynamic scenes, although it seems the symmetries exploited here are simple. - The problem this paper is solving is very important and meaningful for the community. Although I do see more promising ways to solve the problem (see weakness), I do like this paper's perspective on this problem. - The model learned on cats and dogs is somehow generalizable to new categories, verifying their effectiveness.

Weaknesses

- I do like the equivariance part, but given the current advances in the community, I do find that maybe 3D-track or exploiting depth models may lead to better performance instead of learning everything from scratch. - Solving a single scene efficiently: one important loss that supports the model learning is the reprojection error, which seems to conduct an implicit BA when training the model over many sequences, the reviewer is curious whether the same techniques can work on a single video efficiently. This is to justify the necessity of the feedforward network if the same technique can fit a single video efficiently as well.

Questions

Please see weakness

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

discussed in last paragraph.

Summary

This paper presents TracksTo4D, a feed-forward approach for estimating 3D structure and camera poses from 2D point tracks. Authors propose a novel architecture that directly processes 2D point tracks, takes into account symmetries present in the data, and assumes movement patterns can be represented by a low-rank approximation. Notably, this approach achieves similar performance to state-of-the-art methods with significantly reduced runtime. Despite training on videos of animals, TracksTo4D generalizes to unseen videos and unseen semantic categories at inference time.

Strengths

Problem Motivation. This paper clearly describes the problem they are trying to solve and explicitly outlines the inputs and outputs of their method. Learning to predict camera poses and 3D structure from 2D point tracks is a challenging and relevant problem of interest in the community. Few methods currently solve both subproblems using feed forward approaches. Ablations. Authors extensively ablate different losses used in training their method and highlight the impact of training on different subsets of COP3D (e.g. cat vs. dog vs. cat + dog) Clear Writing and Informative Visuals. The paper is well written and easy to follow. The provided visuals in the supplement clearly demonstrate the effectiveness of the proposed method.

Weaknesses

Limited Evaluation. Although many methods do not tackle jointly predicting 3D structure and camera poses, authors evaluate methods that support a subset of these tasks. However, these evaluations can be supplemented. MiDaS is quite old at this point, and there has been significant improvements in monocular depth estimation in recent years. Instead, one should evaluate multi-view or video depth estimation methods. Other simple baselines can be included to provide more context like Chained RAFT3D [1] and Lifted CoTracker [2]. Small Test Set Size. Authors evaluate on a subset of COP3D and the NVIDIA Dynamics dataset, both of which only have a limited number of videos. Instead, authors should consider evaluating on synthetic datasets like PointOddessey [3], which can allow authors to explore other dimensions of their method like data scaling (which is unique to this method because it is much faster than competitive methods). [1] Teed et. al. Raft3D: Scene Flow using Rigid Motion Embeddings. [2] Karaev et. al. CoTracker: It Is Better to Track Together.

Questions

Evaluating Robustness to 2D Point Tracking Noise. Although authors use CoTracker in this work, it would be interesting to characterize how this method works when evaluated using a different point tracker at test-time. Evaluating Robustness to Speed of Dynamic Motion. Although authors highlight that their method does not perform well given rapid motion, it would be useful to quantify this to benchmark performance for future methods. It would be useful to show off performance on automotive datasets that have ground truth depth (via LiDAR) and precise camera poses. Impact of Data Scaling. Given the speed and generalizability of this approach, it would be interesting to explore how this method scales with more data, particularly since this method doesn't require labeled 3D data. Defining Out-of-Distribution. Since this paper takes 2D points tracks (instead of raw images) as input, its I would not consider different datasets and semantic classes to be out-of-distribution, since the 2D point tracks don't have any notion of semantics. Instead, it would be useful to benchmark on data with different speed profiles, and provide breakdown analysis by speed buckets. Contextualizing Runtimes. Since this method runs CoTracker before processing, run times should take this into consideration as well. Although the proposed method is quite fast, I believe the 2D point tracker can be a limiting factor on real-world performance. Supplemental Baseline Comparisons. Although SpaTracker and DuST3R (CVPR 2024) are considered concurrent work, I think this paper can be further strengthen by comparing against such methods. Note that a lack of comparison to these approaches does not influence my ratings.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, authors highlight that their approach can't handle rapid motion and is limited by the tracking noise imputed by CoTracker.

Keep my original recommendation

After reading the reviews and rebuttals, I think this paper makes enough contribution to the community, I keep my original recommendation.