Depth Anything V2

Summary

The paper tries to answer two important questions in current monocular depth estimation: 1. whether generative-based models can generate more fine-grained depth than discriminative models. 2. The effectiveness of synthetic data and large-scale pseudo data. The results show that discriminative models can also generate fine-grained depth estimation when trained with synthetic data and gradient matching loss. The paper provides several depth estimation models with different sizes, which may promote various downstream applications.

Strengths

1. The paper proposes to enhance the performance of the existing discriminative depth estimation model, DepthAnything, with synthetic datasets and large-scale pseudo labels, which is simple yet effective. 2. The experiments are extensive and the writing is good.

Weaknesses

1. How do you define the fine-grained depth quantitatively? There seem to be no concrete metrics to evaluate it, except qualitative visualization in the paper. So it is still hard to fairly compare the fine-grained depth with the generative-based models. 2. The pseudo label can only be relative depth, which may not be effectively directly used for metric depth estimation. 3. There are no dataset ablation experiments to verify the necessity of using the total 62M pseudo labels in the paper. If one random samples a subset from the pseudo-labeled real images, e.g. sample 10% data from all the 8 real datasets, what is the performance of the student model? Whether the 62M data is redundant or not?

Questions

1. What does "+unlabeled real image" mean in Figure 6? 2. What is the backbone in Table 11? 3. In Section C.10, the authors find that synthetic data is important for fine-grained predictions, I am curious about the differences in quantitative results when training the teacher model with and without real labeled images. 4. Can 62M pseudo labels improve the performance of the teacher model (ViT-Giant backbone)? 5. For metric depth estimation, the authors fine-tuned the model from pre-trained relative depth estimation, what is the performance when fine-tuning the model from the original DINOv2 pre-trained backbone? (e.g. using the same ViT-large backbone with different weight initialization) 6. What is the performance of the proposed metric depth estimation model compared to recent SOTA models, e.g. Unidepth and Metric3Dv2?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness and questions

Summary

Depth Anything V2 is an advanced monocular depth estimation model that improves upon its predecessor by utilizing synthetic images, enhancing the teacher model's capacity, and leveraging large-scale pseudo-labeled real images for training. This approach results in significantly faster and more accurate depth predictions compared to models based on Stable Diffusion. This model also offers a range of model sizes to support various applications and demonstrates strong generalization capabilities, which are further fine-tuned with metric depth labels. Additionally, a versatile evaluation benchmark, DA-2K, has been developed to address the limited diversity and noise in current test sets, facilitating more accurate future research.

Strengths

The paper offers several noteworthy advantages. Firstly, it provides a comprehensive analysis of recent depth foundation models, detailing their features, strengths, and weaknesses. The comparison between generative and discriminative models in Table 1 is particularly insightful. Secondly, the approach is novel, utilizing pseudo-labeled real images to harness the advantages of both synthetic and real datasets. This method is impressive and appears beneficial for scaling up foundation models, similar to SAM. Thirdly, the paper demonstrates strong generalization performance, as evidenced by various tables and figures. The real-time performance showcased in the CVPR conference demo was particularly impressive. Lastly, the extensive additional experiments and analyses in the appendix offer valuable insights, supporting the validity of the proposed methods and experimental protocols.

Weaknesses

Data Volume and Performance Relationship: According to Table 10, there seems to be a linear relationship between the amount of data and performance. It would be interesting to see how the model performs with even more data added. Metric Depth Estimation Comparison: Table 4 shows that the results of Metric Depth Estimation are similar to or worse than other Monocular Depth Estimation models (UniDepth[1], Metric 3D[2]). Additional analysis on this would be beneficial. ##### [1] UniDepth: Universal Monocular Metric Depth Estimation ##### [2] Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation

Questions

These points are included in the Weakness.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations of Depth Anything V2 include a dependency on data volume for performance, as suggested by Table 10, which raises questions about scalability and diminishing returns with more data. Additionally, Table 4 shows that its Metric Depth Estimation results are comparable to or worse than other models like UniDepth and Metric 3D v2, indicating that while the model excels in some areas, it does not consistently outperform competitors across all metrics.

Summary

The authors introduce Depth Anything V2, a powerful monocular depth estimation model. This model relies entirely on synthetic data to train a teacher depth estimation model, which is then used to generate pseudo-labeled real images. These pseudo-labeled images are subsequently fed into the training pipeline. Depth Anything V2 achieves state-of-the-art results in both quantitative and qualitative evaluations. Additionally, the authors offer models in various scales, accommodating different scenarios. To address the limitations of existing monocular depth estimation benchmarks, the authors also provide a new benchmark with precise annotations and diverse scenes, facilitating future research. However, compared to depth anythingv1 and other affine-invariant depth methods, the novelty is limited.

Strengths

1. The authors dedicate considerable space to discussing the choice of datasets, highlighting the disadvantages of labeled data, the advantages and limitations of synthetic images, and the role of large-scale unlabeled real images. This provides a clear rationale for the proposed pipeline. 2. A new benchmark is introduced to address the limitations of existing benchmarks. This benchmark encompasses a wide variety of scenarios and includes comprehensive, high-resolution depth estimation results. 3. The results are impressive. The method demonstrates significant improvements in zero-shot relative depth estimation, particularly on the proposed DA-2K benchmark. The qualitative results also reveal the fine-grained detail of the predictions.

Weaknesses

1. Lack of Novelty: The article primarily focuses on analyzing the choice between synthetic and real data, while the pipeline remains unchanged compared to the original Depth Anything V1. V2 used more high-quality data to achieve better performance, but it has no new methods contributions and insights. 2. Lack of Detail: Although some methods may be well-known within the community, it would be beneficial for the authors to elaborate more on the description of the pipeline and the formulation of the loss function. 3. More Comparisons Required: In Table 2, the authors argue that the method predicts affine-invariant inverse depth and compares it with Depth Anything V1 and MiDaS V3.1. MiDaS is not an up-to-date SOTA method. It should enclose more recent works, such as UniDepth, ZoeDepth, Marigold, Metric3D, and Metric3Dv2. Similarly, in Table 4, which compares metric depth methods, Unidepth, and metric3dv2 achieve state-of-the-art performance in these two benchmarks, but they are not listed in tables.

Questions

1. Why can the proposed method be compared with other methods on the proposed dataset, but not on other zero-shot relative depth estimation datasets? 2. The performance of the pipeline seems to be limited by the performance of the teacher model, as the training data for the student models is provided by the teacher model. Why, then, do some student models outperform the teacher model in Table 5?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper does not present limitations in detail.

Summary

A system is proposed for scaling up large monodepth estimation models to achieve very strong zero-shot performance, focusing on finely detailed depth prediction. Essentially, a teacher model is trained on synthetic depth datasets with perfect ground truth, and thereafter distilled on a large in-the-wild dataset.

Strengths

* The qualitative results are really stunning. * The distillation-based framework is novel (as far as I know), and provides a valuable insight into the training of depth estimators more generally.

Weaknesses

* It would be ideal to show the results back-projected into 3D as a point cloud or mesh for some example cases. It is difficult to tell the depth map quality from colormaps alone. * The comparison with MIDAS feels like it leaves out an important limitation of the proposed the method. The proposed high-quality synthetic datasets appear to consist mostly of indoor and outdoor scenes. How will the technique stack up against MIDAS for unusual images that are not likely to be well-represented in such synthetic data, but may be available in 3D movies, such as of moving animals or people? Does the distillation on in-the-wild images just solve this problem? * The paper appears to sort of conflate image-conditioned generation and regression, which are different problems even for image-conditional depth estimation. While "blurry" depth maps may be suboptimal for users looking for perceptual quality and seeking to back-project a 3D pointcloud or mesh, they may be in some sense metrically optimal since even scale and shift-invariant depth estimation is often ambiguous. Making predictions "fine-grained" could just be a problem of learning to sample from the distribution of image-conditioned depth, rather than heuristically applying losses such as gradient-based matching losses to a regression-based framework which is inherently a flawed way to look at the problem of perceptually high quality depth estimation.

Questions

The performance on scenes which are not among the original synthetic data (animals, people, etc.) is unusually impressive. I understand that distillation is the proposed explanation, but can the authors provide a clear intuition or explanation for why this works so well?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The computational limitations are briefly mentioned.

Thanks for your feedback

Thank you for the feedback. Similar to Reviewer RjZq, could you provide a more detailed evaluation comparing the relative depth metrics of existing metric depth models, such as Metric3D, Metric3D v2, and Uni-Depth, against challenging indoor and outdoor benchmarks? Specifically, NYU-D is known for having noisy labels, which raises a concern. It appears that metric depth models trained on large real-world datasets may already exhibit strong generalization performance across diverse scenes, suggesting that pseudo-labeling might not be essential. Consequently, the main benefit of the pseudo-labeling approach proposed in this paper seems to be in achieving more detailed relative depth maps and cannot be applied to improve metric depth estimation directly since pseudo-labels lack scale information.