BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

Summary

This paper proposes a plug-and-play diffusion refiner for pre-trained zero-shot feed-forward MDE (monocular depth estimation) models, so that these generalized models can capture fine-grained details. In this paper, the coarse depths output from a pre-trained feed-forward MDE model are used as additional conditions for the proposed diffusion refiner, which is parameterized into any existing diffusion-based MDE model. In addition, a global pre-alignment strategy and a local patch masking strategy are proposed to ensure the faithfulness of the predicted depths.

Strengths

(1) The paper is well-organized. (2) The proposed diffusion refiner is simple and clear, which could be be easily combined with different existing feed-forward MDE methods. (3) Experimental results on several datasets demonstrate the effectiveness of the proposed method in some cases.

Weaknesses

(1) The motivation of this work is a bit far-fetched: As stated in the introduction, the low accuracy of the existing feed-forward MDE methods is caused by noisy and incomplete depth labels collected in real-world scenarios, while the poor generalizability of the diffusion-based MDE methods is caused by less diverse synthetic training samples. However, if only training data results in limited MDE performance for these existing methods, it seems that a straightforward motivation for alleviating the above problems is to enrich the training datasets, rather than to modify the model architecture as done in this work. (2) Some statements in Table 1 are confusing: (i) To the reviewer’s knowledge, many existing feed-forward methods only use real datasets for training, rather than using real and synthetic datasets together. Why do the authors indicate that feed-forward methods use both real and synthetic datasets for training? (ii) Some existing diffusion-based methods (e.g., "Monocular depth estimation using diffusion models.arXiv preprint arXiv:2302.14816, 2023.") also use real datasets for training. Why do the authors indicate that diffusion-based methods use only synthetic datasets for training in Table 1? (iii) The authors state that feed-forward methods have a better generalizability than diffusion-based methods. But to the reviewer’s knowledge, in many zero-shot learning tasks (e.g., zero-shot recognition), generative models (e.g., GANs and diffusion models) generally show their better generalizability than the feed-forward counterparts. Could the authors give some explanations on why to make the above statement? (3) Comparaive evaluation: As seen from Tables 2 and 3, when the proposed BetterDepth is used together with two early methods MiDaS (published in 2020) and DPT (published in 2021), it could bring some improvements. However, when the proposed BetterDepth is used together with Depth Anything[43] (a SOTA method), it only brings a slight improvement in most of the used datasets (particularly NYUv2 and ScanNet). So it seems that the proposed BetterDepth has a limited effect for boosting the performance of SOTA methods.

Questions

(1) As for the global pre-alignment, since the estimated depth values are derived from their corresponding depth labels, what is the performance of only using them in the local patch masking? And for the local patch masking, the authors simply discard significantly different regions in the estimated depths and real depths, will this lead to the loss of visual world information? (2) In lines 128-132, the author's conclusion is that large-scale datasets lead to the generalizability of the feed-forward MDE methods. Therefore, in order to improve the generalizability of diffusion-based MDE methods, why not to simply train them with large-scale datasets following the feed-forward MDE methods? Why is this still a challenge? (3) In lines 133-137, the author's conclusion is that in the diffusion-based MDE methods, the high-quality labels in the synthetic datasets lead to the ability to capture fine-grained details, but some feed-forward MDE methods also use these synthetic (and real) datasets for training as indicated in Table 1. Why are these feed-forward MDE methods unable to capture fine-grained details? (4) Is Fig. 3 a schematic diagram or a direct visualization of the output distribution? What is the difference between the output distributions X(M_DM,D_syn) and \hat{X}? Moreover, for the middle sketch, why does the global pre-alignment not only affect \hat{X} but also X(M_DM,D_syn)? (5) As indicated in Fig. A2, lower η generally means stricter filtering, but if η is too small, too many local regions would be discarded, which would lead to severe information loss. To this end, I’m wondering how many local regions are discarded in the current version of BetterDepth, and how to balance the strictness of the filter and the loss of information. (6) Is there any diffusion-based MDE method trained on real datasets?

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The proposed method might be improved in two ways: (i) The offsets from the real depths to the estimated depths could be modeled in an implicit learnable manner; (ii) The pairs of significantly different local regions in two depths may not be harmful, but contain useful information for monocular depth estimation.

Summary

The paper proposes a simple approach to improve and refine current monocular depth estimation (MDE) methods. Leveraging the strong geometric prior from a state-of-the-art discriminative depth estimation method, and the strong image prior from a generative model, the authors set a new state-of-the-art in MDE. They condition a pre-trained latent diffusion model on an image and a corresponding depth map from a pre-trained MDE model and fine-tune the diffusion part to obtain higher-fidelity depth maps. Additional loss masking and alignment of affine-invariant depth to the ground truth are found to be crucial for performance.

Strengths

The paper is well-organized and well-written, with a clear common thread. The authors present a comprehensive set of experiments, validating the effectiveness of their method, and clearly ablating their contributions. The idea is simple and achieves very good results on a broad range of depth estimation benchmark datasets with minimal additional training effort, utilizing strong image and depth foundation models.

Weaknesses

- Since the idea is that simple, the general impact or contribution might only be moderate in my opinion. The main contribution is the injection of additional conditioning information into the diffusion model. Since this prior already gives state-of-the-art results, adding a full diffusion model on top with a strong image prior to learn the affine-invariant residual between ground truth and prediction is obvious to improve benchmark metrics. This also explains why there is only few data needed, and why the error bar in appendix D is much lower than Marigold. The task for Marigold is much harder compared to the refinement of a very good depth map. - Further contribution claims incorporate the global pre-alignment and local patch masking. The first, global pre-alignment, is a simple re-phrasing of scale and shifting the ground truth depth map to the predicted depth map of the MDE method, which is a common approach for testing affine-invariant depth estimation methods. Since the least squares fit cannot satisfy all pixels to match perfectly, discarding some portion of it, which the authors termed local patch-masking, is also just a small trick to further boost performance. Hence, these two contributions lack novelty in my opinion, and the only real novelty is the incoporation of a strong depth prior into the diffusion model. Besides the concern about contribution impact I think the idea is very clearly and nicely presented, and the paper very nicely polished.

Questions

- Figure 1: I was wondering why the surface normals still show some wobbles for flat surfaces. Since the strong geometric prior from DepthAnything does not show these artefacts, it seems like the diffusion model inserts these. Do you have any explanation for that? Have you evaluated whether this stems from the first stage? - Line 196: Maybe I am missing something, but is there a reason why you choose max-pooling? As far as I understand that means that if at least one patch in a certain region has a small distance between ground truth depth and scaled and shifted depth, the full region is included for training. Do you have an estimate of how often a full region is rejected? And at which semantic regions in an image that usually is the case (e.g. only for the sky)? - Furthermore, since you exclude non-matching regions via the local patch masking, wouldnt it be better to directly use an outlier-aware method (e.g. RANSAC)? This would directly better align matching parts and non-matching parts can be more easily filtered out. - Since depth refinement alone is in my opinion only a moderate contribution, I was wondering whether you could extend your method to transfer affine-invariant depth to metric depth, since this might be a more challenging and thus more interesting task? - Line 305: Does this time include the MDE model? Maybe I was missing it, but can you give a clear separation of much time goes into which step and how many ensembles you are taking for this inference time measure? - If possible I would appreciate further clarification on appendix C. Is the model without any geometric prior just Marigold? And did you train the diffusion model without image prior completely from scratch? - Lastly, one of your claims is to capture more "fine-grained scene details" compared to other methods. I was wondering whether you know about some quantitative way to show that?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed and discussed the limitation of their method.

Summary

This paper presents a plug-and-play monocular depth estimator with the diffusion model. In the proposed method, the authors first employ the pretrained monocular depth model (MDE) to estimate a coarse depth map as the condition of the diffusion model. Then, a modified diffusion refiner is used to obtain the fine-grained depth map of the scene. Extensive experiments are conducted on benchmark datasets to demonstrate the effectiveness of the proposed method.

Strengths

1. The proposed method is well-motivated and easy to follow. 2. The proposed method is a plug-and-play module and easy to use in different backbone models.

Weaknesses

1. The novelty of the proposed method is somewhat limited. The proposed method is basically a conditional diffusion refiner, which combines the zero-shot MDE such as depth anything and diffusion-based MDE such as Marigold. In my opinion, the predicted depth map with depth anything can provide a “good” initialization to the diffusion model. Although the global pre-alignment and local patch masking modules are developed in the proposed method, I still think it is an incremental contribution. 2. In the proposed local patch masking module, why the patch mask strategy can obtain more refined details of the depth map? How to determine the parameters $\eta$ to control the mask ratio? In the ablation study of the supplementary material, the range of the mask ratio is [0.05, 0.3]. Can we set the range to a big interval? 3. In the experiment section, for the proposed method, different numbers of training samples are used. These samples are real data or synthetic data? For Marigold, the training samples are synthetic data without real data. Therefore, I suggest the authors to highlight the use of the real data or synthetic data. In addition, the authors could provide the reasons why the proposed method is slightly inferior to the existing methods on some datasets.

Questions

Please refer to the weakness section. I hope to address the issuses in the rebuttal.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discussed the limitations in the supplementary material.

Follow-up Discussions

Dear Reviewers, We appreciate your valuable comments and suggestions. In the rebuttal, we have responded to all the questions raised and highlighted the contribution and performance of BetterDepth: - We explain the **contribution and novelty** of BetterDepth. Specifically, BetterDepth efficiently addresses the two main challenges (**performance trade-off** and **resource efficiency**) and achieves robust performance with fine details. - We demonstrate the **state-of-the-art MDE performance** of BetterDepth on zero-shot datasets, where the very recent **Depth Anything V2** [R4] is also included for comparisons. - We provide both **quantitative and qualitative evaluations for fine-grained detail extraction**, further validating the superiority of BetterDepth. We kindly invite the reviewers to check our response and give us a chance to clarify the remaining concerns. Please let us know if you have any unsolved or additional concerns before **Aug 13**. Then, we have enough time to provide further feedback. Thanks. Best, Authors

Rebuttal Response

Thank you for your response and the detailed answers to my questions, which addressed most but not all of my concerns. I still have doubts about the contribution of the method. As mentioned earlier, providing a diffusion model with a very strong depth prior, which is already state-of-the-art, is highly likely to improve metrics. The proposed training strategies - global pre-alignment and local patch masking - are common techniques in the MDE literature. The global pre-alignment seems to be a rephrasing of the well-known approach of scaling and shifting the ground truth depth map to match the predicted depth map, which is standard practice for testing affine-invariant depth estimation methods. The local patch masking, which involves discarding certain portions to boost performance, appears to rather be a trick to circumvent the issues of non-matching scale and shift parameters. In my view, the real novelty lies in incorporating a strong depth prior into the diffusion model, but the other contributions seem to lack originality or novelty. Nevertheless, I will maintain my score, since the small incremental step of adding a strong prior is well ablated and justified.

Please discuss

Dear reviewer, The discussion period is coming to a close soon. Please do your best to engage with the authors. Thank you, Your AC

Please discuss

Dear reviewer, The discussion period is coming to a close soon. Please do your best to engage with the authors. Thank you, Your AC