RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions

Summary

This paper proposed a novel approach to leverage the language inputs to retrieval the scaling factor between relative depth prediction and ground truth metric depth. The task and the approach is novel and interesting.

Strengths

(1), To the best of my knowledge this is the first work to use language (text) as input to retrieve the metric depth scale. (2), The presentation of this paper is clear and easy to follow. (3), Authors conducted extensive analysis of the text input design and sensitivities.

Weaknesses

(1), My primary concern is the practical applicability of this problem. Is there a significant use case for depth prediction using text descriptions as input? For autonomous driving or AR applications, the user might not be always speaking and describing the environment. (2), For the case of automatically generating text from a detection model, essentially, this is leveraging prior knowledges from the pre-trained detector and CLIP model. What if you just extract the detector/CLIP features as input to the RSA model?

Questions

(1), It's better to have more comparison with other works that are trying to retrieve metric depth scaling factor. (2), As mentioned above, in my understanding, this approach is leveraging prior knowledges from the pre-trained detector and CLIP model. The text generation and encoding is more like a human-designed bootstrapping of "how to extract prior knowledge". It would be interesting and greatly helpful if the author could have comparison to direct fusion of detector/CLIP feature into the RSA model to prove the effectiveness of this bootstrapping.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The text description of images could be hard to obtain in autonomous driving or AR scenarios, limits the potential application of this method.

Summary

The paper proposes a method for learning the scale of single-view observation of scenes from embeddings of text descriptions of the scenes. The method (referred to as RSA model) is of a simple design of MLP layers and is trained on text embeddings extracted from single images. Evaluation is done on various datasets and combination of in-dataset generalization and few-to-zero-shot settings. The paper is able to demonstrate the superior capabilities of the method when compared with baseline methods in various settings both quantitively and quantitatively, as well as to include ablation study on design choices and statistics of the outputs.

Strengths

[1] The method is simple and novel. The paper illustrates a simple yet effective finding that the scale of a single-view scene can be learned from the text description of the image. The idea is intuitive and not limited to specific domains and is proven to generalize well to unseen domains given the text description is a high-level abstraction of the scene appearance. The findings may be inspiring to the community on the potential of using text prompts as an image representation which is useful to yield high-level scene properties. [2] Extensive evaluation. Based on the assumption, the paper is able to include relative extensive experiments on abundant datasets to demonstrate the method is applicable to a wide range of modern depth estimation methods to recover the scale, and to outperform methods predicting absolute depth. The paper also includes ablation study that answers to several questions including important design choices of the text description, and analysis of the results across domains. [3] The paper is well-written and easy to follow. The results are well presented.

Weaknesses

[1] Robustness of the method. Despite various design choices and sample images are discussed in the paper, it is not clear how well the method generalizes to difficult images, where there is less context or objects present (e.g. an empty corner of a room, or empty section of a street), or when the off-the-shelf object detectors make mistakes. In this case, methods which rely on intermediate representations including RSA may not be as robust as methods that directly operate on the image appearance. [2] Limited evaluation. Despite evaluation is done on various dataset on different settings, it still feel the scale of evaluation is limited, given the claim that the proposed method is not domain specific. In this case, there should ideally be more datasets involved to represent domains in the wild (e.g. indoor/outdoor/driving, real/synthetic), yet the paper only picks a few to evaluate on. [3] Simple model design and prompt design. The paper picks a very simple MLP-based model without discussing on alternative choices. Moreover, the prompt design can be further elaborated, to include e.g. coordinates of objects, intrinsics of the camera, configuration of the objects in the scenes, as those properties may add additional information on the scale of the scene and to help further disambiguate the issue (for instance, existing methods for geometric estimation are known to perform worse when the FoV of the camera changes). [4] Minor questions and suggestions. (1) What is the method termed RSA? (2) L199, how does the model take CLIP image and text features respectively? What are the dimension of features and layers of the model in both cases?

Questions

Please see above comments for questions raised.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper proposes a method for metric-scale monocular depth estimation called RSA. RSA aims to recover metric-scaled depth maps through a linear transformation, leveraging language descriptions of objects in scenes. The proposed method is validated on multiple standard datasets.

Strengths

The use of language descriptions to convert relative depth predictions into metric scale depth maps; The proposed technique outperforms previous methods;

Weaknesses

The paper could benefit from a deeper theoretical analysis of why the proposed method works and its limitations; Additional experiments on a wider range of datasets and scenes would strengthen the evidence for the method’s generalizability and robustness; The description of the linear transformation process and the role of text captions need more detail to fully understand the approach.

Questions

How does the method perform in real-world scenarios where text captions may not be readily available or may be noisy?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

N/A

Summary

This paper tackles the challenge of scale ambiguity in monocular scenes. The idea is to leverage the priors in LLMs to estimate the scales of an image. Evaluation is performed on NYU and KITTI datasets in comparison to baseline heuristics.

Strengths

- The value of LLMs is shown in a novel domain - code will be made available

Weaknesses

- The proposed solution seems unreasonable, at least how it is presented in the paper. Specifically, the LLM does not utilize the image but only text alone. Admittedly, the text is generated from image input, but is limited to existing object detection and only tested in 2 domains/datasets. Since the image data is not used (or the relative depth prediction) the LLM prediction seems to learn dataset biases rather than from real image data. - This concern is also shown in the evaluation, were only a limited evaluation on 2 datasets is performed. Results are insignificantly improved. The comparison is limited so simple baselines, while work that explicitly uses object scales to refine relative depth are not shown. - The value of extracting text from images to then compute scales from text seems limited in real applications. A comparison to finetuned GPT4v (or equivalent) with image data alone to directly estimate scales is suggested.

Questions

see above

Rating

6

Confidence

4

Soundness

1

Presentation

3

Contribution

2

Limitations

see above. - Only few datasets - Code not yet available - Insignificant improvements - Soundness of solution is not convincing

Thanks authors for the response

Dear authors: Thank you for the prompt response! For the user input text, it might be hard for the user to say "XXX object occupies YYY% of the image". For some user-friendly text design like “An image with c(1) , c(2) , c(3) , ... ", seems it works worse than directly using CLIP features. For the detector knowledge, I can not find the "tables 5 and 6". Looking forward to the further discussion!