Summary
The paper presents a framework for learning 3D indoor scene representations from CAD models and language commands. The goal is to learn better scene representation for scene understanding tasks, mainly perception tasks such as 3D object detection and scene segmentation. There exist two main challenges in solving this task using a dataset of CAD models – (1) the existence of CAD_models-to-3D_scenes domain gap, and (2) synthetic to real domain gap. It should be noted that the 3D indoor scenes are represented as point clouds in this work.
These challenges are addressed by three main parts: data pre-processing, latent space regularization using Deep Convex Hull regularization, and designing an objective function that allows for using language embeddings to better learn point feature embeddings in the 3D scene. Note that the points can be sampled on the surface of a CAD model (see below).
The data pre-processing step is done to reduce the model-to-scene gap. Here, CAD models are placed on the floor of an existing scanned scene (represented as a point cloud) to simulate a “crowded” scene (as is referred to in the paper; see P2, second line in the first paragraph). The CAD models and the 3D scene scans are rotated randomly, scaled randomly by a scaling factor in the range [0.9, 1.1], and a model is randomly placed in a scene.
The latent regularization step is done by projecting point features from the input CAD models and the input scene into a common space constrained to be convex. This step is called a Deep Convex Hull Regularization (DCR).
To summarize, Input: Labeled CAD models and a 3D scene scan (i.e., a point cloud)
Task: Learn scene representations
Dataset used:
ModelNet , ScanNet and S3DIS datasets
Underlying Neural Network:
The paper uses two types of neural frameworks.
One is MinkowskiNet14 whose backbone is made of 3D Conv Layers. The other framework that is employed is a MLP-based contrastive learning setting (this is where visual-language connection is introduced).
Loss function:
Cross Entropy Loss in learning the visual-language embeddings (see Eq 3)
Quantitative Metric:
3D object saliency detection -- mAP (mean average precision),
Zero-shot 3D semantic segmentation – mIoU
Strengths
1) The paper provides experiments on real scene datasets to better evaluate the proposed framework.
2) Ablation studies seem to be thorough.
Weaknesses
1) The paper is poorly written, especially the Abstract, Intro, and parts of the Method Sections. For example, in Sec 2.2, the first line says “Since the network is only trained….” – there has been no mention of the “the” network before this in the Method section. The clarity of thought in writing this is lacking. The readers can understand that there is a neural network that is employed but the first sentence does not document the research well. Please provide a context first and then follow it up with the first sentence of Sec 2.2.
The same goes for the first reference of DCR on P2. It is difficult to understand what point features are referred to in this sentence “to project point features into a unified convex hull space that further ….”. Prior to this sentence, there is no info/mention on what kind of representation is used. Okay, the reader has to understand that it is point cloud representation. But which specific points are being sampled/referred to? You get my point. The paper lacks serious clarity of writing. This is also seen in the Section numbering, where the word “Introduction” as a section heading is missing.
2) I am not sure what a “crowded scene” means, but I am guessing the paper meant to say that it is to simulate a real-world scene. This, as is claimed in the paper, is to reduce the CAD_model-to-scene gap. I agree to some extent on this, but it can also reduce the synthetic-to-real gap. Why is this not the case?
3) It would help the readers if a single sentence is written to introduce “prototypes” before starting with the formulations on P5. I am still confused as to what is a “prototype”. Is it a template/basis function?
4) It is unclear as to why there is a need to introduce visual-language embeddings. I am having a hard time understanding the motivation for going from the DCR step to the language embedding step. Moreover, it is not described as to how these language prompts for the CAD models/ scenes are obtained. The overall framework of the paper is not convincing.
5) Fig 4 – what is the notion of saliency? Is it just object detection? If so, how does the algorithm understand saliency in a scene composed of multiple objects? Plus, how is the saliency visualized?
6) Not sure if PointContrast is the only work that needs to be compared to. There are follow-ups of PointContrast that are more recent, which could be discussed. This is a weak weakness, so, I can overlook this if the paper fared in other aspects.
7) No discussion of the limitations
Questions
1) Table 3 – This table represents quant results for semantic segmentation on the S3DIS dataset. The caption says “improved accuracy”, but as I understand, the evaluation metric for semantic segmentation is mIoU. There is a discrepancy here. Using the word “accuracy” is misleading.
2) The same goes for Table 4, but it is a little clearer in this Table that “accuracy” is being referred to as mAP scores. Again, using a better term, like “improved performance”, is apt.
3) As explained in Sec 2.1, a CAD model is placed on the scene floor. How is this done? Can you briefly discuss this?
4) Also, it is mentioned that the CAD models are scaled to match real scenes’ object size. I am not sure how this is done. The scene scan contains objects of different sizes, say, TV stand, Lamp, Coffee Table, etc. and as I understand, these scene semantics are not known. Only the CAD model labels are known during training. So, how is the scaling done? One needs to have a reference object (i.e., its point cloud) from the scene to obtain the scaling factor by which the CAD model needs to be scaled when it is being placed in the scene.
5) Random rotation transformation on CAD models – This cannot be true. If you do this in the SO3 space, rotated objects will be implausibly placed in the scene.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.