PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds

Summary

This work incorporates images into point cloud pretraining since images contain richer semantic information. Instead of adopting the back propagation strategy which can not handle the misalignment between camera and LiDAR, this paper leverages the neural rendering technique to injecting the semantics into the representation learning process.

Strengths

(1) This paper is well-written and presents a clear and concise idea. (2) The motivation of leveraging the neural rendering to overcome the mismatch of pixel and point is compelling. (3) The experimental results demonstrate consistent improvement on different datasets with varying baselines. And the ablation studies validate the contribution of each component in the pipeline.

Weaknesses

(1) The technical contribution is limited. For example, the BEV conditioned semantic neural rendering strategy, and the masking strategy have been studied in previous works. (2) Although the occlusion problem can be alleviated by assigning a reduced weight to occluded points, the rendering process will introduce additional noise since it assigns multiple semantic labels to each point, leading to the semantic ambiguity. (3) The rendering process allow every points in the ray receive the gradients which leads to many irrelevant points being optimized. (4) The authors claims that the neural rendering is superior to the point-to-pixel projection, but there is no performance comparison between these two methods in the experiments. I think the latter may also bring an obvious improvement since the inambiguous projections. (5) This work takes the semantic labels as supervision for neural rendering. I think it would be better to use the 2D feature vectors as the supervision.

Questions

My major concern lies in the superiority of this rendering process compared to the point painting process. While the author address the occlusion issue, I firmly believe that it can be effectively resolved by accurately determining the object boundaries. By precisely locating the boundaries of objects, the occlusion challenge can be easily mitigated, potentially diminishing the need for a complex rendering approach.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The author address the limitations.

Summary

This paper investigates weakly-supervised representation learning for outdoor LiDAR point clouds. To start, the authors point out that the inherent incompleteness of outdoor LiDAR points would reduce the effectiveness of self-supervised representation learning approaches. To mitigate this, the authors propose to use synchronized images as additional signals to supervise the representation learning process. Observing the lack of color information in the point cloud and the mismatch between points and pixels due to occlusion, the authors propose to use neural rendering of pixel semantics as a pre-text task to circumvent the two problems. Specifically, with a slight modification to NEUS’s weighting function, the authors build an implicit neural representation on top of the BEV LiDAR features, positing that good BEV LiDAR features for implicit neural representations could be good representations for any downstream recognition tasks. Extensive experiments have been conducted to show the effectiveness of the approach.

Strengths

- Novelty - Using neural rendering as a pre-text task for point cloud representation learning is a novel idea the reviewer has never seen in the literature. - Beautiful figures - Figure 1,2,3 are aesthetically pleasing, with figure 2 clearly illustrating the core problems in representation learning for outdoor LiDAR point clouds. - Strong performance improvement over baselines that do not use images as additional signals. - Thorough experiments - Experiments thoroughly demonstrate the strength of the approach.

Weaknesses

Overall, this is a strong paper but the reviewer has to point out three weaknesses — two related to presentation and the other related to baselines. - (Presentation 1) One of the contributions of the paper is to show that images could be valuable signals for pre-training representation. However, it is unclear which baselines in table 1 and 2 actually use additional images as signals. Also, it is unclear what type of signals are being used (i.e. raw pixel values vs pixel semantics). - (Baseline) Another contribution of the paper is to show that pixel semantics is a useful signal for pre-training representation. Although the proposed approach did not converge when trying to render raw pixel value (line 101-103), SLidR (table 2) is a baseline that actually leverages color information to form superpixel for pre-training. Given the big difference between the no-pretrain variants for SLidR and ours, it is difficult to judge whether pixel-value is a weaker signal or there is some other difference (such as model architecture or optimization recipes) that is causing SLidR to underperform PRED. - (Presentation 2): Since using pixel semantics is part of the core contributions, it is important for the authors to acknowledge/mention whether there are any overlaps between the pixel semantic label space and the downstream task label space. Without this, it is difficult to tell whether the proposed approach is a self-supervised approach or a weakly-supervised approach.

Questions

The reviewer believes this is a good paper. However, the reviewer thinks it is important to properly address the weaknesses (especially on the SLiDR baseline and presentation 2). For pre-rebuttal, the reviewer would give “borderline reject” to implore the authors to properly address the weaknesses. If the weaknesses are sufficiently addressed, the reviewer is more than happy to give this submission a “strong accept”. Suggestions: - To fix the first weakness in the presentation, the reviewer recommends adding a column in table 1 and 2 to indicate whether and the type of pixel signal used. Also, it would be great to emphasize this in section 4.2. - To fix the second weakness in the presentation, the reviewer recommends adding a paragraph indicating label overlap between the pixel semantics and downstream tasks in section 4.1. - For the baseline, the reviewer would like to know why the “no-pretrain” SLiDR baseline is much worse than the "no-pretrain" OURS baseline is table 2.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

Yes, the authors have adequately addressed the limitations.

Summary

This paper proposed a novel point cloud pre-training framework, PRED, which leverages the semantic information consistency between the LiDAR point clouds and the camera images to improve the point cloud pre-training performance. The author proposed (1) a novel semantic rendering module for decoding the semantics from the BEV feature maps and (2) a point-wise masking mechanism to alleviate the reconstruction ambiguity. The proposed pre-training method demonstrates superior performance according to the experiments section.

Strengths

(1), To the best of my knowledge, the proposed point cloud pre-training framework with semantic rendering is novel and reasonable. (2), point cloud pre-training is an important task for academia and industry. (3), According to the experiment section, the proposed framework PRED has achieved superior performance on multiple benchmarks.

Weaknesses

(1), Dealing with occlusion is claimed as one of this paper's major contributions and advantages. However, why the occluded points will be allocated a lower weight is not illustrated clearly in the paper. The author could add more presentation, analysis, visualization, and evaluation. (2), The pre-trained semantic model is required for the proposed approach. This point should be marked and compared with other methods in Tables 1 and 2.

Questions

(1), The proposed method relies on well-trained 2D segmentation models, which limits its generalization ability.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

(1) Is there anything special or novel for handling occlusion compared to [46]? If not, then this point should not be highlighted. (2) How would the proposed method perform if the 2D segmenter fails?

Summary

This work proposes a new pretraining algorithm for outdoor 3D perception tasks, where images are utilized to provide comprehensive semantic information. The main algorithm is to leverage the semantics of images for supervision through neural rendering. The authors also apply point-wise masking with a high mask ratio to further enhance performance. The pretraining brings notable performance gains on multiple benchmarks.

Strengths

1. The authors provide a novel insight into the exploitation of image semantics by combing off-the-shelf image segmenter and neural rendering, which I think is well-motivated. 2. Extensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed method. 3. The paper is well-written and easy to follow. The illustrations are straightforward and helpful.

Weaknesses

1. I acknowledge the motivation and the method design of this paper. Different from neural rendering for RGB as NeRF does, the high-level idea of rendering semantics from point-cloud is quite novel. However, I have heavy concerns about the use of Cityscape pre-trained segmenter as it is not general enough, e.g., Cityscape pre-trained segmenter can perform well on nuscenes and Once but can't on Waymo. Despite the authors explaining the reason Waymo experiments do not perform that well on Line101 of the appendix, I think the poor generalizability of pre-trained segmenter maybe also a reason. So I suggest replacing the pre-trained segmenter with SAM. 2. This work shares a similar spirit to Ponder [1]. Despite [1] for indoor scenes, the paper aims at outdoor scenes, the authors should carefully discuss the intrinsic differences. [1] Ponder: point-cloud pre-training via neural rendering.

Questions

Please refer to the weakness part, I would raise my score if the two concerns can be addressed.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer above.

Summary

1. The paper proposes PRED, a novel pre-training framework for outdoor point clouds that leverages image semantics through neural rendering. The paper addresses the challenges of incompleteness and occlusion in point clouds, which are common in outdoor LiDAR datasets for autonomous driving. 2. The paper uses an encoder-decoder architecture to extract a BEV feature map from the point cloud and render semantic maps from image views, supervised by image segmentation and depth estimation. 3. The paper incorporates point-wise masking with a high mask ratio (95%) to enhance the pre-training performance and avoid losing semantics of small objects. 4. The paper conducts extensive experiments on nuScenes and ONCE datasets, showing that PRED significantly improves various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks.

Strengths

1. The paper presents a novel pre-training framework for outdoor point clouds that integrates image semantics through neural rendering. This is a creative and effective way to address the incompleteness and occlusion issues in point clouds, which are often overlooked by previous pre-training methods. The paper also introduces point-wise masking with a high mask ratio, which is different from the conventional patch-wise or group-wise masking strategies and preserves more semantics of small objects. 2. The paper is technically sound and well-motivated. The paper provides a clear and detailed description of the proposed method, including the encoder-decoder architecture, the semantic rendering pipeline, the loss functions, and the masking strategy. The paper also conducts extensive experiments on two large-scale datasets, nuScenes and ONCE, and compares with various baselines and state-of-the-art methods on 3D object detection and BEV map segmentation tasks. The paper reports significant improvements over previous methods, demonstrating the effectiveness and generality of the proposed framework.

Weaknesses

1. The paper does not conduct ablation studies on the choice of image segmentation model and its impact on the pre-training performance. The paper uses DeepLabv3 as the image segmenter but does not justify or evaluate this choice. It is unclear how the quality and accuracy of the image segmentation model affect the semantic rendering and supervision. 2. This paper does not report the computational cost or time complexity of the pre-training framework. Semantic rendering involves sampling and aggregating points along multiple rays for each pixel, which may be computationally expensive and memory-intensive. It would be helpful to provide some statistics or benchmarks on the pre-training speed and resource consumption.

Questions

Please see the comments on 'weaknesses'.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Please see the comments on 'weaknesses'.

Response

The reviewer thanks the authors for the detailed response. The reviewer's concerns are sufficiently addressed and decides to raise the rating to strong accept.

Keep my rating

Thanks to the reviewer for responding to my comments with further comparative studies and analyses. The practice of utilizing 2D output to aid in training 3D tasks is widely accepted. The remaining issue pertains more to the constraint of denoising and enhancing the generalizability of the 2D views, which are restricted. The authors have indeed taken into account my comments, yet the novelty of the paper remains somewhat constrained. I intend to maintain my current rating.