LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modeling

Summary

This paper proposes a locally constrained compact point cloud model (LCM), which consists of a locally constrained compact encoder and a locally constrained decoder based on Mamba. The encoder replaces the self-attention layer with a local aggregation layer, thus achieving a perfect balance between performance and efficiency. The locally constrained Mamba-based decoder is introduced considering the different information densities between masked and unmasked patches in the MPM decoder input. Extensive experimental results show that the LCM greatly outperforms existing Transformer-based models in terms of both performance and efficiency.

Strengths

The novelty of this paper is evident in proposing a locally constrained compact point cloud model by revisiting the shortcomings of the commonly used Transformer-based point mask modeling approach. By acting on five classical point cloud self-supervised methods, it revolutionizes the point cloud self-supervised learning technique in terms of methodology and results. The theoretical analysis and experimental results in this paper are quite detailed. By localizing and spatially statefulizing the structure of common encoder-decoder architectures, it allows existing self-supervised methods to achieve significant improvements. The quality of the presentation in this paper is generally excellent, but some of the illustrations and content still need to be improved for clarity. See “Weaknesses” for more details. The method proposed in this paper is aimed at self-supervised learning of 3D point clouds and has considerable application value in practical applications.

Weaknesses

1. The quality of some figure illustrations in the paper concerns me; The fonts in Figures 4 and 5 are too small and both are in bold, which does not seem to fit the human eye. 2. The elements in Sections 3.3 and 3.4 appear to be improvements over the existing technology and Equation 4-7 should be interpreted differently. 3. Some parts of this paper conflict and it is recommended that the word “section” be capitalized and standardized. 4. The experimental analysis shows that the decrease in the number of parameters of the LCM is extremely obvious, which should be attributed to the Mamba model. It is suggested that the authors analyze the Transformer and Mamba network parameters in detail.

Questions

The significance of this work seems to me and whether the authors will choose to make it fully open source? I hope this work can contribute to the development of the field.

Rating

8

Confidence

5

Soundness

3

Presentation

2

Contribution

4

Limitations

No obvious limitations, but some elements require more detailed explanation.

Summary

This paper first proposes a locally constrained compact encoder, which leverages static local geometric constraints to aggregate the most relevant information for each patch token, achieving an elegant balance between performance and efficiency. Moreover, this paper also proposes a locally constrained Mamba-based decoder for masked point modeling. The authors verify the effectiveness and efficiency of the proposed model on multiple pre-training strategies and downstream tasks.

Strengths

1. The model LCM proposed by the authors is novel and very efficient, achieving leading performance with only 2.7M parameters, which is much lower than existing mainstream models. 2. The authors provide a rational explanation for the model design, i.e., an encoder design based on redundancy reduction and a decoder design based on mutual information maximization. 3. Extensive experiments show the effectiveness of the proposed method. 4. The paper is well-organized. The tables, figures and notations are clear.

Weaknesses

Some experimental comparisons are insufficient, and certain details are not clearly described. 1. I noticed that the authors only compared the results of the proposed method with PointGPT-S (NeurIPS 2023) and PointGPT-B. In fact, PointGPT-L offers more powerful performance. Can the author provide a specific explanation for this? 2. The ShapeNet dataset is relatively small, with a limited number of 3D models. Could the author provide results of the proposed method pre-trained on a larger dataset, such as the unlabeled hybrid dataset used in PointGPT (NeurIPS 2023)? This would have a significant impact on demonstrating the generalization ability of the proposed method.

Questions

See the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors' description of the limitations of their work is reasonable and effective solutions are given.

Summary

To address the issues of quadratic complexity and constrained decoders in existing masked point modeling methods based on Transformers, this paper proposes a locally constrained compact point cloud model. First, to tackle the complexity problem, the paper presents an observational experiment with top-K attention to demonstrate the importance of redundancy reduction in point cloud analysis. Based on this redundancy reduction idea, the paper then introduces a locally constrained compact encoder by replacing the self-attention layer with a local aggregation layer. Finally, to overcome the limited reconstruction potential of the original Transformer decoder, the paper proposes a locally constrained Mamba decoder and demonstrates its superiority through both experimental results and information theory analysis.

Strengths

1) The motivation of this paper is strong, and the mentioned efficiency and reconstruction potential issues of Transformer is important in point cloud analysis; 2) The paper presents a locally constrained compact point cloud model (LCM) in the field of point cloud self-supervised learning. This model design is quite novel. 3) The proposed model is universal and achieves impressive results, reaching state-of-the-art effects with a minimal number of parameters. 4) The paper also offers many well-reasoned and insightful explanations for observed phenomena.

Weaknesses

1) In the model design, as shown in Figure 5, I noticed that the proposed method seems to perform KNN at each layer. In fact, KNN is a computationally intensive operation. Considering practical applications, this operation could significantly increase the actual inference time of the model. I hope the author can provide a reasonable explanation or experimental validation to address this concern. 2) The results in Table 1 appear to show the highest performance. However, due to the influence of random seeds, this is not sufficient to reflect the true effectiveness of the model. The authors need to explain how these experimental results were selected, and also report the average performance over multiple trials. 3) There is an error in the citations. Both the Transformer and PointMamba references in Table 1 under "supervised learning only" point to the paper "Pointmamba: A simple state space model for point cloud analysis."

Questions

My concerns and suggestions have already been outlined in the weaknesses section. I hope the author can provide further explanations on these issues.

Rating

6

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The author adequately explains the limitations of their work. This static importance of the model may somewhat restrict its actual performance. I hope this limitation can be addressed in future work.

Summary

This paper proposes LCM, a locally constrained compact point cloud model, to improve the efficiency and performance of point cloud processing tasks. It consists of a locally constrained compact encoder and a locally constrained Mamba-based decoder. A locally constrained compact encoder utilizes the proposed local aggregation layer to replace self-attention to achieve an elegant balance between performance and efficiency. A locally constrained Mamba-based decoder introduces LCFFN after each Mamba SSM layer, maximizing the perceived point cloud geometry from unmasked patches. By focusing on local geometric constraints and leveraging SSM, the model achieves a balance between performance and efficiency.

Strengths

1. Introducing a locally constrained compact encoder and Mamba-based decoder is a creative solution that improves both performance and efficiency. 2. The paper offers a thorough explanation of the proposed model, including the local aggregation layers and the integration of state space models, making the methodology clear and reproducible. 3. The paper provides comprehensive experimental results demonstrating the effectiveness of the proposed method across multiple tasks and datasets.

Weaknesses

1. ***[Static Importance Perception]*** The reliance on static local constraints may limit the model’s ability to dynamically identify and focus on important regions of the point cloud, potentially missing critical information. 2. ***[Long-Range Dependency Modeling]*** While the local constraints improve efficiency, they might not capture long-range dependencies as effectively as self-attention mechanisms, which could be a limitation in some applications. 3. ***[Scene-level semantic segmentation]*** It would be better if the author could provide fine-tuning performance on semantic segmentation with scene-level point cloud datasets like ScanNet or ScanNet++ to make the claim stronger.

Questions

1.The authors have validated the effectiveness of the proposed LCM model in scene data through indoor scene detection tasks. However, is the proposed model also effective for other scene tasks, such as indoor semantic segmentation? 2.In the supplementary materials, the authors used the proposed mamba-based decoder as an encoder to verify the impact of patch order and LCFFN. I have the following two questions: 1) Firstly, the performance listed in Figure 8(a) seems to show a significant disparity compared to the classification performance listed in Figure 2(c) and Table 1. What is the reason for this disparity? 2) Secondly, does this disparity indicate that the effectiveness of the proposed mamba-based decoder is inferior to that of the Transformer and the proposed LCM encoder?

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitations presented by the authors are reasonable. However, the solutions to these limitations appear to lack specificity. I would like to inquire whether the authors have any concrete solutions regarding "efficient dynamic importance." This is merely to ask if the authors have any reasonable and specific ideas without requiring actual experimental results.

Reviewer's Response

Thanks to the authors for their careful responses. All my concerns have been addressed. In return, I will firm up my positive assessment of the work and upgrade the rating. As a side note, it is recommended that the authors add some excellent work on self-supervised point clouds. [1] Wu, et al. Self-supervised intra-modal and cross-modal contrastive learning for point cloud understanding. IEEE TMM 2023. [2] Liu, et al. Inter-modal masked autoencoder for self-supervised learning on point clouds. IEEE TMM 2024.

Discussions

Dear Reviewers, Thank you for your efforts. Please review the rebuttals, engage in the discussions, and provide your final ratings. Thank you again for your valuable contributions. AC

After the rebuttal

Thank the authors for your rebuttals carefully. My concerns are well addressed. Thus, I keep my original rate to accept this paper.