Spherical Frustum Sparse Convolution Network for LiDAR Point Cloud Semantic Segmentation

Summary

This paper proposes a Spherical Frustum Sparse Convolution Network to address the challenge of LiDAR point cloud semantic segmentation. Traditional approaches often project point clouds into 2D images and apply 2D convolutional neural networks (CNNs) or vision transformers, leading to quantization information loss due to the overlap of multiple points projected onto the same 2D location. To overcome this limitation, the paper introduces a novel spherical frustum structure that allows for direct processing of the 3D point cloud data, preserving spatial and geometric information. The proposed Spherical Frustum Sparse Convolution Network achieves more accurate semantic segmentation of LiDAR point clouds, offering a promising approach for robot perception and scene understanding.

Strengths

1. **Direct 3D Processing**: The Spherical Frustum Sparse Convolution Network processes the LiDAR point cloud data directly in 3D space, preserving the spatial and geometric information of the points. This avoids the information loss that occurs in traditional 2D projection-based methods. 2. **Reduced Quantization Loss**: By operating directly on the 3D point cloud, the proposed network significantly reduces quantization loss compared to projection-based approaches. This leads to more accurate semantic segmentation results. 3. **Efficient Sparse Convolution**: The network employs sparse convolution, which is tailored for sparse data such as point clouds. This allows for efficient computation and reduced memory usage, enabling the network to handle large-scale point cloud data. 4. **Spherical Frustum Structure**: The novel spherical frustum structure enables the network to capture contextual information from different orientations and distances, enhancing its ability to identify objects and regions in the point cloud. 5. **Improved Performance**: Experimental results show that the proposed Spherical Frustum Sparse Convolution Network achieves superior performance compared to existing methods on various LiDAR point cloud datasets, demonstrating its effectiveness for semantic segmentation tasks. 6. **Flexible and Extensible Framework**: The network framework is flexible and can be easily extended to incorporate additional components or techniques, providing opportunities for further improvements and adaptability to different applications.

Weaknesses

1. **Lack of Conceptual Novelty**: The use of spherical structures, while effective, appears to have been partially explored in SphereFormer. This limits the novelty of the proposed SFCNet in terms of the core concept. 2. **Performance Gap**: Based on the information provided, the performance of SFCNet for semantic segmentation of LiDAR point clouds lags far behind that of SphereFormer, whose radial window transformer structure may achieve higher recognition accuracy, especially for distant objects. 3. **Potential for Further Optimization**: Since SFCNet based on 2D projections lags behind SphereFormer based on 3D voxels, there may be room for optimization in terms of data input or network architecture. This paper does not discuss potential future avenues to address this performance gap or to work on different application scenarios.

Questions

Despite the fact that the paper is full of content, it always gives a discomfort in the presentation. Obviously, the authors are requested to resize Figure 1, remove the captions in the figures of Figures 2 and 3 and optimize their presentation. It is recommended that the internal spacing of Table 1 be increased appropriately. The authors are invited to review more carefully. In addition, similar to the work on semantic segmentation of large scenes, the authors should compare it with state-of-the-art methods such as DRQNet (ACMMM'2023) and JoSS (INFFUS'2024).

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper proposes a method SFCNet for point cloud semantic segmentation by directly projecting 3D point clouds into 2D spherical frustums, the main limitation of which lies in the large performance gap. Of course, the proposed method still has a large potential in specific scenarios.

Summary

This paper introduces SFCNet, a spherical frustum sparse convolution network designed for semantic segmentation of LiDAR point clouds. Traditional 2D projection methods suffer from quantized information loss when multiple points project onto the same pixel, leading to sub-optimal segmentation. To address this, the authors propose a spherical frustum structure that retains all points within the same pixel by using a hash-based representation. This structure is then utilized in spherical frustum sparse convolution, which considers both neighboring frustums and the nearest point within each frustum for convolution. Additionally, Frustum Farthest Point Sampling (F2PS) is used to sample points stored in spherical frustums. Experimental results on the SemanticKITTI and nuScenes datasets demonstrate that SFCNet outperforms conventional 2D projection-based methods, particularly in the segmentation of small objects, while maintaining complete point cloud information.

Strengths

1. The paper is well-written and easy to follow. The introduction to the Spherical Frustum Sparse Convolution and Frustum Farthest Point Sampling is clear and comprehensible. It provides a clear narrative structure supported by comprehensive figures and tables. The experimental settings are reasonable and well-documented. 2. The motivation to address quantized information loss during projection is well-founded.

Weaknesses

1. The experimental results do not demonstrate a clear superiority of the proposed method. While the SFCNet shows some mIoU improvements over prior projection methods on the SemanticKITTI and nuScenes datasets, these gains are marginal. Furthermore, the performance of SFCNet is still significantly below that of SOTA 3D voxel-based methods. 2. The paper does not provide an analysis of the computational efficiency of the proposed method. Given that SFCNet involves specific point sampling for convolution, assessing the computational efficiency is crucial to understand its practical significance and potential trade-offs in real-world applications.

Questions

In Table 1, it is clear that 3D voxel-based methods achieve significantly better results compared to point-based and 2D projection-based methods. Does this suggest that 3D voxel-based methods are inherently superior? Could you provide a comparison of the computational efficiency of SFCNet against other baseline methods? This information is crucial to evaluate the practical applicability and performance trade-offs of SFCNet.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

No limitations are discussed in the paper.

Summary

The paper introduces a Spherical Frustum Sparse Convolution Network (SFCNet), a novel approach for LiDAR point cloud semantic segmentation that addresses the quantized information loss existing in 2D projection-based methods. By utilizing a spherical frustum structure, the SFCNet preserves all points within a 2D projection area, thus maintaining the complete geometric structure of the data. This model improves the segmentation accuracy, particularly for small objects, through the innovative use of hash-based spherical frustum representation and a custom sampling method called Frustum Farthest Point Sampling (F2PS). Extensive evaluations on the SemanticKITTI and nuScenes datasets demonstrate superior performance over existing methods.

Strengths

S1. Originality of Spherical Frustum Structure. The spherical frustum structure is a novel approach that effectively preserves all the points within a given 2D projection area, mitigating the common issue of quantized information loss. S2. Efficiency of Representation. The proposed hash-based spherical frustum representation ensures memory efficiency and computational feasibility. S3. Performance Boost. The enhanced accuracy in segmenting small objects and maintaining geometric integrity significantly contributes to the field, especially for applications in autonomous driving and robotic navigation.

Weaknesses

W1. Performance Decrease on Large Classes. While SFCNet introduces significant advancements in semantic segmentation of LiDAR data, the model demonstrates slightly weaker performance on wide-range classes, such as roads and parking areas. This suggests that while the model excels in handling small object segmentation, its techniques might not be as effective for broader landscapes or larger objects.

Questions

Q1a. Performance Change on Large Classes. Could the authors provide insights into why SFCNet shows slightly weaker performance on wide-range classes, such as roads and parking areas (see also W1)? Q1b. Maintaining performance on large Classes. Are there any thoughts how the model could improve the small classes while maintaining performance on larger classes? Q2. Performance Comparison toward tail-end of Distribution. Given a different performance w.r.t. class sizes, additional tests on accuracy w.r.t. data changes might be helpful. How does SFCNet adapt its segmentation accuracy in non-standard environments (like in night settings, heavy rain, etc.)? Insights here could greatly influence its applicability in real-world settings.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Investigating how the system performs in unpredictable, non-standad settings would offer a clearer picture of its real-world readiness. The paper can also benefit from a deeper exploration of SFCNet's slightly weaker performance on wide-range classes such as roads and parking areas. Understanding the factors that contribute to these discrepancies could lead to improvements in the model's overall accuracy and applicability in real-world scenarios.

Summary

This paper introduces a novel spherical frustum structure for LiDAR point cloud semantic segmentation, addressing the quantized information loss in 2D projection-based methods. By preserving all points within frustums and employing a memory-efficient hash-based representation, the Spherical Frustum sparse Convolution Network (SFCNet) achieves superior segmentation accuracy, particularly for small objects. The proposed Frustum Farthest Point Sampling (F2PS) method ensures uniform sampling and computational efficiency. Extensive experiments on the SemanticKITTI and nuScenes datasets demonstrate SFCNet's improved performance over existing methods.

Strengths

- **Innovation in Data Representation**: The spherical frustum structure preserves all points, avoiding quantized information loss and improving segmentation accuracy, especially for small objects. - **Memory Efficiency**: The hash-based representation efficiently stores spherical frustums, minimizing memory usage compared to dense grid storage. - **Improved Sampling Method**: The Frustum Farthest Point Sampling (F2PS) method ensures uniform point cloud sampling, enhancing the retention of key information while maintaining computational efficiency.

Weaknesses

- The proposed methods, including the spherical frustum structure and hash-based representation, add layers of complexity that might pose challenges for practical implementation. Interesting to see the analyze on memory usage, parameter size, and inference time. - The paper focuses primarily on comparisons with 2D projection-based methods. While it mentions the performance gap with 3D voxel-based methods, a more detailed analysis and direct comparison with state-of-the-art 3D voxel-based approaches would provide a clearer picture. - The paper lacks a sensitivity analysis of key parameters, such as the stride size in the F2PS, the number of points in the spherical frustums, and the impact of different hash table configurations, etc. Understanding how sensitive the model performance is to these parameters would be valuable for tuning the model for specific applications. - LiDAR data often contains noise and outliers. The robustness of the proposed method to such imperfections is not thoroughly evaluated. Experiments that introduce varying levels of noise and assess the impact on segmentation performance would be beneficial in demonstrating the method’s robustness. - The qualitative results, although helpful, are somewhat limited in scope. Providing a more diverse set of visualizations, including different types of scenes (e.g., urban, rural, complex intersections), would offer a clearer picture of the method’s practical performance across various real-world scenarios.

Questions

I overall like the idea of this paper. However, my concerns are focused on the experimental results. If the authors can address my concerns regarding the experiments, I am inclined to raise my score.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

no potential negative social impact

Response from Reviewer

Thanks to the reviewers for their serious response! A lot of my questions and concerns have been addressed. I hope the authors will make the appropriate changes on the revised version of the paper. In return, I will further improve my rating.

Raised my rating to ACCEPT

I appreciate the authors' rebuttal and additional experiments. I have read the comments from other reviewers and the related rebuttal. The rebuttal has effectively addressed my main concerns. Additionally, I highly appreciate the responses to Q3 and Q5, and I suggest incorporating them into the supplementary material. Overall, I believe this is a technically solid paper, and I am inclined to accept it - I have raised my rating from borderline accept to **ACCEPT**.