Annotator: A Generic Active Learning Baseline for LiDAR Semantic Segmentation

Summary

This work benchmarks several point selection approaches for label-efficient LiDAR point cloud semantic segmentation. Their proposed criterion leads to better results than existing selection approaches and good generalization with few annotations. They use several settings depending on the accessibility of auxiliary data to overcome the "cold-sart" problem.

Strengths

- the proposed criterion is sensible and leads to good results - the experiments are extensive - the definition of three different setting (no auxiliary data, pretrained model, auxiliary annotated data) is helpful - active learning is a critical problem for the industrialization of ML methods and is not studied nearly enough

Weaknesses

- the entire contribution boils down to the "voxel confusion degree" selection criterion. Yet the authors present the entire field of active learning as a contribution - the quality of writing is quite low, with many misused technical terms (softmax cross-entropy) and vague statements (active learning [is] an optimum paradigm). - the authors only compared their approach to 2 selection criteria [27,76], but could have used many other approaches, such as [25, 35, 58, 59, 78, 84]. Otherwise, they need to explain why these approaches do not apply - The results seem dubious. Annotating only 5 voxels (how many points total?) leads to such strong performance in a setting with many classes. Even when using 40 million parameters- networks trained from scratch and no particular measures taken to avoid the extreme overfitting bound to occur?

Questions

Q1) How do you explain that large networks can be trained with cross entropy and only a handful of points? How many epochs are you using? Q2) Why not compare to any of the other DA approaches [25, 35, 58, 59, 78, 84] S1) proof read the entire text looking for mistakes, vague and imprecise statements.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

1 poor

Limitations

not given

Summary

This paper introduced a baseline for active learning called Annotator for LiDAR point cloud semantic segmentation. The paper includes an analysis of various active selection strategies, including random selection, entropy-based selection, margin-based selection, and a novel strategy called voxel confusion degree (VCD). The study investigates three different setups: active learning (AL), active source-free domain adaptation (ASFDA), and active domain adaptation (ADA). Through experiments conducted on multiple Simulation-to-Real and Real-to-Real benchmarks, the proposed method demonstrates its effectiveness.

Strengths

1. The paper is well-written and easy to follow. 2. Considering the significant challenge of annotating point-wise point cloud data in LiDAR segmentation, exploring active learning approaches for this task is valuable and important. 3. The idea of annotator is simple but effective. 4. The authors conducted comprehensive benchmarks across different setups, including AL, ASFDA, and ADA. This extensive evaluation covers several novel investigations that have not been explored before.

Weaknesses

1. This paper lacks in-depth insights and analysis. What is the fundamental intuition behind it? Why do these active selection strategies contribute to performance gains? Furthermore, what sets the proposed VCD apart and enables it to achieve superior performance improvements? 2. The selection of voxel grids plays a crucial role in determining the final performance, particularly when it comes to random selection. However, this paper did not include error bars in relation to the random seed for multiple experiments.

Questions

1. Why did the annotator achieve significantly smaller performance gains in SynLiDAR->KITTI in Table 2? 2. A minor issue: The per-class IoU performance of PolarMix is provided in its supplementary material, which is missing in Tables 2 and 3. 3. Please also refer to the weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

This paper discusses the limitations of their work in the conclusion. However, it is suggested to have a separate section dedicated specifically to addressing these limitations.

Summary

This work proposes a general and efficient data annotation pipeline, namely Annotator, to label LiDAR data for semantic segmentation. Specifically, the proposed method introduces a voxel-centric online selection strategy to determine which voxel should be annotated by humans. Voxel confusion degree (VCD) is thus proposed based on the classification labels predicted by a semantic segmentation network. To prove its efficiency, the authors also conduct some experiments on domain adaption and they use 1,000 times fewer annotations.

Strengths

This work proposes a new criterion namely voxel confusion degree (VCD) to evaluate the certainty of a voxel. It seems using this criterion the authors only need very few times of annotations to achieve better performance. The experiments support the claim of this paper: the proposed method uses the least annotations and achieves better performance.

Weaknesses

1. In Fig. 2 (ii), it is ASFDA source-free domain adaptation but in (1) pre-trained with source data. This is a bit confusing. 2. The VCD is well defined. How to use it to determine the next annotated voxel is not clear. Should we choose a voxel with larger VCD? Why is that? The motivation or insights are missing. 3. Algorithm 1 is not very informative but more like active learning pipeline. 4. The proposed method Annotator can be only applied to voxelized data. How does the size of voxels affect the final annotation rounds? 5. The compared baselines are all voxel based annotation. How about labeling points in BEV? In other words, there would be many ways to annotate LiDAR data. Why voxel is the best? Or is the proposed method best suitable for voxel based partition?

Questions

1 whether the proposed method can be applied to indoor semantic segmentation? 2. Why choose voxel based partition? 3. In Tables 1, 2, 3 The oracle performance (full annotations) should be provided for comparisons. 4. Can the proposed method be applied to other tasks, such as detection or instance segmentation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have addressed the concerns properly.

Summary

This paper presents Annotator, a general and efficient active learning baseline for LiDAR semantic segmentation, which can adapt to different settings and scenarios with minimal annotation cost. Annotator consists of a voxel-centric online selection strategy that exploits the local topology and structure of point clouds to query the most informative voxel grids for annotation. Annotator can also leverage an auxiliary model to address the cold start problem. The paper evaluates Annotator on two datasets (SynLiDAR and SemanticKITTI) with two network architectures (MinkNet and SPVCNN), and shows that Annotator can achieve on-par performance with the fully supervised counterpart using 1000 fewer annotations and outperform existing methods under various active learning settings. The paper states that Annotator is a simple and general solution for label-efficient 3D perception.

Strengths

**Originality**: The paper proposes 3 ideas, a voxel-centric online active learning baseline, a label acquisition strategy (VCD), and is generally applicable and works for different network architectures. **Quality**: The paper provides a thorough **experimental evaluation** of Annotator on several simulation-to-real and real-to-real LiDAR semantic segmentation tasks, using different network architectures and baselines. The paper also conducts **ablation studies** to analyze the impact of different components of Annotator, such as voxel size, selection strategy, and auxiliary model. The paper demonstrates that Annotator can achieve **on-par or superior performance** with the fully supervised counterpart using 1000 fewer annotations, and significantly outperform other state-of-the-art methods. **Clarity**: The paper also provides sufficient background information and related work to situate the contribution of Annotator in the context of existing literature on LiDAR perception, active learning, and domain adaptation. The Supplementary Material is well-written and contains enough details and information as needed. **Significance**: The paper addresses an important and challenging problem of label-efficient LiDAR semantic segmentation, which has many applications in autonomous driving, robotics, and 3D scene understanding.

Weaknesses

**Major Issues:** - **Insufficient novelty and contribution**: this paper perform an analysis of several common selection strategies, e.g., Random, Entropy and Margin, which are all proposed in previous research. The newly proposed VCD strategy lacks justification for its design, and contains too few details (L195-L199). The pipelines of distinct active learning settings (e.g., AL, ASFDA, and ADA) seem natural and basic. - **Insufficient results for experiments:** Although the authors state in the main text, "***Experimentally***, we find that the large voxel grid is also more robust to the noise and sparsity of the point clouds", they provide no experimental results of different $\Delta$ in their main text and Supplementary Material. - **Insufficient justifications**: For example, about **Voxel-centric** selection, some justifications are missing in this paper. Why choosing voxel-based selection rather than other methods, e.g., point-based, range-image-based, BEV, etc (L101-102)? Any advantages and limitations of **Voxel-based** methods? **Minor Issues:** - **Readability**: Some figures and text in figures (e.g., Figure 6 and Figure A3-A4) are too small for readability. Authors should attach more detailed images/visualization to the supplementary material. - **Variables undefined** although they are obvious in meaning: e.g., $X_t$ in L155, $X_s$ and $Y_s$ in L157.

Questions

- Is the term '*cell*' (L190) the same meaning as '*voxel*' (L189)? If they're interchangeable, it might be better to use the same word '*voxel*'; if not, it's a good idea to explain their differences, relations, or the definition of '*cell*'.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Authors should be rewarded that the limitations and potential negative societal impact are explicitly mentioned in the Conclusion.

Summary

This paper proposes a voxel-centric online active learning baseline that efficiently reduces the labeling cost of enormous point clouds and effectively facilitates learning with a limited budget. The contribution of this paper can be summarized in three aspects: 1. A voxel-centric online active learning baseline that efficiently reduces the labeling cost of enormous point clouds and effectively facilitates learning with a limited budget. 2. A novel label acquisition strategy, voxel confusion degree (VCD), that requires 1000 times fewer annotations while reaching a close segmentation performance to that of the fully supervised counterpart. 3. A generally applicable and works for different network architectures (e.g., MinkNet, SPVCNN, etc.), in distribution or out of distribution setting (i.e., AL, ASFDA, and ADA), and simulation-to-real and real-to-real scenarios with consistent performance gains.

Strengths

1. active learning is not entirely new in this field, but reducing the data volume requirement at the order of 1000 times is still very impressive 1. active learning coupled with domain adaptation has great practical significance, but little work has been done to consider the problem in 3D domains. The Annotator aims to minimize human labor in a new domain, regardless of whether samples from an auxiliary domain are available or not. 2. Methodologically, Annotator adopts a voxel-based representation for structured LiDAR data, which is different from the scan-based representation in UniDA3D. 3. Furthermore, Annotator can perform online active sampling that is more efficient than UniDA3D in terms of computation cost.

Weaknesses

1. The cost of developing an AI based solution include multiple sectors: 1) data collection; 2) data annotation; 3) model training; 4) model deployment and integration. One of the most costly stage, both in time and money, is annotation. The AL paradigm, instead, proposes to use computation to reduce the total cost, by using computation to reduce the amount of data needed for human annotation. In this paper, an additional pre-training phase is added, it is unclear how to evaluate the cost balance, eg, increased cost of computation vs reduced cost of human annotation. In some sense, this is not the weakness/limitation of this particular paper, but rather applies to the whole AL paradigm.

Questions

1. voxel based methods is one of the major and popular representations for lidar perception tasks, however there do exists other representations such as range view and point based methods, where the concept of voxel does not exist. question is, how does the proposed method apply to non-voxelization based methods? 2. segmentation tasks require dense point-wise annotation, but detection tasks only require, to some extent, much sparser box level annotations, how does the proposed method apply to detection based tasks? one would expect that the data volume reduction persists but not as impressive as 1000x

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

no potential negative societal impact is noticed by the reviewer to the best of their knowledge

Thank you

The authors have addressed all my queries and reservations. While I initially perceived the contribution as too limited, several points have swayed my opinion: - the authors make a good point relative to the lack of standardized baseline, and their method makes a good stepping stone in that direction. - they added more comparisons that still show the merits of their approach. - the method, while simple, yields superior results. This makes its simplicity an asset. I also recognize that some of my earlier concerns arose from my own minsunderstanding (specifically regarding the 5 voxels equating to 35 points). Considering the above factors, I am now leaning towards endorsing this paper and have subsequently revised my rating.

A good paper

Thank you for additional experiments. The authors have addressed all my concerns and questions. After reviewing the rebuttal, I am satisfied with the justifications provided for the voxel-centric online active learning approach. The additional ablation study on voxel size is useful for understanding the impact of the hyperparameter. I agree that the voxel-based selection strategy provides an efficient way to query salient regions in large 3D point clouds. I believe this is a solid contribution. The voxel-centric online active learning approach could be valuable for label-efficient LiDAR segmentation. Therefore, I am inclined to accept this paper and have raised my final rating.

Thanks for the response

The response has addressed most of my concerns. Given that the authors emphasize the significance of this work as a baseline for future studies, could you please provide information regarding the potential release of the code? If so, when is the expected release date?