Is Your LiDAR Placement Optimized for 3D Scene Understanding?

Summary

This paper proposes Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. The framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, this paper introduces a Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, this paper propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi LiDAR perception, the authors collect a 364,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using palce3D outperform various baselines.

Strengths

- This paper first attempt at investigating the impact of multi-LiDAR placements for 3D semantic scene, which is seldom explored in LiDAR-based detection and segmenation areas. - This paper proposes a more comprehensive metric M-SOG. Experiments show M-SOG is more relevant to the model performance than S-MIG, demonstrating its effectiveness to represent the scene coverage. - Guided by M-SOG, a LiDAR placement optimization methods is proposed in this work which can provide a solid baseline for LiDAR placement optimization.

Weaknesses

- The representation may be a little confused. The M-SOG and the optimization method, under my understand, is task-independent. However, there are two optimized LiDAR placement related to Det and Seg tasks, which is a little confused. - To demonstrate the effectiveness of M-SOG, it's recommended to draw a scatter plot of M-SOG and model performance in Tab2 and Tab3, which can be more intuitive.

Questions

- I'm curious about the influence of the LiDAR numbers. It's better to add a experiment to explore the most cost-effective LiDAR number. - M-SOG seems to be a global metric. Can It also reflect the scene coverage of a specific local area?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Nan

Summary

The paper targets an overlooked but important aspect of 3D scene understanding -- the influence of LiDAR sensor placements for the success of 3D perception tasks, such as 3D detection and 3D segmentation. The authors introduce Place3D, a comprehensive pipeline for optimizing LiDAR placement in autonomous driving scenarios to enhance 3D scene understanding, especially under some adverse conditions. Indeed, current single-LiDAR systems and datasets tend to fail to capture the complexities of real-world environments. This work proposes to address this by using the Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. M-SOG measures the information gain and scene understanding capability by calculating the entropy of semantic voxel distributions. The optimization strategy employs the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to iteratively improve LiDAR configurations. The objective function is the M-SOG score, ensuring that the optimized placements are close to the global optimum. Compared to the existing S-MIG, this work makes a significant contribution by extending the binomial occupancy distribution to multinomial occupancy distribution through multiple semantic labels for better scene understanding. Place3D also introduces a dataset of 364,000 frames collected under various conditions, including severe weather and sensor failures. The optimized placements show superior performance in LiDAR semantic segmentation and 3D object detection tasks compared to existing baselines.

Strengths

1. This work focuses on a less-explored but rather important perspective of 3D scene understanding. The established LiDAR placement evaluation benchmark and the proposed placement optimization method could open up some new and interesting research directions in related research areas. 2. An innovative metric, M-SOG, for evaluating LiDAR placement quality by incorporating semantic information, which improves accuracy over previous methods. Additionally, this work contributes a full-cycle approach from LiDAR placement optimization to data generation and downstream evaluation. 3. The proposed Place3D dataset includes several adverse conditions for realistic and practical evaluation, demonstrating significant improvements in robustness. Additionally, the use of CMA-ES ensures that LiDAR placements are close to the global optimum, with strong theoretical optimality certification.

Weaknesses

1. In the experiments, although the baseline LiDAR placement methods are based on existing configuration of current autonomous vehicle systems (Fig. 7), it is better to compare the proposed method with some randomly sampled LiDAR placements. This can further justify the effectiveness of CMA-ES compared to a wider range of scenarios. 2. It is not clear how the density of the occupancy grid (Sec. 3.3) will influence the surrogate metric, as the estimation of semantic distribution depends on the number of samples. 3. The experiments are conducted in the simulation. Although CARLA is a rather realistic simulator and has been widely adopted in academia, it is unknown how well the proposed method works in the real-world setting. Perhaps it is not feasible to conduct on-site experiments, but more justifications regarding this aspect are needed for further discussion. 4. The optimization process and data generation might require substantial computational resources. Either the main body or the appendix omitted such details. It is highly recommended to supplement such details as they are critical for follow-up works. 5. Can the proposed benchmarks and optimization strategies be extended to other 3D scene understanding tasks, such as 3D object tracking, semantic occupancy prediction, etc?

Questions

Please refer to the above section for detailed questions.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have included some discussions of the potential limitations and societal impact in the appendix.

Summary

The authors present a very interesting study of optimal lidar placement on autonomous vehicles. They introduce a novel measure of the quality of lidar placement called M-SOG, and also discuss an optimization approach to find the best placement. They measure the impact of the placement on the important vision tasks in autonomous vehicles, such as object detection and semantic segmentation, showing that the placement metric is aligned with the improved quality. Lastly, in addition to clear weather they also analyze the impact on data when the conditions are degraded (such as fog or faulty sensors). Moreover, they also provide the data and the code that supports their work.

Strengths

- Very interesting and timely work. - The new metric and the results could be of interest to the self-driving community, and the methods should not be difficult to implement. - The presented results are quite promising.

Weaknesses

- The paper writing is can be improved significantly, as the method is not actually clearly explained. - The authors tried to stuff too many things in a single paper (metric and degraded sensors), which degraded focus and impact of the work. - The experiments can be improved significantly. (detailed comments on the above weaknesses can be found in the section below)

Questions

- Line 107, "voxelized grips" -> "voxels"? - Line 110, "where each voxel is assumed to be independently and identically distributed", what does this actually mean? The voxels are clearly not independent of each other. - The notation can be improved, as it is confusing at times and not properly introduced. E.g., line 112, what is T, 1. - Along similar lines, H is not properly introduced a few lines below. - Line 143, should P_SOG also have k-index in its name? As it is defined for a particular k, as per the given equation. - In the paragraph starting with line 150, the authors discuss the metric. However, given that the metric is computed using only a subset of voxels, is there an invalid setup that the metric would not capture properly, such as putting lidars below the car or something similar? - Fig 2, would be good to zoom in on the roof of the car to make it more visible, as that is the relevant part in this figure. - The optimization in Section 3.3 is not well explained, and the notation is also not properly introduced. - E.g., the authors do not properly explain what is the lidar configuration space, or what are the constraints. Some details can be gleaned here and there, but given how critical this part is for their work the authors should do a much better job explaining it. - Along these lines, line 173 mentions cov-mat of distribution, distribution of what? This is very poorly explained. - Also, in this paragraph the authors use k-index for samples, but k was already used for classes? - How significant and how practical is the theoretical analysis? The authors introduce C_M as a gap, but this constant is not discuss anywhere, or how large it is. This makes the theoretical discussion quite weak. More discussion and more clarifications are needed. - Line 238, Tables 2 and 3 don't actually show this. - The detectors used in these experiments are not clearly discussed. - In general, it seems that the authors tried to stuff too many things at the same time in the work and in the experiments. This made them relegate many results and discussions to the appendix, even those that are critical to the work. It seems that it would be much better to focus on one aspect first (e.g., metric + impact on vision results in clear weather) and only then expand from there. - What are the points in Fig 4, not clear and not discussed. - The adverse aspect is very important, yet the authors do not spend a lot of time/space on it, and the results are just given. This is unfortunate, and weakens their argument around this part of their work. - Line 253, this was already mentioned in line 238? - Line 263, why is the placement good for adverse weather? Any deeper insights? How generalizable are these placements? All these important questions are not discussed at all, and the authors just skip them. This is another symptom of stuffing too many things in one paper and losing focus and depth. - Line 270, the authors just move all important ablation results to the appendix, which is not acceptable. At this point the authors are just abusing the appendix. Please note that the appendix is not meant to increase the page limit, and critical experiments/results/discussions should NOT be moved there. - More discussions and insights about the found placement should be added, such as around 2D or roll solution. Any advice for the designers, any interesting findings? This is not provided at all. ################### FEEDBACK AFTER THE REBUTTAL ################### I would like to thank the authors for responding to my comments, and also to the other reviewers who helped me further understand the paper. However, after reading through all the other responses/discussion it seems that my main concerns were not addressed. E.g.,while I said that the paper is too stuffed and poorly explained, the authors just said that they will add more explanations and that's that. First, I'm not sure where would this be added since there is no space and the paper is overstretched as is, and second, the required changes are significant and just saying that it will be addressed is not enough, the paper would need to be re-reviewed. Moreover, the authors marked my comments under "Other Minor Modifications", which just shows that they did not appreciate the severity of my comments. I do like the idea and I do think that the work has future, but with the current execution I just can not see how can I increase my recommendation. The paper needs quite a lot of attention and rewriting, so much so that I am not confident that the authors would not be able to do that with quick updates, especially given their responses that made light of some of my more important comments.

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors do not discuss the limitations in a separate section.

Summary

This paper proposes a framework called Place3D to investigate the placement of multiple LiDAR sensors for semantic segmentation and object detection task under various weather conditions. Place3D mainly introduce a Surrogate Metric of Semantic Occupancy Grids (M-SOG) to evaluate the perception performance with different sensors placements. Besides, an optimization approach is applied to refine the LiDAR placement. The method is evaluated on the simulated dataset with CARLA, showing the advantages of the refined placements.

Strengths

- The motivation is interesting and the paper is well-written. - A new metric and a dataset are proposed for the evaluation of the placement. - The experiments are extensive, three typical models are evaluated on 7 weather scenarios. - Comparison with 7 popular placement methods show the advantages of the refined placement with the proposed methods.

Weaknesses

- Although the proposed method investigates LiDAR placement for more tasks under more weathers, the core formulation is similar to that in [34] as cited in the paper. - Only one type of 16-beam LiDAR sensor is investigated. Actually, sensors with more laser beams (64 or even 128) are becoming low-cost and popular. It is important to evaluate the effect of the placement as the point cloud data get dense. - During the companions in Tab.2,3, different refined placements are applied for different tasks. How is the performance if applied a Seg-optimized placement to the Det task? It is impractical to change the placement for different task. Is that possible to find an optimal placement for both tasks?

Questions

- In the dataset comparison part, it is written that no adverse weather for Nuscenes and Waymo. They may include rainy weather and even more (e.g., Snow). Although in the appendix part, please also have a check on that. - How is the solution spaces for the LiDAR placement? Are the displacements along z-axis or even rotation are considered during the optimization? - Is there any intuitive interpretation about why refined sensor placement are beneficial to different tasks?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed the possible limitation in data collection.

Looking Forward to Discussion with You

**Dear Reviewer `DpmT`,** We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback. --- In response to your insightful comments, we have made the following revisions to our manuscript: - We have improved the explanation of the optimization approach using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and added more details to the theoretical analysis. - we have clarified our experiments for comparison between our M-SOG and existing placement method S-MIG in Fig 4 and provided several findings and insight into LiDAR placement in our experiments. - We have improved the clarity of our figures, such as zooming in on the relevant parts in Fig 2. To better explain Tabs 2 and 3, we have added a scatter plot of M-SOG and model performance, as shown in the attached single-page Rebuttal PDF. - We have clarified notations and improved the overall writing to make the method more understandable. For typos, We corrected "voxelized grid" to "voxels". For notations, we properly introduced notations $T$ and $H$, included the $k$ index in $P_{SOG}$, and revised the $k$ index in optimization to avoid overlap. We also did a thorough check to ensure all notations were properly introduced and clearly explained. --- Additionally, we provide a list of **Summary of Notations** as follows: |Notation|Explanation| |-|-| |$L$|Length of the Region of Interest (ROI) |$W$|Width of the Region of Interest (ROI) |$H$|Height of the Region of Interest (ROI) |$\delta_L$|Resolution of voxelization in the length dimension |$\delta_W$|Resolution of voxelization in the width dimension |$\delta_H$|Resolution of voxelization in the height dimension |$S$|Set of voxels in the Region of Interest (ROI) |$N$|Total number of voxelized grids |$K$|Total number of semantic labels |$v_i$|Voxel $i$ |$p(v_i)$|Probability of voxel $v_i$ being occupied |$y_t$|Frame $t$ |$T$|Total number of frames |$H(v_i)$|Entropy of voxel $v_i$ |$H(v_{i} | L)$|Conditional entropy of voxel $v_i$ given LiDAR placement $L$ |$H_{POG}$|Total entropy of POG (Probabilistic Occupancy Grids) |$p_{POG}$|Joint probability of all non-zero voxels in the ROI |$p_{POG | L}$|Conditional probability of POG given LiDAR placement $L$ |$H_{POG | L}$|Conditional entropy of POG given LiDAR placement $L$ |$\Delta H$|Information gain of 3D scene understanding |$S_{SOG}$|Semantic occupancy grid |$s_{y_k}^{(t)}$|The set of voxels occupied by semantic label $y_k$ at frame $t$ |$p(v_i = y_k)$|Probability of voxel $v_i$ being occupied by semantic label $y_k$ |$\mathcal{P}_{SOG}$|Conditional joint distribution of SOG given LiDAR placement $L$ |$\mathcal{H}_{SOG}$|Entropy of SOG |$L_j$|LiDAR configuration $j$ |$\mathcal{H}^{L=L_j}_{SOG}$|Conditional entropy distribution of P-SOG over the voxel set $S | L_j$ |$\mathbf{M}_{SOG}(L_j)$|Normalized surrogate metric of SOG |$\mathbb{E}$|Expectation operator |$\mathbf{u}_j$|LiDAR configuration $j$ in objective function $F(\mathbf{u}_j)$ |$F(\mathbf{u}_j)$|Objective function for LiDAR configuration $j$ |$P(\mathbf{u})$|Physical constraint for LiDAR configuration |$G(\mathbf{u})$|Function representing the surrogate metric |$U$|Set of all potential LiDAR configurations |$\mathbf{u}^*$|Optimal LiDAR configuration |$\mathbf{m}^{(k)}$|Mean vector of the distribution at iteration $k$ |$\sigma^{(k)}$|Step size of the distribution at iteration $k$ |$\mathbf{C}^{(k)}$|Covariance matrix of the distribution at iteration $k$ |$\mathcal{N}(\mathbf{m}^{(k)}, (\sigma^{(k)})^2\mathbf{C}^{(k)})$|Normal distribution for sampling ($k$ will be updated to $g$ to avoid overlap with semantic class) |$\delta$|Density for discretizing the configuration space $U$ |$\mathbf{u}_i^{(k)}$|Sampled candidate configuration at iteration $k$ |$M_k$|Number of best solutions used to update $m^{(k+1)}$ |$w_i$|Weights based on solution fitness |$\mathbf{p}_\mathbf{C}^{(k+1)}$|Evolution path accumulating information about the direction of successful steps |$c_\mathbf{C}$|Learning rate for updating the covariance matrix |$p_{\sigma}$|Evolution path for step size adaptation |$c_{\sigma}$|Learning rate for updating the evolution path $p_{\sigma}$ |$d_{\sigma}$|Normalization factor for step size adaptation |$E\|N(0, I)\|$|Expected length of a standard normally distributed random vector |$\| \cdot \|$|Euclidean norm --- We hope these revisions adequately address your concerns. We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made. --- Once again, we sincerely thank you for your contributions to improving the quality of this work. Best regards, The Authors of Submission 1401

Looking Forward to Discussion with You

**Dear Reviewer `PfQe`,** We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback. --- In response to your insightful comments, we have made the following revisions to our manuscript: - We have included additional experiments comparing the proposed method with randomly sampled LiDAR placements to justify the effectiveness of our method. - We have included details on the computational resources required for the optimization process and data generation. - We have discussed the feasibility of extending our benchmarks and optimization strategies to other 3D scene understanding tasks. - We have supplemented details on the influence of the density of the occupancy grid on the surrogate metric. --- We hope these revisions adequately address your concerns. We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made. --- Once again, we sincerely thank you for your contributions to improving the quality of this work. Best regards, The Authors of Submission 1401

Looking Forward to Discussion with You

**Dear Reviewer `iXBb`,** We sincerely appreciate your time and effort in reviewing our manuscript and providing valuable feedback. --- In response to your insightful comments, we have made the following revisions to our manuscript: - We have clarified the task-independent nature of M-SOG and the optimization method and explained the variations in optimized placements for different tasks. - We have added experiments to explore the influence of the number of LiDARs on performance, showing the most cost-effective LiDAR numbers. - We have included discussions on the local area coverage using M-SOG, adapting the metric to reflect specific regions. - We have added scatter plots of the M-SOG and model performance, which can be found in the attached single-page Rebuttal PDF file. --- We hope these revisions adequately address your concerns. We look forward to actively participating in the **Author-Reviewer Discussion** session and welcome any additional feedback you might have on the manuscript or the changes we have made. --- Once again, we sincerely thank you for your contributions to improving the quality of this work. Best regards, The Authors of Submission 1401

Summary of Author-Reviewer Discussion Session

**Dear Reviewers, Area Chairs, and Program Chairs,** As the Author-Reviewer Discussion session comes to a close, we would like to express our sincere gratitude for your time, effort, and thoughtful feedback throughout this review process. We are pleased that our rebuttal and revisions have addressed many of the reviewers’ concerns, and we appreciate the positive feedback we received. --- Our revisions were guided by your insightful suggestions, resulting in several **key improvements**: - We clarified the core formulation and highlighted the improvements over the work cited in [34], specifically addressing critical limitations in S-MIG, as suggested by Reviewer `8eRR`. We also improved the explanation of our optimization approach using CMA-ES and expanded the theoretical analysis as per the Reviewer `DpmT`’s advice. - We included a scatter plot illustrating the relationship between M-SOG and model performance, making our results more intuitive and easier to interpret, as suggested by Reviewer `iXBb`. Additionally, we clarified our comparisons between M-SOG and S-MIG and added further insights into LiDAR placement based on Reviewer `DpmT`’s suggestions. - We included higher-resolution LiDAR experiments, demonstrating the continued importance of sensor placement with denser point clouds, as recommended by Reviewer `8eRR`. We also included experiments on the influence of the number of LiDARs, as suggested by Reviewer `iXBb`. - We provided intuitive interpretations of why refined sensor placements are beneficial for various tasks, such as enhancing the field of view and capturing richer features, as suggested by Reviewer `8eRR`. We also discussed the feasibility of extending our benchmarks and optimization strategies to other 3D scene understanding tasks, following Reviewer `PfQe`’s advice. - We addressed specific concerns regarding adverse weather conditions in datasets, clarified notations, and improved the overall clarity of our figures and writing. We also included additional experiments and discussions, such as local area coverage using M-SOG, reflecting the valuable input from all reviewers. --- We would like to highlight again the **technical contributions** of this work: - To the best of our knowledge, Place3D is the first attempt to investigate the impact of multi-LiDAR placements for 3D semantic scene understanding across diverse conditions. It addresses the limitations of existing datasets that often lack adverse weather and sensor failure scenarios. - We introduce the Surrogate Metric of Semantic Occupancy Grids (M-SOG), an innovative metric designed to evaluate the quality of LiDAR placements effectively for both detection and segmentation tasks. This metric overcomes the limitations of previous methods by incorporating semantic information and handling occlusions. - Our novel optimization strategy, utilizing the M-SOG metric, refines LiDAR placements to enhance performance in both LiDAR semantic segmentation and 3D object detection. This approach demonstrates superior robustness and accuracy, outperforming various baselines by approximately 9%. - We contribute a large-scale, 364,000-frame dataset that includes both clean and adverse conditions, along with a comprehensive benchmark for evaluating LiDAR-based 3D scene understanding. This dataset is crucial for testing the robustness of 3D perception systems under realistic conditions. --- We hope these revisions have strengthened our manuscript and fully addressed your feedback. We are encouraged by the **positive recognition** of our work’s potential impact, as highlighted by Reviewer `PfQe`’s comments, who has upgraded the rating from *"6: Weak Accept"* to *"7: Accept"*. --- We sincerely thank you for your constructive suggestions and for helping us improve the quality of this work. Best regards, The Authors of Submission 1401