TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight

Summary

The paper introduces TALoS, a novel test-time adaptation method for Semantic Scene Completion (SSC) that leverages information from driving environments. The key idea is to use observations and predictions from different moments as self-supervision signals to adapt the pretrained model. Specifically, LiDAR point clouds collected at various timestamps are used to create binary occupancy labels, and semantic predictions from these timestamps are fused based on their confidence levels. Additionally, a dual optimization scheme is proposed to exploit observations from both past and future frames. Experimental results on the SemanticKITTI dataset validate the effectiveness of the proposed method.

Strengths

1. The key idea of exploring the LiDAR observations and model predictions across frames during inference is novel and interesting. The Line of Sight binary occupancy and confidence-aware prediction fusion are effective and interesting. 2. The proposed dual optimization scheme is also useful and interesting. Such a strategy can make sure to exploit both historical and future frames for test-time adaptation. 3. The method shows improvements in both geometric and semantic segmentation tasks on the SemanticKITTI dataset, demonstrating its effectiveness. The paper includes extensive experiments, ablation studies, and cross-dataset evaluations, providing a thorough analysis of TALoS's performance and robustness.

Weaknesses

1. My major concern lies in the evaluation. SemanticKITTI is a small-scale dataset and does not adequately address dynamic objects in the creation of ground truth SSC labels. The authors are suggested to conduct experiments on larger and more diverse datasets, such as KITTI-360, nuScenes, and Waymo, to better validate their approach. The authors are suggested to consider using SSCBench [1], which employs the same format as SemanticKITTI. Cross-dataset evaluation on SSCBench would significantly strengthen the paper. Furthermore, Occ3D [2] could also be utilized for evaluation, though it uses a different format than SemanticKITTI. 2. My second concern pertains to the computational overhead introduced by the proposed method. The test-time adaptation involves multiple modules, such as creating binary occupancy labels and training two different models at each timestamp, which can significantly increase computational demands. I recommend that the authors conduct a detailed computational analysis, including an evaluation of the computation-performance tradeoff and the frames per second (FPS) achieved. 3. The literature survey is notably incomplete. Several important SSC works are missing from the related work section, including both datasets and methods [1-7]. A more comprehensive review of the existing literature is necessary to provide proper context and background for your study. Meanwhile, the idea of Line of Sight has been studied in prior works, such as 3d object detection [8] and point cloud registration [9,10]. The authors are suggested to add the missing references. 4. The current approach is tailored to LiDAR data, which may limit its applicability to other sensor modalities or combined sensor data. The authors are suggested to add more details about how to use their method in the camera-based SSC methods. [1] Li, Y., Li, S., Liu, X., Gong, M., Li, K., Chen, N., Wang, Z., Li, Z., Jiang, T., Yu, F. and Wang, Y., 2023. Sscbench: A large-scale 3d semantic scene completion benchmark for autonomous driving. arXiv preprint arXiv:2306.09001. [2] Tian, X., Jiang, T., Yun, L., Mao, Y., Yang, H., Wang, Y., Wang, Y. and Zhao, H., 2024. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36. [3] Cao, A.Q. and De Charette, R., 2022. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3991-4001). [4] Shi, Y., Li, J., Jiang, K., Wang, K., Wang, Y., Yang, M. and Yang, D., 2024, March. PanoSSC: Exploring Monocular Panoptic 3D Scene Reconstruction for Autonomous Driving. In 2024 International Conference on 3D Vision (3DV) (pp. 1219-1228). IEEE. [5] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C. and Anandkumar, A., 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9087-9098). [6] Zhang, Y., Zhu, Z. and Du, D., 2023. Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9433-9443). [7] Huang, Y., Zheng, W., Zhang, Y., Zhou, J. and Lu, J., 2023. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9223-9232). [8] Hu, P., Ziglar, J., Held, D. and Ramanan, D., 2020. What you see is what you get: Exploiting visibility for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11001-11009). [9] Ding, L. and Feng, C., 2019. DeepMapping: Unsupervised map estimation from multiple point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8650-8659). [10] Chen, C., Liu, X., Li, Y., Ding, L. and Feng, C., 2023. Deepmapping2: Self-supervised large-scale lidar map optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9306-9316).

Questions

Can the authors discuss potential extensions of their method to other sensor modalities, such as camera data, and how their approach could be adapted to handle combined sensor inputs? Can the authors provide additional details and clarification on the implementation and functioning of the dual optimization scheme (such as a pseudo algorithm)? Can the authors discuss the scalability and practicality of their approach in more detail, including potential challenges and solutions for real-world implementation?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed several limitations in their work, but there are areas where additional detail and clarity would be beneficial. Here are some suggestions for improvement: ### 1. Computational Overhead **Current Limitation:** The method involves significant computational overhead due to the creation of binary occupancy labels and the training of two models at each timestamp. **Suggestion:** Provide a detailed analysis of the computational demands, including the computation-performance tradeoff and the impact on frames per second (FPS). This would help to understand the practical feasibility of the approach. ### 2. Dataset and Dynamic Objects **Current Limitation:** The evaluation primarily relies on the SemanticKITTI dataset, which is small-scale and does not adequately handle dynamic objects. **Suggestion:** Extend the evaluation to larger and more diverse datasets such as KITTI-360, nuScenes, and Waymo in SSCBench would provide a more comprehensive validation of the method. ### 3. Applicability to Different Sensor Modalities **Current Limitation:** The current approach is tailored to LiDAR data, which may limit its applicability to other sensor modalities or combined sensor data. **Suggestion:** Discuss potential extensions to other sensor modalities, such as camera or radar data, and how the approach could be adapted to handle combined sensor inputs. This would enhance the method's applicability and robustness in real-world scenarios. ### 4. Literature Survey **Current Limitation:** The literature survey is incomplete, missing several important SSC works. **Suggestion:** Expand the related work section to include a more comprehensive review of existing SSC literature, including both datasets and methods [1-7]. This would provide better context and background for the study. ### 5. Potential Negative Societal Impact **Current Limitation:** The potential negative societal impacts of the work have not been explicitly addressed. **Suggestion:** Discuss any potential negative societal impacts, such as privacy concerns or misuse of the technology, and suggest mitigation strategies. Being upfront about these aspects would align with the NeurIPS guidelines on broader societal impacts and contribute to responsible AI research.

Summary

This work proposes a test-time adaptation method for 3d semantic scene completion from point clouds. Using the point cloud and predictions of past and future frames in a sequence the authors propose a self-supervision strategy to adjust the model weights during test time. They propose a completion loss based on the line of sight of each measured point and a semantic loss using pseudo labels from predictions at different points in time. Furthermore they propose to use two versions of the model, one that updates the model based on the information at one moment in time and another that gets updated continuously throughout the sequence of test images.

Strengths

The authors propose a novel test time adaptation (TTA) framework for semantic scene completion based on the temporal line-of-sight consistency. The presentation of the method is clear.

Weaknesses

The instant adaptation of F_M(oment) seems like a indirect fusion of previous point clouds. The method should then also be compared to using the fused point cloud as for the SCNet teacher and fusing the predictions at i and j directly. Related to this it is also unclear what distance is used between i and j and how this effects the method or if using multiple frames is even better. The fusion of p_m and p_G is not explained in detail, while the intuition of using only the static parts from p_G makes sense the actual implementation remains unclear. It is only mentioned that the last "few" layers of the network are updated at test time. For reproducibility it would be good to provide the details. The authors write "Considering that SCPNet also involves knowledge distillation using future frames during training, this performance gain confirms that our TTA-based method is more effective for leveraging future information". Could you elaborate on this argument? As from my point of view in SCPNet they use samples from the training data for the teacher model, here the samples could actually overlap the same spatial area and provide additional measurements.

Questions

The main question is how much of the TTA actually related to temporal fusion and how much is statistical fine-tuning of the model. It seems that for F_M it can only be fusion, as the model is optimized for each frame from the original model, but for F_G it would be interesting to instead of playback the same sequence use a different sequence to see if it also helps statistically. As it seems to be more of a fusion method, it would be good to compare against other temporal/multi-view scene completion baselines e.g. simply fusing measurements or predictions.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

I think the limitations were addressed.

Summary

The manuscript proposes an method for test time adaptation of semantic scene completion models. During test time the method updates two models: one causally based only on previous times and one (in a delayed way) on future times. The models are updated through harvesting freespace and occupied annotations from lidar point clouds, as well as confident semantic predictions as annotations from the other timestamps. This method improves the base pretrained model without TTA on the in-domain test set and can also be used to improve the base model on out-of-domain sequences to some degree.

Strengths

- The test time losses for TALoS are well motivated and sensible. Leveraging not only point but also the freespace information from LiDAR rays makes use of all available information from it. The Shannon entropy-based uncertainty measure for selecting which semantic voxels to use for TTA is well motivated in Fig 2. The idea of leveraging pose tracking to enable leveraging information from previous timesteps makes sense and seems to help (it is a bit questionable how much of a completion problem this then still is see weaknesses) - Overall the paper is well written and well illustrated. The figures (Fig 1 and 2 for example) are very helpful in conveying the message and ideas in the paper. The qualitative and quantitative experiments are comprehensive and clarify the choices made in designing TALoS as well as the improvement over the pretrained base model. It is good to see that other more generic TTA approaches are less performant that the proposed one (but the communication could have been more clean with a table see weaknesses) - It is great to see that TALoS only needs 1-3 optimization iterations to adapt to a new timestep. I do wonder how long that optimization takes per step? (Table 5)

Weaknesses

- Approach is only shown to improve one method (SCPNet) in the experiments which weakens the (implicit) claim that TALoS is a universally useful approach for any SSC model. - The approaches that TALoS was compared against are not communicated well. I am not sure what I should compare the performance of the TENT and CoTTA adapted models too. Please put those into a table. Similarly the OccFiner results are just mentioned in text but not propperly reported in the table. To make the OccFiner result comparable to TALoS one could simply "replay" TALoS on the same sequence after TTA was performed, no? (Like in Sec 4.6) - Leveraging additional information from previous timestamps makes SSC less of a completion problem since it is quite possible that surfaces that are not observable in the current time are actually observed in the previous timesteps.

Questions

I do wonder how long that optimization takes per step? (Table 5)

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation that the method needs point cloud information is addressed in the conclusion.

Rebuttal Answer

Thank you for providing the additional information and evaluations. I think the comparison with the temporal fusion baselines and the evaluation on two different sequences provide interesting insights. The results showing that the simple temporal fusion actually hurts the performance are somehow unexpected. What do you think could be the reason for this? Might it be related to the use of only the previous frame which maybe does not cover so much additional volume, but rather creates a domain gap due to the different point cloud density? Because intuitively in the extremum when combining all measurements at least for the static parts (assuming good sensor pose estimation) it should also improve the semantic metrics, or not? However, the insight that the TTA temporal fusion outperforms the other two baselines is actually very interesting. What do you think could be the underlying mechanisms enabling this? Thank you.