SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding

Summary

Currently，2D semantic maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain， in order to solve these problem, the author proposes fusing features from different input modalities to obtain a 2D neural map that is robust over time and retains sufficient geometric and semantic information. The effectiveness of the method was demonstrated by using the 2D neural map generated by SNAP for other tasks.

Strengths

This paper proposes to fuse various input modalities and use camera pose estimation to generate a 2D neural map, which is the originality. Moreover, the paper delves into this methodology, related work, and conducts a plethora of experiments, culminating in high-quality output. However, the clarity of the paper requires further refinement, and its significance remains somewhat limited.

Weaknesses

The author demonstrated the effectiveness of using self-supervised pre-training in the article, but there is no relevant experimental evidence to prove whether the fusion of multi-mode features and the proposed supervision method are reasonable. In addition, it is unfair to only compare with OrienterNet in Experiment 1, as the generation of neural graphs involves aerial images, which are more similar to [1][2]. [1] Shi Y, Li H. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 17010-17020. [2] Lentsch T, Xia Z, Caesar H, et al. SliceMatch: Geometry-guided Aggregation for Cross-View Pose Estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17225-17234.

Questions

1.It is necessary to supplement experimental evidence for the effectiveness of fuse and contrastive learning supervision in the generation process of neural maps. 2.Can other satellite-based localization methods generate neural maps? The authors need to conduct experiments to demonstrate the advantages of their method compared to other satellite-based localization methods in generating neural maps. 3.Does the method also perform well on other general datasets such as KITTI?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The author discussed the limitations of the method on accuracy and the impact of the method on privacy.

Summary

This paper proposes a self-supervised learning approach to create a neural representation of the 3D environment from aerial-view and ground-view images. It utilizes contrastive learning to align neural maps based on the same pose and distinguish maps with inconsistent poses. Large-scale pretraining enhances the model's capabilities for visual positioning and semantic mapping. Experimental results on two datasets validate the effectiveness of the approach.

Strengths

1. The overall idea of learning a neural scene representation from multimodal images via pose contrast is both elegant and effective. This approach allows for a comprehensive understanding of scenes by leveraging different modalities, resulting in a robust representation. 2. The implementation of the overall framework is detailed and non-trivial. The monocular inference, multiview fusion, multimodal fusion, and differentiable pose estimation are well executed and contribute together to the final performance. 3. The large-scale pretraining enhances the neural map with high-level semantics. This pretrained model can be further fine-tuned for tasks such as visual positioning and semantic mapping. This versatility makes it a strong candidate for serving as the foundation model in areas like robotic localization and mapping. 4. The writing in the paper is clear and easy to follow. Additionally, the visualization provided is informative.

Weaknesses

1. **Pose supervision**: The authors should discuss the influence of pose accuracy on SNAP's performance and address how reliable pose information can be obtained to improve SNAP's learning capabilities. Exploring the impact of pose accuracy on the model's performance would provide valuable insights. 2. **Overall illustration**: It would be beneficial for the authors to include a pseudo-algorithm to better illustrate the notations used and clarify the training and inference procedures of SNAP. This addition would enhance the understanding of the model's implementation. 3. **Training and inference time**: Given the large scale of the training set, it would be helpful for the authors to discuss the training time of SNAP and justify the necessity of such a large-scale dataset (will reducing the training data significantly reduce the performance of SNAP?). Additionally, considering the practicality for mobile robotics, it would be appropriate to provide information on the inference time on a more affordable GPU. This discussion would provide insights into the computational requirements of the model. 4. **Depth estimation**: The authors should clarify how the depth model handles the large distortions present in the input images. Providing details on how these distortions are addressed and mitigated would enhance the understanding of the model's performance. 5. **Ablation studies**: The reviewer suggests conducting ablation studies to investigate the influence of Equation (1) and the max-pooling used for multimodal fusion. Including these studies would demonstrate the effectiveness of different modules and contribute to a deeper understanding of the model's performance. 6. **Details of semantic mapping**: More information is needed regarding the input, scale, and output of the semantic mapping process. Additionally, since the semantic classes appear to be limited, the authors are encouraged to test the model on publicly available datasets such as SemanticKITTI SSCBenchmark to provide a comprehensive evaluation. 7. **Dataset and code availability**: It is essential for the authors to make the datasets used for pretraining and semantic mapping publicly available. Furthermore, open-sourcing the code and data would greatly benefit the field, allowing researchers to replicate and build upon the presented work. **Minor** 1. The authors are suggested to show easy, medium, and hard examples’ visualizations respectively. 2. In the literature review section (line 196-line 204), it would be beneficial for the authors to include references to works related to semantic occupancy mapping, specifically addressing voxel representations for large-scale scene understanding (not only just object-centric applications). Adding these related works will provide a broader context. Some relevant works to consider include MonoScene [1], VoxFormer [2], TPVFormer [3], and SemanticKITTI [4]. [1] Cao, A.Q. and de Charette, R., 2022. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3991-4001). [2] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C. and Anandkumar, A., 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9087-9098). [3] Huang, Y., Zheng, W., Zhang, Y., Zhou, J. and Lu, J., 2023. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9223-9232). [4] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C. and Gall, J., 2019. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9297-9307).

Questions

Overall, the paper presents a novel and well-executed approach to learning neural scene representations from multimodal images. Its technical implementation, combined with the potential for practical applications, makes it a valuable contribution to the field. However, there are some concerns listed in the weakness section. This reviewer will consider raising the score if the concerns are well addressed.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

There seems no checklist.

Summary

This paper presents a method for self-supervised learning of BEV features, by using a contrastive loss that links features computed from perspective-view images to features computed from aerial images, where BEV correspondences are given by poses. Interesting method details include depth-weighted splatting in the style of Lift-Splat-Shoot, local decoding/fusion with an MLP, and hard negative sampling via RANSAC. The method provides good localization especially in "hard" cases where the camera moves a lot, and the features are useful pre-training for downstream semantic segmentation tasks.

Strengths

This paper is very well written. The problem and the method are very clearly articulated, the figures and sectioning and style are all helpful to getting the key ideas across efficiently. Self-supervised feature learning from the combination of car-mounted and aerial data is a new and exciting area, and the approach here is well motivated. The method makes sense. The approach achieves fairly good results, and outperforms SfM methods in some cases. Using featuremetric similarity to guide the sampling is a very nice detail.

Weaknesses

The pre-training experiment would be more convincing if it were benchmarked agaginst some existing approach. I am particularly interested to see if the pre-trained model outperforms some SOTA on some BEV segmentation task (e.g., on nuScenes). The danger is something like: as you add more supervised data, the impact of the pre-training goes to zero. The interpolation operator used in eq 4 is confusing for me. Can you write it or explain it in another way? (One additional question here, beyond the main confusion of that equation, is: are you really doing a plain sum here? Typically a mean makes more sense here, so that the scale of the term does not depend on a hyperparameter like resolution.) The overall map seems like it may be huge. I need more implementation details on this aspect to really understand what is going on, especially with respect to generating pairs (and non-pairs) for the contrastive loss.

Questions

"Overhead images are ortho-rectified, such that all world points along the z axis project onto the same pixel coordinate." This sounds odd to me, so I want to double check. I am imagining the aerial imagery is captured from very high up, and the images cover a wide FOV, and only a few pixels will be well-aligned with the z axis (defined as the gravity direction). Fixing this seems like it should require depth information (e.g., especially for tall buildings). "we define K horizontal planes at heights {zk}, which are uniformly distributed around the camera heights and points of interest" What exactly are the "points of interest"? It's interesting that the architecture does not have any BEV CNN involved. To my understanding, it is often this BEV computation which resolves much of the 3D structure of the scene, since depth information is so difficult to estimate from RGB alone (without supervision). Is it really just an MLP applied on each cell independently? The citation here (SimpleRecon) seems not as simple as what's presented here -- it uses a cost volume, various geometric "metadata", and another 2D CNN. Can you comment a bit more here, to help me get a sense of what's going on in the architecture? "Our approach tightly combines 3D geometry and representation learning instead of relying on expensive Transformer [60] or 3D CNNs [25]." OK but I think the more reasonable baseline is a 2D CNN applied in BEV (e.g., [26]). This is lightweight also. "We build mapping segments only from car sequences" Why? And why discuss the data collected from pedestrians/backpacks, if it is never used?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Looks good

Summary

This work develops neural 2D maps in an self-supervised manner by matching multi-view ground level imagery with aerial images. To construct a neural map (2D map of features), they encode an image into a feature frustum (similar to lift-splat-shoot), then pool vertically - for multi-view images, they perform a similar fusion method and interpolate between image features at multiple defined heights to construct the map. For supervision, they use known 3-dof poses to align two neural maps. They use a contrastive loss and mine for negative samples using RANSAC. They train on a large scale streetview dataset with aerial imagery and evaluate performance on a visual positioning task - given 36 views, predict the 3-dof pose of where a query image was taken. Results show improvements over traditional SFM and more recent learning based methods.

Strengths

* Innovative and novel idea with real world applications. * Technical section fairly clear to follow. * Overall the work is well written and descriptive. * The multi-view fusion could be applied to other fields (3D object detection)

Weaknesses

* There are many simplifying assumptions (which the authors have provided discussion for) - relative poses which they "claim can easily be obtained", gravity direction, pose in only 3-Dof. * Quantitative evaluation is not as fair as I would like. Especially when comparing streetview inputs only, the performance improvement is not particularly strong. * All ablations are in the supplementary. The system is consists of many design choices and it would be useful to present which choices were critical and which did not matter much. * All baselines are fairly strong in the "Easy" group. I imagine this is actually the most important category for real world applications since the pose estimation would be applied in the loop.

Questions

* Are the neural maps biased towards features that are greater in height? During training with multi-modal inputs, the aerial tiles are biased towards features that are maximum in height (all other stuff is occluded) and so in theory there is no need for the feature encoder to learn anything below the highest cell. * Line 105: I didn't understand this. I thought lift-splat-shoot normalized the distribution. * Is the data used public?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are provided (accuracy + method assumptions). CO2 usage is specified in supplementary.

Summary

This paper proposes SNAP, a novel architecture to obtain 2D neural maps from street-view and top-down images. The model works by encoding street-view and top-down images separately using corresponding encoders, projecting features to the top-down 2D space, and fusing them via max-pooling to obtain the final features. The paper also proposes to train the model using a contrastive map alignment objective, where a query map is aligned with a reference map at the ground-truth pose, and differentiated from negative poses. The learned representations are evaluated on single-image positioning and semantic mapping applications. The results show strong improvement over prior approaches.

Strengths

* The problem of learning semantic map representations in a self-supervised way from raw images and camera poses is interesting and well-motivated. While the approach shares some technical similarities with prior work in terms of the mapping functions, the large-scale nature of training, self-supervised learning, and use of top-down + street-view images is exciting. * The proposed approach is sensible, the training methodology is simple yet effective, and is clearly described in the paper. * The experiments are well designed and show strong improvements over baselines, and demonstrate the semantic quality of the learned representations.

Weaknesses

* L41 - 42 - Will the "50M street view images" dataset be shared publicly for reproducibility? * L73 - 77 - How reasonable are the assumptions made? E.g., Is the projection function accurate and noise-free in practice? How easy is it to get ortho-rectified overhead images? How are they obtained? * Clarifications needed * L92 - "picks the best estimate ... handles partial observability" -- how exactly is this achieved? * L163 - 165 - How is inference performed to generate pose hypotheses? How scalable is this as map sizes grow larger? * Figure 6 - The classification setup for the center plot is not clearly described. Is there quantitative evaluation for the segmentation shown in the right? * Similarity to prior work not discussed. Please compare and contrast. * The mapping approach described in L99 - 101 is very similar to [R1, R2]. * The objective of learning features by localization is demonstrated in R1, albeit in a classification approach instead of the proposed contrastive learning. [R1] Henriques, Joao F., and Andrea Vedaldi. "Mapnet: An allocentric spatial memory for mapping environments." proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. [R2] Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. "Geometry-aware recurrent neural networks for active visual recognition." Advances in Neural Information Processing Systems 31 (2018).

Questions

Please address the weaknesses listed.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, this is discussed well.

OK

**Obtaining training labels for 3D semantic tasks is very expensive. [...] Getting enough training data for global generalization (across countries, sensors, times of the day) is unfeasible** This doesn't make much sense to me. NuScenes is already labelled. There's also KITTI and Waymo. Running the experiment only costs GPU time. Since as you point out, NuScenes is not very large compared to StreetView, and especially considering the compute capacity ("each model is trained with 16 A100 GPUs"), it seems like the test might only take a couple hours. **Aerial images are captured by planes and are ortho-rectified using a coarse 3D model derived from the same imagery. [...] There is no conceptual difference between ortho-rectified aerial images and satellite imagery** This still sounds odd. My specific complaint was the claim that *"all world points along the z axis project onto the same pixel coordinate"*. Does the 3D model include buildings, street signs, and trees? Figure 4 suggests not: we can see street poles at an oblique angle, instead of top-down like we should expect if the gravity axis is aligned with the camera axis. **a BEV CNN easily introduces boundary artifacts due to padding and breaks the rotation equivariance of the mapping process. This CNN would increase the complexity of the model and it is unclear what problem it would solve. Our experiments show that the MLP-only design already performs well.** This seems missing the point. It was claimed that the approach is an alternative to "expensive Transformer or 3D CNNs", but those architectures are not the right comparison. The current popular approach is BEV CNNs, which are much cheaper and work well. It may in fact be necessary to quantify this comparison -- is the per-cell MLP really cheaper than a 2D BEV CNN? In any case, the 2D issues pointed out here seem inaccurate or trivial to resolve. Problems with boundary artifacts can be resolved with minor architecture changes (e.g., alternate padding schemes), and anyway I don't think this is a common issue for SOTA; rotation equivariance can be approximated by training at many rotations (as SOTA methods actually do already). Anyway I didn't realize rotation equivariance was a claim of the work -- those words never appear in the paper. Are you planning to add some discussion on this? The rebuttal also missed the very first point of my review: "The pre-training experiment would be more convincing if it were benchmarked against some existing approach."

Reviewer response to rebuttal

I thank the authors for their response and for addressing the majority of my concerns. I have a couple of comments based on the rebuttal responses. I trust these will be resolved sufficiently, so I will retain my rating at Accept. * My impression post-rebuttal is that the computational cost for inference and its implication for real-world applications is not represented adequately in the main paper (e.g., an A100 GPU for on-device deployment may not be available). This is a crucial component of such a system that needs to work real-time and on-device. Even if it is computed on the cloud, the resource consumption of such a system is critical if it needs to be scaled to billions of users (e.g., a google maps-like application). * For Fig. 6., it is important to add some more details in the main paper. It is okay to allude to the appendix for complete details, but some minimal information + reference to appendix in the main paper would help.

Reviewer response to rebuttal

Thank you for answering my concerns. Please find my following comments. **1. Effectiveness of multimodality.** In my opinion, the author uses more modalities as input and wishes to demonstrate that the proposed method is superior to any single-modality approach. In Figure 1 of the paper, the author mentions the combination of street view and aerial images. However, in the experimental data presented in Table 1, the author demonstrates that their generated semantic maps are better than using the street view method and also better than using the semantic method, but they do not compare other methods with using only aerial images. Although the author responses that such a comparison is not necessary, I would prefer to see some quantitative results. **2. Unfair comparison.** In the visualization results of Figure 4, the representation of pillars and trees on the map is very clear, which is crucial for successful localization and image segmentation. I believe this is mainly due to the utilization of aerial images and street views. On the other hand, OrienterNet only uses semantic maps as input, making it almost impossible to learn the positions of pillars and trees in such maps. Therefore, I think the author should pay more attention to comparing methods that employ aerial images and street views as inputs.