ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splattings

Summary

This paper proposes a method of rasterizing and optimizing 3DGS for omni directional cameras. Built on top of the original 3DGS perspective camera rendering, this paper approximates the rendering of each 3D Gaussian as a perspective camera pointing towards each Gaussian. By limiting the size of individual Gaussian, this effectively approximates the omni-directional camera model applied to 3D Gaussians.

Strengths

**Motivation** * Unlike NeRF which operates on ray-level, 3DGS by its nature is limited by the perspective projection applied to 3D Gaussians and is not easily migrated to different camera models. Extending the 3DGS method to omni directional camera model is useful in many real-world applications due to the data collection efficiency of omni-directional camera. **Method** * The method proposed by this paper is simple but effective. Without introducing very complicated omni-directional camera models to project the 3D Gaussians, this paper proposes to approximate the projection with individual perspective cameras pointing towards each Gaussian. This effectively approximates the rasterization without changing too much on the pipeline. * The method proposed by this paper is inspiring, not only in the context of omni-directional camera, but arbitrary camera models. Most cameras can be approximated with a similar manner to handle 3D Gaussians using local tangent perspective cameras. * The proposed method is quite general and is not limited to the type of scene, compared to the previous 360-GS. **Experiments** * The quantitative rendering quality is consistently higher than the previous NeRF counterparts and the wrapped perspective version of NeRF and 3DGS. The qualitative results also demonstrates considerable improvements. * The high rendering speed of 3DGS is well preserved to a large extent.

Weaknesses

**Method** * The error of the approximation depends on the size of the Gaussian (or the size of the projected 2D Gaussian). It is mentioned in the paper that the maximum size of the Gaussian is limited to minimize the error, but I could not find the details about this limit. I think more details should be included. A hard limit in 3D might not be the best way of performing this limit. **Experiment** * Although the flexibility of this method is preferred over 360-GS, it is better to include the performance comparison between these two works as they are very similar. **Typo** * On Line 138 "distribution of after projection".

Questions

I am very interested in one thing and hope the author can provide me with some answers: can a 3DGS model trained with Omni-directional images with this method be directly used to render using perspective camera, while preserving the rendering quality without fine-tuning? The reason I think this is important in practice is that omni cameras are usually useful in data capturing stage, but not the inference stage. It is ideal to train with panorama images but render with perspective cameras during application. I think adding the relevant results in the paper can further strengthen the usefulness of this paper.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

There is no societal impact involved. One of the main technical limitations is the size of the Gaussian causing approximation error. This paper chooses to limit it but there might be a better way of handling it such as splitting the Gaussians during approximation to handle more complex scenarios efficiently.

Summary

This paper implements a rasterization module for 3D Gaussian Splatting (3DGS) for omnidirectional images. The rasterizer assumes local affine approximation and projects Gaussian primitives to the unit sphere centered by the camera position, which may be reasonable for relatively small Gaussians. Based on the CUDA implementation, experiments show faster and more accurate rendering results compared with the omnidirectional extension of NeRF (e.g., EgoNeRF).

Strengths

## First attempt to omnidirectional 3DGS This paper would be the first attempt to extend 3DGS to omnidirectional images in general (360-GS [2] is available but for the specific purpose of room modeling requiring layout information, and OmniGS [a] is considered to be a concurrent work uploaded just before NeurIPS deadline.) [a] Li, Longwei, et al. "OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images." arXiv preprint arXiv:2404.03202 (2024). ## Technical correctness The method seems to be adequate. Under affine approximation, this paper correctly formulated the projection and implemented it with CUDA. This is a good engineering effort, and many people will benefit from it. It is good news to know that the omnidirectional version of 3DGS generally achieves better rendering than omnidirectional NeRF methods.

Weaknesses

## Novelty I am struggling to evaluate the technical novelty of this paper. This paper implements a useful tool, while the paper's contribution is rather in its engineering effort, and the scientific (or technical) novelty may be limited. I understand it is quite important to release those useful tools to the community, while the idea of extending 3DGS to omnidirectional images is somewhat natural. Locally affine approximation may be straightforward if someone may extend 3DGS to omnidirectional images.

Questions

Are there any specific technical novelty we (readers and reviewers) should care about? I would rather like to support the usefulness and technical correctness of the paper, but I think others may not.

Rating

6

Confidence

3

Soundness

4

Presentation

3

Contribution

2

Limitations

I did not find notable negative social impacts.

Summary

The paper introduces a novel approach for 3D scene reconstruction from 360 omnidirectional imagery that the authors make compatible with a 2D Gaussian Splatting representation [x]. The proposed method is evaluated on 3 public datasets against 4 Radiance Field variants including NeRF derivatives and 3D Gaussian Splatting [25]. [x]: Huang, B., Yu, Z., Chen, A., Geiger, A., & Gao, S. (2024). 2d gaussian splatting for geometrically accurate radiance fields. Siggraph 2024.

Strengths

### + Practical usefullness of the considered problem. The paper addresses a narrow yet valuable problem in the scope of data acquisition from imaging-equipped vehicles in urban areas. ### + Structure, and Organization of the Contents. Each dedicated section of the paper is properly balanced. The use of text real-estate is fair. ### + Amount of discussions on the experimental evaluation is relatively good. ### + Fair results of the proposed contribution, overall. ### + Level of implementation details, in particular in the appendix is good.

Weaknesses

### - (1). Writing, and consistency between key sections and the contents, and overall maturity of the presentation. #### (i) Writing and readability. In its current state, the paper is in a relatively dense state, with grammar issues and typos, even in key sections. E.g., typo in the title of the paper: "ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian **Splattings**" -> "ODGS: 3D Scene Reconstruction from Omnidirectional Images with 3D Gaussian Splatting" The writing of the abstract section for example is very dense and hard to understand. #### (ii) Text vs. Contents - Inconsistencies. Title and paper suggest the proposed paper builds on the 3D Gaussian Splatting framework (eg, title, l7), while it in fact uses 2D Gaussian Splatting (Eq (9)). ### - (2). Comparative evaluation lacks depth. In particular, additional comparative and standard baselines could have been envisioned to better help the reader unstand the relative positioning of the proposed conceptual contribution, perfromance-wise. E.g., COLMAP (standard SfM), Neus, NeusFacto, NeuralAngelo: - Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger. Sdfstudio: A unified framework for surface reconstruction, 2022 - Schonberger, J. L., & Frahm, J. M. (2016). Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4104-4113). - Schönberger, J. L., Zheng, E., Frahm, J. M., & Pollefeys, M. (2016). Pixelwise view selection for unstructured multi-view stereo. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14 (pp. 501-518). Springer International Publishing. - Li, Z., Müller, T., Evans, A., Taylor, R. H., Unberath, M., Liu, M. Y., & Lin, C. H. (2023). Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8456-8465). - Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., & Wang, W. (2021). NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems, 34, 27171-27183. In particular, the unified codebase makes it very easy to produce results for the aforementioned baselines. ### - Other minor issues and suggestions. - Missing reference (2DGS): Huang, B., Yu, Z., Chen, A., Geiger, A., & Gao, S. (2024). 2d gaussian splatting for geometrically accurate radiance fields. Siggraph 2024.

Questions

I do not have more questions or suggestions than the ones underlying the aforementioned weaknesses.

Rating

3

Confidence

5

Soundness

3

Presentation

1

Contribution

2

Limitations

The authors provide one dedicated paragraph that reasonably addresses such considerations.

Summary

This submission tackles the problem of extending 3D Gaussian Splatting (3DGS) to omnidirectional imagery. 3DGS and specifically its proposer rasterizer is limited to perspective camera. While omnidirectional images can be decomposed into perspective cameras, this typically introduces severe distortion artifacts that impacts the applicability and the quality of the reconstruction results. To address this, the authors present a number of contributions: - A fast omnidirectional rasterizer (implementable in CUDA) that relies on a local affine approximation while preserving optimization and rendering speed. Each Gaussian is projected for rasterization on the tangent plane of a sphere centered at the camera origin, thereby decreasing the amount of distortion to a minimum. - The pruning and spiltting heuristics of 3DGS are adapted to account for the omnidirectional projection. - An array of qualitative and quantitative evaluations on egocentric (OmniBlender, Ricoh360, OmniPhotos) and roaming (360Room, OmniScenes, 360VO) datasets is shared with comparison against 3DGS and NeRF-based baselines (NeRF, TensoRF, EgoNeRF).

Strengths

- With more and more image contents coming from egocentric cameras, identifying and proposing ways to address the limitations of an established approach like 3DGS to work on this new type of contents is quite relevant to the research community. - The proposed changes to cope with omnidirectional imagery are very simple: essentially projecting Gaussians to use a per-Gaussian rasterization frame that limits distortion while still allowing an efficient implementation on graphics hardware. - The quantitative and qualitative evaluations do a good job of demonstrating the benefits of introducing a custom omnidirectional rasterizer. Performance is measured against relevant baselines (3DGS of course but also NeRF-based ones, including one targeting egocentric contents: EgoNeRF) across multiple datasets. The approach of the author is shown to significantly outperforms in Table 1 in terms of quality (generally) all other approaches (using the typical PSNR, SSIM and LPIPS metrics) while actually being by far the fastest one. Figure 3 and 4 also share convincing qualitative results with significant artifacts in other approaches that ODGS does not generate.

Weaknesses

- I believe there is a major gap in the current presentation of the approach: how to implement alpha blending of Gaussians appears to be missing. It is in particular unclear to me in which common coordinate frame Gaussians are being composited on top of another to produce a final image. As this is, in my opinion, essential to understand (and reproduce) the approach, until resolved, this gap is dragging the proposed rating of the paper down despite the strengths above. - Decomposing an omnidirectional field-of-view into six perspective cameras to allow comparison against approaches that only work on perspective imagery makes sense. Such comparison could however be improved and pushed much further: how does the comparison hold with - the omnidirectional field of view being decomposed into gradually more perspective cameras, - the rasterizer properly handling the different perspective coordinate frames (compositing all the relevant Gaussians overlapping in the current perspective field-of-view)? The areas of introduced distortion due to the rectification could be visualized and super-imposed on the omnidirectional images of the qualitative comparisons to better understand and correlate with observed artifacts. - The abstract and introduction indicates some adjustments being applied to the densification heuristics, which are not apparently covered in the paper (not in section 3 or 4) apart from a mention of limiting the maximum size of a Gaussian. - Some reference (and possibly comparison) to (very) related work appears to be missing: [OmniGS: Omnidirectional Gaussian Splatting for Fast Radiance Field Reconstruction using Omnidirectional Images](https://arxiv.org/abs/2404.03202) (April 2024) - Some minor typos to correct: - l.111 available To -> available to - l.285 often misses -> it often misses - l.289 ODGS overcome -> ODGS overcomes

Questions

- Please clarify how alpha-blending of Gaussians is implemented with per-Gaussian rasterization (see Weaknesses above). While the presented results otherwise do look good, this is a critical gap in the presentation of the approach that prevents me from proposing a higher rating. - Could the authors indicate how ODGS achieves to be significantly (more than 3x) faster than 3DGS applied on perspective images?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Thanks for the rebuttals

I read the others' reviews and rebuttals. Indeed, implementing the omnidirectional rasterizer and splatting policy would be a new part. I am now leaning a bit toward acceptance, but I would also like to discuss it with other reviewers.

Response to Rebuttal

Dear Authors, Having read (all of) the feedback and having revisited the paper, here is my response. Thank you for providing meaningful details. There was indeed a confusion on my end in my initial take on Q1 in particular. Regarding Q2 though and the relative lack of experimental depth, I do maintain that the relative positioning of the proposed contribution performance-wise, can not be properly and completely assessed in absence of more competitive baselines such as the following: - Mip-NeRF360: Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: unbounded Anti-Aliased Neural Radiance Fields. In CVPR, pages 5470–5479, 2022 Its absence even from the related work discussion and references is also puzzling, especially given it is very competitive against EgoNeRF in the original CVPR paper [10] To a lesser extent, the following ones: - Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video. NIPS, 35:32653–32666, 2022 - Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi. RobustNeRF: Ignoring Distractors with Robust Losses. In CVPR, pages 20626– 20636, 2023 I will ponder and discuss with other reviewers regarding the remaing issue regarding experimental evaluation depth and the relative positioning of the proposed contribution in terms of its performance. Warm regards.