GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation

Summary

The primary objective of this paper is to develop a geometry-aware large reconstruction model. Previous approaches either predict tri-planes or per-pixel Gaussians for reconstruction from multi-view images. However, these methods lack an explicit correspondence between 2D image features and 3D representations. To address this issue, the authors employ a Proposal Transformer to identify grids with valid occupancy and use these tokens to perform Deformable Cross Attention for aggregating image features and subsequently predicting Gaussians. The results demonstrate that the proposed methods achieve state-of-the-art performance and adapt well to different numbers of views.

Strengths

- This paper presents an alternative approach for building large reconstruction models by first predicting occupancy grids. This method may be beneficial in reducing memory consumption as the number of input views increases. - The results show that the proposed method achieves state-of-the-art performance, with performance improving as the number of input views increases.

Weaknesses

- The authors compare their model with InstantMesh under different input view settings, which is unfair. InstantMesh fixes the number of input views at six during training, so testing with varying numbers of input views may degrade its performance. In contrast, GeoLRM is trained on varying input view numbers. Therefore, the main claim of this paper is not well-verified. - The authors use a training and rendering resolution of 448x448, while other works render 512x512 images. It would be better to maintain the same settings for a fair comparison. - Introducing the Proposal Transformer complicates the process. An inference time comparison with other methods would be beneficial. - The comparison is only conducted on the GSO dataset. Evaluating and comparing with other methods on additional datasets, such as OmniObject3D, would be more convincing. - The authors claim superiority compared to per-pixel Gaussian prediction methods and compare performance with LGM. However, another method, GRM [1], released around the same time as LGM and sharing a similar architecture but showing much better performance, could also be included for a more convincing comparison. - The paper is not clearly written, especially the method section: details of the Proposal Transformer are missing; In Figure 2, the term "Reconstruction Transformer" is used, but in Section 3.2, "Anchor Point Decoder" is used, which is inconsistent. [1] Xu, Yinghao, et al. "Grm: Large Gaussian reconstruction model for efficient 3d reconstruction and generation." arXiv preprint arXiv:2403.14621 (2024).

Questions

Questions: My main concern is that the comparison with previous methods using different numbers of input views is unfair, and this paper employs a different rendering resolution. The experimental results do not adequately support the authors' claims. Suggestions: - One of the main advantages of this paper is its potential to reduce GPU memory usage as the number of input views increases. However, there is no clear comparison with other methods. Adding GPU memory and inference time comparisons in Table 2 would be helpful. - What if the number of input views is further increased beyond 12?

Rating

4

Confidence

5

Soundness

1

Presentation

2

Contribution

2

Limitations

The authors have discussed their limitations and provided potential solutions.

Summary

This paper introduced a sparse reconstruction model based on the LRM. This paper try to use the projection between the 3D point with the pixel position to save computational consumption and this paper use the deformable attention to lift the 2D feature to 3D, and finally propose a two-stage pipeline to generate Gaussian.

Strengths

1. Using the projection correspondence between 3D point with the pixel coordinate is reasonable to save consumption. 2. Deformable attention is also a useful solution to lift 2D multi-view image features to 3D.

Weaknesses

1. Some confusions about the method and the experiment. - Why cannot train the first stage and the second stage together? Is it because of the memory consumption? - The advantage with more input images is good but seems to be unimportant in the sparse reconstruction model, which is designed to handle the situation with only few inputs. And there also a confusion in this experiment: will the number of Gaussian primitives increase as the input increases in your method? - The 3DGS-based method has advantages in novel view rendering quality, but the advantage over InstantMesh is not obvious and the results shown in this paper seem to be much worse than the existing 3DGS-based methods [1,2]? - How does your method generate the mesh? And there seems lack this comparison experiment. Is there still advantage in the mesh generation with Gaussian representation? 2. The contribution seems to be not novel enough. - The strategy of using projection strategy is a common solution, and has been widely used in existing sparse reconstruction methods like [3,4,5]. - Using the deformable attention to lift 2D image features to 3D space is also a widely used strategy like [6,7], and the two-stage pipeline that first generating the sparse queries and then only processing in these sparse queries is also the existing solution such as [7]. [1] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation [2] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting [3] SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. [4] C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction [5] GenS: Generalizable Neural Surface Reconstruction from Multi-View Images [6] SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving [7] VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

Questions

Refer to weaknesses for details. And due to these doubts, I tend to give the borderline and hope to see the sufficient response of the author.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Declared in the paper.

Summary

The paper proposes a geometry aware large reconstruction model, that represents the scene as 3D Gaussians. The paper proposes a novel architecture for multi view reconstruction that first generates a proposal occupancy grid for 3D gaussians with a proposal transfomer, and then refines them with a reconstruction transformer. The model uses hierarchical image encoders that encode both the semantic features (dino2), and rgb values and plucker coords. The paper uses a first proposal transformer that classifies which dense 3d tokens will have occupancy, and based on that samples sparse tokens, which are denoted as "anchor points", that are processed by the reconstruction transformer, to gaussian tokens that are decode a lightweight MLP. The network uses deformable cross attention to the hierarchical image features, which allows for recovering higher resolution features, and also uss a 3d version of Rotary Positional Encoding RoPE. The model is trained in two stages, first the proposal transformer, and then reconstruction model. The model is trained on Objaverse and evaluated on GSO against LGM, CRM, and Instant Mesh. The model is evaluated across different number of input views, and also some of the contributinos are ablated: types of features, 3D RoPE, and training with fixed number of views.

Strengths

- The paper proposes a novel architecture to perform 3D reconstruction from multiple views based on transformers and 3D Gaussians that improves upon previous work. The main contributions are the use of higher frequency information thanks to Deformable Cross Attention, and selective computing thanks to a proposal network that computes anchor points around which 3D gaussians are generated. - The paper compares results with a number of highly relevant recent papers (LGM, CRM, InstantMesh), and showcases the strength of their method. - The method seems to be more robust to increasing the number of input views compared to previous works, which seems like a win. - The paper is well written and easy to follow.

Weaknesses

- I think the deformable cross attention is not ablated properly -- yet it seems a key to the success of this method. Similarly, understanding the sampled points would be quite interesting. - The paper could be a bit more clearly written. See questions. I think it it confuses the reconstruction transformer, and the anchor point decoder. I think it is also not clear the architecture of the proposal transformer, until one reads the supplementary and realizes that it's the same architecture as the reconstruction transformer (cool!). I think adding a more clear structure of when the proposal network and the reconstruction transformer are explained would make the paper more readable. - The paper could be a bit better if it showed that deformable attention allows the model to be robust to slight pose noise. As well, scaling experiments of the different models would be quite useful.

Questions

- What is the feature dimension of the low-level image features (eg after the conv)? - The Anchor Point Decoder L125 is missing in Figure 2, and is labeled as Reconstruction Transformer. - How much do the plucker coords in the low-level image features matter? - What is the typical number of sparse tokens? Is it even an issue when there are too many? - L159: I did not got these subtle point. The proposal anchor points are upsampled to 128^3, right? And then each of the active becomes a token for the reconstruction transformer, with max seq length of 32k?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Yes.