MVGamba: Unify 3D Content Generation as State Space Sequence Modeling

Summary

This work introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). The Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity. With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only 0.1x of the model size.

Strengths

1. The paper is of good written quality. 2. The inference speed is relatively fast compared to optimization-based methods. 3. This work introduces the new architecture, Mamba, into the field of 3D generative models, which is novel. 4. The computation cost of the training is lower than LGM and LRM. 5. The method is robust to multiview inconsistency.

Weaknesses

1. In Fig. 6(b), the performance is continuously increasing as the token length increase. Although the authors' motivation is to lower the computation cost, Fig. 6(b) indicates the the performance grows as the computation cost grows. 2. No failure cases are shown to demonstrate the limitation of this work. 3. Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity. 4. The video results in the supplementary materials are only marginally better than LGM's results.

Questions

See the weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See the weaknesses.

Summary

This paper introduces MVGamba, a feed-forward, sparse reconstruction model. This model takes a small number of images (e.g., 4 views) to infer 3D Gaussians. Basically, it is a Mamba version of multi-view large reconstruction model. The authors have implemented several strategies to ensure stable training and optimal performance. Experimental results demonstrate that the MVGamba model can reconstruct 3D Gaussians with higher quality than the Large Gaussian Model (LGM).

Strengths

(1) The experimental results are somewhat promising, showing better Gaussian reconstruction quality than LGM, although the quality is not as high as the concurrent GRM and GS-LRM works. (2) I like the experiment design in Section 5 Q1; it provides some motivation for why we need a non-pixel-aligned architecture. It seems this kind of architecture is more robust to multi-view inconsistency. (3) The paper is clearly written and easy to understand.

Weaknesses

(1) In the abstract and also in Lines 40-41, the authors state that the generated 3D models often suffer from multi-view inconsistency and blurred textures, which is why MVGamba is proposed. However, it seems that this issue is more likely due to whether the architecture is pixel-aligned or not. Therefore, I believe a non-pixel-aligned transformer model can also address these issues. Did the authors try this? How does a non-pixel-aligned transformer model perform? (2) I cannot find any tables or numbers in the paper to support the conclusion “0.1× of the model size.” Did the authors forget to include them? (3) The reconstruction results look in lower quality than the concurrent GRM and GS-LRM works.

Questions

(1) Line 53: Could it be more specific? What does 'post hoc' operation specifically mean? (2) What is the number of output 3D Gaussian primitives? Does it equal N (the number of tokens)? Is there any way to increase the number of Gaussians for better reconstruction quality? (3) Given the advantages of Mamba in computational efficiency for long sequences, did the authors try to train an MVGamba with a larger number of views, such as 10 views?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The method is limited in object-level reconstruction with clean background. I encourage the authors explore on scene-level reconstruction as an interesting future work.

Summary

The authors propose MVGamba, a general and lightweight Gaussian reconstruction model for unified 3D content generation. By replacing the previous LRM work’s transformer architecture with the recent model Mamba. MVGamba can generate long Gaussian sequences with linear complexity in a single forward process, eliminating the need for post hoc operations. Experiments demonstrate that MVGamba outperforms the current open-sourced LRM work in various 3D generation tasks with a smaller model size( 0.1× ). The authors also claim the proposed model has the potential to be applied to scene-level and 4D generation.

Strengths

1. The paper is well-written and easy to understand, and the experiments are sufficient and well-analyzed. 2. The motivation of this paper is sound. The current LRMs papers have achieved impressive results in 3D generation, but their architecture is highly computationally intensive, especially when handling long-sequence inputs. The recent Mama has the potential to solve it. 3. The proposed pipeline achieves better results than the selected baselines.

Weaknesses

1. The paper’s experiments do not align with its claims regarding the inefficiencies of transformer-based LRMs in handling long-sequence inputs. The primary issue highlighted is that these transformer-based models are not efficient for long-sequence input. However, most experiments in the paper are for single or sparse image input 3D generation. In this context, the Mamba-based architecture does not demonstrate significant speed improvements over transformer-based architectures because the input tokens are relatively short. Table 1 shows no obvious speed enhancement compared to other feedforward methods, supporting this point. Therefore, I question the necessity of Mamba for most tasks discussed in the paper. The Mamba-based reconstruction model appears to be more suitable for dense view reconstruction tasks. 2. The performance of paper doesn’t achieve the state of the art. Although in Table 1, the paper achieves better quantitative results over all baselines, some previous LRMs papers have better performance according to their paper. For example, GS-LRM attains 8 dB higher PSNR than LGM according to its table. 1, while MVGamba is just 2 dB higher than LGM. I can understand this paper didn’t compare with GS LRM because GS LRM doesn’t release its code. I am just not sure if Mamba is important for quality improvement as claimed by the authors, when some transformer-based LRMs can also achieve good (or even better) quality.

Questions

- In lines 38-41, the authors state that previous Gaussian-based LRMs fail to achieve multi-view consistent results because they rely on local or mixed attention across limited multi-view image tokens or process each view separately before merging the predicted Gaussians. However, some prior works, like GS LRM, employ self-attention on all input image tokens. Despite merging the Gaussians afterward, they are predicted collectively. It is unclear why this approach leads to multi-view inconsistency. - In lines 281-286, the authors note that performance improves with increasing sequence length, and they model different sequence lengths by varying the patch size. If decreasing patch size enhances final performance without significantly impacting efficiency, as the authors claim, why didn't they choose a smaller patch size than 14, as indicated in the appendix? 14 is actually bigger than some transformer-based LRM’s patch size. (For example, GS LRM chose 8.)

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been well discussed.