MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

Summary

This paper introduces MeshFormer, a sparse-view reconstruction model designed to generate high-quality 3D textured meshes from sparse RGB images and their corresponding normal maps. By leveraging voxel representation, 3D inductive biases, SDF loss, and normal information, the model shows comparable inference performance to concurrent methods, while the entire training process can be completed using only 8 GPUs within a week (concurrent methods typically require around 100 GPUs). Experimental results demonstrate the effectiveness of the design.

Strengths

1. The authors provided a detailed explanation of the motivations behind the model designs (including the introduction of voxel representation, the introduction of 3D full (or sparse) convolution, and so on) and demonstrated the reasonableness of these choices. 2. Compared to baseline methods, this model is simpler to train and demonstrates better qualitative and quantitative results. 3. The ablation study demonstrates the effectiveness of normal input, SDF supervision, geometry enhancement, and other methods proposed in the paper.

Weaknesses

1. Although the authors provide detailed textual descriptions in the method section, it would be better if more mathematical symbols and equations were used, which could explain the entire pipeline more clearly and unambiguously. 2. For reproducibility, the authors should provide more implementation details, including a more detailed model architecture, the values of hyperparameters (e.g., \lambda in the loss function), and other relevant information. 3. The authors don’t report the comparison of inference time and memory usage between the proposed model and the baseline models.

Questions

1. Can the normal maps of the mesh be completely consistent with the normal maps predicted by the model after the post-processing algorithm?

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the authors addressed limitations, potential negative societal impact, and mitigation.

Summary

The paper proposes a high-quality feed-forward 3D object reconstruction method from sparse view RGB images. It uses an explicit voxel structure for better geometric inductive bias, auxiliary inputs such as 2D diffusion generated normal images and SDF representation for better geometric details, and an end-to-end trainable pipeline that eliminates the need for multi-stage refinement. The method gives high quality reconstruction results, especially in terms of fine-grained and smooth geometry.

Strengths

1. Although the network architecture and 3D representations are more complicated than previous methods, they are end-to-end differentiable and alleviate the training burden of multi-stage refinement. 2. The idea of using 2D diffusion generated normal images as input to the reconstruction pipeline is interesting and insightful. 3. It is more computationally efficient to train (Line 73). 4. The qualitative results are impressive, especially the mesh normals.

Weaknesses

1. In original LRM the only supervision signal needed is RGB images. The proposed method, however, needs access to the full 3D shape for supervising the occupancy. It is fine for hand-made 3D assets but might poses some difficulty when trying to scale to real datasets.

Questions

1. Table 3 row (a) shows the impact of normal input. When you remove the normal input, do you also remove the normal output and the normal loss? I ask this because in section 3.3 you say learning from RGB to geometric details directly can be difficult, so it makes more sense to just remove the normal input but preserve normal supervision to compare.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

1. It requires 2D diffusion models to generate auxiliary inputs, which can drastically slow down the reconstruction speed.

Summary

In this work, the authors propose a sparse view reconstruction model that utilizes a set of images (with camera poses) and corresponding normal maps to produce a reconstructed textured mesh. The primary contribution lies in adopting voxel-based 3D representation and employing a network architecture that integrates both 3D convolution and attention layers. Moreover, direct geometry supervision (SDF loss) is applied during the training process, alongside rendering-based losses. Experimental results demonstrate that the generated 3D shapes achieve state-of-the-art performance when compared to existing works on the single-view to 3D task. However, as highlighted in the weakness section, there are potential misclaims regarding the technical contributions. It is highly recommended to revise the manuscript to cite and discuss these related works. Despite this, I am currently inclined towards accepting the paper and would be happy to champion it if the aforementioned issues are addressed in the final version.

Strengths

- The writing is clear and easy to follow. - The combination of SDF loss and rendering losses appears novel for training a feed-forward based network. Additionally, the ablation study in Table 3(b) clearly indicates that SDF supervision is crucial for achieving good geometry, as evidenced by the significant CD difference between (b) and (g). - Although [33] has explored using normal maps for the reconstruction task, it seems new to employ normal maps as inputs and supervision for a feed-forward reconstruction network. - Experimental results demonstrate state-of-the-art performance over existing baselines, as shown in Table 1 and Figure 3. Furthermore, it is illustrated that existing methods cannot achieve similar performance given the same computational resources (Table 2). - The ablation study confirms that various components are essential for the final performance, including considering normal input and SDF supervision.

Weaknesses

Possibly Misclaimed Technical Novelties: However, the current manuscript may contain several misclaims regarding its technical novelties. One claimed novelty is the adoption of a 3D voxel representation. However, the use of 3D voxel-like volumes in reconstruction is not a new idea and has been well-explored in various works, including: A. Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process, CVPR 2023 B. SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation, CVPR 2023 C. Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, SIGGRAPH 2023 D. One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion, CVPR 2024 E. Make-A-Shape: a Ten-Million-scale 3D Shape Model, ICML 2024 Additionally, the use of convolution + transformer layers to process grid input seems to be standard procedure in 2D generation tasks, as seen in: Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021 Similar architectures have also been widely adopted in some of the aforementioned 3D reconstruction works, such as [A, C, D, E]. Regarding image conditioning, the cross-attention with image patch features is also well-explored in various works mentioned above, such as [C, D, E].

Questions

Some suggestions: - Considering the above existing and concurrent works (Weakness Section), it is difficult to be convinced that some of the proposed modules are novel. It is highly recommended to cite and discuss the differences with these prior works and adjust the claims accordingly. - Although it is acknowledged in the limitation section that the reconstruction performance will be affected by the errors of 2D models, it is recommended to include this as one of ablation case in Table 3 to better visualize this limitation. - Furthermore, as no real-world images have been tested within the proposed framework, it is advisable to avoid from using the term "open-world" (L384) to describe the current framework in order to prevent overclaims.

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

The main limitation is well described in Section 5.

Summary

This paper proposes an improved framework for feed-forward reconstruction models. The authors advocate a number of improvements over the initial design of Large Reconstruction Model, including model architecture and training schemes. Experiments show that the method reconstructs better geometry and texture on Google Scanned Objects and OmniObject3D datasets.

Strengths

- The paper is focused on ablating different components for feed-forward sparse-view reconstruction, and in-depth analyses are provided for each design choice. Although there are no complicated new method proposed, such analysis bring value for understanding how and why each component works. - The proposed method is evaluated on (preprocessed) real-world multi-view datasets, showing improvements over baselines on all metrics. Extensive ablative analyses are also provided to better understand the behaviors of the proposed method.

Weaknesses

- Since this is more of an analysis paper, it would be good if the authors could also document the other components that were tried/ablated but did not see significant differences. - Since training resources was discussed and compared, it would be nice if there could be an analysis on the mesh generation/reconstruction quality over training time.

Questions

Please see the questions in the weakness section.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Summary

In this work, the authors propose MeshFormer, a sparse-view reconstruction model that explicitly leverages 3D native structure, input guidance, and training supervision. They leverage 3D sparse voxels as their representation and combine transformers with 3D (sparse) convolutions to inject 3D prior. Additionally, they propose to take the corresponding normal maps together with sparse-view RGBs as input and also generate them as output, which could be used for geometry enhancement. Extensive experiments show that MeshFormer can be trained efficiently and outperforms state-of-the-art methods in terms of generating high-quality textured meshes.

Strengths

- MeshFormer is able to generate high-quality textured meshes with fine-grained geometric details. - The authors find that using normal images together with RGB images greatly helps in predicting geometric details. Additionally, the model outputs a normal map, which can be used for geometry enhancement. - The proposed method explicitly leverages 3D native structure, input guidance, and training supervision, resulting in faster convergence speed and better geometric details.

Weaknesses

- Pixel-based 2D methods (e.g., LGM) can preserve thin details, while 3D-based methods often smooth these details. How do you justify that? For example, in Figure 3 Column 4, the loose thread of the toy is captured by LGM, while MeshFormer ignores it. - The proposed name "VoxelFormer" seems improper to me. It seems more like a 3D UNet with a deep bottleneck composed of multiple transformer layers. - The projection-aware cross-attention layer projects 3D voxels onto the m views to interpolate m RGB and normal features. However, in the object case, one 3D voxel usually only corresponds to one view (due to occlusion). This cross-attention is projection-aware but not truly 3D-aware. Have you tried some occlusion-aware attention in your sparse model? Since you already have the coarse structure of the object, it could be used to filter out unneeded features. - According to Table 3 (d), you mention "we replace the cross-attention with simple average pooling and observe a significant performance drop." Could you also try max-pooling? Additionally, do you concatenate the 3D feature voxel at every level of the network, as done in One-2-3-45++?

Questions

- Do you use a shared backbone (trainable DINOv2) for both RGB and normal images? Do you use Plücker embedding here? - Could you provide a more detailed description for the Sparse VoxelFormer architecture? For example, how many sparse convolution layers are used in each resolution? - Instead of joint training, have you tried splitting the dense model and sparse model for two-stage training? - The output voxel resolution is $256^3$, while the SDF supervision is $512^3$. I notice that there is an interpolation step in Figure 2. It would be better to add a short text description for this. - Do you use GT multi-view normals for the teaser? If you use the GT normal images, please include that in the caption. - I suggest discussing XCube [a] in your literature review. XCube also utilizes sparse voxels as their 3D representation and leverages 3D sparse UNet with transformer layers. Additionally, they generate 3D shapes in a coarse-to-fine manner and use tiny MLPs to predict various attributes, such as normals, semantics, and SDF. [a] XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. CVPR 2024.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors already include limitations and broader impact in the paper.

Comments on Rebuttal

Thanks for the additional experiments and comments. They have addressed my previous concerns. Regarding the technical contribution, it is understood that the authors would like to highlight the contributions of applying a voxel representation and corresponding networks instead of a tri-plane representation in an open-world reconstruction task. I agree that this work has shown good evidence for the necessity of this choice, and this should be recognized. Despite this, I believe that a comprehensive discussion of related works, regardless of in-categories setting ([A-C]) or open-world setting (D-E), would help readers understand the field's development. I am glad the authors agreed to include these discussions in the final revision and adjust my rating to acceptance.

Comments on Rebuttal

I am glad the authors agreed to include these discussions in the final revision and provided more experimental results in rebuttal. I will keep my positive score.