Tetrahedron Splatting for 3D Generation

Summary

This paper proposes a method for 3D mesh generation from a text. The idea is to improve gradient propagation ability of DMTet framework by splatting multiple tetrahedra instead of rendering only one surface triangle per pixel. Qualitative results on 3D mesh generation show good quality results with competitive processing time.

Strengths

- The proposed idea is interesting. Splatting multiple tetrahedra around the surface allows to backpropagate gradients further in the model and avoid getting stuck in local minima. - Figure 3 is interesting and shows advantage of the proposed framework. - I am not so familiar with the datasets and tasks of 3D mesh generation from text but qualitative results look good and competitive with state-of-the-art

Weaknesses

- Some explanations are missing to fully understand the experiments section. - Results are few and maybe cherry picked. Only qualitative cannot prove better performance of the proposed method. Again, more explanations about the generation process is required to understand advantages. For example, in figure 6 the 2D images are different, with different level of details in the 2D images. Then it is expected than the 3D models are also different and level of details also different. Then does it really make sense to compare these output 3D meshes? The difference may come from other parts in the framework than the proposed model. - The authors motivate many of text for computational speed. But in the end advantage in computational power is not clear.

Questions

- How many tetrahedra are used in the experiments? The proposed idea to splat tetrahedra and improve gradients propagation ability of original DMTet is interesting. Qualitative results show good performance. However, the explanations in the paper of the 3D generation process is lacking. Quantitative evaluation is few and only few qualitative results are maybe not enough to confirm advantage of proposed method.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

There is no specific section or discussion about the limitations of the proposed method

Summary

The paper presents a novel method for generating 3D models from the given inputs. The authors, for that purpose, propose the use of tetrahedra instead of 3D Gaussians in the recent proposed Gaussian Splatting framework.

Strengths

The paper proposes the use of TeT Splatting, that are well formulated and demonstrated throughout the paper. The method allows for precise and fast mesh extraction, that is a well-known limitation of 3d gaussian splatting. The motivation to use TeT Splatting against the well-known DMTet is also well tackled.

Weaknesses

1. For the task of 3D Reconstruction, 3D Consistency is highly important. However, lack of videos of the generated result is a major drawback that highly affects my decision since just two views are not enough to judge the quality of the output and created mesh. This is also a standard in the baseline papers such as DreamFusion that the authors compare with. I am happy to revise my scores otherwise. 2. The paper uses hash grids as described in sec 4.1. Is it really necessary to use that, or how well the method will work without a hash grid or some other encoding such as positional encoding?

Questions

Please see weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors provide limitations and potential societal impact in the supplementary pages of their work.

Summary

The paper presents a new differentiable 3D shape representation based on DMTet. It borrows the efficient rasterization-based rendering techniques from Gaussian Splatting to improve the global optimization of DMTet. The new TeT splatting is shown to perform well in the task of text-to-3D.

Strengths

- The paper presents an interesting new differentiable representation for 3D meshes. It is more efficient than DMTet in optimization. - The representation works well in text-to-3D generation.

Weaknesses

- The comparison in the experiment section is mostly qualitative. A quantative comparison with DMTet on a 3D mesh reconstruction task would be appriciated. - The quality of underlying mesh connectivities is unclear to me.

Questions

- How to ensure the sharpness of texture obtained without extracting mesh during optimization? - Can the proposed method be used in a feedforward generative model?

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

Yes, the limitation and societal impact discussions look good to me.

Summary

This submission is proposing a new representation for 3D content generation combining the three following benefits: - easy to optimize, - real-time rendering, - allowing the extraction of precise meshes. Recent advances in novel view synthesis, i.e. NeRF and then 3D Gaussian Splatting (3DGS) are eventually bringing two of these benefits but the lack of quality output meshes still limits a number of applications. The proposed approach follows the deformable tetrahedral grid approach of DMTet while using it instead for volume rendering and making it suitable for splatting as in 3DGS. Meshes can be easily extracted which facilitates chaining a first geometric optimization stage based on splat rendering with a later mesh-based texturing stage for complete 3D generation of geometry and appearance. The paper makes the following contributions: - using a deformable tetrahedral grid rendered via tetrahedra splatting as a new 3D representation, - adapting 3DGS to work with such new representation with the implementation of a fast differentiable rasterizer for tetrahedra-based splatting, - leveraging this representation and optimization in a two-stage 3D generation pipeline, first to optimize tetrahedra geometry before optimizing mesh texturing, - An array of qualitative evaluations against other generative baselines (with some quantitative evaluations), either on image-to-3D or text-to-3D tasks using vanilla RGB diffusion priors or with rich diffusion priors.

Strengths

- This paper proposes to tackle a known shortcoming of 3D generative approaches when leveraging 3D representation from modern novel view synthesis techniques: poor meshed outputs. Identifying a suitable representation that is easy to optimize, enables real-time rendering while allowing mesh extraction is very relevant to the research community, especially to better bridge 3D and generative work. - The presented approach elegantly solves the issue of DMNet where only part of the representation (near the zero level set) can effectively be updated. This is addressed by turning the optimization into a full volumetric rendering of tetrahedra via an adaptation of 3DGS to different primitives. - The qualitative evaluations using vanilla RGB-based diffusion priors against the selected comparable techniques: Magic3D, Fantasia3D and DreamGaussian but based on NeRF, DMTet or 3DGS respectively do a rather convincing job to demonstrate the superiority and versatility of the proposed approach (Figure 4) that maintains high mesh quality with real-time rendering speed. Similarly the qualitative results with rich diffusion priors show pretty high surface quality with a reasonable training time,

Weaknesses

- Apart from the CLIP scores of Table 2 (or the training time and rendering speed), the shown results are essentially qualitative. For a stronger comparison against other baselines, it would have been more helpful to try to integrate measures of the quality of the generated surface to demonstrate the benefits of the approach (instead of relying only on normal maps and renders). Also note that the presentation of these qualitative results could be improved by zooming on specific areas with artifacts (or lack or thereof) to better highlight them and guide the reader. - It seems there is no information on how the tetrahedral grid is initialized (besides its spatial resolution and Figure 3 hinting at a sphere). - Some suggestions for improvements: - l. 107-109: the formulation make it sound like DMTet invented the established marching tetrahedra algorithm - rephrase - Figure 2: Geometry Opimization -> Geometry Optimization - Figures 4 and 6: it would be helpful to match the orientation for normals across objects (i.e. have consistent coloring for the whole figure)

Questions

- To address the above weakness, do the authors have some measures of surface quality to provide some form of quantitative evaluation against baselines?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A