MeshXL: Neural Coordinate Field for Generative 3D Foundation Models

Summary

The paper is about 3D mesh generative models. The generative method is done by generating triangles one-by-one (auto-regressively). Each face is represented by 3 vertices (9 coordinates).

Strengths

The idea to generate faces sounds interesting than previous methods. Unlike PolyGen which needs to generate connections (faces) between vertices, the method seems to be more elegant and easier for programming. The ShapeNet results are also convincing.

Weaknesses

The major limitation of the method is the sequence length. The method is only able to handle 800 faces with the current resources. This makes the method is nearly impossible to generate complicated meshes. The method also looks a bit brutal. Some faces share vertices. But the method needs to generate the shared vertices multiple times. The method can also generate intersecting faces.

Questions

How is resolution defined? The number N (in L103) seems to be missing in the context. In L111, the authors mentioned it should be an unsigned integer. But it is still not clear what the number should be. In my experience, the number should be large (or the classification will be very difficult when training the autoregressive transformer). If the number is too small, then the resolution is limited. I didn't find the sampling time and hardware. Sampling 7200 tokens takes lots of time. I would like to see a time analysis (and maybe memory consumption). The training is done on Objaverse-XL but the main results are mainly about ShapeNet. The comparison is not fair. Also I can only find Fig 6 is about general objects instead of Shapenet objects. The quality seems to be very limited. Even the authors call the method neural coordinate fields, but it is nothing about field. A field should be like a function with continuous domain (like nerf and signed distance fields). The method is similar to PolyGen. All vertices are discretized.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper proposes MeshXL, a mesh generation model based on the Neural Coordinate Fields(NeurCF), which encodes discretized coordinates of mesh vertices into a sequence of tokens. Then a decoder only transformer is trained to generate meshes unconditionally/conditioned on modality. The model is trained on multiple datasets for better performance. In terms of quantitative and qualitative results, the method outperforms other baselines.

Strengths

The method proposes an end-to-end transformer model for mesh generation based on its neural coordinate field. On the performance side, it produces 3D meshes with better quality.

Weaknesses

The weaknesses of the paper mainly lies in the technical part and the comparison part. To my knowledge, the main difference between previous discrete mesh generation methods like MeshGPT is that MeshGPT first encodes the mesh with VQVAE and then trains a generation model, while the paper does it in a single stage. The paper does produce great results but seems miss a reason why single stage can give performance. In the introduction part (lines 21–28), the paper discusses previous generative methods but does not explicitly point out why they are missing. I think adding more analysis can help readers understand the paper better. I am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it on a specific category. I believe using the same dataset for training is more convincing. For reference, I think template-based deformation methods and command-based methods can be added to the discussion: TM-NET: Deep Generative Networks for Textured Meshes Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation DeepCAD: A Deep Generative Network for Computer-Aided Design Models Computer-Aided Design as Language

Questions

1. Since the MeshGPT is closely related, I think comparisons should be included. https://github.com/lucidrains/meshgpt-pytorch 2. As mentioned in the weakness part, I am really wondering why end-to-end training can generate better results compared to two-stage method like PolyGen and MeshGPT, especially considering 2D SOTA generation model like the latent diffusion model is two-stage. 3. I am not sure the comparisons between MeshXL and PolyGen/GET3D are fair in Table. 2 since it uses pre-trained MeshXL and fine-tunes it for a specific category. I believe using the same dataset for training is more convincing. 4. Why does the scaling law seem not to apply to the lamp as shown in Table 2? 5. Normals in Fig. 7 seem worse (Columns 1, 3, 4) compared to GET3D. 6. will you release the code and pre-trained models since training MeshXL takes a lot of resources?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Limitation is discussed, I do not see any issues.

Summary

This paper proposes a way to use LLM to generate polygon meshes. The key idea is to model mesh generation as the next coordinate prediction, using a strategy similar to that of prior work like PolyGen and MeshGPT. The paper has shown the capability to create a mesh with reasonable quality. I think the key contribution for the paper is that it shows the potential to use LLM style scaling to achieve better mesh generation, while the weakness is that it's unclear how much technical contribution given most techniques are already been explored in prior works like MeshGPT and PolyGen.

Strengths

The paper shows the potential to use large-scale LLM models to produce better mesh generation. Specifically, with more compute and larger models, one can achieve stable training and better mesh generation quality.

Weaknesses

My main concern about the paper is its potential lack of novelty. The key idea of using an Autoregressive model to generate a mesh has been explored in many prior works, such as PolyGen and MeshGPT. However, the writing and the results show that this paper differs technically from existing methods other than running on larger datasets with larger-scale models. At the same time, it's not clear from the writing how this paper handles many technical challenges of modeling mesh as a sequence in a new way, such as permutation invariance of the faces. Also, I think the paper lacks a proper evaluation of mesh quality. The introduction is set to claim that the generative mesh has better quality and is potentially more suitable for downstream applications compared to modeling other representations, such as point clouds or voxels. However, few metrics are indicating that the generative mesh is of good quality that the artist can edit. For example, what's the ratio of the generative mesh being watertight? How well is the triangulation? Most generative metrics this paper reports concern the shape this mesh represents but not how well this mesh triangulates.

Questions

* Question about the ordering. Is the ordering for creating the sequence unique? how does it related/differ from the ordering polygon? I think that in L103 the paper should cite PolyGEN for the partition idea. * L29-35 I believe this positioning risks overclaiming. The auto-regressive model can also have cumulative error issues. It's not entirely clear why auto-regressive mesh generation does not have "great redundancy when representing flat surfaces". * L105 - what is "polynomial face"? I think L101-106 requires much more detailed descriptions for readers to find it reproduciable.

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

See the weakness section. Other limitations include restricted context length (i.e. mesh has very long context lengths, as many meshes have trillions of triangles).

Summary

This paper addresses the challenge of generating high-fidelity 3D meshes by introducing Neural Coordinate Field (NeurCF), an effective representation for large-scale sequential mesh modeling. The authors present MeshXL, a family of generative pre-trained auto-regressive models, which applies modern large language model techniques to 3D mesh generation. Extensive experiments demonstrate that MeshXL produces high-quality 3D meshes and outperforms existing methods on various tasks. Key contributions include validating NeurCF as a viable representation for 3D meshes, presenting MeshXL as robust base models for conditioned 3D mesh generation, and showcasing MeshXL's superior performance in generating detailed 3D meshes compatible with current texturing methods.

Strengths

1. This paper trains a foundational mesh generation model using extensive datasets from ShapeNet, 3D-FUTURE, Objaverse, and Objaverse-XL, with the addition of data augmentation. 2. It proposes a novel 3D mesh representation that can be encoded as a token sequence, effectively leveraging the capabilities of autoregressive large language model approaches. 3. The paper establishes a fair evaluation metric, considering both the generation score (as shown in Table 2) and the 3D mesh quality from a graphics perspective (as shown in Table 3).

Weaknesses

1. This method does not appear to incorporate domain knowledge from traditional remeshing techniques to ensure correct connectivity between different components, avoid self-intersections, and prevent flipping. 2. In the user study, more objective metrics for measuring mesh quality should be considered. For instance, in downstream tasks like ray tracing, long thin triangles should be avoided, and aspect ratio can be used to measure how thin these triangles are.

Questions

1. How does this method address common mesh surface artifacts in modeling, such as ensuring correct connectivity between different components, avoiding self-intersections, and preventing flipping? 2. In Section 4, we generate triangles within “<tri> · · · </tri>” and quadrilaterals within “<quad> · · · </quad>”. However, what is the form of the output in the results presented in the paper? Should these sequences of triangles and quadrilaterals be generated separately or can they be combined in the final meshing result?

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Please check weakness

Reply to the rebuttal

Thanks for your great efforts! After reading the response, some major issues have been addressed well, so I still lean towards positive for the submission. I encourage the author to add these clarifications to the main paper. Thanks!

Thanks for the response

Thanks for the detailed response. After reading other reviews together with the rebuttal, I lean to believe that the paper does carry some technical novelties to make mesh generation more scalable - some of which help simplify the pipeline, which is under-appreciated by the community. With that being said, I am willing to change my score toward acceptance. I strongly encourage the authors to revise the paper to bring out the technical contribution needed to simplify the pipeline and make mesh generation more scalable. I would appreciate more detailed discussion on the difference between MeshXL's ordering method and PolyGen's. The evaluation on mesh quality is still unconvincing since the presented metrics can be biased by the fact that the generated meshes are outputted by a model which only sees short meshes. This means that the model can sacrifice other aspects of the mesh qualities, such as being watertight or without self-intersecting faces, just to generate a subset of faces well. I believe this is still the most significant limitation of the current method and would appreciate it if the authors make appropriate acknowledgment of that.