DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation

Summary

The paper proposes a framework to generate animated surfaces from input videos. It has a static step to generate a base triangle mesh, and textures represented by 3D Gaussians. A subsequent dynamic step changes the positions of the mesh vertices and the gaussians. The framework leverages on Zero123 and SuGaR to generate the mesh and gaussians, integrating loss terms from those approaches in its training mechanism. Tha dynamic step uses geodesic distances to create a deformation graph and include As-Rigid-As-Possible and Normal Consistency loss terms.

Strengths

(1) This paper is very informative and well contextualized. I am a specialist in geometry but not in Diffusion models, but I was able to use the Introduction and Related Work sections to familiarize myself with the theme. That was very fun. There are lots of the references from very recent papers, so good job on keeping track of the field! (2) The results are probably ready for current Computer Graphics pipelines, since it uses meshes and skinning. (3) I liked the mathematical modeling. Explaining everything in terms of the losses made a good paper structure. I have reservations about the specific mathematical notation (see Weaknesses), but reading was good overall. (4) I liked the proposed method. All losses terms make sense and are intuitive. Knowing that the SuGaR SDS would work well in a more complex pipeline was informative. (5) I liked the use of geodesic distance to compute the deformation graph. It is more robust than euclidean distance indeed.

Weaknesses

(1) I missed a video in the supplementary material. As a 4D approach, the paper would benefit greatly from showing one. It shows metrics about temporal coherence, but a video would avoid any possible doubt. (2) I'm missing references about 4D neural implicit surfaces in the 4D Representations paragraph in the Related Work section. They also deal with dynamic surfaces in neural pipelines. I believe a sentence including all of them would be sufficient. Novello, Tiago, et al. "Neural Implicit Surface Evolution." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021. Mehta, Ishit, Manmohan Chandraker, and Ravi Ramamoorthi. "A level set theory for neural implicit evolution under explicit flows." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. (3) The mathematical notation is a little bit polluted, impacting reading in my opinion. Subscripts and superscripts are used too much. The text would be cleaner by using more symbol letters instead. A good rule of thumb is to avoid using subscripts and superscripts unless they represent indices. The same letters are also used for very different contexts. N is used for number of vertices and control point set, for example. Even though different fonts are used in each context, the cognition when reading is to remember N and forget the font used. It would be cleaner to use different symbols.

Questions

(1) I would like to know more about the temporal coherence in the proposed approach, since a video is not available. Even though the paper show metrics in that respect I would also like to see qualitative results. (2) Even though the results are probably ready for a CG pipeline, I believe the training and inference must be computationally expensive. I would like to know the training and inference times, and the hardware used. (3) Does only changing positions, rotations, and scaling of the gaussians in the deformation step result in accurate view-dependent effects? In theory the spherical harmonics should also be recalculated. (4) Why 6 gaussians per triangle in the static stage? (5) Equation 10 should come with a small explanation to give intuition. Mentioning that the logarithmic rotation is a mapping to the Lie Algebra and that the exp is a conversion back to the rotation matrix form would at least give a direction for a reader that is not a specialist. (6) I believe the references in line 38 [29, 54, 17, 41, 9] are not associated directly with score distillation sampling. Is that a typo?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the paper has a specific section describing limitations.

Summary

DreamMesh4D is a novel framework for transforming static images into 4D dynamic meshes. It refines object geometry and texture using a Gaussian-mesh hybrid and a geodesic-based deformation graph. A new skinning algorithm combines the best of LBS and DQS for enhanced deformation. The method excels in creating high-quality 4D objects with improved rendering and consistency.

Strengths

1.The results show an improvement in both quantitative and qualitative metrics compared to previous methods. 2.The use of points to control motion is an interesting approach. 3.The writing is clear.

Weaknesses

1.The novelty of this work is somewhat limited in comparison to previous methods. The training strategy, the structure of the loss function, and the design of deformation are quite similar. The main difference lies in the representation, which is a predictable technical combination. 2.I find that the detail of the results is superior to those of previous methods. Therefore, in section 4.2, the author should elucidate which aspects of the pipeline design have contributed to the improved quality compared to the previous method's design. 3.The paper argues that previous works have significant time and memory complexity, but there is a lack of comparison regarding time and GPU usage costs. It would be beneficial to include such a comparison to demonstrate the efficiency of the proposed method.

Questions

The current article lacks interesting insights. I suggest that the author should focus more on the advantages of this representation and explain why it is important for the 4D generation task.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

1

Limitations

Refer to the Weaknesses and Questions section

Summary

This paper proposes DreamMesh4D that combines mesh representation with sparse-controlled deformation technique to generate high-quality 4D objects from a monocular video. The authors bind Gaussian splats to the surface of the triangular mesh for differentiable optimization of both the texture and mesh vertices. The method begins with a coarse mesh from single image based 3D generation. Sparse points are then uniformly sampled on surface of the mesh to build a deformation graph, which drives the motion of the 3D object. For each step, transformations of sparse control points are predicted using a deformation network. The mesh vertices and the bound Gaussians are deformed via a geometric skinning algorithm that combines LBS and DQS. Reference view photometric loss, score distillation loss, and regularization losses are used in a two-stage learning. Experiments are performed on the consistent4D dataset.

Strengths

- The methods address a challenging problem to generate dynamic objects from a monocular video. - A new method is proposed that generates dynamic meshes through a static-to-dynamic optimization process. By employing a Gaussian-mesh hybrid representation, the authors simultaneously refine both the geometry and texture of the object. This approach allows the static object to serve as an excellent starting point for dynamic learning. During the dynamic stage, a deformation graph is used on the object’s surface using geodesic distance. - Experiments show the superior performance of this method in generating high-fidelity 4D objects. - The method use dynamic mesh with good compatibility with modern geometric pipelines in the 3D gaming and film industries.

Weaknesses

- One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, which was originally proposed in the work of SuGaR. The technical difference that is designed to address the special setting of this paper’s task is not clearly explained. - In figure 1, the input is a composited scene video. But in the rest of the paper, the inputs are well segmented object video, which makes the setting of the addressed task confusing. - The method combines some existing methods such as SuGaR, deformation graph, LBS and DQS based skinning. It’s not clear to identify the author’s innovative contribution.

Questions

1. One of the main idea of this paper is to use a Gaussian-mesh hybrid representation, what is the difference between the proposed method and SuGaR? 2. In figure 1, the input is a composited scene video. Is a segmentation method needed to extract each object region to use as input to the method? There seems no mention on this in the rest of the paper. How is the mutual-occlusion between the objects solved by the proposed method for the scenario in figure 1? 3. What is the computational cost of the proposed method?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Not applicable.