We present a differentiable representation, DMesh, for general 3D triangular meshes. DMesh considers both the geometry and connectivity information of a mesh. In our design, we first get a set of convex tetrahedra that compactly tessellates the domain based on Weighted Delaunay Triangulation (WDT), and select triangular faces on the tetrahedra to define the final mesh. We formulate probability of faces to exist on the actual surface in a differentiable manner based on the WDT. This enables DMesh to represent meshes of various topology in a differentiable way, and allows us to reconstruct the mesh under various observations, such as point cloud and multi-view images using gradient-based optimization. The source code and full paper is available at: https://sonsang.github.io/dmesh-project.
Paper
Similar papers
Peer review
Summary
This paper proposed a novel differentiable representation of a mesh. It takes advantage of two probabilities, $\Lambda_{wdt}$ and $\Lambda_{real}$, to generate the mesh, such that the mesh is differentiable.
Strengths
Based on weighted Delaunay triangulation and its dual power diagram, the paper proposed a novel approach to compute $\Lambda_{wdt}$, which makes the WDT generation differentiable. It seems that the accuracy of the conversion from mesh to DMesh is high enough. The paper applied the novel representation for various applications, such as, point cloud and multi-view reconstruction.
Weaknesses
The efficiency of the method seems some what slow due to the WDT construction. In the experiments, it is better to show the deformation process of the a mesh with different topologies.
Questions
As there are spurious non-manifold errors in the mesh, what are the ratio of the non-manifold errors? Is the method robust against noises and holes? Current experiments did not test point clouds with noises or holes.
Rating
7
Confidence
4
Soundness
4
Presentation
3
Contribution
4
Limitations
The limitations should discuss the potential limits to process indoor/outdoor scenes. These data are more challenge due to noises, holes and complicated topology.
Summary
This paper proposes a novel differentiable mesh representation, which focuses on both the vertex position and connectivity. To achieve this, authors come from the probability of face existence, and decompose the probability into two parts: 1) the WDT probability; 2) a probability of vertices' existence on surface. Further, to accelerate the WDT process, authors adopt a novel relaxation operation, making the whole design applicable.
Strengths
+: The probabilistic approach to determining the existence of faces in the mesh is innovative and allows for gradient-based optimization of both vertex positions and their connectivity, which reveals the potential application in many optimization or learning-based reconstruction tasks. +: Both the efficient CUDA implementation and the relaxation design further enhance the usability. +: Sufficient evaluations on settings: mesh reconstruction from ground truth, point clouds and multi-view images, verifies the effectiveness of the paper. And the quantitative comparisons with other methods are well-presented and solid.
Weaknesses
-: Although the method is efficient compared to traditional approaches, the computational cost is still significant, especially when dealing with large datasets. This prevents the application beyond small object level task. -: While the method is tested on synthetic datasets and controlled environments, there is a lack of demonstration on real-world datasets, or texture-rich dataset. Applying DMesh to more complex scenarios would strengthen the paper's contributions. -: No ablation on the influence of \lambda_WDT and \lambda_real.
Questions
I think this is a good work and I believe further improvements as mentioned in the weakness are not easy to complete during the rebuttal. I suggest the author to add experiment on ablation of the influence of \lambda_WDT and \lambda_real to make the core contribution more convincing.
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes, authors discussed limitations.
Summary
The paper presents a differentiable mesh representation for mesh reconstruction from input meshes, point clouds, and for multi-view reconstruction. It builds off ideas of Rakotosaona 21, and uses a weighted Delaunay triangulation framework to predict simplex probabilities. They borrow the notion of distance to reduced power cell, but consider lower bound estimations and precomputation to speed it up. Their method is the first to give a differentiable mesh representation that is flexible enough to handle open/closed meshes, and accommodate changes in mesh topology and connectivity. The method is tested against several competing SOTA works and performs better or comparably in all scenarios (save for producing exactly manifold output).
Strengths
1. As noted, the framework is the first to directly represent meshes in a differentiable way that can handle topology change. 2. They extend (in a straightforward way) the framework of Rakotosaona et al, to the setting of surfaces in 3D, and note the possibility of its use for other k,d (though do not demonstrate use in these scenarios). 3. They provide computational speedup methods that help to alleviate the computational burden of power diagram computation. 4. The authors provide extensive validation and implementation details in the supplementary, including some ablation studies.
Weaknesses
1. The method does not guarantee manifoldness of the end mesh result, which is acknowledged by the authors. 2. The matter of computational cost is swept under the rug a bit. It would be nice to present detailed statistics of some sort (over meshes/point clouds of various sizes, etc.) in the main paper. 3. It is unclear to me how much technical novelty is contained in the estimation step of 4.1 over the prior framework. The steps seem relatively natural and easy to consider once you realize that the dual simplex at hand is no longer a simple point. 4. The earlier parts of the article could be compacted to include at least an overview of the technical pipeline used in the experiments. There are a lot of details that are omitted entirely in each use scenario, especially for the multi-view reconstruction. 5. There is a small, confusing technical mistake above equation (3). The signed distance function should be positive inside, and negative when outside for this to make sense. This is as done in Rakotosaona.
Questions
1. As noted I had some technical questions about S4.1: * For Eq. (5), can you give an instance when this lower bound is not sharp? I thought that if the simplex does not exist in the WDT, then the power cell would remained unchanged under deletion of the other two points? * How is it that one approximates the reduced power cell? This is not explained at all, unless I missed something? 2. Many of the mins,maxes, absolute values, lengths, etc. that appear in the regularization terms appear to be non-differentiable. Are these simply smoothed, or am I missing something? 3. What does the optimization do when there is not a shot showing a particular side of the model? E.g. from the bottom of a model? 4. Why were "reality" weights vertex-based? Was there any consideration for edge-based weights? 5. How often does non-manifoldness occur?
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors appropriately acknowledged their limitations, some of which I listed in Weaknesses above.
Summary
The work introduces a differentiable mesh representation where both the topology of the surface and the connectivity are differentiable. The paper builds on previous work titled 'Differentiable Surface Triangulation,' which suggests using a soft relaxation of Weighted Delaunay Triangulation (WDT) to differentiate through meshing and vertex positions. DMesh highlights that the original work cannot be used as a differentiable mesh representation in a straightforward way and would be very inefficient due to the quadratic scaling of the time complexity with the number of vertices. In light of this, the paper suggests approximating the weights in Delaunay Triangulation using a tight lower bound, which significantly accelerates the algorithm, making it practically linear. Additionally, the work introduces another per-vertex parameter that defines the probability of existence of each face that includes this vertex (given that it exists in WDT). Extensive validation shows that DMesh is capable of reconstructing complicated meshes, including open surfaces and non-orientable surfaces. Furthermore, DMesh outperforms the baselines on the tasks of surface reconstruction from a point cloud and multi-view rendering, while also being a more flexible and general representation.
Strengths
The paper is easy to follow, and despite the complicated nature of the topic, the narration remains clear and formal; main concepts are well illustrated. The proposed modifications to Differentiable Surface Triangulation are significant and not straightforward. Extensive evaluation of the proposed method proves its effectiveness and significance for the field. *Additional notes*: + Many regularization terms can be easily expressed natively for this representation, as demonstrated in the paper. + To the best of my knowledge, the paper compares against all the required baselines. + The supplementary material provides seemingly all the details needed to reproduce the algorithm. + The paper provides an efficient CUDA implementation of the algorithm (though I did not test it). + The demonstrated results are very impressive, and the resulting meshes look very clean. + The limitations of the paper are discussed in great detail. Given all of the above, I believe this paper is well-positioned to significantly contribute to the community and potentially return attention to explicit representations in many fields, such as differentiable rendering, physics-based optimization, and so on.
Weaknesses
I am a bit confused why the paper states that the running time of “Differentiable Surface Triangulation” growth exponentially with the number of vertices, while in fact it is quadratic? I would like to see either an explanation or a corrected version by the rebuttal. While the method is very well evaluated and ablated, I think a few additional results will be interesting to see. Here they are in the order of decreasing significance: 1. The biggest question to me so far is if the interpolations between two DMeshes are possible, especially if there is a topology/connectivity change. All the presented experiments showcase an optimization process that can start in a good initialization of the DMesh. My concern is that, after convergence the DMesh should get somehow more binary in $\Lambda_{\text{wdt}}$ and $\Lambda_{\text{real}}$, what will make it hard to take it out of this local minima in the further optimization process. By the rebuttal I would like to see a simple demonstration of a mesh-to-mesh optimization with a surface topology change, or a discussion of this in the limitation section if this is not trivial. 2. In the limitations it is mentioned that sometimes the resulting mesh is non-manifold. It would be interesting to see some examples of failures and an estimation how often it happens and why? (at least I did not find this in the paper). 3. (Minor) The point cloud reconstruction results are demonstrated for 100k sampled points, which is very dense. It would be interesting to see how the quality of mesh reconstruction depends on the number of sampled points. 4. (Minor) Table 2 is evaluated only on 11 meshes, which is not much. Would be great to see a bigger test set by the camera-ready deadline. *I would be ready to even further increase my rating if these questions are addressed.* Minor Notes and Typos: - Line 91: “it is not ..” seems to miss something - Would be great to increase the resolution of illustrations. In particular, the pixelation is very visible on Figure 6., at least in my PDF viewer - Footnote 3 on page 5 – seems to be a typo around “and p_j and p_k”
Questions
Here I have a few questions that don't affect the rating, but are rather out of curiosity. Including them in the final manuscript is up to the authors: 1. It looks like the representation allows the user to freeze the connectivity of the mesh if they want to. Is it also possible to "freeze" surface topology ? 2. Did you have to prevent the optimization process from converging to binary values of the probability ? (which I believe will make the mesh effectively non-differentiable ?) 3. is there anything we can say about the existence and uniqueness of the additional two parameters ($\Lambda_{\text{wdt}}$ and $\Lambda_{\text{real}}$) when reconstructing a mesh with the reconstruction loss from section 4.3.1. ? After the authors addressed the questions in the rebuttal, increasing my rating to strong accept.
Rating
8
Confidence
3
Soundness
4
Presentation
4
Contribution
4
Limitations
The limitations are discussed in great detail, apart from the lack of examples of non-manifold meshes as mentioned in Weaknesses. The broader impact discussion does not seem to be necessary here.
Thank you for the detailed response. For Q8: 1. when you say "we can simply adjust the real values stored in the 3 vertices", do you mean that the optimization process naturally has gradients that minimize $\Lambda_\text{real}$ or that you need to embed this behavior separately? 2. In the case of the probability of face existing = 1, are there any cases when we would prefer the optimization process to decrease $\Lambda_\text{wdt}$ instead? I am curious to understand if the over-parametrization caused by having two parameters ($\Lambda_\text{real}$ and $\Lambda_\text{wdt}$) might cause any kind of dead-locks in the optimization process. As you mentioned in Q9, for a fixed tesselation there is a global minimizer in terms of $\Lambda_\text{real}$, but I believe that to efficiently optimize from one surface to another we will need to change tessellation first and only then optimize $\Lambda_\text{wdt}$. 3. Thank you for providing the mesh-to-mesh optimization experiment. This is very interesting. Do I understand correctly that the optimization loss was defined through multi-view inverse rendering? I noticed that the tessellation has changed and that in both cases the meshes seem to get bigger triangle size. Do you have any intuition why this happens?
Thank you for the great questions, it inspires us a lot. To answer your additional questions, --- 1. We have not embedded any specific behavior to prefer $\Lambda_{real}$ over $\Lambda_{wdt}$, or manually optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$ in the optimization process. Technically, $\Lambda_{real} = 1$ and $\Lambda_{wdt} = 1$ would get the same amount of gradient, because they are just multiplied together to get the final face existence probability (Eq. 2). However, note that $\Lambda_{wdt}$ includes sigmoid function in its formulation (Eq. 4). Therefore, the parameters that are fed into the sigmoid function could get very small gradient when $\Lambda_{wdt}$ is almost binary. In contrast, $\Lambda_{real}$ does not have sigmoid function in its formulation, which means that we do not have to worry about vanishing gradient problem for $\Lambda_{real}$. In this case, the optimization process would "naturally" optimize $\Lambda_{real}$ instead of $\Lambda_{wdt}$, because $\Lambda_{wdt}$ would not be very optimizable. So we intended to say that even when $\Lambda_{wdt}$ is binary, optimizer would be able to adjust $\Lambda_{real}$ to escape the binary state. --- 2. As described above, if $\Lambda_{wdt}$ is almost binary, optimizer would automatically adjust $\Lambda_{real}$ instead of $\Lambda_{wdt}$. And as far as we understand, your concern is about only manipulating $\Lambda_{real}$ instead of $\Lambda_{wdt}$, even when we could get better optimization result when we change $\Lambda_{wdt}$ instead to get better tessellation. Correct us if we are wrong, but if our understanding is correct, our current approach does not care about this possibility yet. Our current algorithm minimizes the loss function in blind manner right now, regardless of specific $\Lambda_{real}$ and $\Lambda_{wdt}$. Therefore, we would have to say it finds the local minimum near current tessellation, instead of finding the global minimum near the optimal tessellation. However, we believe that the coarse-to-fine strategy in Appendix D.2 is a very good method to re-initialize $\Lambda_{wdt}$, thus tessellation, near the global optimal tessellation. Therefore, we'd like to suggest to use that method to find the global minimum. --- 3. Yes, as you mentioned, the optimization loss was defined with multi-view inverse rendering. The reason about the triangle size is related to the answers above. Because $\Lambda_{real}$ is usually easier to optimize than $\Lambda_{wdt}$, it is optimized faster. That is, while the entire tessellation does not change a lot, only the faces that we select change fast. To elaborate, at the end of the first optimization to fit DMesh to a single torus, most of the points are gathered around the torus surface. Since there are not so many points located at the places far away from the torus surface (as we remove unnecessary points using regularization), the faces in that region become larger than those located near the torus surface. Then, we start from that DMesh to fit different shapes like double torus, and in that process, rather than moving (single) torus surface toward the target shape, the optimizer just removes the undesirable parts on the torus surface by adjusting $\Lambda_{real}$. And then, it makes the probability of desirable faces larger, and they usually have larger faces, because they were located far away from the original single torus. After that, $\Lambda_{wdt}$ is optimized slowly to fine-tune the shape. After this process, if we re-initialize DMesh using the coarse-to-fine strategy, then it would get much better initial tessellation to start from. Then, it would end up in much smaller, fine-grained faces in the end. We believe it would be very interesting to explore if we can impose any kind of guidance to select (or prefer) between $\Lambda_{wdt}$ and $\Lambda_{real}$ during the optimization. Thank you again for the inspiration.
Thank you for the fast reply. This is very interesting. It will be interesting to look if stopgradient or just rescaling the gradients will help to define the balance between $\Lambda_\text{wdt}$ and $\Lambda_\text{real}$. I think the mesh-to-mesh example will be a good addition to support the claims of the paper, so please consider including this in the final version. At this point, I don't have any more questions and will gladly adjust my recommendation.
Thank you very much, we agree that we can consider such balancing techniques! Also, we will definitely add the additional experimental results in the paper, thank you for the suggestion. It would be really big help for us if you could give a raise to our rating, we will really appreciate it :)
Summary
The authors first propose a differentiable mesh representation, DMesh, which can represent a wider variety of mesh types. They then introduce a computationally efficient approach to differentiable weighted Delaunay triangulation that can run in approximately linear time. Following this, an efficient algorithm is provided for reconstructing surfaces from both point clouds and multi-view images, with DMesh used as an intermediate representation. Finally, an effective regularization term is proposed for mesh simplification and enhancing triangle quality.
Strengths
The method proposed by this paper aims to optimize the geometry and topology of the reconstructed mesh. It presents a computationally efficient approach to differentiable weighted Delaunay triangulation, which can achieve this goal. The methods in this work are partly valuable to the NeurIPS community.
Weaknesses
In this work, the inner structure is removed via a depth test, meaning that any reconstruction method producing an inner structure can be improved by using the depth test to remove the unseen parts of the mesh. While this post-processing step does help to improve the quality of the result, it is not a feature of the proposed method itself.
Questions
1. During the post-processing step, the inner structures are removed via depth testing. How does the method handle the boundaries caused by the previous operation? 2. In line 91, "... and it is not." should be completed for clarity. Maybe the authors missed some words at the end. 3. The DMesh is constructed from the normal mesh. However, in real applications we often do not have access to the mesh. How does the DMesh help in reconstructing the result?
Rating
6
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Yes.
About global response
Sorry for your inconvenience, but we just found out that the global response is not visible to the reviewers. We will make it visible to reviewers as soon as possible. --------- update It is visible now, thank you.
Thank you
Thank you for the extensive responses. I will be keeping my score as is entering the discussion phase. Reviewer
Thanks for the authors' response with details, it solved my questions and I'd like to change my rating to weak accept.
We're happy to hear that our response solved the questions, thank you for the raise!
Decision
Accept (poster)