Shape Non-rigid Kinematics (SNK): A Zero-Shot Method for Non-Rigid Shape Matching via Unsupervised Functional Map Regularized Reconstruction

Summary

The paper proposes a zero-shot method for non-rigid shape matching based on unsupervised functional map regularisation and non-rigid shape deformation regularisation. The proposed method combines both intrinsic information from the functional map regularisation and the extrinsic information from shape deformation regularisation (i.e. PriMo energy) leading to a better matching performance compared to methods solely based on either intrinsic or extrinsic information from the 3D shapes. In the standard shape matching benchmarks, the proposed method demonstrates better matching performance compared to axiomatic approaches and competitive results in comparison to both supervised and unsupervised learning-based methods.

Strengths

1. The paper is well-written and easy to follow. Both the motivation and the proposed method are well explained in the main paper. The technical details are also well illustrated in the supplementary material. 2. Unlike most existing shape matching methods solely based on functional map regularisation, the proposed method also considers spatial alignment and adds the shape deformation regularisation to incorporate also the extrinsic/spatial information leading to a better matching performance. 3. Instead of using common deformation regularisation such as ARAP, Laplacian, the paper utilises the PriMo energy leading to a more smooth and natural shape deformation, as demonstrated in the supplementary material.

Weaknesses

1. The paper claims that the method is a zero-shot method for non-rigid shape matching. But the main difference between the proposed method and the previous unsupervised methods is not well discussed. In theory, all previous unsupervised methods can also be used as a zero-shot method to optimise only for a single shape pair. More explanation and experiments should be provided. 2. In the supplementary material, it shows the average runtime on the FAUST dataset is about 70s, which is much slower compared to existing learning-based methods. Prior unsupervised works (e.g. Cao et al. Unsupervised Deep Multi-Shape Matching, ECCV 2022) typically train on the training data and utilise test-time-adaptation for a few iterations to reduce runtime. More discussion and experiments about the runtime is expected to be provided. 3. Unlike purely intrinsic/spectral methods, the proposed method also aims to spatially align two shapes. Therefore, more insights about the rotation robustness of the proposed method should be given.

Questions

1. Even though in the limitation part, the paper claims that the proposed method is specifically tailored for complete shape matching, it is also recommended to include some (potential failure) matching results of partial shapes in the supplementary material. This will benefit follow-up works addressing partial matching. 2. What is the number of the LBO eigenfunctions for functional map computation? Prior methods like Smooth Shells/Deep Shells optimise the matching with gradually increasing number of the LBO eigenfunctions - is the proposed method also robust to the choice of the number of LBO eigenfunctions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

1. The proposed method is specifically tailored for complete shape matching, so more adaptations should be taken for partial shape matching. 2. The runtime of the proposed method takes 70s for shapes with ~5000 vertices, which is much slower compared to existing learning-based methods. 3. The proposed method is tailored to shapes represented as triangle mesh, so the extension to other data representation like point cloud could be the future work.

Summary

A zero shot method for computing correspondences between meshes is proposed. The core idea is to use DiffusionNet to produce features which are then used to produce a functional map from which a p2p mapping is produced. Using it, a deformation module using Primo is employed to deform the meshes to one another. This deformed shape is then used to reevaluate the functional maps and the process continues, training an autoencoder to predict the features that drive the correspondence and deformation.

Strengths

Zero-shot correspondence computation is an important problem to solve in cases where no object categories are known. The combination of an extrinsic deformation module with an intrinsic functional maps approach is appealing and novel as far as I can tell. The paper is written in a clear manner for the most part (see below).

Weaknesses

I have several concerns: * In this context, "zero shot" essentially means "no use of semantic information" (as opposed to, e.g., zero shot stemming from using some existing module that has semantic knowledge and applying it to the correspondence problem without further training). Thus, the term used in the paper for "axiomatic" methods simply means ones using geometric priors rather than semantic ones. As such, I am uncertain why use the word "zero shot" at all -- this is simply an optimization algorithm for shape correspondence. As such it lies within a different research area than learning which encompasses many other works. Two recent ones are ENIGMA and Neural Surface Maps. The advantage of this approach of theirs is not immediately apparent and they should be discussed at least, if not compared to. * It is unclear to me what is the exact prior being enforced here. At first I thought it's a an isometric-distortion minimization prior but the authors show results on non-isometric datasets, so it is unclear to me what is the prior here and what is the justification for it. Without a justified prior nor semantics, it seems concerning the method solely relies on empirical success. It is further unclear what specific class of deformations is Primo regularizing for. It does seem to be for reducing isometric distortion here, thus I do not fully understand the claim to work on a non-isometric case. I suspect that the method is still working on rather-isometric cases - SMAL's 4-legged animals are still rather isometric. Again in this context, ENIGMA shows success and seems like a good comparison. * Additionally, the method is currently limited to matching meshes that are homeomorphic and are made up of one connected component - this is not mentioned in the paper and is somewhat limiting. * in terms of novelty, the method is simply a mix of several techniques - DiffusionNet, Functional Maps, Primo, which indeed work well together, but there does not seem to be a great novel insight beyond combining them to have a method to predict correspondences and then update a deformation prediction, which is a very classic idea (in essence, a non-rigid ICP approach where the closest point is replaced with a functional map prediction).

Questions

how is the use of the low eigenspectrum of the laplacian justified without assuming near-isometry? why is the autoencoder needed? why couldn't the functional maps and deformation module be optimized in block-descent manner directly?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

yes

Summary

The paper proposes a novel zero-shot method for non-rigid shape matching that eliminates the need for extensive training or ground truth data. The proposed method, Shape Non-rigid Kinematics (SNK), operates on a single pair of shapes and employs a reconstruction-based strategy using an encoder-decoder architecture, which deforms the source shape to match the target shape closely. SNK demonstrates competitive results on traditional benchmarks, simplifying the shape-matching process without compromising accuracy.

Strengths

The paper introduces a novel method for non-rigid shape matching that does not require extensive training data, making it more practical for many applications. The proposed method also combines the benefits of axiomatic and learning-based approaches and addresses their limitations. Moreover, the paper introduces a new decoder architecture that facilitates the generation of smooth and natural-looking deformations. The proposed method is thoroughly evaluated and achieves state-of-the-art results on the SHREC dataset.

Weaknesses

While the paper highlights the strengths of the proposed method, it lacks a thorough comparison with closely related work. For instance, although several learning-based methods are presented, there is no detailed comparison, such as a visual comparison and evaluation of other datasets (recommend moving some results to the main paper). Also, the limitations of the proposed method are mentioned but need to be adequately discussed. For performance, the proposed method can not achieve the best average, especially for some unsupervised methods. For the SOTA - Deep Shells, why do the proposed methods have the similar performance on Faust and Scape datasets instead of Shrec? Could there be some analysis and discussion of the results? The description of the Prism decoder is not clear, and I cannot find the corresponding components in Figure 3. I suggest the author re-illustrate the figure and add more details for easier understanding. Some mathematical notations are repeatable: > F_i and F_j in eq3, F_1 and F_2 at line 1130

Questions

1. For the Shape reconstruction module, why not directly use the vertex feature to predict the deformation on the vertex rather than using the features on the faces to predict the rotation of the faces? BTW, I think the optimization of eq. 5 is not trivial and needs more time to get optimal. 2. The proposed method is currently tailored towards complete shape correspondence. Have you thought of adapting it for partial shape-matching scenarios? 3. In the limitations section, you mentioned exploring ways to further expedite the training process. Could you provide more details on what you mean? 4. In the loss function, how do you determine the weight for each term and evaluate the importance of each term for the target?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper adequately addresses the limitations of the proposed method but could benefit from a detailed discussion of the potential negative societal impact. Besides, I think the running time should be evaluated. I think the optimization of Eq. 5 is too slow and takes other steps. The paper also mentions it in the last section.

Summary

This paper works on learning a point-to-point mapping between two sets of deformable mesh vertices in a self-supervised manner. To this end, the authors extract features of two meshes from a DiffusionNet, solve for functional maps by an optimization problem, and finally convert the functional maps to a point-to-point map iteratively. In the meantime, a per-face rigid transformation is generated from a shape decoder and is used to transform the source shape to the target shape. To demonstrate the effectiveness of the proposed method, the authors conduct experiments on both near-isometric and non-isometric datasets, and achieve reasonable performance as a self-supervised method.

Strengths

- The paper is well organized and written. The problem is well defined in Sec.1, while the previous works and preliminary knowledge are also well introduced in Sec.2 and Sec.3. - The two-stream (implicit shape transformation decoder and explicit functional map) pipeline is a reasonable design: - The low-rank functional map estimation is not only efficient but also facilitates the learning of feature extraction. - The MLP shape decoder and the PriMo energy regularize the predicted transformation by both network structure and training losses. - As a self-supervised method, the performance is strong on both near-isometric and non-isometric data.

Weaknesses

I do not see major weaknesses except for some details: - The feature extractor should have more discussion, since it is key for the functional map prediction. Specifically, the author could either visualize or quantitatively measure the consistency of the extractor features. In addition, it would be better to explicitly indicate that the DiffusionNet is not pre-trained (in spite of lines 36~37) to avoid confusion with existing models. - The pipeline figures could be better illustrated, such as the module blocks can be colored based on whether they are learned or have explicit formulation.

Questions

The shape encoder might be redundant, and similar to DeepSDF the authors can directly learn the latent code jointly with other module parameters.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been adequately addressed.

Reply

Thanks for the authors' detailed responses and clarifications. After reading the rebuttal and other reviews, most of the concerns have been fully addressed. I suggest all of the revisions should be presented in the revised paper. Besides that, I am happy to see some visual comparison with other baselines and the visual/numerical results on the diverse datasets.

Response to the authors

Dear authors: Thank you for the response. The rebuttal has addressed all my questions.