In this paper, we propose a learning-based framework for non-rigid shape registration without correspondence supervision. Traditional shape registration techniques typically rely on correspondences induced by extrinsic proximity, therefore can fail in the presence of large intrinsic deformations. Spectral mapping methods overcome this challenge by embedding shapes into, geometric or learned, high-dimensional spaces, where shapes are easier to align. However, due to the dependency on abstract, non-linear embedding schemes, the latter can be vulnerable with respect to perturbed or alien input. In light of this, our framework takes the best of both worlds. Namely, we deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps (DFM). In particular, the correspondences are dynamically updated according to the intermediate registrations and filtered by consistency prior, which prominently robustify the overall pipeline. Moreover, in order to alleviate the requirement of extrinsically aligned input, we train an orientation regressor on a set of aligned synthetic shapes independent of the training shapes for DFM. Empirical results show that, with as few as dozens of training shapes of limited variability, our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching, but also delivers high-quality correspondences between unseen challenging shape pairs that undergo both significant extrinsic and intrinsic deformations, in which case neither traditional registration methods nor intrinsic methods work. The code is available at https://github.com/rqhuang88/DFR.
Paper
Similar papers
Peer review
Summary
The method proposes an unsupervised pipeline to solve 3D shape-to-shape registration. Firstly, the two shapes are aligned by a pre-trained orientation regressor. Then, a soft-correspondence is obtained by a Point Feature extractor, optimized using a Deep Functional Maps schema, and considering several unsupervised regularizations (e.g, bijectivity, orthogonality). Finally, the source shape is iteratively deformed using a Deformation graph (using Chamfer distance, the learned correspondence, and as-rigid-as-possible), and the correspondence is updated every 100 iterations. The method is tested on humanoid datasets (FAUST_r, SCAPE_r, SHREC19_r, SHREC07-H, DT4D-H), and extensively compared against many other approaches, both supervised and unsupervised. obtaining interesting results. == FINAL RATING == In the discussion phase, authors provided much further evidence and clarified my concerns. I notice, however, that while a majority is leaning toward acceptance, there is not a consensus among the reviewers. Going through the reviewers that assigned negative scores, I do not find significant ground to change my rating. In particular, Reviewer xFDL mainly criticizes the novelty of the work but does not point to other works to support this claim. While I can understand that the work does not seem particular novel in its component (and the general principle of registration + correspondence in feature embedding can be found in previous works (e.g., more SmoothShell, DeepShell than DPC, while both of them are designed for meshes), I do not find itself a proper ground of rejection (as also reported in the Reviewing Guidelines; also combinations of existing techniques are valuable). The experimental evaluation is appreciable, the performance convincing, and I do not see this as incremental w.r.t. any other previous work. I find this an interesting contribution to a research field that counts only a limited amount of work. Instead, I see the detailed review and discussion of Reviewer iB8H, and I think it contains many valuable observations that could improve the paper. However, summarizing the reported criticisms seems to be: A) novelty w.r.t to DPC; which in the last comment looks solved, or at least significantly tuned down B) general motivation of the work; while I see that the underlying principle and positioning of the work in literature can be improved, overall, I do not see other works that perform similarly, and I struggle to see this as a "follow-up" of a specific methodology to consider this an incremental contribution of something in particular. Also, the obtained results seem to be already a reasonable justification for the proposed approach (since I think we all agree that the paper effort is beyond engineering work, and hence, they communicate a promising research direction) C) Other details (e.g., clarifying the role of hyper-parameters, missing citations, rephrasing); that I think make sense, but they can be easily addressed in the camera ready, and I do not consider a sufficient ground for rejection. For these reasons, I lean toward acceptance. I suggest authors incorporate the suggestions (especially about suggested experiments and paper positioning in literature), and I wish them the best of luck with their work.
Strengths
1) Not many methods are available to solve shape-to-shape correspondence in an unsupervised way; the proposed approach smartly combines existing techniques and well-established methods to obtain good results. 2) The proposed approach outperforms existing direct competitors, even some supervise approaches. I am sure it would be of interest for the shape-matching community. 3) The paper is well presented, clear and direct, I enjoyed the reading, and I had no problem understanding the main components. I am sure the method can be re-implemented with limited effort.
Weaknesses
1) The general applicative context of the method is unclear. From the introduction and the pipeline figure, I had the impression that the method is designed to obtain correspondence between point cloud. However, the method is about registering a mesh to a point cloud, which can be extended to the point cloud case, using the source mesh as a bridge. This, however, is not properly tested: while other approaches also show noisy point clouds with different levels of noise (e.g., [30]), in this case, only point clouds sampled from meshes are reported, with a gaussian noise that is not detailed nor in the main manuscript or in the supp.mat. . Given the nature of the used feature extractor (which I expect to be quite sensitive to real noise and clutter), I suggest including more details on the considered Gaussian noise, and test on real raw pointclouds (which could also be another failure case, but would point the reader to interesting future directions) 2) While I appreciate the simplicity of the proposed approach and the combination, I struggle to see a general message or insight. Also, I do not see the method as directly applicable since it is trained only on a few shapes, and we do not know how it would scale on larger datasets (e.g., AMASS), and I guess it would fail to solve registrations in real contexts (e.g., noisy scans), since it would ruin the learning on the underlying surface. It is also tested only on a domain (humans) with a significant amount of labelled data. So, without these elements, I wonder if the paper could be of significant impact or if it is just a carefully designed pipeline that will be just another matching method without much influence in the field. I suggest discussing the main message, analysing the scalability of the approach, and testing on domains in which labelling data are much more complicated to obtain (e.g., chairs, which also would make stronger the claim about topological noise in the supplementary)
Questions
1) In Table 1 some results are not reported. Why? 2) The orientation regressor module resembles the input transform module of PointNet. Wouldn't a rotation augmentation for the feature extractor work (and remove the need for a further module)? Another possible alternative is given by the method proposed in [A]. 3) Given that overall the pipeline assumes some degrees of bijectivity, how would it perform in the presence of a significantly different number of vertices between the two shapes (e.g., by a factor of x100)? Even considering two complete shapes which have a bijectivity on the surface, the losses are defined vertex wise, and might lead to a degradation of the regularization impact. Minor: 1) I see that reference [30] is incorrect in the bibliography, and [31] has names compressed. I suggest double-checking the references. 2) Relevant references not discussed: [B], [C], [D] [A]: Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes, Zhou et al., 3DV 2022 \ [B]: NeuroMorph: Unsupervised Shape Interpolation and Correspondence in One Go, Eisenberger et al., ECCV 2021 \ [C]: NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching, Attiki et al., NeurIPS 2022 \ [D]: 3d-coded: 3d correspondences by deep deformation, Groueix et al., ECCV 2018
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Limitations are discussed in the main paper.
Summary
In this paper an unsupervised non-rigid shape registration method is proposed. The proposed method combines intrinsic spectral mapping (i.e. based on the deep functional map framework) together with extrinsic deformable shape registration (i.e. a deformation graph) to enable unsupervised 3D deformable shape matching. In many challenging benchmark datasets, the proposed method demonstrates competitive matching performance, better cross-dataset generalisation ability, and robustness against noise and rotation of input shapes.
Strengths
1. The paper is well-written and easy to follow. The main contribution and methodology are well illustrated. 2. The paper integrates shape matching and shape registration into the same framework. The shape matching part is based on the deep functional map framework to obtain point-wise correspondences. The shape registration part is based on the deformation graph to non-rigid align two shapes to refine the final correspondences. 3. In order to enable matching shapes with different orientation, the paper proposes an orientation regressor to align shapes into a canonical frame.
Weaknesses
1. The novelty of the method is limited. The proposed method consists of three components (rotation regressor, feature extractor, shape registration) and each component is derived from prior works without large modifications. 2. The interconnection of the components in the method is missing. The first stage is to train a rotation regressor to align shapes into the canonical frame. The second stage is to train a feature extractor to obtain correspondences. The third stage is to optimise the deformation graph to align two shapes. Every stage is somehow separated during the training stage, while it is desired to see the connection of shape matching and shape registration like Deep Shells or NeuroMorph.
Questions
1. Since the proposed method is based on test-time optimisation, what is the convergence speed of the proposed method? 2. Since the deep functional map can be fully intrinsic (i.e. with intrinsic input features), what if we use it to obtain the initial correspondences and rigidly align the shapes into the canonical frame without using rotation regressor?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
1. The proposed method is tailored to complete shape matching, so it cannot achieve desirable matching results for partial shapes. 2. The proposed method is based on iterative optimisation to align two shapes, so the runtime is slower than other learning-based methods.
Summary
The paper describes a method for corresponding a 3D triangle mesh to a point cloud of a similar (possibly articulated) shape. The two-stage process first corresponds the two shapes in a high-dimensional feature space, then corresponds them again using geometric features while deforming the source closer to the target.
Strengths
The approach makes sense, and the results seem strong compared to previous work, both qualitatively and quantitatively.
Weaknesses
In general, the presentation could be improved. There should be a figure analogous to Figure 1, but showing qualitative correspondences for cases where train/test datasets do match. Only showing severe failures of competing methods on very disparate shapes does not convey the full picture. The supplementary video showing the progress of the correspondence algorithm is very helpful. It would be even more informative if the mesh was texture mapped with the checkerboard as in Figure 1. Equations, Figures, Tables need more thorough descriptions. Equation 1: Even though the text says that one can optimize the feature function F when it introduces equation 1, it is unclear how the function F is optimized. If not including an equation involving F, you could at least point to a specific equation number in [11]. Equation 2: Parameter alpha should be defined after this equation, rather than after equation 4. Equation 3: What does the cross symbol represent? Equation 4: Parameter n2 is not used in the expression. Should the denominator summation go up to n2? Table 1 & 2: What are the two sections separated by a horizontal line (they each have a bold set of numbers)? Train/Test column contains no values. Table 2: What is "Ours-CRYPTO", this acronym is not discussed/referenced in the text? Table 3: What is "Ideal PC"? Algorithm should also initialize "Flag" (presumably to Stage-I).
Questions
Could this approach be implemented without learned functions? The intro mentions that spectral mapping can be geometric (rather than learned), and the second stage is akin to non-rigid ICP. Would it make sense to run Stage-I again after Stage-II, and repeat both stages a few times?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The method requires a triangle mesh for the source shape, which is not discussed in the limitations section. The exposition would be stronger if the authors took the source shape as a point cloud and ran some automatic meshing on it for their experiments. Another limitation that the authors mention is that the method only works on "full shapes". I assume this means that the whole source and target surfaces are adequately sampled. It would help if the authors suggested some ideas on how one would go about lifting this requirement.
Summary
This paper proposes an unsupervised framework for non-rigid shape registration. The proposed method deforms source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps. Empirical results show that the proposed method achieves state-of-the-art results on several benchmarks for non-rigid point cloud matching.
Strengths
1. The overall writing is fluent; 2. The proposed method outperforms state-of-the-art approaches on several benchmarks;
Weaknesses
1. The organization can be further improved (the organization of Methodology does not follow Fig. 1), which makes the paper sometimes hard to follow (probably due to the complexity introduced by assembling 3 stages for tackling the problem); 2. The novelty of this paper would be a major concern. It seems the proposed algorithm in this paper simply assembles three stages, each with an existing method. The orientation regressor is from [9]. The feature extractor is the modified DGCNN proposed in [22]. The non-rigid registration mainly follows [18]. 3. The submission and main paper have different titles. Please fix it.
Questions
1. Why is the proposed method unsupervised? It seems the orientation regressor requires the ground-truth pose to the canonical space. And the PointInFoNCE loss (Eq. 4) also requires correspondence labels? Or the authors call the generalization from one dataset to the other datasets the unsupervised learning? 2. How good is the orientation regressor? Can it deal with large translations? e.g., the translation introduced by a walking human. From my experiences, this kind of pose regressor can only overfit to a specific model and hard to generalize. Also, it is usually not very accurate (correct me if I was wrong); 3. Why all the models are called "pre-trained"? Should it be generalizing a model trained on the same task to new data?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Limitations have been discussed in the main paper.
Summary
This paper studies the problem of non-rigid shape matching. The problem is first decomposed into learning (rigid) orientation of shapes and then learning shape matching on aligned shapes. For the later, it proposes to learn it with a sequential pipeline, consisting of various modules, that is also optimized in a two stage manner. the main idea is to facilitate learning by similarity in both ambient space (R^3) as well as high dimensional learned feature space. To this end, it trains a DGCNN feature extractor based on combination of several loss functions (DFM prior loss, ARAP loss, Chamfer/cosine loss). The method is validated on isometric as well non isometric benchmarks. Post Rebuttal: The rebuttal provided additional experiments to support some of its claim as mentioned below. I lean towards rejection since the work is a sequential concatenation of existing individual modules (also noted by reviewer CNuX and xFDL ) without any conceptual justification to why those modules need to be combined e.g. why do we need to combine a DFM and ARAP loss. The rebuttal also contains several unsubstantiated statements detailed below. Combining two lines of research is not a technical contribution or contribution to community if there no conceptual justification given. Majority of the introduction section needs to be rewritten and contextualised *correctly* wrt prior work. Besides, submission had several hyper-parameters whose values were missing and there is no mention of model sensitivity towards these hyper-parameters even though the approach is unsupervised.
Strengths
- Experimental results on various near isometric benchmark look promising (Table 1) and compares extensively with existing work. - the paper identifies the generalization problem of some embedding based approach [30,8] to unseen shape data.
Weaknesses
- Presentation: Section 1 contains several factually incorrect or unsupported claims. Moreover, the motivation of the work is misplaced since their is no validation of it later in the experimental section. Please support all the intuitive claims either with a prior reference or by explicitly mentioning results from this paper. a) why do we need to decompose the shape matching problem into learning an alignment first and then matching aligned shapes? Is it because [8], [22] do so or is there a scientific justification behind it as shown empirically in rigid shape matching or non-rigid shape matching with DFM. e.g. it is shown to provide an extrinsic supervision that helps to disambiguate symmetry issues in DFM. Please motivate the problem/solution accordingly. b) Line 46 < prior work with resulting high dimensional embedding lack intuitive geometric meaning> Since this is used as one of 3 reasons behind this paper/formulation, please demonstrate this on an example where in contrast, this paper obtains an intuitive embedding with geometric meaning. c) Line 54-57: Please provide a reference to support these claim or prove it later with visualization and examples in experimental section. d) [30] requires shapes to be pre-aligned at train or test time. This is not true. You can train and test [30] without such alignment and it makes no such assumption on input requirements in the paper[30]. - Novelty: the main conceptual idea/key insight (Line 52) in this work is to enforce similarity in both ambient space as well as learned feature space. DPC [b] proposed the exact same idea with DGCNN, cosine/chamfer loss in the most simplest possible way for non-rigid shape matching. The idea is already well known that others have even built upon it for non-rigid shape matching e.g. [a]. So claiming this as a key insight is not justified IMO. - Missing references and comparison of a very similar work[b]: authors should also compare their work with DPC on their benchmarks. this will show what gains (if any) are brought by DFM prior or ARAP loss. a. Learning Canonical Embeddings for Unsupervised Shape Correspondence with Locally Linear Transformations b. DPC: Unsupervised deep point correspondence via cross and self construction, 3DV 2021. - formulation without a principled approach: the network consisting of (DGCNN, DFM prior, ARAP loss, chamfer loss, pointinfo loss etc) combines modules from different frameworks without any principled reason. e.g. DFM prior from DFM literature, DGCNN feature extractor, chamfer & cosine loss from DPC etc. in a two stage optimization procedure. why do we need a DFM prior when we are deforming a source and a target shape with a ARAP loss ? - Too many hyperparameters in an unsupervised approach : The paper should mention in the main body how many hyperparameters overall does this approach has and how were their values chosen? I count more than 10. Moreover, how is it justified to choose different values for the same hyperparameter (weighing scalars) in an algorithm and call the resulting approach *unsupervised*? there is a two order magnitude of difference in hyperparameter values ($\lambda_{cd}$ and $\lambda_{corr}$) between different runs of the algorithm (Stage 1 and Stage 2).
Questions
- for Non-isometric, SMAL is considered the main benchmark and there is already a large literature that benchmark their results on SMAL. why ignore such standard benchmark?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
yes
Post-Rebuttal
I thank the authors for their reply to my concerns. I appreciate the amount of experiments and different settings provided in the rebuttal. I would just say that probably the dependence on bijectivity can be elaborated on in the limitation section. This is because even if it is true that "down-sampling the underlying smooth surfaces, in general, would not lose many high-frequency signals", ideally, some applications would obtain the highest possible precision. Matching could have more "scales" (i.e., we seek a global but also a local coherence), and at higher frequencies, the details are (often) more prone to vary across different instances of the same class (and bijectivity does not hold much). Of course, it is not a request to solve also this, and I will not consider this a weakness, but more a discussion point to enhance the conclusion/limitation discussion. I do not have other questions, and I maintain my positive opinion. Looking forward to hear from other reviewers their feedback.
We thank you for the reply and positive opinion. Regarding the problem of bijectivity dependency, we think that the challenging cases you mentioned are beyond the scope of the current submission and that it can be indeed an interesting future direction (e.g., matching high-resolution human faces with the proposed pipeline). We will add the respective discussion in the future revision.
follow up on rebuttal
thank you for your time and effort. The reviewer has gone through the supplement and main submission multiple times as suggested in rebuttal. Comments below: \ - Algorithm 1 (pseudo-code) not same in Main submission and supplement: There is an additional early stopping count condition in Line14-17 of Algorithm 1 in supplement. no such thing exist in algorithm 1 of main submission. I assume the results shown in main submission are based on the algorithm from supplement? if so, please explain what this early stopping (count <15) criteria is. why do we need it especially if the algorithm has another stop criteria (max_iteration =100) in the underlying optimization. (Line 183 in main submission). why $E_i -E_{i-1} < eps$ would not suffice for termination of any algorithm? \ - Claims made in Line 54-57: The rebuttal repeatedly points to the table 1 and texture transfer figure 1 to justify these intuitive claims.These table and figures contain the end result of this approach ( a concatenation of 5 full fledged siggraph/TOG/CVPR papers (DGCNN, DiffusionNET, orientation regresssor, modified DFM, ARAP) along with geodesics to tackle a single problem). the comparison with NDP/AMM to justify these claims is unfair since this submission takes a SOTA non rigid shape matching method (DiffusionNet+modified DFM) as a head start. a fair way to justify these intuitive claims of tackling large deformation with superiority over NDP/AMM or any other baseline would be to also initialize them with the same SOTA non rigid shape matching (e.g. w/o registration baseline in ablation table). \ - Too many hyperparameters and heuristics for an unsupervised approach: In addition to the count, iteration parameters outlined above, the rebuttal/submission also misses out on critical hyper parameters and their values etc: -- Threshold parameter in correspondence filtering and its value, how it is chosen -- Rebuttal mentions $\lambda_{cd}, \lambda_{corr}, \lambda_{arap}$ were chosen based on 2 training shape pairs but does not mention how even though their values are critical to understand what contributes to overall performance in shape registration pipeline (details below). Since these parameters are different for different stages, they should also be indexed accordingly to distinguish better. \ - Stage 1 and Stage 2 in Shape registration: Based on the $\lambda_{cd}, \lambda_{corr}, \lambda_{arap}$ values in two different stages (and two order of magnitude difference between them), stage 1 does not rely on chamfer distance whereas stage 2 does not rely on correspondence filtering. Does it not imply the network absolutely needs geodesic based filtering in stage 1 ? and that chamfer distance based on feature similarity( & NN) is not effective/needed in Stage 1? this is speculative since submission/rebuttal does not show the relative strength of different loss terms to gauge individual contribution within these individual stages. \ - alignment and matching: robustness is different than requirement of pre-aligned shapes. the submission should replace [30] with [38] (first work to show results with this requirement) or change the text accordingly. - Geometric meaningful embeddings: please clarify the same in Introduction.
Responses to further comments
Thank you for the detailed reply. Below we address your new comments: **Inconsistencies in algorithm descriptions in the main submission and Supp. Mat.:** Due to the lack of space, we defer some algorithmic details to the more complete version in Supp. Mat. The algorithms are essentially the same. First, the 'converged' condition in Line 10, 11 of Alg. 1 in the main submission is described in details in Line 12-16 of the version in the Supp. Mat., which is the only stopping criteria in our algorithm. Second, Line 183 in the main submission is not an early stopping criteria. It says that we update point-wise correspondences between deforming template and the target with the learned embedding every 100 iterations during stage-I. **Ablation study on utlizing DFM output as initialization to NDP, AMM:** According to your suggestion, we have performed ablation studies to compare our pipeline with NDP and AMM based on the same initial correspondences. In particular, we train the DFM on the training set of SCAPE_r and use the SCAPE template (see Fig. 3 in the main submission) in all experiments. In the following table, we report the average errors of the initial maps computed by DFM, and that of output maps of Ours, NDP, AMM, which are all based on the same initial maps. It is evident that, across three different test sets, our method consistently improves the inital maps (at least 37% error reduction), while NDP and AMM can even lead to deteriorated maps than the initial input. These results shows the advantage of our proposed pipeline, especially the registration part. Test set | Ini. | Ours | NDP | AMM SCAPE_r | 5.5 | **2.6(-52%)** | 5.4(-2%) | 11.4(+107%) SHREC19_r | 8.1 | **5.1(-37%)** | 11.4(+40%) | 10.7(+32%) SHREC07-H | 11.5 | **5.9(-48%)** | 8.9(-22%) | 8.8(-23%) **Dissusion on hyper-parameters:** Thank you for the suggestions on clarifying the roles and choices of hyper-parameters. We would be happy to include a detailed discussion on it in the future revision. **How exactly the hyper-parameters (e.g., threshold in correspondence filtering, $\lambda_{cd}$, $\lambda_{corr}$, $\lambda_{arap}$) are chosen:** As we mentioned in the rebuttal, performing shape registration allows to evaluate the resulting registration/maps without ground-truth correspondences. In particular, we perform grid search with respect to the weights used in the final optimization to seek for the combination that leads to the the best registration results (quantitatively in terms of Chamfer distance and qualitatively by visual inspection). We seek for the threhold in correspondence filtering in a similar way, and it is set to 0.01 across all experiments. We emphasize again that (a) The above hyper-parameter selection is quite loose and may be *suboptimal*, since only two pairs of training shapes (in fact, the template source shape is the same) are involved; (b) We use the same hyper-parameters for *all* of our experiments. **The roles of $\lambda_{cd}$, $\lambda_{corr}$ at different stages:** Essentially, the registration procedures in stage-I and -II are respectively guided by proximities in the high-dimensional embedded space and the ambient space. On the other hand, the loss $E_{corr}$ (measuring how the deformation agrees with the maps induced by the learned embeddings) and $E_{cd}$ (measuring the discrepency between the deformed template and the target in R^3) quantify them respectively. Therefore, it is natural to put more weight on the resepctive loss in the corresponding stage. The exact ratio, 100:1 and 1:100, is determined as described above. In fact, we have not tried to turn off either $E_{corr}$ or $E_{cd}$ in any stage before. This is motivated by our initial motivation -- to enforce the deformed shape to be close to the target in *both* the high-dimensional embedded space and the ambient space. According to your suggestion, we have also performed ablation study on setting $\lambda_{cd}$ and$\lambda_{corr}$ to be zero independently in the two stages. The following table reports the scores. We find that most of the time, turning off either leads to worse results, especially when the test shapes are heterongenous (see, e.g., SHREC07-H). Test set | Ours | $\lambda_{cd}$ = 0 in Stage-I | $\lambda_{corr}$ = 0 in Stage-II SCAPE_r | 2.6 | **2.5(-4%)** | 2.7(+4%) SHREC19_r | **5.1** | 5.5(+8%) | 5.6(+10%) SHREC07-H | **5.9** | 6.9(+17%) | 6.8(+15%) **Revising arguments on alignment & matching and geometrically meaningful embeddings:** We would be happy to revise these in the future version.
Final remarks based on reviews, rebuttal and follow up
Thank you for your answers. The rebuttal has clarified my concerns about unsupported claims in the submission with further experiments. However, many of the new claims on contributions/novelty in rebuttal are handwavy , unconvincing or presented without justifications (details below) and therefore my doubt remains on the presentation of future/revised versions (unlike journal, a reviewer will not see the revised version?) I revise my my final rating but keep it slightly negative, same as reviewer xFDL. \\\\ - *Key conceptual idea contextualized wrt DPC*: As mentioned in the initial submission (Line 52), key is to enforce similarity in both ambient space as well as learned feature space. Closest work on matching unstructured point cloud is DPC with cross reconstruction/self reconstruction modules. While the cross reconstruction module is very similar conceptually in enforcing similarity via latent space, I agree self reconstruction is different than this work. In my opinion, submission should stick to this (Shape registration Stage 1 and 2 ) while revising future versions instead of the following: - unconvincing/handwavy claims on contribution in rebuttal: \\\\ a) *combining learning based and optimization-based shape matching*: the two terms are not exclusive : every single follow up paper on DFM is a learning based paper with underlying optimization (autodiff) of DL frameworks. The fact that this one uses an explicit ADMM based optimization is not a contribution. b) *first to address the problem of unstructured point cloud matching via a hybrid approach*: why do we need a hybrid approach when there are already existing frameworks for unstructured point cloud matching that does not require a mesh/laplacian. emphasis on 'first to do hybrid matching' is not convincing. the submission can argue that it is less rigid in requirement than some purely mesh based methods but it sill relies on DiffusionNet to start with (underlying bottleneck). c) *generative modelling*, *lifting* alignment requirement etc: Such terms should be avoided since generative models in ML is a well defined term and this submission did not formalize a generative model in submission. \\\\ - Back to Shape registration Stage 1 and 2 and supporting its claim in future version: \\\\ a) Contextualizing the key conceptual idea and citing DPC (as mentioned above). b) Proper referencing and a motivation for alignment and matching. c) Including rebuttal experiments to support claims made in line 54-57. d) clarifying Geometric meaningful embeddings claim. e) supporting Shape registration Stage 1 and 2 with an ablation study as done above. f) clearly mention this approach introduces 8 hyperparameters (6 in stage 1/2+ corr. filtering threshold + early stoping count) and how critical they are.to overall performance and model sensitivity to them. In my view, above modifications are must to support the key idea and resulting claims. \\\\ - nice to have that will make submission stronger: --why do we need DFM prior with ARAP loss: rebuttal mentions an empirical answer. a more convincing could be show if their output somehow are uncorrelated so combining them makes atleast a statistical sense if not geometric one. -- generalisation of this shape registration to other methods: reviewers understands it requires effort but it only makes the approach more effective if shape registration module/DFM prior can be taken and plugged in other methods and improves their performance as well. -- rename correspondence filtering to Geodesic filtering. -- similarly rename feature extractor to something more specific that conveys student-teacher feature extractor
Clarification on novelty.
We sincerely appreciate your detailed comments and suggestions. We are glad that the rebuttal has clarified some of your concerns and would be happy to incorporate the revisions and experimental results in the future version. Below we mainly address your concerns about novelty and contributions (presented in our rebuttal): 1. Our pipeline is based on two series of works on estimating correspondences between non-rigid shapes: 1) shape matching via learning a high-dimensional embedding (DFM-like works [8, 22, 30, 38], DPC); 2) non-rigid shape registration (NDP, AMM). To the best of our knowledge, the two lines have long been developed **in parallel**. We have discussed the respective limitations: 1) high-dim embeddings are obtained in an uninterpretable way, which is often *hard to analyze* and can *suffer from poor generalization*; 2) non-rigid shape registration methods are typically designed for non-rigidly aligning consecutive scans of the same instance, therefore they can fail when the shapes are of *large deformations or of distinctive styles*. 2. Following (1), our **novel hybrid approach** combines these two lines of work. Rather than a random combination, we put them in a unified framework to compensate for each other’s limitations -- The learned embeddings provide more suitable proximities than the extrinsic coordinates in matching shapes, while by performing explicit non-rigid alignment we can introduce regularizations such as ARAP so that the alignment is more reasonable and plausible, which is easier to analyze and more reliable in matching/registrating unseen shapes. 3. At this stage, it may seem trivial to put them together. We emphasize that we aim to deal with **unstructured point clouds**, which is the key challenge. Overall, the embeddings have to be computed on raw point clouds, ideally without mesh/graph construction (for both efficiency and robustness regarding topological noise). To this end, our novel teacher-student learning scheme is general, flexible, and effective. 4. Finally, we highlight that the meshes and DiffusionNet are only needed during training the teacher network. Remarkably, with as few as 80 training meshes from standard datasets, our pipeline in the end achieves state-of-the-art results on matching raw point clouds which can be *non-isometric* (DT4D-H), *heterongeous* (SHREC07-H), or *with topological noise* (TOPKIDS).
Thanks for the rebuttal
My rating remains.
Thank you for the reply and the positive feedback to our work.
Reviewer Feedback
Thanks for the feedback! That helps me understand your work better. As pointed by all the reviewers, the major flaw of this paper is the limited novelty. I think this is the most important part in evaluating a paper, and based on that, I will keep my initial rating, although it is a little bit negative. Moreover, I hope the authors could carefully revise their paper afterwards, as there are some places in the current version unclear and misleading. Cheers!
Thank you for the reply, and we are glad that our rebuttal helps you to understand our framework better. We would be happy to revise our paper for a better and clearer presentation. **Novelty discussion:** We have actually devoted length to the discussion of our main contributions and novelty in the Author Rebuttal (see at the top of the page) and would be more than happy to address your *further comments/questions* regarding it. Our key novelties are two-fold, as re-iterated below for ease of discussion: 1. To the best of our knowledge, we are the first to address the problem of matching **unstructured** point clouds undergoing significant deformation via a hybrid approach. The relevant prior works depend on either heavily mesh structure (NeuroMorph, Deep Shells), or correspondence supervision (TransMatch). In contrast, our method can deform a template to a raw point cloud in a *direct and unsupervised* manner. 2. We propose a novel self-supervised learning scheme to infer intrinsic-aware features from unstructured point clouds effectively and efficiently. Compared with the relevant and concurrent work [8], our design is more general (not adhered to DiffusionNet architecture, not based on graph Laplacian construction) and flexible (can be extended to more powerful and/or more tailored-for backbones).
I thank the authors for the clarifications. I will keep my initial rating.
Decision
Accept (poster)