Jigsaw: Learning to Assemble Multiple Fractured Objects

Summary

The authors address the problem of reassembling fractured 3D objects from its pieces, where each piece is represented as a point cloud. The main challenge lies in the fact that one has to simultaneously solve a segmentation problem (which part of an object is fracture surface and which part is the original surface) and a multi-matching problem. The authors propose a learning-based approach that nicely take into account these different aspects. Their approach is supervised, which sounds reasonable given that it is relatively straightforward to simulate object fractures from complete objects.

Strengths

+ The authors address a difficult, relevant and relatively novel problem and set the new state of the art + The proposed solution is a neat combination of learning and traditional optimisation techniques + All pieces are matched simultaneously which really is key to address this difficult problem + The idea of using a primal-dual feature descriptor pair to account for the complementary geometric properties of features is novel and useful

Weaknesses

- Although the presented results look appealing, they only show a small subset of all results. Given the relatively large angular errors (up to 52.4 degrees), it is unclear whether results are cherry-picked or representative. Given the overall improvement over previous methods I do not see this as critical for acceptance, yet, it is important to understand this. - There a few things that leave some room for clarification (see below).

Questions

1) The intro also discusses furniture assembly - however, this is quite a different problem since furniture does not have fractures. I suggest to smoothen this. 2) In related work it is mentioned that most graph matching methods focus on pairwise matchings. There are some recent multi-matching methods which should be discussed (see e.g. Universe Points Representation Learning for Partial Multi-Graph Matching, AAAI'23 and references therein). 3) Please make the font and visuals in all figures larger (ideally the fonts match the font size of the caption). 4) How many V100 GPUs are in one machine? Please comment on the training and inference time. 5) I think there is some ambiguity in the notation around lines 161-165: $N$ is already introduced before as the number of object pieces; also, the index $i$ is used both for the object piece index, and as point index. 6) How do you ensure that the primal-dual feature pair is really complementary/consistent? Is this enforced by construction or does the learning take care of that? Also, the formulation "the surface looks different from a different viewpoint" is not ideal, since this suggests there is some camera (with projection) involved (at least that was my first thought). I suggest to rephrase this. 7) Is matrix $A$ in eq. (6) positive semidefinite? It looks a bit like the Mahalanobis distance in a Gaussian kernel, but then we would also need a negative sign. Please comment on this. 8) Eqn. (9) misses that (R,t) are elements of SE(3). Similarly, the solution below finds (R,t) in E(3) - one needs to ensure the determinant of the rotation is +1. 9) How do you ensure that the blocks of the matrix X in eqn. (13) are consistent? In 3.4 it is explained that pairwise transformations are computed first (representing blocks of X), and then RANSAC is applied. Does this guarantee global consistency? Do you apply some variant of permutation synchronisation, or is consistency only guaranteed for the rigid-body transformations via the Shonan-averaging? 10) What is the scale/unit of the point clouds? Absolute values of transformation error are not very meaningful.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are adequately addressed.

Summary

This paper proposes a novel framework for the task of 3D fracture assembly. The proposed approach leverages hierarchical features of global and local geometry to match and align the fracture surfaces. Experiments are conducted on the Breaking Bad dataset to verify the effectiveness of the proposed method.

Strengths

1. This paper is well-written and easy to follow;

Weaknesses

1. I am not the guy from the same area and therefore I cannot judge the novelty of the proposed mehtod well. However, from my experiences in my research field, this task is quite close to pairwise and multi-frame point cloud registration, and the major difference is the utilization of surface segmentation and the significantly lower overlap ratio between parts. Based on that, some techniques, especially the geometric learning and the matching, are well-explored and widely-adopted in the aforementioned registration tasks, and therefore I consider the proposed solution trivial. 2. For the experiments, I think there are some problems: 2.1 The experiments are conducted on a single dataset, and I am not sure whether it is a synthetic or real one. To this end, I think it is necessary to also evaluate the proposed method on a second dataset. If the leveraged dataset is synthetic, it would be better to include experiments on real data; 2.2 In Tab. 1, the latest baselines are from 2020 which is 3 years ago. I have no idea why some latest ones are not included. Does it mean that there have been no work in this topic for 3 years? I would like to see the comparisons to some more recent works during rebuttal; 2.3 In Tab.1, the only point cloud registration baseline is Predator which is a work in 2021. Some more recent works, for example, CoFiNet (NeurIPS 2021), GeoTransformer (CVPR 2022), Lepard (CVPR 2022), RegTr (CVPR 2022) should be considered as the baselines from the field of point cloud registration (GeoTransformer is the state-of-the-art). 2.4 As the task is similar to multi-frame point cloud registration, I also think it reasonable and necessary to include baselines from that area, like (1) and (2). 3. Fig. 1 carries overly much information and is hard to focus on each specific part. I would suggest the authors modifying it to make it clear. Also, in Related Work (line 90), [30] is not based on the overlap prediction. --------------------------------------------------------------------------------------------------------- (1) Gojcic et al. Learning Multiview 3D Point Cloud Registration, CVPR 2020 (2) Yew et al. Learning Iterative Robust Transformation Synchronization, 3DV 2021

Questions

1. As in the cross-attention, all the points from all the parts are leveraged for the attention computation. Therefore, the efficiency heavily relies on the size of point clouds. So how many points are included in each part and the whole shape? Is the proposed method still applicable if the number of points raised to >10k? 2. What is the main difference between the assembly task and the task of pairwise/multi-frame point cloud registration? 3. What is the overlap ratio (probably in average) between different parts?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have been discussed in the main paper.

Summary

The authors present a learning framework for assembling physically broken 3D objects with point cloud observations. There are three components in the framework, surface segmentation to identify the fracture points, multi-parts matching to find correspondences among the fracture points, and the global alignment to recover the global poses of the pieces. The main contributions are the joint learning of segmentation and multi-parts matching and the design of the multi-parts matching including the primal-dual descriptor and the losses. The contributions are validated by superior performance on the Breaking Bad dataset compared with other methods.

Strengths

The task is new with just few methods already developed. This work may serve as a baseline for future work in this line. The overall framework is clearly clarified and organised. The adding of self-attention and cross-attention is well justified and illustrated for the front-end feature extractor. The primal-dual descriptor is introduced in the multi-part matching part, which is new for this task to my knowledge. A further analysis of the primal-dual descriptor is also provided in the appendix to provide better understanding of the effect. The experiments and results are also solid.

Weaknesses

The absence of the statistics of the number of pieces in the dataset may lead to unfair comparison against the baseline methods as points are sampled at different level as shown in Table 3 in the appendix. A further analysis of the multi-parts matching w.r.t. the number of pieces could better show the effect of multi-part matching against pairwise matching. The notations are sometimes a bit confused, especially for the tilde and hat. For example, the hat_P_i at line 258 seems to be not defined. The experimental settings are not clear for all the involved methods. A table to list and compare the conditions and data/label requirement for each method would be helpful. The introduction of the rigidity loss is not well justified and analyzed, especially it is only applied to the model training after 200 epochs. Texts in the figures are too small to recognize.

Questions

Does the generation of the segmentation labels require additional knowledge about the data compared to other methods? Although I don't think so, a clarification from the authors would be helpful. Is only one precise match at line 193 an assumption? The assumption of only one precise match may not be satisfied as the points sampled on neighboring pieces may be quite far and no matching can be done simply based on point clouds. Addressing the weaknesses listed above could change my opinion.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors deals with the accuracy drop of pose estimation when the number of fractured pieces increases.

Summary

This paper proposes Jigsaw, a novel framework to assemble 3D fragments of a shape to complete the whole 3D shape. Jigsaw first segments the fracture surface, since only the points on the fracture surface are involved in the physical assemble between fragments. Jigsaw then performs multi-part matching to find correspondences among the fracture surface points, which is finally used to recover the global pose of each fragment via global alignment. Jigsaw exhibits state-of-the-art performance on the Breaking Bad dataset. The authors propose Jigsaw to be the first learning-based method to assemble 3D fractures over **multiple** pieces.

Strengths

* First learning based method to tackle the task of multi-part assembly of 3D fragments. * Well-motivated, novel learning pipeline to facilitate multi-part assembly of 3D fragments. * Strong performances on the Breaking Bad dataset. Considering that a two-fragment assembly problem can also be seen as a zero-overlap (or surface-overlap) point cloud registration, the baseline methods in the table are also appropriate.

Weaknesses

* Lacks self-containedness. While it is understandable that Shonan-averaging was used as the off-the-shelf global alignment method, but the paper does not go on to explain how this global alignment functions at the minimum, nor how the global pose values from Shonan-averaging is used for evaluation against the given ground-truth global pose values. * Jigsaw samples a fixed number of points across all the fragments of a single shape. In that case, Jigsaw is expected to fail given a particularly small fragment, from which a very small number of fragments would be sampled. This may result in not a single point on the fracture surface being sampled, and this lacks emphasis in the manuscript. * The manuscript lacks explanation on how the primal-dual descriptor works. The authors state that their 'system learns surface features that capture the characteristics of a local surface from both directions' - which lacks the evidence the substantiate this claim. Is it the **aim** of primal-dual descriptors to function as mentioned, or is there a theoretical evidence to prove that the primal-dual descriptors actually captures such characteristics? * While the idea of surface segmentation to narrow down the points to consider is viable, it is expected that the fracture surface is rather easy to segment due to its irregularity in comparison to other parts of the shape. How would the proposed method perform if the fracture is not irregular, but of regular cuts (e.g., straight lines, smooth curves) such as in Neural Shape Mating? [1] [1] YC Chen et al., Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors, CVPR 2022

Questions

Please refer to the above weaknesses section. It would be appreciated if the authors could include additional details to improve the self-containedness of the paper, and to clear any potential misunderstandings.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The limitations of Jigsaw have been discussed by the authors.

Summary

This work provides a Joint Learning of Segmentation and Alignment Framework(Jigsaw) framework for 3D fracture assembly task. Experiments are evaluted on Breaking Bad dataset and show better performance than other existing methods.

Strengths

1. The proposed approach applies a joint learning framework specific for multi-part fracture assembly, which appends an attention-based network module to capture local geometric information. To further capture viewpoint-dependent features for surface matching, the proposed method applies the primal-dual descriptor to achieve better matching. 2. The authors incorporate fracture point segmentation to capture intrinsic features and introduce a multi-part matching approach for internal automatic piece positioning, with global pose alignment capability. 3. The proposed method shows better results than baseline methods evaluated on the multi-part assembly Breaking Bad dataset. The ablation analysis also validates the efficacy of the proposed network structure.

Weaknesses

1. The motivation of using primal-dual descriptor seems not sufficient, it would be better to append a more description of it. 2. The ablation study part is considered a bit insufficient. e.g. The authors already mention matching loss and rigidity loss. The related ablation analysis about the impact of segmentation loss, matching loss and rigidity loss is missing.

Questions

The running time analysis of the proposed method or the comparison to other methods is missing. How about this metric?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

In the experiments, evaluation on extra dataset rather than the Breaking Bad dataset is much desired.

Reviewer Feedback

Dear authors, Thanks for your feedback and I really appreciate your effort. However, I still have questions on some specific aspects: 1. Regarding Q 2.3, GeoTransformer is a strong basline that has much better performance compared with Predator for the point cloud registration task, while the authors claimed that "its performance aligned with that of Predator". Moreover, the author also said the training of GeoTransformer on their target dataset is unaffordable. Then, how was the conclusion "its performance aligned with that of Predator" drawn? 2. Why is the training of GeoTransformer on the target dataset way slower than that on 3DMatch/3DLoMatch (GeoTransformer's target benchmarks)? From my understanding, your data is synthetic and with fewer points. Does it have more training cases? If yes, then comparing the proposed method with GeoTransformer on 3DMatch/3DLoMatch should be appliable. Considering the similarity of the two tasks, I think this experiment should be included. 3. Regarding Q 2.4, I think from your explanation, the assembly task is an easier version of the multi-frame registration. Therefore it would make sense to apply some of the state-of-the-art multi-frame registration methods to your task as additional baselines. 4. Considering that only a synthetic dataset is used in this paper and the also the similarity between the assembly task and point cloud registration, I would like to see comparing the proposed methods with the registration baselines, e.g., GeoTransformer, on 3DMatch/3DLoMatch to further validate it. If GeoTransformer requires 15days to be trained on your data, I think this experiment is suitable to conduct and affordable to your GPU. I have carefully read other reviewers' comments and found I am the only guy that leans towards the negative side. However, after reading authors' feedback, let me be honest, I have a feeling like the authors tried to find excuses to avoid the comparison with the registration baselines. Also, there are some statements from the authors that I still disagree. According to the previous comments, I will insist my evaluation on this paper. If I was wrong, please correct me. I will reconsider my score afterwards. Best, Reviewer jmgU