Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation

Summary

The paper describes a method to simultaneously segment rigid parts and estimate part-level transformations for multi-body point cloud inputs. Given the input point cloud, equivariant features are first extracted and aggregated to invariant features for part segmentation. To estimate the transformations between two point clouds, an equivariant motion estimation head is applied to pairs of point clouds to obtain SE(3) motions. The entire model is trained in an unsupervised fashion using the consensus between 3D scene flow, motion segmentation, and transformations. Experiments show that the model is generalizable in two aspects: open-set motion and category-agnostic part segmentation. The authors test the model on SAPIEN, OGC, and KITTI datasets, and show superior performance in comparison to past baselines, with much better accuracy and much fewer FLOPs, even surpassing the supervised method.

Strengths

1. The identification of two generalization problems, i.e. invariant segmentation and equivariant motion estimation, is insightful. The use of equivariant backbones and their invariant counterpart is well-suited for such a problem. 2. The use of initial scene flow to bootstrap (or cold-start) the learning process and to get rid of local minima is novel. By utilizing the consistency between the output attributes which is inherent in the multi-body registration task, the method could be learned in an unsupervised manner. 3. Experiments are extensive and the results are compelling. The method is tested on four different datasets ranging from articulated objects to outdoor driving scenes, justifying the wide applicability. In the meanwhile, the fact that the algorithm is able to achieve a much higher accuracy with a much lower cost is impressive and set up a strong waypoint for future works.

Weaknesses

1. The unsupervised training strategy lacks clarity. What is the loss function? Is it eq.(8) plus eq.(7) in the supplementary with some balancing factor? This should be clarified in the main paper. In Line 213, the authors mention the weighted-Kabsch algorithm, what is the 'weight' here? 2. The generalization of the segmentation head is questionable -- this is my most important doubt about the paper. Although the generalization to category-agnostic part segmentations is demonstrated through experiments, the reason why an equivariant network backbone could achieve that is not explained. What adds to the confusion is that the method is trained in an unsupervised manner, which means that no canonical segmentation labels are defined. Given a novel multi-body shape that is not seen from the training set (e.g. a cabinet with four drawers), how could the model give a unique index to each of the drawers in a consistent fashion (i.e. the drawer index stays unchanged across multiple articulation states)? What is the maximum number of parts supported in the algorithm? Is it predefined or optimized/given for each test sample? The doubt also links to the experiment section -- In Fig.4 (a,b), the gap between training and validation is not shown, and it's unclear whether Fig.4 (c) is based on ground truth or predicted segmentation. 3. The experiments are not explained very clearly. In Sec4.2, why 'point-level invariance of segmentation' helps 'open-set motion' generalization, and why 'part-level equivariant of motion' helps 'category-agnostic part' generalization? Shouldn't they be swapped? It is preferable to move Sec4.4 ahead of Sec4.3. 4. More visualizations and challenging cases should be shown. There are only 3 visualizations provided in the paper+supplementary, on SAPIEN datasets. These visualizations only contain segmentation labels with two parts. To make the results more convincing, the author should provide more visualizations on all the datasets they've tested on, with transformations shown, and on more challenging cases, e.g., large scenes with >4 rigid parts.

Questions

1. Does the current training scheme leverage the cycle consistency among multiple inputs? If not how could this be possibly incorporated into the algorithm? 2. How does the method generalize to different point densities/distributions? What is the maximum number of points could the method process? 3. Some typos: - Line 144, 'There are mainly comprised of...' - Line 237, 'even given the global since...'

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are discussed in the last paragraph of the main paper.

Summary

This paper studies the problem of rigid part segmentation and motion segmentation. The proposed network first uses a global SE(3) equivariant backbone to extract point features, then a part segmentation head will predict invariant zero-order segmentation and a motion head will predict one-order transformations. The network can be trained on pairs of point clouds before and after deforming with noisy scene flow. The multibody equivariance is approximate because of 1.) descretization in EPN, 2.) the global backbone can’t handle deformation.

Strengths

- The paper exploits strong geometry inductive bias — Equivariance, into the segmentation problem, which should be highlighted and emphasized more to the community. - The system is tested across different settings from articulated objects to outdoor scenes and the experiments are convincing.

Weaknesses

- The main weakness of this paper is that the “equivariance” is not exact, i.e. because there are pooling and information aggregation in the point cloud feature backbone when the target is deformed (not necessarily non-rigid, but rigid transformed, articulated, this is a word from equivariant learning theory), the overall point cloud changes in such cases, but the information can be propagated across rigid parts so the features of each point do not transform purely following its parts transformation. This is because without knowing the mask, the group action of SE(3)xSE(3)x…xSE(3) on the point cloud can not be defined. - Another weakness of this paper is that the system should take in at least a pair of point as input during inference (as far as the reviewer currently understand, please correct him if this is wrong). So the task is relatively easy because by comparing two point clouds (even by brutal-force enumerating and classical geometric algorithms ) the segmentation of rigid moved parts may be trivially found in most cases. And the system seems only to focus on moving parts rather than semantic or instance parts that may not be moved. - The reviewer is also curious why the author chooses to use EPN as the backbone, is there a specific reason for such a design? Note EPN is only approximate equivariance due to the discretization. In other words, will such a framework still work for other backbones like TFN or VNN?

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is explicitly highlighted in the paper, but the reviewer hasn't found the social impact claim.

Summary

This paper proposes a unified framework to simultaneously handle rigid segmentation and motion estimation. Specifically, an probabilistic SE(3)-equivariant network is presented to obtain transformation-invariant features, which is followed by two heads for rigid segmentation and motion estimation. Due to the interdependent nature between the above two tasks, the authors utilize a unified training strategy to jointly optimize the two heads. Extensive experiments on four datasets show that this method can yield good performance with low computational complexity.

Strengths

-Significance and originality Unsupervised rigid segmentation and motion estimation are challenging. Taking the generalization requirement into account, the authors introduce a probabilistic part-level SE(3)-equivariant feature encoder to capture transformation-invariant features, which is different from the global SE(3) equivariance. And it is simple but makes sense. A unified training strategy to jointly optimize the outputs of two heads in an online fashion is well motivated. Although many other tasks also employ joint optimization, this is the first one for this task. -Clarity The paper contains enough details and definitions of the proposed contributions. -Experiment Quantitative results show a clear improvement compared to previous methods.

Weaknesses

-Time cost of the proposed methods and other methods. -The author should systematically analyze the robustness on initial scene flow. Does the performance drop a lot with poor initial scene flow results? (add some noise or with insufficiently trained scene flow results). -This paper relies on rigidity constraint and works well on finite datasets. The SAPIEN and OGC-DR datasets are without background and well-separated. The authors can conduct some experiments on large outdoor scenes, which are more complicated.

Questions

please see the weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

-In related works, deep learning on dynamic point clouds is too highly summarized. The authors can present more details. -The authors can visualize more segmentation results and intermidiate results so that the proposed method can be better understood.

Summary

This paper proposes a part-level SE(3)-equivariant framework for multi-body rigid motion modeling. The two heads that take in the SE(3) features, segmentation and motion estimation, are carefully crafted based on their inherit invariant and equivariant characteristics. The relationship of scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. It achieves state-of-the-art in multiple datasets while significantly reducing the parameters and training computations required.

Strengths

The structure of this paper is very clear, providing a comprehensive explanation of the research problem, motivation, background, principles, methods, and experiments. The text and figures have been meticulously polished. This work effectively utilizes SE(3) equivariance in rigid segmentation and multi-body motion estimation, and is outstanding in terms of experimental results and model size.

Weaknesses

Rigid segmentation and multi-body motion estimation are intrinsically coupled. I wonder if this coupling might also have negative effects on the model, for instance, inaccurate segmentation leading to poorer scene flow estimation results.

Questions

Please refer to the weaknesses section, refute views or answer questions or improve the paper.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations has been discussed.

Summary

Modeling multi-body rigid movements involves - rigid segmentation which is category-agnostic and motion estimation which is open-set. This work proposes a part-level SE(3) equivariant network that takes in point-clouds corresponding to video frames and can effectively estimate category-agnostic open-set motion.

Strengths

The work leverages the point level feature equivariance from existing works and uses it to obtain invariant representations for the rigid segmentation and then further uses it to estimate motion using an equivariant motion head. The approach is able to provide state of the art performance with much lesser flops. It makes note of the interdependence between the tasks of rigid segmentation and motion estimation and uses that interdependence for unsupervised training.

Weaknesses

The paper writing could be better so that the high-level idea is more easily understood. The contributions of the work could be summarized in fewer words in the introduction section making the work much easier to follow. SE(3) has been used multiple times in the paper including the title but it has not been described in the paper anywhere.

Questions

Please address the issues mentioned in the weakness section

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have discussed limitations of their proposed approach.

Summary

This paper presents a method that estimates the part segmentation, and the rigid motions of the parts, for an articulated object (with multiple rigid parts), given as input a sequence of point clouds of that object. The method works by first computing SE3-equivariant features for the points, and then processing with an SE3-invariant head for segmentation estimation (pooling across rotation groups), and then processing with an SE3-equivariant head for rigid motion estimation (pooling across points within estimated segments). The method also initializes a scene flow estimate from a pre-trained off-the-shelf model, and then iteratively updates this to match the estimated rigid motion. Finally, the model uses the weighted-Kabsch algorithm to estimate transformations for the parts according to the flow and segmentation, which yields optimization targets that encourage the estimates to be self-consistent.

Strengths

This paper makes a well-motivated case for using a combination of equivariant and invariant features to jointly solve rigid part segmentation and motion estimation. The proposed network has fewer parameters and runs faster than the baselines, and also achieves higher accuracy, in both the supervised setting and the unsupervised setting. Jointly optimizing the motion, segmentation, and scene flow, makes sense.

Weaknesses

I found this paper very difficult to follow. I am quite sure that the content in the main text is not a complete description of a reproducible method. Nearly every section defers to the supplementary material for additional information. In my reading of the guidelines, the supplementary material is for supporting the paper, not for describing key details of the method. The method in the paper should make sense on its own. 
Because my main complaint is on clarity, I'll put a lot of questions, but please interpret some of the questions as reflecting weaknesses, because these are things that should be clear in the work.

Questions

Do the two point clouds correspond? I suppose not, because then motion can be computed just from taking a difference of the positions. The problem statement in 3.2 suggests a sequence, but later (e.g., in 3.3) it seems like the sequence length might just be 2. Is the sequence length ever not 2? The w(p^i_k,g_j) used in equation 2 are described as "derived through a 1x1 convolution". What does this mean exactly? The paper talks about "fusing such invariant representations u^i_{k(1)},... from h layers" -- how is this fusing done exactly? The paper says "the segmentation head outputs a probabilistic prediction M^k of the rigid mask" -- what makes it probabilistic exactly? This idea of "probabilistic" masks is repeated many times through the paper, and at some point there are also "probabilistic SE3 features" (actually only in Figure 3). In my experience, the word "probabilistic" is usually used to indicate something non-deterministic, but in this case it seems like it might actually be deterministic. What exactly is meant by a probabilistic mask or probabilistic feature? The paper says "the motion head is supposed to *handle these uncertain category-agnostic parts*" (italics in the paper). Why are those words in italics? What is meant by "handle" here? It seems like a special usage of the word. Equation 3 looks like it might just be depicting an element-wise product. Is this correct? It would be great to write this more efficiently. The paper says "the specific category labels can be agnostic to the model". What does it mean for category labels to be agnostic to a model? The paper says "Based on the correlated feature Ckl, the motion head estimates rotation R^s_{kl} and translation t^s_{kl} of each rigid part s." This is a critical part of the model. How are the correlations used exactly, what is the predictor, how are the rotation and translation outputs represented, and how are the outputs supervised? Under equation 6, it says "In this manner, improved scene flow is capable of providing enhanced supervision to learn segmentation masks and motion estimates." I don't see how this follows from the equation. The equation only produces an updated scene flow, using a convex combination of the previous estimate and the rigid estimate. I see no "supervision" (or "enhanced supervision") applied to segmentation masks here. In the final paragraph of the method, it says "the motion head is optimized by the rotation component of T^s_{kl}, and estimates corresponding translation by minimizing our probabilistic part-level distance." What does it mean to optimize the motion head by rotation? What does it mean to estimate translation by minimizing "probabilistic part-level distance"? It seems like these are critical details in the procedure.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Looks OK

Response to Reviewer sjc7 (Q9-Q11)

**Q9:** The paper says "Based on the correlated feature $C_{kl}$, the motion head estimates rotation $R^s_{kl}$ and translation $t^s_{kl}$ of each rigid part $s$." This is a critical part of the model. How are the correlations used exactly, what is the predictor, how are the rotation and translation outputs represented, and how are the outputs supervised? **A9:** Originally, due to the page limitation and its close similarity following the approach of EPN [9], we included the details in the supplementary material (Lines 38-46). We realize that this may cause slight ambiguity and will briefly describe this method towards the end of Line 183 in the main paper. The summary is as follows. The predictor of rotation is based on a 1×1 convolution. The anchor $g_{kl}^s$ is chosen from $\mathcal{G}$ by minimizing the registration error and then optimizing the residual $r_{kl}^{s}$. The rotation is computed as $\hat{\mathbf{R}}^s_{kl}=g_{kl}^{s}r_{kl}^{s}$. The translation is derived from the minimal weighted distance between the transformed frame of $P_k$ and the origin $P_l$, where $\hat{\mathbf{t}}^s_{kl}=argmin_{\mathbf{t}} d(\hat{\mathbf{R}}^s_{kl}P_k + \mathbf{t}, P_l)$ and $d$ is the chamfer loss weighted by the mask predictions. **Q10:** Under equation 6, it says "In this manner, improved scene flow is capable of providing enhanced supervision to learn segmentation masks and motion estimates." I don't see how this follows from the equation. The equation only produces an updated scene flow, using a convex combination of the previous estimate and the rigid estimate. I see no "supervision" (or "enhanced supervision") applied to segmentation masks here. **A10:** Thank you for bringing this to our attention. We apologize for any confusion caused by the sentence in question. Instead of only continuing from the previous text (Equation 6), this sentence mainly serves as a transition to the next two paragraphs (Lines 201-216), which subsequently describe how the improved scene flow supervises the segmentation (Lines 201-211) and motion estimation (Lines 212-216). **Q11:** In the final paragraph of the method, it says "the motion head is optimized by the rotation component of $T^s_{kl}$, and estimates corresponding translation by minimizing our probabilistic part-level distance." What does it mean to optimize the motion head by rotation? What does it mean to estimate translation by minimizing "probabilistic part-level distance"? It seems like these are critical details in the procedure. **A11:** Based on the original supplementary material (Lines 54-60), we create a summary that will be added to the revised paper (we will make our contributions in Introduction more concise to compensate for the space needed). The summary is as follows. Based on the prediction of segmentation masks, the part-level rigid transformation can be computed by the weighted-Kabsch algorithm [32, 27]. By minimizing a mean SO(3) loss (described in the original paper of EPN [9]) from the rotation component of this rigid transformation, the estimated rotation ($\hat{\mathbf{R}}^s_{kl}$ in **A9** ) is optimized in a differentiable manner. The corresponding translation is then estimated by minimizing the part-level chamfer distance (through the formula of $\hat{\mathbf{t}}^s_{kl}$ described in **A9**), which aims to find the translation that best aligns the two point clouds. We will make sure to revise our paper to ensure that the method is fully described and further understandable. We appreciate your questions and will address them in detail to clarify any weaknesses or misunderstandings in the revised manuscript. Thank you again for bringing these issues to our attention.

Response to Reviewer ZttT (Q3)

**Q3:** This paper relies on rigidity constraint and works well on finite datasets. The SAPIEN and OGC-DR datasets are without background and well-separated. The authors can conduct some experiments on large outdoor scenes, which are more complicated. **A3:** Thank you for this insightful suggestion! Following the same evaluation protocol of OGC [61], we further evaluate the generalizability of our model from KITTI-SF to two larger outdoor datasets, i.e., KITTI-Det [1*] and SemanticKITTI [2*], and compare the performance with the results reported in OGC: (1) KITTI-Det | | AP$\uparrow$ | PQ$\uparrow$ | F1$\uparrow$ | Pre$\uparrow$ | Rec$\uparrow$ | mIoU$\uparrow$ | RI$\uparrow$ | |----------------------------|--------------|--------------|--------------|---------------|---------------|----------------|--------------| | OGC$_{sup}$ | 51.4 | 41.0 | 49.1 | 43.7 | 56.0 | 66.2 | 91.0 | | Ours$_{sup}$ | **52.5** | **43.3** | **51.8** | **47.5** | **57.0** | **68.0** | **92.6** | | | | | | | | | | | OGC$_{unsup}$ | 40.5 | 30.9 | 37.0 | 30.8 | **46.5** | **60.6** | 86.4 | | Ours$_{unsup}$ | **41.3** | **32.9** | **38.8** | **35.3** | 43.1 | 60.2 | **87.2** | (2) SemanticKITTI | Sequences | Methods | AP$\uparrow$ | PQ$\uparrow$ | F1$\uparrow$ | Pre$\uparrow$ | Rec$\uparrow$ | mIoU$\uparrow$ | RI$\uparrow$ | |-------------------------|----------------|--------------|--------------|--------------|---------------|---------------|----------------|--------------| | 00 - 10 | OGC$_{sup}$ | 53.8 | 41.3 | 48.1 | 40.1 | 60.0 | 68.3 | 90.0 | | | Ours$_{sup}$ | **60.1** | **47.6** | **55.4** | **48.6** | **64.4** | **71.9** | **93.4** | | | | | | | | | | | | | OGC$_{unsup}$ | 42.6 | 30.2 | 35.3 | 28.2 | 47.3 | 60.3 | 86.0 | | | Ours$_{unsup}$ | **46.9** | **31.6** | **36.9** | **29.0** | **50.6** | **63.2** | **88.7** | | | | | | | | | | | | 00 - 07 & 09 - 10 | OGC$_{sup}$ | 55.3 | 41.8 | 48.4 | 40.1 | 61.1 | 69.9 | 90.3 | | | Ours$_{sup}$ | **60.5** | **48.1** | **55.6** | **48.8** | **64.7** | **73.2** | **93.8** | | | | | | | | | | | | | OGC$_{unsup}$ | 43.6 | 30.5 | 35.5 | 28.1 | 48.2 | 62.1 | 86.3 | | | Ours$_{unsup}$ | **47.4** | **31.7** | **36.8** | **28.7** | **51.0** | **64.8** | **89.3** | | | | | | | | | | | | 08 | OGC$_{sup}$ | 49.4 | 39.2 | 46.6 | 40.0 | 55.8 | 60.3 | 88.3 | | | Ours$_{sup}$ | **58.4** | **46.0** | **54.4** | **47.8** | **63.1** | **65.8** | **91.7** | | | | | | | | | | | | | OGC$_{unsup}$ | 38.6 | 29.1 | 34.7 | 28.6 | 44.0 | 51.8 | 84.3 | | | Ours$_{unsup}$ | **44.2** | **31.0** | **37.3** | **30.0** | **49.1** | **55.8** | **86.1** |

Response to Reviewer XLXX (Q3-Q7)

**Q3:** In Sec4.2, why 'point-level invariance of segmentation' helps 'open-set motion' generalization, and why 'part-level equivariant of motion' helps 'category-agnostic part' generalization? Shouldn't they be swapped? It is preferable to move Sec4.4 ahead of Sec4.3. **A3:** Thank you for helping us improve the paper writing. We will revise the paper to clarify the experiments and their explanations with the following three details: (1) Point-level invariance of our segmentation head makes the model robust to unseen pose changes (feature remains relatively consistent despite pose changes), which improves the generalizability to open-set motion (unknown pose variations). Part-level equivariance of the motion head helps generalize to category-agnostic parts because it allows the generalization based on the part-level motion, even if the category is unknown. Nevertheless, by improving the segmentation, the part-level equivariant motion head naturally obtains better supervision from the accurate mask, and vice versa. (2) Your suggestion of moving Sec4.4 ahead of Sec4.3 indeed creates a smoother logical flow from the intact model (Sec 4.4) to the ablations on separate modules (Sec 4.3). We will make this amendment to our paper accordingly. **Q4:** To make the results more convincing, the author should provide more visualizations on all the datasets they've tested on, with transformations shown, and on more challenging cases. **A4:** We provide additional visualizations (attached as PDF in the global response) across four datasets: OGC-DR, OGC-DSRV, SAPIEN, and KITTI-SF with multiple rigid parts. We also show some failure cases in challenging scenes on SAPIEN, as circled in green on the final two rows of the SAPIEN visualization. Please refer to the main response for more details on visualization. **Q5:** Does the current training scheme leverage the cycle consistency among multiple inputs? If not how could this be possibly incorporated into the algorithm? **A5:** Yes, we attempt to make use of cycle consistency from two aspects: (1) For a pair of frames $(P_k, P_l)$, scene flow is updated and obtained for both forward ($\hat\delta_{kl}$) and backward ($\hat\delta_{lk}$) directions. (2) The estimate of our motion head of the original input pair $(P_k, P_l)$ is rectified by averaging from the inversion of the predictive transformation from $(P_l, P_k)$. For readability， we simplify the description of cycle consistency; we will make sure to release the open-source implementation with these training details. **Q6:** How does the method generalize to different point densities/distributions? What is the maximum number of points could the method process? **A6:** (1) We follow standard protocols on corresponding datasets and evaluate our model under various point numbers. Specifically, the point number of one frame is set to 512, 2048, 2048 and 8192 on SAPIEN, OGC-DR, OGC-DRSV and KITTI-SF, respectively. (2) As for the generalizability to different point densities, we include additional experiments on KITTI-Det & SemanticKITTI, of which the points are sparser than those on KITTI-SF. Please refer to **A3** in our response to Reviewer ZttT for more details. **Q7:** Some typos. **A7:** Thank you for bringing these typos to our attention. We will make sure to correct the typos in Sec 3.3 & 4.2 in the revised manuscript.