Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance

Summary

This paper considers an important problem in learning on point clouds -- the equivariance under the SE(3) group. Namely, the authors address the requirement of inter-part equivariance, essential for handling real-world scenarios, where an object can consist of multiple moving parts, or a scene can contain multiple objects that undergo different rigid transformations. The key observation is that segmentation is necessary to define such per-part equivariance, which the authors are the first to do in a strict manner. Furthermore, the proposed fixed-point framework with one-step training and iterative inference is used to demonstrate that the per-step equivariance induces an overall equivariance upon convergence. The developed inter-part equivariant message-passing network with stable convergence is experimentally shown to have strong generalization under different scene configurations, even those changing the point cloud geometry/topology. Overall, the paper provides a sound theoretical framework for a complex practical problem. --------------------------------------- POST REBUTTAL The authors comprehensively addressed my questions and concerns. Given the provided answers and rebuttal overall, I maintained my positive assessment of the paper.

Strengths

S1. The paper is very well-written, and the structure is sound. The illustrations are clear and instructive, which helps to comprehend the presented theory. S2. The proposed view of inter-part equivariance as a co-evolving interplay between geometry and segmentation is novel and compelling. S3. The experimental validation supports well the claims stated in the contribution.

Weaknesses

W1. A discussion on the (time-)complexity of the proposed method is missing. - It is unclear how efficient the iterative inference is. - Besides, a comparison of the proposed model complexity and the baselines is missing. W2. The details of the hyperparameter choice, including the iterative inference part, are missing. - Crucially, the optimal number of iterations $\textbf{k}$ is unspecified, nor is its effect on the complexity with regards to W1. - The same applies to the motivation of the chosen size of the radius of the ball query, $r$, and the maximum number of points in the local neighborhood, $k$. W3. Experiment Section 5.1: the selected Shape2Motion shapes have at most 3 parts with two different semantic labels. - I wonder how the proposed method would perform on Shape2Motion shapes with more than 2 different part labels, e.g., motorbike or bicycle.

Questions

Q1. The first part of the title is often associated with negative connotations in English; see, e.g., [0]. The authors might want to revise it so as not to cause unnecessary misunderstanding. Q2. How can the proposed method be used to perform point cloud classification? Q3. The declaration of $φ$ in (14) is missing. Q4. What is the number of runs in the experiment statistics presented in the tables in Section 5? Especially important for Table 2. Q5. Can the authors explain the large standard deviation of the results in Table 1? Why does the proposed model, in most cases, perform better on Unseen states + unseen instances than Unseen states (see Table 1)? Q6. Ablation studies: Was the inherent ambiguity of PCA-based canonicalization (see, e.g., [1]) taken care of? Could authors elaborate on the details of this experiment? Q7. In the theory of the VN framework [2], which the authors' method is based on, the pose-effect cancelation is achieved by means of the inner product of the equivariant features, which also cancels the effect of reflections. Could the authors run a simple experiment showing if their method is actually E(3)-equivariant and discuss it? Q8. Further noise-stability analysis of the method, where the noise is applied to the input point cloud (not just the segmentation mask), would be beneficial. Q9. It would make the paper more accessible to a broader audience if the authors included an informal motivation of using the Banach fixed-point theorem and iterations. A non-exhaustive list of typos: - Line 148 " and we" --> ", we" - Line 277 "espite" --> "despite" ------------------------------------------------- [0] "banana." Farlex Dictionary of Idioms. 2015. Farlex, Inc 14 Jun. 2023 https://idioms.thefreedictionary.com/banana [1] Li et al. (2021), A Closer Look at Rotation-Invariant Deep Point Cloud Analysis [2] Deng et al. (2021), Vector neurons: A general framework for SO(3)-equivariant networks

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors adequately addressed the limitations and broader societal impact.

Summary

The authors introduce a method for object part segmentation of point clouds that is equivariant/invariant to SE(3) part transformations. The core of their method is a neural network with points and segmentation as input and segmentation as output. The network is assumed to be contractive and is then used to perform Banach fixed point iterations towards the correct segmentation during inference. The Banach fixed point network is embedded in a larger architecture using Vector Neurons. For implementation, a message passing network is used, which uses the input segmentation for determining message strength. The proposed method is evaluated for articulated object part segmentation (on Shape2Motion) and for segmentation of multi-object scans (on DynLab), where it compares favorably against baselines.

Strengths

- The theoretical framework of Banach iterations is elegant and an interesting perspective on iterative networks during inference. In practice, similar ideas on point clouds have been there before, understanding such networks as reweighting functions for iterative reweighting least square schemes, EM iterations, etc. However, I think there is value in this specific perspective, especially since it comes with a convergence guarantee under Lipschitz constraints. To my knowledge, this is a novel contribution. - The presented algorithm seems to solve the given tasks well, outperforming the given baselines clearly (however, there are some concerns regarding missing baselines, see below). - The paper is nicely presented and easy to follow, given the slightly more complex nature. - The result of generalization from synthetic to real chairs is strong

Weaknesses

- The first part of the paper is all about the consequences of a Lipschitz-constraint network and then there is no Lipschitz constraint in practice. The authors claim that the local message passing serves as some type of Lipschitz regularization. However, there is no reference given for this claim and I don't know one either. I think it would make the paper stronger if this aspect would be supported by experiments or theory, which is isn't. The authors only ablate on model performance under different noise levels, which does not give a full picture about convergence. - In articulated object part segmentation, the paper seems to leave out important comparisons to previous work, e.g., [1,2,3]. All three papers are referenced in related work but not compared against. It would be good if the authors would provide a comparison or a discussion, why the comparison is not necessary. I think this is a very interesting paper with a new perspective on iterative networks. However, the above points are a bit concerning, which is why I am not fully convinced to give a better score. [1] Kawana et al.: Unsupervised pose-aware part decomposition for 3d articulated objects. [2] Kawana et al.: Neural star domain as primitive representation [3] Chen et al.: BAE-NET: Branched Autoencoder for Shape Co-Segmentation

Questions

- I suspect that equation 1 wants to express linear blending of part transforms. However, I don't think it is correct: (1) sum of matrix $\mathbf{R}$ and vector \mathbf{t}, (2) $\mathbf{t}_p$ not weighted by $\mathbf{y}_{np}$ . Could the authors clarify? - Please provide further evidence that the network indeed behaves contractive or a theoretical justification for the Lipschitz regularization via local message passing. - Please discuss or provide missing comparisons with previous methods for part segmentation on Shape2Motion.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors discuss limitations and societal impact.

Summary

The paper proposes an equivariant network for part-based (or multi-object) point cloud segmentation. The approach is equivariant to separate SE(3) transformations of each part/object. This is ensured by introducing a Banach fixed-point network. The network takes the point-could and the current segmentation as input, and iterates till convergence. The proposed part-aware equivariant network employs the Vector Neurons method to achieve equivariance for the individual parts. A segmentation-weighted message passing then adds communication between the different parts. Experiments are performed on objects from the Shape2Motion and DynLab datasets.

Strengths

- Novel work based on very interesting and elegant ideas. - Theoretically sound. - Very well written and clear. - Good illustrations. - Comparison with several methods.

Weaknesses

1. The experimental evaluation is limited to very small-scale datasets. It is not clear how the method would scale to larger datasets and more complex scenes, e.g. in an automotive setting, or from terrestrial Lidar scans. It would be good if the authors could discuss this in more detail. 2. It seems that the authors always train on the exact instance which is also encountered during inference. How would the method perform if it encounters a new type of, e.g., oven, after being trained on a dataset of different ovens. 3. I did not find how the other methods in table 1 and 2 were trained. Is data augmentation used? How would data augmentation impact the performance of e.g. PintNet++ or MeteorNet? 4. I did not find discussion on inference and training time. 5. It would be very interesting to see some plots of the IoU w.r.t. the number of iterations, to understand the convergence behavior of the model. In summary, I think that this is a very interesting and solid work. I find no significant weaknesses. Although it would be appreciated if the authors can answer and address my comments in a rebuttal to further strengthen the paper.

Questions

See weaknesses. Moreover: Which other tasks could be suitable for this method?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

They are well addressed.

Summary

The paper propose a Banach, an approach for part-based point-cloud segmentation. In particular, the authors propose an approach to enforce equivariance of the part-segmentation, by construction. They propose a fixed-point framework with one-step training and iterative inference. They propose a part-aware segmentation network.

Strengths

- The results are very convincing - the proposed approach seem to significantly outperform previous approaches

Weaknesses

I do not see any direct weaknesses but I have very little knowledge about this field.

Questions

- what is the run-time of the approach? How does it compare to previous approaches?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

yes

Summary

The paper proposes a neural network architecture that is equivariant to transformations in SE(3) for each object part independently. The network is part of a fixed-point framework where the network is trained with a single step but during testing an iterative approach is used that converges to the desired segmentation.

Strengths

- The paper presents a fresh idea to solve the problem of part segmentation that takes into consideration the different movable parts that compose an object and makes the network equivariant to transformations of those. - The paper provides theoretical derivations that motivate the solution proposed in the paper. - The paper shows the effectiveness of the solution by providing several experiments.

Weaknesses

Although I like the paper, I believe the evaluation could be improved by including other types of equivariant networks: - For example, neural networks based on equivariant operations such as group convolutions or steerable convolutions have by construction equivariance wrt the object parts too. Although these methods would allow information flow between object parts, most of the object parts would remain equivariant to transformations of those. - Moreover, network architectures that work only with the intrinsic information of the shape should also be included. Graph Convolution networks or equivariant mesh convolutions would also maintain equivariance of object parts. These networks are commonly used to segment people in different poses, which is a related problem to the one addressed in the paper. - As the paper states, there is no guarantee that L < 1. The paper states that weight truncation could restrict the upper bound of L but harm the expressivity of the network. An experiment where this is studied would improve the paper. - Lastly, I do not like the acronym Banana since it is not really an acronym of the title. There is no need to include an acronym on the paper.

Questions

See the weaknesses section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are well addressed.

Answer to rebuttal

Thank you for the thorough rebuttal. All my concerns have been addressed and I would encourage the authors to include these discussions in the paper and/or supplementary material. Regarding the acronym, I was not aware of this since mathematics is not my background. I would not oppose leaving it in the paper. I keep my initial positive assessment of the paper.

Thanks

Thank you for explaining the differences in setup with respect to related works [1 - 4]. I would encourage to make this distinction also clear in the related work section of the paper. I guess one could still compare the methods in a setup where all articulation states are used as training data but I agree that this might not be essential to support the main argument of the paper. I appreciate the theoretical justification for message passing networks. It provides a good framework of how to think about it. Also, the additional results shown in the PDF are good additions to the paper. One concern I have regarding the Lipschitz losses: If they are zero everywhere when applied as additional loss, doesn't this just mean they are weighted to strongly? If this is the case it is also no wonder that they have strong negative impact on the IoU. Wouldn't it be more interesting to show these plots in a scenario where they are not used in the loss in order to show the behaviour of the network without additional regularization? All in all, I am satisfied with the answers to my question and increase my score. I think even if this method is not directly relevant in practice, the theoretical framework is interesting and investigates iterative networks from a new perspective.