SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation

In this paper, we introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as a denoising diffusion process, which progressively refines the pose of the source point cloud to obtain a precise alignment with the model point cloud. Training our framework involves two operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process gradually perturbs the optimal rigid transformation of a pair of point clouds by continuously injecting noise (perturbation transformation). By contrast, the SE(3) reverse process focuses on learning a denoising network that refines the noisy transformation step-by-step, bringing it closer to the optimal transformation for accurate pose estimation. Unlike standard diffusion models used in linear Euclidean spaces, our diffusion model operates on the SE(3) manifold. This requires exploiting the linear Lie algebra $\mathfrak{se}(3)$ associated with SE(3) to constrain the transformation transitions during the diffusion and reverse processes. Additionally, to effectively train our denoising network, we derive a registration-specific variational lower bound as the optimization objective for model learning. Furthermore, we show that our denoising network can be constructed with a surrogate registration model, making our approach applicable to different deep registration networks. Extensive experiments demonstrate that our diffusion registration framework presents outstanding pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LINEMOD datasets.

Paper

Similar papers

Peer review

Reviewer 9j1P5/10 · confidence 4/52023-06-25

Summary

The paper tackles model-based 6D object pose estimation using SE(3) diffusion model-based point cloud registration. Point cloud registration is trained with diffusion and denoising processes on SE(3), which gradually perturb the optimal pose and learn a denoising network to refine the noisy transformation progressively. The SE(3) optimization objective is derived from a 3D registration-specific variational lower bound and the denoising network is trained with a surrogate registration model. The perturbation and interpolation on SE(3) are performed with Lie algebra. Experiments on TUD-L, LINEMOD, and Occluded-LINEMOD datasets show the validity of the method.

Strengths

1) The paper reformulates 3D point cloud registration as a diffusion and denoising framework, and the results validate the effectiveness of the framework. 2) The paper is fairly well-written and easy to follow.

Weaknesses

1) The title of the paper says the method is "robust". However, it seems there is a dilemma in sample diversity and sample efficiency for the proposed method. That means a fixed set of hyper-parameters might not work well for different kinds of datasets. In other words, the method is not as robust as claimed in the title. 2) Only mAP is used for evaluating 6D pose estimation, other popular evaluation metrics such as ADD and the BOP metrics should be considered for a more comprehensive understanding of the method's performance. 3) [a] also learns a SE(3) diffusion model. Maybe it should be discussed as a related work. [a] SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. ICRA 2023.

Questions

1) Could the authors provide the training time cost comparison of the proposed method and the compared methods? Does it require significantly more time for training compared to existing methods? 2) How about performing the interpolation and perturbation of the transformation on so(3) + R^3 (which is also a linear vector space) instead of se(3)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Reviewer NS5b8/10 · confidence 3/52023-06-30

Summary

The authors propose an approach to address the problem of 6D pose estimation on real-world data based on a denoising diffusion process. They introduce a novel diffusion process on the SE(3) manifold, leveraging Lie algebra se(3) to shift the process from the linear Euclidean to the nonlinear SE(3) space. Furthermore, a registration-specific variational lower bound is derived as optimization objective for model-based learning. By reformulating the transformation following the posterior of the denoising matching term, the prior mean can be adapted to incorporate a surrogate registration model, enabling the application of existing deep registration models. The experimental results show that not all registration models can deal with real-world data and emphasize the significant effectiveness of the novel SE(3) diffusion process.

Strengths

Originality: A novel diffusion process on the nonlinear SE(3) space for point cloud registration is introduced. Quality: The paper is well written in terms of language and organization, and a solid mathematical foundation for the proposed diffusion process is provided. The extensive experiments highlight the superior performance of the approach compared to the respective baselines and a variety of other related work. The ablation studies demonstrate the impact of the approach and the influence of different configurations on multiple datasets. Clarity: Despite the emphasis on the mathematical background, all mathematical derivations are explained concisely and comprehensibly. The figure of the framework overview and the provided pseudo codes of the training and inference algorithms allowing to grasp the underlying concepts easily. Significance: The novel denoising diffusion process significantly outperforms the baselines on real-world data, showing the effectiveness of the approach compared to related work.

Weaknesses

I could not find any major weaknesses. The authors: - Explain the method well - Conduct extensive experiments and compare their approach to multiple related works on different datasets (only real-world datasets, but this is intended) - Provide quantitative and qualitative results and analyzed which part of their approach contribute to the improvements - Include ablation studies of the diffusion process (forward & backward) to further demonstrate the impact of their approach - The Appendix contains the mathematical derivations in detail, additional visualization, and a section on broader impact, limitations and future work

Questions

Where do the evaluation results for the related methods in Table 1 come from? Did you evaluate the methods yourself? If so, which code base did you use, or where did you find these results? Most methods do not provide them in their papers. Why are only the qualitative results for RPMNet listed in the Appendix and not also for DCP? Since DCP was used in the ablation studies, it would also be interesting to see some visual results of it. Appendix A.1 $\rightarrow$ equation 5 $\rightarrow$ penultimate term: distribution $p$ is used in the numerator, but should it not be distribution $q$?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper addresses the broader implications in terms of societal and academic impact. Moreover, it discusses the limitations of their work and indicated possible future directions based on these limitations.

Reviewer WNiN7/10 · confidence 4/52023-07-03

Summary

This paper introduces a point cloud alignment method based on a diffusion model on SE(3). For this, the forward and reverse diffusion processes are performed in the lie group se(3). The method is evaluated on challenging real datasets, showing significant improvements over its baselines.

Strengths

Relevant Application and Challenging Method: The paper addresses the highly relevant topic of 6DoF pose estimation using diffusion models, which is an area of significant interest. Applying diffusion models to complex parameter spaces like SE(3) poses a challenging problem that has yet to be fully resolved. Experimental setup: The method is evaluated on challenging real datasets that are often used in the literature regarding 6DoF pose estimation and refinement. The comparison with a significant number of existing methods is fair and comprehensive. Multiple scenarios and thresholds are evaluated, allowing for a better insight into the strengths and weaknesses of the different methods. Results: The results validate the method, showing significant improvements over the baseline methods. Several ablation studies allow for insights into the method's parameterizations. Method: The method can be used with different backbones, increasing its impact as an add-on method that potentially strengthens any existing baseline. Writing: The paper is well written and guides the reader through the traditionally notation-heavy math of the diffusion model. The ideas are well motivated and explained. **Rebuttal** The reviewer appreciates the clarifications and additional experiments provided in the rebuttal and maintains the initial rating.

Weaknesses

Some related work on diffusion models over SE(3) / SO(3) could be cited. Examples include: [R2] Leach et al., Denoising diffusion probabilistic models on so(3) for rotational alignment, ICLRW 2022. [R3] Urain et al., SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion, ICRA 2022.

Questions

Q1: 179: There are several different ways for interpolating rotations; I've often seen quaternions used for this. Is there a motivation for using the exponential map over other representations? Similarly for the perturbations. Q2: 268: How are 512 points sampled if the objects are smaller than that (especially in real scenes)? (Suggestion for future work, no effect on rating, no need to respond): The authors might want to look at SYMSOL [R1], data synthetic dataset especially for dealing with pose ambiguities. [R1] Murphy, Kieran A., et al. "Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold.", ICML 2021

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Limitations and impact are adequately discussed.

Reviewer VaQE4/10 · confidence 5/52023-07-05

Summary

This paper proposes a SE(3) diffusion model-based point cloud registration framework for robust 6D object pose estimation, which formulates point cloud registration as a denoising diffusion process and enables progressive refinement of the transformation between the source point cloud and the model point cloud. Experiments are conducted on TUD-L, LINEMOD, and Occluded-LINEMOD datasets, and state-of-the-art results are achieved on these datasets.

Strengths

1. The paper is overall well written and easy to follow. 2. Experiments are conducted on three different datasets and comparisons are done to different methods. High performance is achieved.

Weaknesses

1. Since this paper aims for 6D object pose estimation, the compared methods should mainly focus on 6D object pose estimation methods obviously, not just some point cloud registration methods. 2. The proposed method is trained and tested on 6D object pose estimation datasets, However, the compared methods are point cloud registration methods. Did the authors retrain these learning-based point cloud registration models on the datasets in order to fairly compare with these learning-based methods? I am concerned that this is the key to the improved performance. 3. The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison, e.g., point cloud registration methods (Predator[1], Cofinet[2], Geotransformer[3], and so on) and 6D object pose estimation (Fs-net[4], Ove6d[5], Gpv-pose[6] and so on). [1] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In CVPR, 2021. [2] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic. Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration. NeurIPS, 2021. [3] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In CVPR, 2022. [4] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Shen Linlin, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In CVPR, 2021. [5] Dingding Cai, Janne Heikkila, and Esa Rahtu. Ove6d: Object viewpoint encoding for depth-based 6d object pose estimation. In CVPR, 2022. [6] Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry-guided point-wise voting. In CVPR, 2022.

Questions

See Weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

The authors mention the limitation of the proposed method in supplementary material.

Reviewer 5ZGT5/10 · confidence 3/52023-07-06

Summary

The paper proposes and SE(3) diffusion model for pose estimation i.e. rotation R and translation t for the use case of registration two pointclouds. The paper addresses an important problem in 3D vision and which has applications in Robotics, AR/VR. The author claim that the convential diffusion models won't work for this task of SE(3) diffusion and proposes several changes to constaint the transformation transitions during the diffusion and reverse processes. Results are shown on competing and challenging baselines.

Strengths

The technical contributions look sound and the author addresses an important and challenging problem in 3D vision. Adapting conventional diffusion models to SE(3) using a thorough mathematical formulation is convincing. Results show strong improvement on quantitative metrics.

Weaknesses

1. I wonder why the authors didn't compare or show experiments on more challenging categorical 6D pose and size estimation benchmarks nor discuss those in related works [1][2][3]. Is it a limitation of current work? This should at least be discussed in literature review for the work to have more merit since conventional instance-based pose estimation which assumes known CAD models has mostly been solved achieving higher accuracies. Might this be one of the reasons the author gets smaller improvements in some of the metrics? 2. Relating to my point above, the author only evaluates their approach on a single and very simpler benchmark. I understand this is what the authors are proposing i.e. point cloud registration but adding more benchmarks (within the point cloud registration works such as in the wild pose estimation etcetera) could further reinforce the author's results since it is hard to fully validate the efficacy of the results when evaluated on a single benchmark. Recently, the shift has been more toward the wild pose estimation[4] and zero-shot pose estimation and I wonder if this work can directly be useful to some of the other more complicated problems as well. 3. One of the pain points of traditional ICP is parameter tuning which can improve results although takes a lot of hand-tuning. Does the author's approach suffer from the same problem? like a number of denoising steps and etc. 4. Diffusion models are slow to train and infer. Can the authors comment on their inference speeds vs accuracy i.e. if they use a smaller denoising timestep? In essence, I saw no comparisons to state-of-the-art timing results for 6D pose estimation i.e. [2][3], or a direct discussion of it. Timing is crucial for robotics applications and the work's strong improvement might be downplayed if this is very slow and not real-time. [1] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, Wang et al. [2] CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation Irshad et al. [3] ShAPO : Implicit Representations for Multi Object Shape Appearance and Pose Optimization, Irshad et al. [4] Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset, Fu et al.

Questions

Please see the remarks in the weakness section. I am willing to improve my score if the authors address my concerns about 1. more complicated benchmarks i.e comparisons to categorical 6D pose and size estimation or in the wild pose estimation, discussion/comments about parameter tuning for the author's approach, discussion/comparison to timing i.e. speed vs accuracy tradeoff relation to some of the state of the art, fast 6D pose and size estimation approaches.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author's work focuses on point cloud registration only and requires a depth map or depth sensor and wouldn't work in a monocular i.e. RGB only case

Reviewer WNiN2023-08-17

Thank you for the additional details and clarifications.

Reviewer 9j1P2023-08-19

Thanks for the clarifications.

Reviewer VaQE2023-08-19

Thank you for the author's feedback. This paper is dedicated to the task of point cloud-based 6D object pose estimation, and there are so many depth-only-based 6D object pose estimation methods, e,g, OVE6D, StablePose, CloudAAE, CloudPose, and so on. I think it is necessary to include these methods in the comparisons. Otherwise, I worry that the proposed method is not convincing enough. I think a thorough comparison with existing methods is necessary to verify the effectiveness of the proposed methods. Furthermore, the listed point cloud registration methods also could perform well on object-centric datasets. Besides, more state-of-the-art object-level point cloud registration methods have emerged recently, and comparison with the latest state-of-the-art methods is necessary.

Authorsrebuttal2023-08-21

We thank the reviewer for reviewing and replying to our rebuttal. * Regression-based vs Matching-based Methods: We note that most previous depth-only-based methods (e.g., CloudAAE, CloudPose, and StablePose) on 6D pose estimation are based on direct pose **regression**, disregarding the utilization of object models for pose reference. As a result, these methods face limitations in their ability to generalize to unseen objects, primarily due to the absence of model-informed pose references. Although OVE6D exhibits the potential to generalize to unseen objects, it necessitates the additional encoding of new instances into the viewpoint codebook, which could potentially increase its application costs. By contrast, our method follows a drastically different research trajectory that leverages **3D matching** between the scanned object point cloud (PC) and the model PC for pose estimation. Therefore, its inherent model generalization ability in 3D matching potentially facilitates the pose estimation of unseen objects. To validate it, we choose objects with IDs=1~8 from LINEMOD to train our Diff-DCP and test its generalization performance on the unseen object with ID=9. Also, we directly generalize the Diff-DCP trained on LINEMOD to the TUD-L object with ID=3. We report the seen results vs unseen results in the table below. We observe that our method Diff-DCP can still achieve meaningful and encouraging estimation precisions on unseen objects without any additional model fine-tuning. | |LINEMOD (ID=9) | TUD-L (ID=3)| |:-----|:-----|:-----| | | mAP (5°/10°/1cm/2cm)|mAP (5°/10°/1cm/2cm)| Diff-DCP (seen) | 0.12/0.46/0.80/0.96 | 0.92/0.98/0.89/0.97| Diff-DCP (unseen) | 0.10/0.39/0.68/0.82 | 0.53/0.67/0.57/0.65| * Core Contribution to Matching-based Methods: As mentioned in lines 30-41, current object-level registration methods (e.g., MN-DCP in ECCV’2022) still suffer from limited matching robustness for pose estimation due to real-world challenges (e.g., full-range rotation and severe occlusion). It promotes us to develop an SE(3) diffusion registration model as a **plug-and-play** method to advance current registration methods for more reliable pose estimation. As a general framework, our method can integrate with different deep registration models to boost their performance (Ref: DCP vs Diff-DCP and RPMNet vs Diff-RPMNet in Table 1), thereby increasing our impact as an add-on method to enhance existing registration models. Theoretically, we can also include Predator, GeoTransformer and CoFiNet as our surrogate registration models (see Eq.12) into our framework for their performance boosting. We are highly interested to validate it in our future paper version. Also, we think the significant performance improvements achieved by Diff-DCP and Diff-RPMNet over their baselines (see Table 1) have sufficiently verified the effectiveness of our framework. * Summary: Different from most previous **regression**-based depth-only-based methods, our method is based on **3D matching**, enabling our method to enjoy the promising generalization ability to unseen object pose estimation. Furthermore, our SE(3) diffusion registration model is a general, **plug-and-play** framework. We can take different deep registration methods (such as the listed Predator and GeoTransformer) for their performance improvement. Finally, we regret that due to time constraints, we have to perform **a thorough comparison** with existing methods, e.g. depth-only-based methods and more point cloud registration methods, in our final paper version. Finally, we sincerely hope the reviewer will reconsider our paper for following aspects: (i) We pioneer the SE(3) diffusion registration model for robust 6D object pose estimation, with two innovative components: a transformation interpolation-based SE(3) diffusion process and a surrogate registration-driven SE(3) reverse process; (ii) In this context, we derive an effective registration-specific variational lower bound for model optimization (Detailed proof can be found in Appendix A.1); (iii) Our method is a general, plug-and-play framework, which can effectively integrate with different deep registration models to boost their performance, significantly increasing our impact as an add-on method to enhance existing registration models; (iv) Extensive experiments on real-world 6D object pose estimation datasets validate that our diffusion variants (Diff-DCP and Diff-RPMNet) consistently achieve significant performance improvements over their baseline models (DCP and RPMNet).

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC