DiffComplete: Diffusion-based Generative 3D Shape Completion

Summary

The paper tackles the shape completion task by introducing a diffusion-based technique. To do so, additional design choices have been made such as hierarchical feature aggregation and an occupancy-aware fusion strategy to better reproduce the shape and respect the details. Based on numerous quantitative and qualitative experiments, the proposed method outperforms others by a considerable margin.

Strengths

The paper clearly outperforms other SOTA. The visual quality of the completion is good. Occupancy-aware fusion and hierarchical feature aggregation that are the main task specific designs make sense and the authros have successfully shown their importance in the design through ablation studies. I like the adoption of control net in the design. I believe it makes sense to learn gradually and at different scales. The fact that the data is from the actual scan and it is not synthesized by adding noise and removing parts make the method very useful and interesting.

Weaknesses

In terms of writing, the paper could use simpler and more understandable and short sentences. For example, this sentence could be improved by making it shorter or breaking it into two sentences: To improve the completion accuracy, we enable effective control from single-input condition by hierarchical and spatially-consistent feature aggregation, and from multiple ones by an occupancy-aware fusion strategy. This happens many times throughout the paper. Missing reference: Multimodal shape completion with IMLE: https://openaccess.thecvf.com/content/CVPR2022W/DLGC/html/Arora_Multimodal_Shape_Completion_via_Implicit_Maximum_Likelihood_Estimation_CVPRW_2022_paper.html

Questions

When you pro-process the input and the incomplete shape through a convolutional network and add them, why do you call it aligning distributions? As the computation and time are two limitations of this method, would a part-based completion task be a better approach? In fact, my question is whether a part based approach in which a model is first broken into parts will save computation time and understand the geometry better?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The method is mostly trained category specific. Considering the time and computational cost of this approach, this makes it hard and expensive to be adopted and modified. easily. However, I weigh the quality of the results more than the time and computation so I am okay with this limitation.

Summary

The authors present a diffusion-based neural network for 3D shape completion. They introduce a hierarchical and spatially consistent feature aggregation scheme to fuse partial and complete shapes, and an occupancy-aware fusion strategy to fuse multiple partial shapes. They demonstrate a new SOTA performance in shape completion for both new instances and completely unknown object categories.

Strengths

1. The authors successfully apply the diffusion-based model to the shape completion task and achieve a new SOTA performance. 2. The stepwise addition of partial scans to predict a complete shape is interesting and makes completion tasks interactive and adaptable. 3. The proposed model supports multimodal completions that can predict multiple plausible results.

Weaknesses

1. The writing of this paper can be improved. 2. The technical designs, such as the two feature aggregation schemes, are not well motivated. As a main contribution, I expect more explanations of the design ideas. 3. The experimental design is not very clear. Does DiffComplete train on a single category? Since PatchComplete trains on all categories, DiffComplete should also train across categories to allow a fair comparison. 4. The results of multiple conditional inputs are unpredictable. A better example is adding and editing a semantic part of the object from a partial input. 5. Does the proposed model require much more time than its competitors?

Questions

1. I wonder if the resolution can go up to 64^3 or even 128^3? 2. The diversity of multimodal complement is worse than other methods. So is it because the diffusion model gives more deterministic results than others?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The limitation is not addressed.

Summary

The paper tackles the problem of probabilistic shape completion using diffusion models, learning from range scans. The main contribution of the paper comes from proposing 2 novel techniques: hierarchical feature aggregation for strong conditioning and occupancy-aware fusion technique. The method is tested on ShapeNet dataset and achieves better completions compared to the baselines. Also, the proposed method exhibits robust generatlization for out of distribution inputs.

Strengths

* I enjoyed the idea of hierarchical feature aggregation. To best of my knowledge, conditioning a diffusion model is hard, and the ablation study in Tab. 7 clearly shows that the multi-level conditioning acts as intended. * Strong empirical results on completion on unseen categories. The results in Tab2, 3 shows that the method achieves strong completion results on unseen categories and it makes sense due to strong hierarchical conditioning.

Weaknesses

* Similar works on 3D completion using diffusion models exists. As mentioned in the related sections, the difference compes from the representation used to represent the state in the diffusion model. For example, point-voxel uses point cloud, while this work uses TSDF (TUDF) representation. While I do not think that the novelty of the work degrades even there are other diffusion based 3D completion models, I want to hear in detail how the method differs from other diffusion based models. Please look at the question section. * My main concern is that some of the important recent works are not mentioned and compared. For determnisitc completion methods, convocc completes the given partial point cloud using occupancy fields. For multimodal completion methods, point-voxel [2] (although mentioned in the related-section) and GCA[3] completes point cloud in point cloud/sparse voxel representation. ShapeFormer[4] and cGCA[5] completes the point cloud in implicit neural representations. Although I realize that comparing all of these baselines requires alot of effort, these baselines should have been tested since the core contribution of the work stems from achieving SOTA results on an existing benchmark. [1] Peng et al. Convolutional Occupancy Networks. ECCV, 2020 [2] Zhou et al. 3D Shape Generation and Completion Through Point-Voxel Diffusion. ICCV, 2021 [3] Zhang et al. Learning to generate 3d shapes with generative cellular automata. ICLR, 2021 [4] Yan et al. ShapeFormer: Transformer-based Shape Completion via Sparse Representation. CVPR, 2022 [5] Zhang et al. Probabilistic implicit scene completion. ICLR, 2022

Questions

* Regarding the related work with diffusion based models, the authors mentioned that “. Due to the absence of meaningful ground truths in these completion scenarios, they could also face completion accuracy challenges like the above generative approaches” in line 114. Does that mean that the diffusion methods require the ground truth while the proposed method does not? To the best of my knowledge, the diffcomplete model requires ground truth TUDF for training. I think that the assumption that we can acquire the ground truth TUDF means that we can acquire ground truth point cloud and implicit function (mostly UDF based) as well. If so, then other baseline methods mentioned in the weakness section should be compared.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Authors have addressed the limitations in the supplementary material.

Summary

The paper introduces a diffusion-based approach, DiffComplete, to generate complete shapes conditioned on partial 3D range scans. The condition is represented as volumetric TSDF (truncated signed distance function), while the complete shape is represented as volumetric TUDF (truncated signed distance function). Inspired by 2D ControlNet, the authors devise a hierarchical feature aggregation mechanism to inject conditional features in a spatially-consistent manner. Besides, they propose a fine-tuning strategy to adapt the model trained on a single condition to multiple conditions. The trade-off between multi-modality and high fidelity can be controlled through the network level for feature aggregation between conditions and denoised complete shapes. The results on 3D-EPN and PatchComplete show the superiority of DiffComplete. The authors also demonstrate its zero-shot ability.

Strengths

- The paper is clearly written and easy to follow. - The ControlNet-style design to inject features from the condition is reasonable. - Fine-tuning the network trained with a single incomplete shape for multiple incomplete shapes is a good strategy. - The generalizability looks good according to Fig. 4.

Weaknesses

It is unclear why some baselines (especially point-cloud-based methods) are missing. For example, "3D Shape Generation and Completion Through Point-Voxel Diffusion" and "SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer".

Questions

1. Is there a typo in Figure 7? Is it an MMD curve? 2. For the ablation study on multiple conditions, can the authors first fuse the scans and compute TSDF, then extract features instead of averaging features extracted from individual scans?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations (e.g., failure cases) in the appendix.

Summary

The proposed method aims to tackle the TSDF shape completion problem using diffusion models. That is, given one or several incomplete TSDFs obtained from partial scans of a single object produced by range sensors, the method generates a TSDF of complete shape while trying to preserve the geometric details of the incomplete one. The method achieves this by using the same technique in ControlNet: adding a control branch to the vanilla UNet module in diffusion models. Also, in order to better incorporate the partial scans from different views, the authors propose an "occupancy-aware fusion" module which performs a weighted feature fusion for the multi-view scans considering their geometric reliability.

Strengths

In general, the paper is easy to read and is technically sound. The following are specific points: - Single probabilistic framework for all tasks. As more and more recent papers suggested, generative models are not only doing great in generation problems, but are also competitive in deterministic problems where usually a unique optimal solution is desired given the input. This method shows this point for the 3D completion problem: Even when the input shape has little to no ambiguity, the generated completion has better quality than previous deterministic approaches. - Incorporate multi-view scans. This special consideration is useful in real-life application, where the scanners are constantly capturing new scans and the small errors in registration will make it non-trivial to fuse features from different views.

Weaknesses

I have concerns on baseline & references, unsatisfactory contricution and writing. Hence I believe this paper does not reach the NeurIPS bar and would like to rate "Reject". Here are detailed points: - Missing baselines and references. Comparison with multiple very related works are omitted. ConvOccNet can perform shape completion very well in higher resolution (64 and higher, see the Fig. 4 in their supplementary). The model is very good at deterministic completion. For multimodal setting, two very related work, ShapeFormer and 3DILG, are not mentioned or discussed. And the former work is especially designed for multimodal completion. - Contribution is not satisfactory. The proposed method seems to be just deploying ControlNet on the PatchComplete problem and benchmark. The used resolution (32) is also not impressive comparing to previous works like DiffusionSDF or 3DShape2VecSet. I would be skeptical that training diffusion models directly and keep the partial tsdf unchanged during sampling (as what PVD does for completion) can achieve better results. - Many minor writing problems. L99: It is confusing to classify AutoSDF to be a type of autoencoder. It is an autoregressive approach. L114: "Due to the absence of meaningful ground truths, ...". In the mentioned works, they all have meaningful ground truths shapes. It is unclear what is "meaningful" here. L325: "Effects of feature aggregation manner". A better way is to write: "Effects of the manner in which features are aggregated" or "Effects of our feature aggregation mechanism"

Questions

- Please show advantages over these baselines: (1) Convolutional Occupancy Network. Train ConvOccNet to map partial TSDF grid to complete grid. Then compare. (2) ShapeFormer. TSDF can be converted to point cloud. Train ShapeFormer to map point cloud to implicit TSDF fields. (3) TSDF diffusion with PVD style of completion sampling. That is, in the sampling time, always replace the noisy to the actual TSDF value in the place of partial scanned cells. This way, only the "missing" regions are denoised and completed.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Authors include a comprehensive discussion on method limitations and potential societal impact in the supplementary material.

Revision

I indeed thank the authors for the clarification and all the experiments conducted. I feel that the paper is much stronger for it. It would be nice if the authors could visualize the qualitative results for the newly added baselines, but I think that can be done in the camera-ready version. I am convinced that the method works well compared to other baselines and I will raise the score.

Author Response to Reviewer 9z1Q

Dear Reviewer 9z1Q, We sincerely appreciate your positive feedback and consistent support. Your suggestions are instrumental in enhancing the quality of our paper.