Point-based Instance Completion with Scene Constraints

We thank reviewer gWgp for their suggestions and questions. Our responses to the feedback are provided below. --- **Subject: Discussing completion quality under different scenarios (e.g., incompleteness, noise)** As suggested, we have conducted an experiment where we see how the completion quality of our method varies under different levels of incompleteness in the partial input. In Figure 12a of our appendix, we present a histogram breaking down the object instances in the test set of our dataset by how much of the complete shape is present in the partial object scan. We find that a majority of our partial object scans contain between 30-60% of the complete geometry before being input to our completion model. In Figures 12(b-d), we plot the average completion metrics for each bin in the histogram. We find that our method outperforms the baseline approaches at each completeness level by a large margin, which is consistent with the large gap in performance observed in Tables 1 and 2. However, we do observe that RfD-Net and DIMR do outperform our method for extremely sparse inputs (e.g., when only 0-10% of the object is present in the partial scan). Qualitatively, RfD-Net and DIMR tend to generate a nondesciptive blob whereas our method usually just returns the partial input, leading to them obtaining better completion metrics in those cases, which we think are probably of less interest because of the lack of object information from the observed areas. In terms of noise, we added Figure 13 where we visualize some example completions when there are errors present in the instance segmentation predictions produced by Mask3D. In the top four rows, we show examples where Mask3D produced segmentations that missed large or important parts of the partial object instance. The regions missed by Mask3D contain important cues for the true geometry and size of the object, and the lack of this information leads our model to produce a completion different from the ground truth. In the bottom two rows, we show examples where Mask3D incorrectly segments two objects that are side by side as a single object, and our completion model simply followed it and completed both objects together as if it were one. --- **Subject: Completing unseen objects** Our method is capable of generalizing to novel objects within the categories we train on, with many of the test scenes in our dataset containing object instances that were never seen during training. We do not evaluate on novel object categories that are not similar to the training categories as we don't believe the method will generalize to those. Instead, in Figure 14 of our appendix, we have additionally shared some examples of how our method generalizes to partial objects from real scans in ScanNet. The objects in ScanNet are novel and their partial scans are considerably less clean than the scans found in our proposed dataset, yet our method is still able to produce plausible completions of the objects.

We thank reviewer eRJX for their comments and questions. Our responses to the feedback are provided below. ---- **Subject: Clarifying contribution about robustness to object pose and scale** Our approach extends prior work on object-level completion methods such as SnowflakeNet [1], SeedFormer [2], AnchorFormer [3] (ln. 53 - 63 of our paper). These approaches produce high quality completions but expect the object to be in a canonical coordinate system. Hence our contribution statement is with respect to those approaches where we lifted the requirement of the canonical coordinate system and additionally leveraged scene constraints. Alternatively, prior instance scene completion methods like RfD-Net [4] and DIMR [5] indeed do not require objects to be in canonical coordinates, but these approaches produce much lower quality completions than our approach (since their network design was inferior to the object-level completion methods), as shown in our experiments. --- **Subject: Effectiveness of scene constraints** For the scene constraint ablation results in Table 4, we note that we do not expect an improvement in UHD (partial reconstruction quality) because the completion model does not need information about free space and occluded space to reconstruct the already observed portion of the object. However, we do find that including scene constraints significantly improves overall completion quality (CD) by 1.63 (7% relative improvement). While we did not observe much of an improvement in the percent of points in collision (%COL) because most of the points are simply far away from other objects, we do observe a very significant 29% relative improvement (2.67 vs 3.75) in how far these points that are in collision are penetrating into other objects (COL). --- **Subject: Performance improvements in object completion model ablation** As mentioned in the response to the first question, the quality of previous instance scene completion methods were significantly inferior to the object-level ones, but object-level ones weren't applicable to the instance scene completion problem before our contributions. From Table 3, we note a 12% relative drop in performance when object-level completion methods are naively used in instance scene completion (20.20 to 22.87). Our approach completely erased this drop and resulted in a significant 12% relative improvement (20.08 vs. 22.87) over the baseline. Note that results in each row contain all the modules in the rows above them. Hence, the final row of Table 3 is the performance of our full completion model. We also note that this ablation was only conducted on the chair category from ShapeNet as training on the 34 ShapeNet categories would be too computationally expensive with the resources we have available. Hence the numbers presented in Table 3 are not directly comparable to the numbers in Table 2. --- **Subject: Concerns regarding performance improvements being due to better instance segmentation** Rather than showing instance segmentation performance, we conducted an experiment evaluating completion quality when the ground truth segmentations are provided to each method (referred to as the "Scene Completion" task in our paper, ln. 449-452). This removes the differences in segmentation quality across methods and isolates the evaluation of completion quality. Results of this experiment in Table 2 show that when all methods are using the same ground truth segmentations, our method significantly outperforms prior work in terms of completion quality. --- **References** $\left [1 \right]$ Xiang, Peng, et al. "Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer." Proceedings of the IEEE/CVF international conference on computer vision. 2021. $\left [2 \right]$ Zhou, Haoran, et al. "Seedformer: Patch seeds based point cloud completion with upsample transformer." European conference on computer vision. Cham: Springer Nature Switzerland, 2022. $\left [3 \right]$ Chen, Zhikai, et al. "Anchorformer: Point cloud completion from discriminative nodes." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. $\left [4 \right]$ Nie, Yinyu, et al. "Rfd-net: Point scene understanding by semantic instance reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. $\left [5 \right]$ Tang, Jiaxiang, et al. "Point scene understanding via disentangled instance mesh reconstruction." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

We thank reviewer TJHm for their comments and questions. Our responses to the feedback are provided below. --- **Subject: Framework and Contributions** We follow the "segment then complete" paradigm as we believe it is a reasonable framework for solving the instance scene completion task, since it allows methods to use state-of-the-art instance segmentation models which might be trained on much larger data. The major issue with existing works is that they are unable to produce high quality completions due to their poor completion network design. Hence, we focused on improving the object completion module. Aside from the proposed improvements in our object completion model, we have also introduced: 1. A way to represent known scene constraints as a sparse set of points, which provide our object completions with context of the scene they reside in. 2. A new dataset for the instance scene completion task, which contains aligned partial scans and ground truth meshes of scenes which are collision free. Unlike previous datasets, which either lack alignment or are not collision free, our dataset is more reliable for evaluating completion quality and plausibility. --- **Subject: Performance against baselines without scene constraints** Table 4 shows the results of our approach without scene constraints on the scene completion task (same task as the results shown in Table 2). Even without scene constraints, our approach outperforms the baseline methods due to a stronger completion network. However, in Table 4 we demonstrate that scene constraints further improves completion quality (CD) and reduces how badly objects in collision penetrate into each other (COL). Qualitative results were shown in Fig. 7.

We thank reviewer M7dZ for their comments and questions. Our responses to the feedback are provided below. --- **Subject: Local attention vs. global attention-based completion** The underlying logic is that one has to see the entirety of the object in order to predict the center. It's difficult for local attention methods to do so with variable object sizes. As mentioned in lines 207 - 210, this was experimentally shown in Table 3, where our "baseline", which uses the local attention-based upsampler from SeedFormer [1], obtains a CD of 20.20 when objects are in canonical coordinates and a CD of 22.87 when they are not. From here, our global attention-based object center prediction results in a 7% relative improvement in completion quality (22.87 to 21.11), and our global attention in upsampling further provides another 5% relative improvement (21.11 to 20.05). --- **Subject: Description of global shape descriptor** The global shape descriptor is generated by passing the final local features $F_{p}^{l}$ through a MLP followed by max-pooling. We originally mentioned this in section $B.1$ of our appendix, but have updated the main paper to contain this detail. --- **Subject: Explanation of Patch Seeds** "Patch Seeds" is a term coined by SeedFormer [1] and we similarly use this term as our method builds off their architecture. We note that we have already provided a high level description of what Patch Seeds are in lines 157-158 and a formal definition in lines 204-206 of our main paper. In particular, Patch Seeds are a set of coordinates $S \in \mathbb{R}^{M_{seed} \times 3}$ and corresponding features $F_{seed} \in \mathbb{R}^{M_{seed} \times C_{seed}}$, which represent a coarse encoding of the complete shape. --- **Subject: Purpose of transposed convolution** We note that the purpose of the transposed convolution is described in lines 236 - 239. In particular, we want to increase the number of points present in our Patch Seeds compared to our downsampled partial points and features that they are generated from. The transposed convolution is used to upsample or "split" the existing set of features we have. We specifically use a transposed 1D convolution with stride 2 and kernel size 2, lifting the features from dimensions $N \times C$ to $2N \times C$ (i.e., we double the number of points). --- **Subject: Generalizing to unseen objects** We assume the reviewer meant during inference time as it is not possible for the model to generalize to unknown objects during training. Our method is capable of generalizing to novel objects within the categories we train on, with many of the test scenes in our dataset containing object instances that were never seen during training. We do not evaluate on novel object categories that are not similar to the training categories as we don't believe the method will generalize to those. Instead, in Figure 14 of our appendix, we have additionally shared some examples of how our method generalizes to partial objects from real scans in ScanNet. The objects in ScanNet are novel and their partial scans are considerably less clean than the scans we trained on from our dataset, yet our method is still able to produce plausible completions of the objects. --- **References** $\left [1 \right]$ Zhou, Haoran, et al. "Seedformer: Patch seeds based point cloud completion with upsample transformer." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

Thanks the authors for the clear clarification. Most of my concerns have been addressed, and I would raise my overall score from 6 to 8.