POC-SLT: Partial Object Completion with SDF Latent Transformers

Dear Reviewer jb2s, thank you for reviewing our paper. In the following, we address your concerns and potential misconceptions about our method: # W1 Concerning the input to our model, we demonstrated that the P-VAE can accurately encode highly detailed geometric information, providing the SLT with a rich representation for shape completion. Concerning the output, as you can see in our results (e.g., Figure 3), there are minor inconsistencies between the predicted patches, producing small seams. However, these seams are small enough that one can potentially remove them in a simple post-filtering process. The SLT naturally learns to predict patches that fit well together during training thanks to its global view in the transformer-encoder. To ensure seamless predictions without post-filtering, one can look into a convolutional decoder, accounting for neighboring patches, or interpolate between overlapping patches or multi-resolution patches. Given our almost seamless results, we did not see the need for applying such methods. # W2 3D positional encoding is necessary to spatially identify the individual patches in the sequence given to the transformer-encoder. The location of a patch within the sequence has no influence on the processing by the transformer-encoder. Therefore, without positional encoding, there is effectively no spatial relationship between patches. Crucially, all masked tokens would complete to the **exact same** result. We apply 3D positional encoding in a standard way, where the channels are equally split into representations of the $x,y,z$ coordinates, encoded as sine-waves of increasing frequency. The exact encoding is unlikely to matter too much in our case. It simply needs to describe the $4\times4\times4$ grid positions of the patches. In other, more common, applications of 3D positional encoding, one would typically represent varying 3D locations of arbitrary points in a point cloud. # W3 We also prefer to report more meaningful metrics for shape completion, which is why we list a wide range of metrics when evaluating our method. However, previous methods did in fact not report these metrics on comparable tasks. Therefore, we had to fall back on reporting on whatever published data was available. The only compatible data turned out to be the UHD data reported by AutoSDF and SDFusion. We have since taken the time to run these evaluations for competing methods ourselves (Table 3, updated). Ideally, these metrics should have already been reported by the original authors to ensure that the best possible configuration of their method is used. # W4 Results for shape completion from randomly removed patches are already shown in Figure 9 in the Appendix, matching the quantitative evaluation in Table 2, and also as part of the supplementary video. # W5 Here, we again compare with all available published data. Previous methods only train and evaluate on very small subsets of data, containing mostly chairs, to keep their training and evaluation costs down. So, what is compared is the smallest common denominator. For our method, we also reported the result across all classes (previously Table 3, now Table 9). We have since taken the time to run the evaluation for AutoSDF and SDFusion to compare all metrics across all classes contained in their test set (Table 3, updated). Our method is the first to scale up to all of ShapeNet, so there is no published data to compare with, as we cannot possibly re-train other methods for comparison. We hope that this answers all of your concerns and allows you to wholeheartedly recommend our paper for acceptance. Best regards, the authors

Dear Reviewer 1tgP, thank you for reviewing our paper. We have noticed some potential misunderstandings and would like to clarify them first and then address your concerns and questions below: # S1 We do not use a "MAE decoder", only a MAE-style transformer-encoder. SDF patches are then decoded using the P-VAE's pre-trained decoder. # S2 We are the first to train a VAE on 3D SDF patches. Other methods only used AE/AD, VQAE, VQDIF, but never a VAE. Unlike prior UNet-style 2D VAEs, we do not use skip-connections between encoder and decoder. Our VAE is free to allocate latent parameters across levels of detail, unrestricted by pre-defined partitions for residuals. LIG [1] explicitly shy away from training a VAE for 3D patch representations due to the difficulties involved in doing so. Our method overcomes these difficulties and we even evaluated autodecoding, which they similarly excluded. # W1 The current design cannot avoid SDFs leaking information about neighboring cells. Our evaluation shows highly detailed completions far away from input patches. This indicates that our SLT's shape completion capabilities cannot possibly derive from potential leaks. Retraining our P-VAE and SLT for TSDFs would be costly. Truncation will necessarily negatively impact performance, since it removes information. However, one could not conclude that this would be due to leakage problems, since the SDF space constrains the P-VAE's latent space: Unlike for TSDFs, introducing surfaces to SDFs requires large changes. Similarly, a small deviation in TSDF latent codes might add unintended surfaces, complicating shape completion. # W2 Our goal is to establish a backbone for shape completion. Solving real-world 3D scanning tasks requires further steps as already outlined in our Limitations and Conclusion sections. # W3 Our P-VAE can easily scale to patches from entire scenes. Training the SLT to on significantly more latent codes might require a convolutional, or multi-scale, approach. We did initially filter out this method among others from the scene-level category. We now discuss this method in our related work. # W4 Actually, we only compared against DeepSDF in the supplemental for SDF autoencoding (Table 7). We also compared shape autoencoding with the recent 3DShape2VecSet (Table 8). We also compared with more recent methods (AutoSDF, AnchorFormer). We have since added comparisons against ShapeFormer and compare more metrics with SDFusion, as recommended by other reviewers (Tables 3,4). # W5 We compare against data published by SDFusion themselves on this particular task. They must have trained their model with this task in mind. Furthermore, partial inputs are still given within a normalized coordinate system where the model can easily figure out missing halves. By masking the entire half, our model also has to figure out where to complete the object. # W6 We also prefer more meaningful metrics for shape completion. However, previous methods did in fact not report these metrics on comparable tasks. We have since run these evaluations for competing methods ourselves (Table 3, updated). # W7 The canonical orientation is derived from ShapeNet. It was shown to improve reconstruction on objects from known classes. Like previous methods, which widely share this limitation, we make use of it for fair comparison. Where not given, the canonical orientation can be estimated by methods like 3DScan2CAD. # Q1 Masking is a required part of the MAE pipeline. The transformer-encoder is always given a full sequence of patches. For every given input patch, an output patch is produced. During training, inputs are masked to train predictions of masked patches from the given patches. During inference, the same mechanism deterministically predicts the completed output. Without a masked patch as an input, there would be no output. # Q2 The VAE is essential for producing a smooth latent space. This is essential for the attention mechanism at the core of the transformer architecture. With VAE, slightly different latent codes maintain the same meaning and receive similar attention. Without VAE, slightly different latent codes could have completely separate meanings, conflicting with the attention. Also, the smooth latent space generally helps with the training and relieves the transformer from having to predict exact latent codes. # Q3 The SLT learns to predict patches that fit together almost perfectly (e.g., Figure 3), given enough training, despite never observing the decoded SDF values and potential differences between patch boundaries. We think that this is due to the rich information contained in the SDF patches (also in contrast to TSDFs), where transitions between patches are inherently smooth. We hope that this answers all your questions and concerns. We believe that your initial score is not appropriate for what we have demonstrably achieved with our method. Please reconsider it. Best regards, the authors

Dear Reviewer mpJj, thank you for reviewing our paper. Your feedback is appreciated. We address your concerns below: # W1 / Q1 The $32^3$ patch size represents a good level of detail for processing: The patch contains information about local geometry beyond just small surface patches, but not the global object. Using $16^3$ patches may still perform well, but $8^3$ patches likely provide too small-scale information to reason about effectively. Importantly, decreasing the patch size cubically increases the length of the input sequence given to the transformer. Using half-resolution patches might increase the cost by a power of $6$, since self-attention is an $O(N^2)$ operation. With our short sequence length, all patches are relevant, allowing for a cheap feed-forward transformer-encoder design using only self-attention between a small number of patches. Otherwise, more complicated methods might be needed, even requiring filtering the input and predicting potentially relevant output patches for querying a transformer-decoder. We aim to compare with a $16^3$ version in the camera-ready paper. Independent of this ablation study, we have demonstrated the effectiveness of our method and can only further validate our hyper-parameter choices. We did not increase the SDF resolution beyond $128^3$ to limit the storage and compute requirements for ground truth SDFs. # W2 We rewrote the caption of Figure 1 and the method intro to make this more clear. # W3 The problem definition of many recent works is incompatible with our method, which works on 3D patches. Comparing with their published numbers would be problematic due to the differences in the evaluated tasks. We can also not exchange inputs between models, since they operate on different resolutions and have not been trained for each other's tasks and type of data. In particular, DiffComplete train on 3D-EPN, containing partial scans, while we only consider complete patches. Also, we operate on $128^3$ SDFs, while they work with $32^3$ TSDFs and TUDFs. In order to meaningfully process their inputs, we would first have to regenerate them at higher resolution. For more recent comparisons, we already compared against AnchorFormer in Figure 3 and Table 10 (previously Table 9). and now also evaluated AutoSDF and SDFusion for all metrics to provide the metrics missing in their publications (Table 3, updated). Additionally, we compare our P-VAE against the VQDIF from ShapeFormer in Table 4 (updated, previously Table 3) and Figure 6 (updated). Our goal is to deterministically generate the best possible completion of the given input, not multi-modal shape generation. Therefore, evaluating TMD would not be meaningful. # W4 See Table 4 (updated) and Figure 6 (updated) for a comparison against the VQDIF by ShapeFormer. In contrast to our method, which processes a full $128^3$ SDF per object, their method only takes 32k surface points per object as input, limiting the expressiveness of the their model. We did already compare our P-VAE's quality against DeepSDF and 3DShape2VecSet as baselines in Tables 7 and 8 in Appendix A.4. # Q2 The current design cannot avoid SDFs leaking information about neighboring cells. We show evaluations given sparse random inputs or single octants, where the completed volume is significantly larger than the volume affected by information leaks. Even when given the bottom half, none of the detail in the top-half completion could be inferred by these leaks. Based on the high amount of detail we observed in the results far from the input, we did not consider this worthy of further investigation. # Q3 Even from partial observations, one can compute partial meshes, from which partial SDFs can be computed. These will contain some artifacts due to missing information. If one has sufficiently complete information about several $32^3$ patches, one can still feed them into the SLT, while masking out all other patches to perform shape completion. E.g., if one has particularly clean information about one side of the object, but did not fully observe the other half. If the scan is too incomplete to extract any complete patches, one could try to estimate matching complete patches from partial inputs, e.g., using the approach presented by DeepSDF. Alternatively, one could try to retrain or fine-tune the SLT on partially corrupted patches. # Q4 We have updated Table 1 to include this information. # Q5 We ran evaluations for AutoSDF and SDFusion to produce the missing metrics on the shape completion task compared in Table 3 (updated). In general, running these comparisons is problematic due to differences in problem definitions, data preprocessing, train/test splits and evaluation metrics and scales. A well-defined set of standard benchmark tasks is desperately needed to allow for meaningful evaluations in this field. We hope this addresses all of your questions and clears any concerns that resulted in a lower score. Best regards, the authors

Dear Authors, Thank you for your explanation, which cleared up my doubts. While it makes sense to use the SDF patch from the full mesh during training time, I am afraid it's an unfair comparison with the point cloud based method in empirical experiments during the test time. The author is concerned that this method is not directly applicable to real-world vision and robotics tasks as it assumes an SDF volume from a full mesh. Due to this limitation, the author is not convinced of the experimental advances in this quantitative comparison and will keep the rating. Best, Reviewer

Dear Reviewer AoP9, thank you for taking the time to review our paper. From your review, we understand that you would have liked to see advancements in the underlying machine learning methodology. Our focus lies on the application side. Based on the call for papers, which lists "applications in vision" and "representation learning for computer vision", this should be considered a relevant subject area. To establish the universal SDF-patch latent space that our SLT operates on, our P-VAE was trained on millions of randomly cropped 3D patches from ShapeNet, which we deliberately selected from various anisotropic scales and with random offsets to incorporate all topological settings. As a result, our latent space allows for autoencoding practically arbitrary 3D patches with very little loss in quality, even when containing intricate geometry. It can be meaningfully processed downstream, e.g., by the SLT, and has shown to generalize well towards ABC and even Objaverse. Our P-VAE has been custom-designed to contain skip connections only within the encoder and decoder, but not across them to compress the large 3D volume down to a small latent code. Using a standard UNet-style VAE like in the 2D image domain would primarily learn residuals and not compress the input down into a single representation. This allows the P-VAE to freely allocate latent parameters to the necessary levels of detail. We are the first method to successfully train a VAE on 3D SDF patches. Previous methods like "Local Implicit Grid Representations for 3D Scenes" explicitly shy away from this, citing "DeepSDF" as "[observing] the stochastic nature of the VAE [making] training difficult". Our results demonstrate that our P-VAE achieves significantly better autoencoding quality than previous work. We already demonstrated this in comparison with 3DShape2VecSet (Table 8) and DeepSDF (Table 7), and now further validated that against ShapeFormer (Table 4, updated). In addition to the high-quality autoencoding, our latent codes are also well-suited for 3D shape completion, as shown by the compelling results of our SLT, which even manages to produce almost seamless results without complicating the architecture with ad-hoc filtering modules. In general, we think that conceptually simple, yet effective methods like ours should be preferred to more complicated methods. Given that our method still outperforms prior work without further tweaking is a result of our deliberate design choices. It provides a solid baseline for further research down this line and thus represents meaningful progress towards better 3D shape completion. Working with SDFs is also technically challenging, since they are costly to compute and require large amounts of storage and high bandwidth for training. Existing software computing exact SDFs is prohibitively slow and would have not allowed us to process all of ShapeNet and 100k meshes from ABC. Competing (T)SDF-based methods still mostly operate at only $32^3$ input resolution (some at $64^3$) or on small subsets of ShapeNet, because they shy away from computing and processing SDFs at high resolutions from high-resolution meshes at scale. E.g., several current methods (such as DiffComplete) still rely on the now 7 years old pre-computed $32^3$ (T)SDFs from the 3D-EPN dataset. For training the SLT, we had to compute the $128^3$ ground truth SDFs for all 150k meshes. For training the P-VAE, we had to compute $32^3$ ground truth SDFs for millions of sampled patches. To facilitate this, we first implemented our own parallelized C++ code, which we have since further replaced with our own CUDA code. Both heavily utilize spatial acceleration structures to speed up the computations. Otherwise, this would have not been possible at this scale. If seen narrowly enough, every new method is just a combination of old methods. This is how we, as the research community, make progress. We hope that you reconsider your decision. Best regards, the authors

Dear Authors, I appreciate your response. After reading the response, let me summarize two key points: 1. the proposed model is trained on millions of patches from 3D shapes, which is at a larger scale than previous methods; 2. the proposed model can achieve high-resolution SDFs, 128^3 rather than 32^3 in previous methods. Therefore, the proposed method can achieve superior performance on downstream tasks, such as shape completion. Indeed, I did notice these two key points during the first round of review. However, I want to see some technical (or methodology) novelty, not just a large pre-trained 3D vision model. I acknowledged the value of the high-resolution SDF model as a strong baseline as mentioned in the response, therefore, I raised my score to 5.