Summary
The paper proposes a method for image-to-3D point cloud prediction. Building on the recent PC^2 method, which conditions the generated point cloud on the image by projecting it onto the image features using known camera parameters, the current method finds it advantageous to explicitly enforce an inductive bias that centers the generated point cloud at each step of denoising. Significant improvements are demonstrated on ShapeNet R2-N2 and Pix3D.
Strengths
- The performance improvement on ShapeNet-R2N2 is impressive. I appreciate that the authors included the oracle evaluation proposed in PC^2. On one hand, this emphasizes the variance in the reconstructions; on the other hand, I believe it makes the comparison more fair.
- The reconstruction improvement over PC^2 and DMPGen on the Pix3D dataset is especially exciting, given that this involves real data.
- The implementation of the centering concept is a straightforward addition to the previous method, PC^2, which is appealing and doesn't incur additional computational cost.
Weaknesses
- Main concern: While the improvement in results is clear and the implementation is simple, I'm currently not convinced by the argumentation. My concern is that the authors propose adding an explicit inductive bias, which assumes that all target models are zero-centered. This assumption may or may not hold for general 3D point cloud generation, depending on how the data is provided. For example, would this be a useful cue if the objects are articulated? In such cases, a folding back action could cause the legs to move, affecting the centering. I'm not convinced by the authors' analysis, motivation, and interpretation. It's crucial that if the paper is accepted based on its strong results, it should also come with a solid understanding of why the results are improving and whether the conditions for improvement will be applicable in other setups.
- Centering in Partial Shapes and Scenes: While centering can serve as an effective canonicalization for complete shapes, it may not be well-defined for partial shapes or scenes. The authors should address this issue. As an aside—while the authors present an experiment with up to 50% missing points, my understanding is that this is merely an augmentation and the ground-truth center is provided. However, this will not be the case in general when training with partial shapes.
- Test-Time Centering: The suggested method modifies both the training and testing processes, suggesting that the improvement is due to better utilization of network capacity. I wonder if this zero-centering could be applied solely as a test-time inductive bias instead? I would be very interested in seeing a comparison that includes this experiment, as it could help validate whether the improvement is indeed due to increased network capacity or the strong inductive bias.
- Clarity: The introduction states that there is a critical problem with the center point shifting during the denoising process. The first issue raised discusses the "wasted" capacity needed to map the center of the Gaussian noise to the center of the final shape. However, the authors do not explain why this mapping is necessary. The assertion that "It is crucial to ensure that the transition of the point cloud center from the initial Gaussian noise state to the final object reconstruction is appropriately managed" requires justification.
- In Section 3.2, the authors open with the statement, "Predicting the center bias is challenging for the network in the reverse process." Why is this the case? What makes it more challenging than any other network prediction? Again, I find the explanations in this section to be somewhat "hand-wavy." Some of the claims appear to require the explicit assumption that each target output is zero-centered—an assumption that is neither guaranteed nor necessarily desirable. Is there any relevance to images in this discussion? I'm not very familiar with the details of 2D diffusion, but I believe there is normalization of the output distribution. However, I don't think the output of each image is assumed to be centered.
- Center of Mass: The authors opted to canonicalize the shapes using the point mean. In cases where the density is non-uniform, this approach could lead to issues. Specifically, very similar shapes that are sampled differently might end up having different "centroids." Did the authors consider other forms of canonicalization, such as using the bounding box?
- Loss of Diversity: The authors mention that there is a sacrifice of diversity in the outputs. This claim needs to be both demonstrated and measured.
- Qualitative results on CO-3D would be nice to see. This data was used in the main baseline work PC^2.
- I’m missing comparison with a NeRF-based methods, like the recent Zero-1-to-3
- I also recommend comparison with point-e
- I don’t see the relevance of the occlusion experiment — it doesn’t seem like the method is proposing anything specific to occlusion.
Minor:
- several statements were unclear to me and would benefit further explanation. like: “each point inside the point cloud and predicted noise is independently modeled” — aren’t the points predicted jointly?
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.