Summary
Novel PointNet-based autoencoder method called KeyGrid, that predicts semantic keypoints on objects, even when objects are subject to deformations. Similar to previous approaches, keypoints are produced as a linear combination of inputs points, according to a learned weight matrix.
The key novelty over related works using PointNet++-based auto-encoders such as SkeletonMerger (SM) [26] is to pass a densely sampled grid of features (3d “point heatmap”) at each decoder layer (which are hierarchical). In this heatmap grid, each cell holds the maximum of the weighted distances to the 'keypoint skeleton segments'.
Also, KeyGrid appears to pass richer encoder information to the decoder than the SM work, although the differences here are less clear.
The method is shown to outperform previous baselines such as SK3D and SM on ShapeNetV2 dataset (more right objects) and on ClothesNet (deformable objects).
Strengths
- Beats KeypointDeformer (KD), SkeletonMerger (SM), SC3K on almost all categories on the ClothesNet dataset and ShapeNetCoreV2. Deformable objects keypoints seem to improve more.
- Ablations in Tab. 4 indicate the importance on 4 (among dozens) of semantic classes how encoder-info, a grid heatmap, farthest point sampling loss, and similarity (i.e. chamfer loss) each boost performance.
Weaknesses
# Significance
- Is the hierarchical decoder setup (excluding the heatmap grid itself) novel, esp compared to SkeletonMerger [26]? Right now there are no specific claims or ablations related to this.
- The primary novelty relative to SM [26] seems to be the use of keypoint grid heatmaps in the decoder, storing max distances from grid cells to the weighted keypoint segments. However, this is just one possible shape descriptor feature, and intuitively it should be quite a weak and ad hoc feature, even if the authors gave some intuition why it may be better than the min function. Would other shape feature with yet more information be more useful could (e.g. why not have K channels with distances to all segments or to the keypoints as opposed to just taking the max, etc)? In general it would help if the paper provided more intuition or experimental results giving more insight on the information contained in this heatmap.
- This class of approaches seems generally limited to cases where entire point clouds were visible. How would you handle cases when some parts of the objects were occluded, such as in real-world scenes? This has not beed discussed.
# Related work
Could discuss other dense representations for correspondence like the pointmaps used in “DUSt3R: Geometric 3D Vision Made Easy” https://arxiv.org/pdf/2312.14132 )
# Clarity
- I found Sec 3.3 particularly difficult to follow and understand. A rewrite, or further details in the appendix clarifying better how these quantities are 'composed' and aligned would help.
- Fig 1 illustrates points heatmap in the decoder by simply showing a point cloud. I was expecting pictures like in Fig 4c.
- L13, L58 Unclear where keypoint pairs come from, when discussed for first time.
- No discussion of how correspondences are established, e.g. as shown in Figure 3.
- The loss terms in Equation 11 are not mathematically defined, would be helpful in the Appendix.
# Experimental results
- KeypointDeformer, SkeletonMerger, and SC3K are tested on KeypointNet dataset (https://arxiv.org/pdf/2002.12687 ). Why do the authors omit it here? Other works such as SC3K state “We use KeypointNet dataset [36 ] in our experiments, considering that this is the standard and most recent dataset used for keypoints estimation”. If the results are not as good on that dataset, that is okay, as long as we provide them and state that our method is specifically designed for datasets with deformable/soft objects. This would provide confidence that the authors ran their eval of official source code correctly on the two new datasets.
- Why no comparisons with supervised baselines?
- Figure 4D: Sun3D keypoints are so small in the figure it appears as if they are randomly distributed
# Some Grammar & Style Nits:
- L453 SC3K bibtex should be ICCV 2023, not Arxiv “
- L148 typo “giving a total of 4096 gird points.” -> “…4096 grid points”
- L229 “mean Intersection over Unions” -> “mean Intersection over Union”
- Grammar: Figure 3: caption: “Different methods on the Hat and Long Pant during” -> prefer “Different methods on the Hat and Long Pant categories during”
- Table 3 caption is too close to Table 3, whitespace seems too shrunken
- L75 “Deformable object dataset” -> “Deformable object datasets”
- Table 1 caption “codes” -> should be singular, not plural (“the results we reproduced based on their official codes” -> “the results we reproduced based on their official code.”)
- Color scheme is jarring in Tables 1-4, prefer tango colors that have a more attractive colormap (https://sobac.com/sobac/tangocolors.htm)
- L3 extraneous article: “focus on the rigid body objects” -> “focus on rigid body objects”
- L6 extraneous article: “for both the rigid-body and deformable objects” -> “for both rigid body and deformable objects”
- L300 “the importance of encoder information and grid heatmap for reconstruction process,” -> “...for the reconstruction process,”
- L302: “Table 4 show that the keypoints detected by Key-Grid which utilizes both two strategies to reconstruct the point cloud have better semantic consistency.” -> “Table 4 shows that when using both input streams to reconstruct the point cloud, the keypoints detected by Key-Grid exhibit better semantic consistency”
- L9-10 Instead of “Unlike previous work, we leverage the identified keypoint information to form a 3D grid feature heatmap 1called grid heatmap, which is used in the decoder section”, prefer to say something like “..to form a 3D grid feature heatmap which we refer to as a grid heatmap. A grid heatmap is…”
- L14 “Into the decoder section” -> “into the decoder model”? Ambiguous what “section” refers to
- L51 “aiming at the semantic consistency” -> “aiming for semantic consistency”, e.g. “an unsupervised keypoint detector on 3D point clouds aiming for semantic consistency under shape variations of both rigid-body and deformable objects.”
- L74 section should be named “Related Work” not “Related Works”
Questions
- Compared to SM[26], is the keypoint heatmap the main difference, how about the use of hierarchical decoder.
- Wouldn't a richer shape descriptor heatmap do even better than a max function? (e.g. K channels, distances to the keypoints, etc).
- How do you get keypoint correspondences across shapes for your model, do they emerge naturally?
- What are the method results on KeypointNet dataset? Can you provide some additional comparisons to supervised baselines on your datasets, if there are such?
- Can this method be applied on datasets where shapes are significantly occluded (e.g. viewed from a specific direction)?
Limitations
Limitations have been sufficiently addressed.