Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images

Summary

Authors propose MHCDIFF (Multi-hypotheses Conditioned Point Cloud Diffusion) to reconstruct 3D human point could from a single image under occlusion. The key idea behind MHCDIFF is the smart use of image projection features and features from multiple SMPL hypothesis, coupled with point cloud diffusion. The method achieves favourable reconstruction performance on CAPE, MultiHuman, and Hi4D datasets.

Strengths

+ The proposed idea to leverage information from multiple SMPL hypothesis is simple and intuitive. + The ablation studies are detailed and show the significance of each component of MHCDIFF.

Weaknesses

Some key details are unclear: - L222: How are SMPL meshes sampled? - Eq. 5: Why use mean occupancy and not mean distance? Is there an advantage of using a combination of distance and occupancy? Experiments: - Authors mostly evaluate their method on in door datasets with controlled settings. What about more “in the wild” data like 3DPW, MS COCO. - Is there any correlation between the SMPL hypothesis and the reconstruction accuracy? How accurate do the hypothesis have to be? Does number of SMPL sampled matter? Does the global translation of the SMPL matter? Minor: - L180: shouldn’t it be argmin if we want to compute distance to closest SMPL mesh?

Questions

Overall I'm positive about the work. It is interesting to see that adding basic SMPL features (average occupancy and distance to closest SMPL point) to the PC2 formulation is a useful combination. The quantitative results also look good. I'm a bit unclear on some experimental and design details as stated above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors discuss limitations and broader impact in thier manuscript.

Summary

The target of this paper is to reconstruct 3D clothed human shape. The conditional point clouds diffusion model is adopted as the main structure of the proposed reconstruction model. This work focuses on designing effective conditioning features, especially for overcoming the occlusions. The conditioning features include the similar projected image features as PC2[55], the local features [signed distance, normal vector, occupancy] extracted from multiple estimated SMPL or SMPL-X human 3D parametric models. The performance of the proposed model is evaluated on synthetic and real datasets, and compared with different SOTA methods. The experimental results show good de-occlusion ability in 3D clothed human shape reconstruction.

Strengths

1. The paper is well written and easy to understand. 2. It is the first work that extends the multi-hypotheses SMPL estimation to pixel-aligned 3D clothed human reconstruction. 3. The experimental results show good de-occlusion ability when reconstructing 3D clothed human 3D shape.

Weaknesses

1. Low reconstruction quality of details, such as hands and feet. 2. The analysis and description of the experimental results are not clear and in-depth enough. For specific details, please refer to the following questions.

Questions

1. Multi-hypotheses was proposed in [8] to model the uncertainty caused by occlusions or depth ambiguities in 3D shape reconstruction. What are the differences between your work and [8] on multi-hypotheses? 2. If one person wears different loose clothes with the same pose, can the proposed model reconstruct their 3D shapes effectively? It should be helpful if the related experimental examples are presented. 3. Table 1 shows that the Chamfer Distances of HiLo [92] and SIFU [100] are 13.711cm and 13.397cm, while the result of the proposed method is 1.872cm, a significant improvement. My question is, both HiLo and SIFU also use SMPL model, which can provide good human 3D shape and pose prior, what is the reason for their poor results? 4. Table 3 shows the results of ablation study. It indicates the Chamfer Distance decreases from 3.640cm (baseline PC2[55]) to 1.872cm (proposed MHCDIFF), totally 17.68mm improvement. It seems that the improvement should be contributed by the added conditioning features extracted from multiple SMPL. But from group A, the results show that each conditioning feature has small contributes, such as ‘w/o occupancy’, the value is 1.893cm vs. 1.872cm, only 0.21mm improvement. I am a bit confusing. A clearer explanation would be better. 5. From Table 3, group B, it seems that using SMPL model (conditioned on single ProPose estimation) is better than using SMPL-X model (conditioned on PIXIE estimation). If it is true, please analyze the reasons. 6. From the Qualitative results (Figure 3, Figure 5), it can be seen that the reconstruction qualities of some parts are not good, such as hands, feet, and so on. Please analyze the reasons. 7. The multi-hypotheses conditioning can take an arbitrary number of SMPL, SMPL-X, and their combined. What is the optimal recommendation?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

As the paper states that proposed method has low efficiency caused by diffusion model, which has limited applications.

Summary

This paper introduces method for reconstructing detailed 3D human shapes from single occluded RGB images. The key contributions are: - A point cloud diffusion model conditioned on projected 2D image features and local features from multiple SMPL mesh hypotheses. - A multi-hypotheses conditioning mechanism that extracts and aggregates local features from multiple plausible SMPL meshes to handle uncertainty due to occlusions. - Training on synthetically occluded images to improve robustness to real-world occlusions. The method is evaluated on the CAPE dataset with synthetic occlusions and MultiHuman dataset with real interactions, demonstrating state-of-the-art performance in reconstructing detailed 3D human shapes from occluded views.

Strengths

- **Novel approach**: The combination of point cloud diffusion with multi-hypotheses SMPL conditioning (from ProPose) is an innovative solution to the problem of 3D human reconstruction from occluded images. - **Robust to occlusions**: Results demonstrate significant improvement over prior work on occluded inputs. - **Detailed reconstructions**: The approach can recover fine geometric details like clothing directly on point clouds, going beyond parametric body models. - **Thorough evaluation**: Experiments on both synthetic (CAPE) and real (MultiHuman) datasets with varying levels of occlusion provide a comprehensive assessment.

Weaknesses

- **Limited Dataset Diversity**: While results on CAPE and MultiHuman are strong, evaluation on additional datasets would further demonstrate generalization. In particular, testing on datasets with more diverse clothing styles and body shapes would be valuable. Suggested datasets include: - THuman2.0: Contains 2500 high-quality human scans with various poses and clothing styles. - AGORA: A synthetic dataset with high realism, featuring multiple people per image. - DeepFashion-MultiModal: Includes high-resolution images with rich annotations for clothing shapes and textures. - **Limited practical applicability**: The method's output is a point cloud, which may not be directly usable in many real-world applications that require mesh representations. While the authors mention the possibility of converting the point cloud to a mesh using Poisson surface reconstruction, this process is computationally expensive (taking about 10 hours per sample) and is not integrated into the main pipeline. This limitation significantly reduces the method's immediate practical utility in applications requiring real-time or near-real-time 3D human reconstruction. - **Texture and color reconstruction and real world usage**: Unlike methods like PHORHUM, this approach does not address texture or color reconstruction, which limits its applicability in photorealistic avatar creation.

Questions

- How does MHCDIFF compare to recent methods like PIFuHD and PHORHUM in terms of reconstruction quality and handling of occlusions? - Have the authors explored more efficient sampling techniques like DDIM to reduce inference time? What speedups might be achievable? - How well does the method generalize to different clothing styles or body shapes not seen during training? - Could the approach be extended to include texture and color reconstruction, similar to PHORHUM? - How does the method perform on real-world, in-the-wild images with multiple humans and complex interactions? Have you considered evaluating on datasets like AGORA or DeepFashion-MultiModal?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. The authors do acknowledge the computational cost as a key limitation.

Summary

This paper investigates the problem of improving the robustness of occluded 3D human reconstruction from a single image. The idea is to achieve better stability under occlusion via two steps: 1. refining pixel-aligned local image feature extraction part with the help of multi-hypothesis human pose and shape estimation; 2. inpainting the invisible body part via a point cloud diffusion process. Experiments on CAPE and MultiHuman datasets show the superior of the proposed method.

Strengths

1. The ablation study in Tab. 3 reveals the effectiveness of each designs. 2. Using point cloud diffusion process to inpainting the invisible part seems reasonable.

Weaknesses

1. Composing the multi-hypothesis human pose and shap with point cloud diffusion seems incremental. The paper lacks solid insight to inspire the readers. 2. The quantitative results on CAPE and MultiHuman don't show obvious improvement, compared with previous methods. The CAPE dataset is processed to "randomly mask the images about 40% in average". While on MultiHuman without any masking, the performance is quite similar to the baselone model. 3. More qualitative results would be helpful to determine the superior of the proposed method.

Questions

What is the key insight behind the two designs presented in this paper? The experiment results on CAPE and MultiHuman may need further explanation.

Rating

3

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

No.

Post rebuttal update

Thanks authors for the rebuttal. It helped clarify my doubts. I don't know why authors say AGORA dataset is not free in reply to reviewer 1vtm. To my understanding it a synthetic dataset available to download. I will keep my rating to 'borderline accept'.