Whole-body pose and shape estimation aims to jointly predict different behaviors (e.g., pose, hand gesture, facial expression) of the entire human body from a monocular image. Existing methods often exhibit degraded performance under the complexity of in-the-wild scenarios. We argue that the accuracy and reliability of these models are significantly affected by the quality of the predicted \textit{bounding box}, e.g., the scale and alignment of body parts. The natural discrepancy between the ideal bounding box annotations and model detection results is particularly detrimental to the performance of whole-body pose and shape estimation. In this paper, we propose a novel framework to enhance the robustness of whole-body pose and shape estimation. Our framework incorporates three new modules to address the above challenges from three perspectives: \textbf{1) Localization Module} enhances the model's awareness of the subject's location and semantics within the image space. \textbf{2) Contrastive Feature Extraction Module} encourages the model to be invariant to robust augmentations by incorporating contrastive loss with dedicated positive samples. \textbf{3) Pixel Alignment Module} ensures the reprojected mesh from the predicted camera and body model parameters are accurate and pixel-aligned. We perform comprehensive experiments to demonstrate the effectiveness of our proposed framework on body, hands, face and whole-body benchmarks. Codebase is available at \url{https://github.com/robosmplx/robosmplx}.
Paper
Similar papers
Peer review
Summary
The paper focuses on improving whole-body pose and shape estimation from monocular images, a task that often struggles with complex, real-world scenarios. The authors argue that the performance of these models is significantly impacted by the quality of the predicted bounding box, such as the scale and alignment of body parts. The difference between ideal bounding box annotations and model detection results poses a substantial challenge to whole-body pose and shape estimation. To address this, the authors introduce RoboSMPLX, a framework to enhance the robustness of whole-body pose and shape estimation. RoboSMPLX integrates three new modules: a Localization Module to improve model awareness of the subject's location and semantics within the image space, a Contrastive Feature Extraction Module that uses contrastive loss with dedicated positive samples to help the model be invariant to robust augmentations, and a Pixel Alignment Module to ensure the reprojected mesh from predicted camera and body model parameters are accurate and pixel-aligned. The effectiveness of the method is showcased through comprehensive experiments on body, hands, face, and whole-body benchmarks.
Strengths
The paper is well-structured and reader-friendly. It begins with a compelling motivation that efficiently highlights the issue of the current state-of-the-art methods' low robustness against varying crops and misalignments. The authors also convincingly justify potential solutions to these problems. The proposed method by the authors is sound even if it lacks complete novelty. Impressively, the authors have performed extensive experiments and provided an in-depth ablation study, substantiating their proposed approach effectively. Although their method doesn't top all the benchmarks, its efficacy is undeniable, particularly in its demonstrated robustness to variations in crop size and alignment - a challenge that other methods fail to meet. The authors provide a thorough justification with robust qualitative and quantitative results.
Weaknesses
The claim made in lines 47-51 and 76-77, suggesting that contrastive learning is not used in the parametric estimation of human meshes, is inaccurate. Previous works, such as [1], have already incorporated a form of contrastive and triplet loss for 3D face shape and pose estimation from monocular face images. Thus, the application of contrastive loss for mesh recovery from monocular images isn't entirely novel. The authors should moderate their claims and include a thorough discussion concerning related work in this field. Furthermore, the pixel alignment loss achieved by projecting the mesh as a mask, a technique used by ICON [2] for improved SMPL pose parameter estimation, is not entirely new. The authors should also include instances of the method's failure during testing within the main body of the paper. [1] Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision. CVPR 2019 [2] ICON: Implicit Clothed humans Obtained from Normals. CVPR 2022
Questions
The paper seems sound to me and I do not have much questions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Although some limitations have been discussed it might not be adequate. Please add some visual limitations of the model during the inference time.
Summary
This paper addresses the task of whole-body pose and shape estimation, including human mesh, hand gestures, and facial expressions, from monocular images. The author identifies the impact of predicted bounding box quality on the accuracy and reliability of existing methods. Based on this observation, a novel framework called RoboSMPLX is proposed to enhance robustness through three modules: a localization module, a contrastive feature extraction module, and a pixel alignment module. Comprehensive experiments are conducted to demonstrate the effectiveness of the framework.
Strengths
1. Overall, the paper is of high quality, with clear motivation and well-organized sections. It is easy to follow and presents novel modules that address existing limitations. The paper includes comprehensive experiments and provides sufficient visualizations of the generated mesh. The author's contributions are highly appreciated. 2. The empirical study on the impact of subject localization, feature extraction, and pixel alignment, as stated in lines 91-100, is crucial in breaking through barriers. The paper provides extensive qualitative and quantitative results, both in the main paper and supplementary material, making it solid and convincing. 3. The three modules work together effectively, aligning intermediate representations such as pose, dense landmarks, segmentation, and silhouette masks. This enhances robustness through proper data augmentation techniques. 4. The proposed framework consistently achieves impressive results for various tasks across multiple datasets, demonstrating superior performance in whole-body pose and shape estimation.
Weaknesses
1. Since the author does not mention code release, it would be beneficial to release the code to contribute to the research community. 2. The paper lacks a discussion on computational complexity and inference speed, which are important evaluation metrics, especially for practical deployment. I suggest reporting total parameters, FLOPs, and fps with a detailed discussion to strengthen the paper.
Questions
See weakness
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
1. Although the proposed framework achieves impressive performance, the overall framework and training strategy is somewhat complex. Simplifying the framework without sacrificing performance would be beneficial. 2. Currently, the proposed framework only allows for whole-body estimation from a single image. Future work could focus on video-based estimation to further improve robustness, alleviate depth ambiguity, and enhance temporal smoothness.
Summary
The paper introduces RoboSMPLX, a method for whole-body 3d human pose and shape estimation from a monocular image. Motivated by the poor robustness of existing methods, especially w.r.t. the quality of bounding boxes, three components are proposed: 1) a localization module, 2) a contrastive feature extraction module, and 3) a pixel alignment module. The robustness of existing methods is evaluated by applying different types of image, location and pose augmentations. By using these augmentations in a contrastive setting during training of RoboSMPLX, RoboSMPLX achieves higher robustness when evaluated on these augmentations. Additionally, the performance of Hand, Body, Face and Wholebody reconstructions are evaluated on typical benchmark datasets.
Strengths
Increasing the robustness of whole-body human pose and shape estimation methods is an important topic. The problem is well motivated, and the evaluation of the performance of existing methods under different augmentations is interesting.
Weaknesses
I have the following main concerns about the paper: 1) Evaluation seems inconsistent and incomplete. It is unclear why some competitors are omitted in different experiments. For example, the performance of PyMAF[54] is reported in Table 1 and 2, but not in Table 3 and 6, even though [54] reports the relevant numbers. OSX[25] is also missing in Table 2. The competitors outperform RoboSMPLX in the experiments where they are omitted. 2) Robustness of RoboSMPLX. The robustness evaluation seems not realistic. It is not surprising that RoboSMPLX shows better performance under the different augmentations the method is trained on. However, most relevant is the performance on in-the-wild scenarios. This is not properly evaluated for the whole-body task. On the contrary, RoboSMPLX performs worse on AGORA than OSX. Although AGORA is a synthetic dataset, it contains realistic and diverse scenes with multiple persons and occlusions. On the other hand, EHF only consists of 100 images of a single subject recorded in a mocap studio. It is therefore questionable, if RoboSMPLX really succeeds in being more robust than its competitors. To better assess the in-the-wild performance, the methods could for example be evaluated on RICH (CVPR'22) (https://rich.is.tue.mpg.de/) or on BEDLAM (CVPR'23) (https://bedlam.is.tue.mpg.de/). Additionally, to show the effectiveness of the localization module, the accuracy of the predicted bounding boxes could be evaluated. Futher concerns: - The contrastive module is not well motivated. Why use the contrastive setting with all its overhead, instead of simply using the different transformations as data augmentation? Especially since adding positive samples only has small influence. - Why are the ablation studies in Table 11 not conducted with the whole body model? Mean+std of multiple runs should be reported since the differences are so small. - Why is the pixel alignment not evaluated quantitatively? This would be more meaningful instead of only showing some examples e.g. in Figure 13. - The experiments section should be revised. It is difficult to read due to the many experiments and referrals to the appendix. - The novelty is limited since the augmentations and the pixel alignment module already have been used extensively in the literature for estimating human pose and shape, e.g. [35] and [45]. - IIn Table 10 7.18 PA-PVE should be in bold, and 14.38 MPJPE in Table 11.
Questions
The following requires clarification: - What is the estimation error reported in Table8? - Table 11: what does DR54 and KS stand for? What is the row named "joints"? - Regarding line 204: why should the model produce consistent representations for the same subject under different poses? - For how long is the model trained on what hardware? What's the inference time? - How are the joints for the body, hand and face determined? How does the 137 joints body skeleton look like?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
More failure cases could be shown in the paper. Also in comparisons to the competitors. The potential societal impact is not discussed.
Summary
This paper proposes a method to improve the robustness of whole-body pose and shape estimation, which mainly contains three components: 1) localization module to give the network awareness of location and semantic part; 2) contrastive feature extraction module to predict consistent representations under different augmentations; 3) pixel-alignment module to ensure alignment between projected mesh and 2d evidence.
Strengths
1) This paper is well-written and easy to understand 2) The topic of robustness of pose and shape estimation is meaningful and the proposed method is effective against augmentations 3) The experiments are comprehensive and visualizations are very nice to help understand the method
Weaknesses
1) Since the main topic is the robustness of whole body pose and shape estimation, the literature review of robustness in vision tasks, especially in pose estimation tasks should be included. For example, [1][2]. [1] Bai, Yutong, et al. "CoKe: Contrastive Learning for Robust Keypoint Detection." [2] Zhang, Yumeng, et al. "Improving robustness for pose estimation via stable heatmap regression." 2) I wonder why the target task is whole-body pose and shape estimation rather than body/hand pose estimation? 3) Why does ‘baseline’ only appear in table2 but not in table1 and 3? Also, the definition of baseline is not clear. The definition of baseline is not clear either in table 9 4) Where is the result of robustness of body subnetwork again augmentations? 5) Where is the details of table 7? 6) Eq,2 typo
Questions
see weaknesses
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
see weaknesses
Summary
This submission proposes RoboSMPLX for whole-body pose and shape estimation. RoboSMPLX incorporates three modules, including a localization module, a contrastive feature extraction module, and a pixel alignment module. The localization module is aware of the location and semantics of body parts so that cropping could be more accurate. The contrastive feature extraction module incorporates a pose- and shape-aware contrastive loss, along with positive samples, for better feature extraction under robust augmentations. The pixel alignment module applies differentiable rendering to inherence of the re-projection alignment of the mesh.
Strengths
The motivation of the proposed method is clear: using a localization module to improve the robustness of the cropping. Overall, the proposed method technically makes sense and can be easily reproduced. The experiments also show comparable or better performances with previous methods.
Weaknesses
The most severe weakness is the novelty of the proposed method and the lack of comparisons with recent state-of-the-art solutions. To be more specific, there are several major issues: - The proposed localization and pixel alignment modules have very limited contributions to the community, as these operations are commonly used in this field. For instance, similar localization strategies are used in [39,56], and differentiable modules are used in [i]. [i] SK Dwivedi, N Athanasiou, M Kocabas, MJ Black, Learning to regress bodies from images using differentiable semantic rendering, ICCV 2021. - There is a lack of discussion and comparison with the recent state-of-the-art method PyMAF-X [54]. An in-depth discussion and comparison with [54] is necessary to support the claim of the proposed method. It is recommended to include [54] in the Related Work section, compare results with [54] in Tables 3,4,5, and Table 7, and show qualitative results of the proposed method and [54] for compressive comparisons. - This paper claims robust performances of whole-body pose and shape estimation, but no video result is provided in the supplementary materials. To convincingly demonstrate the robustness of the proposed method, it is also recommended to include a side-by-side comparison video [54].
Questions
How about the run time of the proposed method?
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The limitations of this paper are mainly the novelty and the lack of comprehensive comparisons with recent state-of-the-art solutions. Given such clear defects in the experimental results, I rate this paper as the one below the acceptance bar.
Clarification of Q2
Dear author, I want to have further clarification on my second question. My question is why the target task is whole-body pose and shape estimation rather than body/hand pose and shape estimation. Let's call this task mesh recovery. I understand that mesh recovery tasks are very different from 2D/3D keypoint detection. My question is usually whole-body is an advanced task compared with body or hand mesh recovery and there are many more existing works focused on body or hand mesh recovery than whole-body. Also, the challenges of whole-body mesh recovery are usually low resolution of hands or the rotation of the wrist. When coming to robustness, whole-body tasks seems the same as purely body or hand tasks for me. Therefore, it would be more meaningful to first study the robustness of body mesh recovery since there are many more competitive works to compare to show the superiority of your method. The proposed method which should be applicable to all mesh recovery tasks also seems less convincing in this way.
Response to Clarification of Q2
Dear reviewer, Thank you for raising valid concerns about our choice of focusing on whole-body mesh recovery. Below we provide explanations that address your queries. **Q1: why the target task is whole-body pose and shape estimation rather than body/hand pose and shape estimation** **A1:** Compared to body mesh recovery, **whole-body mesh recovery is a more prevalent and important task which offers broader applications**. It allows for the accurate modeling of hand gestures and facial expressions, making it particularly advantageous for intricate tasks such as clothed human reconstruction, editing, and animations that utilize SMPL-X predictions. **Q2: the challenges of whole-body mesh recovery are usually low resolution of hands or the rotation of the wrist. When coming to robustness, whole-body tasks seems the same as purely body or hand tasks for me** **A2:** While we acknowledge that whole-body mesh recovery inherently encapsulates the challenges of individual body parts like hands, face, and the body itself, **whole-body mesh recovery presents unique robustness problems, different from body/hand mesh recovery.** Beyond the common challenges like the “low resolution of hands [25] or rotation of wrist [32]”, **the accuracy of face and hand part crops influencing their respective subnetworks is a significant factor.** Two main concerns are the inaccurate localization of part crops by the wholebody network and the robustness of hand and face subnetworks when handling inaccurate part crops. Our approach addresses both these challenges. It's worth noting that in dedicated hand or face-mesh recovery, images primarily showcase the hand or face at the center. In contrast, this is often not the case in whole body estimation (part crops of existing methods are illustrated in Figure 2). The robustness issue is more pronounced and relevant in whole-body mesh recovery. This motivates our study to improve the robustness in every subnetwork of the wholebody pipeline. **Q3: it would be more meaningful to first study the robustness of body mesh recovery since there are many more competitive works to compare to show the superiority of your method** **A3: ** **We demonstrate the efficacy of our proposed modules, namely Localization, Contrastive Feature Extraction, and Pixel Alignment on body mesh recovery** (Table 2, Figure 15). In addition, the modules are also effective for face (Table 3, Table 5, Figure 15) and hand mesh recovery (Table 1, Table 4, Figure 14). Their performance outpaces existing methods (as seen in Tables 1, 2, 3), and they exhibit robustness under various positional augmentations (evidenced by Table 4 and 5). Qualitative demonstrations of this can be found in Figures 14 and 15. **Existing solutions on robustness for body/hand mesh recovery do not work as well.** Previous efforts for robustness in body/hand mesh recovery often relied heavily on data augmentation, leading to domain shifts as seen in [15, 19, 35]. Notably, some found that excessive crop augmentation can degrade benchmark performance (e.g., Tables 4 of PARE [19] and HMR-EFT [15], and Table 8 of [35]). Contrarily, our method leverages supervised contrastive learning, proven to effectively mitigate errors while maintaining robustness. We hope this helps to clarify why we choose whole body mesh recovery instead of body/hand mesh recovery. We are happy to answer further questions.
Dear authors, Thank you for your explanation. Tables are evaluating the performance of body, hand, and face separately against sota whole-body methods. However, the performance on body subnetwork is not very competitive against state-of-the-art body networks, e.g. HybrIK, CLIFF. This makes me doubt whether the gain mainly comes from the more powerful network or robustness strategies. To evaluate the robustness of body-only methods, one can still input inaccurate bounding boxes.
Follow-up 2
Dear reviewer, We will polish the paper and add the clarifications below in the revised version. **Q1: However, the performance on body subnetwork is not very competitive against state-of-the-art body networks, e.g. HybrIK, CLIFF.** Table 1. Evaluation of HybrIK, CLIFF and our network on 3DPW. Our results are also available in Table 2 in main paper. | Method | Backbone | F-T on 3DPW | PA-MPJPE (3DPW) | MPJPE (3DPW) | | ------ | --------- | ----------------- | --------------- | ------------ | | HybrIK | HRNet-W48 | No | 48.6 | 88.0 | | HybrIK | HRNet-W48 | Yes | 41.8 | 71.3 | | CLIFF | Res-50 | Trained with 3DPW | 45.7 | 72.0 | | CLIFF | HRNet-W48 | Trained with 3DPW | 43.0 | 69.0 | | Ours | Resnet-50 | No | 49.8 | 80.8 | | Ours | HRNet-W48 | No | 48.5 | 80.1 | **A1:** There are many factors affecting training,including, but not limited to, the choice of backbone, datasets employed, and specific protocols executed during evaluation. Specifically, with regards to 3DPW, various protocols—ranging from fine-tuning (3DPW Protocol 1), collective training, to omission during training (3DPW Protocol 2)—have a large influence on 3DPW results in the evaluation process. We outperform HybrIK when using the same backbone (HRNet-W48) and not fine-tuning on 3DPW (3DPW Protocol 2). Notably, CLIFF incorporated 3DPW within its training datasets. Given that our approach and that of both HybrIK and CLIFF do not utilize identical dataset combinations, a direct comparison becomes inherently challenging. **Q2: This makes me doubt whether the gain mainly comes from the more powerful network or robustness strategies.** **A2:** For thorough ablation, we presented baselines in Table 2 in the main paper. The baseline is training on the same dataset combination, backbone, and employing consistent training strategies. It's evident that the introduction of our modules improves performance while keeping all other factors (backbone, dataset) the same. Additionally, we provide ablation studies on the hand subnetwork for different modules in Table 11 to showcase their effectiveness when the same backbone and dataset choice is maintained. **Q3: To evaluate the robustness of body-only methods, one can still input inaccurate bounding boxes.** **A3:** We have provided qualitative comparisons of body-only methods under different scale and alignment in Figure 15. Below, we provide quantitative evaluations of our method with HMR, SPIN and PARE. Our method is able to achieve better performance under different scales and alignment. In the future, we will make our robustness evaluation pipeline publicly available for other methods to evaluate on. We believe this can serve as a valuable benchmark for future studies in this domain. Table 2. Evaluated on 3DPW (PA-MPJPE/MPJPE) under different scales and alignment. * denote the same dataset combination | | Normal | Transx +0.2x | Transx -0.2x | Transy +0.2y | Transy -0.2y | Scale 1.3x | Scale 0.7x | | --------------- | ------------ | ------------ | ------------- | ------------- | ------------ | ------------- | ------------- | | HMR | 67.53/112.34 | 77.31/141.70 | 77.06/ 138.51 | 86.57/ 151.15 | 77.26/148.33 | 68.46/ 117.1 | 75.38/ 124.79 | | SPIN | 57.54/94.11 | 70.14/122.56 | 68.67/ 120.04 | 73.08/ 111.33 | 70.64/133.2 | 61.08/ 103.60 | 61.63/ 99.6 | | PARE (HR32) \* | **49.3/81.8** | 74.9/139.2 | 77.1/ 141.7 | 59.1/92.3 | 64.2/ 109.7 | 54.7/86.9 | **50.5/ 83.9** | | Ours (R50) \* | 49.8/80.8 | **67.2/117.2** | **67.7/111.5** | **56.4/90.0** | **62.8/105.6** | **50.2/84.6** | 50.8/ 82.4 | We are happy to answer further questions.
Follow-up
Dear reviewer, We would like to follow up to check if your concerns have been addressed. In the previous response, we have made the following updates/clarification: - Regarding your concern on the novelty of the proposed localization and pixel alignment modules (Q1), we clarified the uniqueness of our modules in contrast to other works. - Regarding your precious advice to compare to PyMAF-X (Q2), we discussed and compared our approach with PyMAF-X in detail and plan to add it to the "Related Work" section. - Regarding your advice to include a side-by-side comparison video (Q3), we've uploaded comparison videos per your advice (Q3). Notably, PyMAF-X's performance is influenced by OpenPifpaf's predictions, evident in `pymafx_openpifpaf.mp4`. We observed improvements with a more robust pose estimator in `pymafx_mmpose.mp4`. Our model's results are in `robosmplx.mp4`. - Regarding the run time of the proposed methods (Q4), we have addressed it in General Concerns 2. We are happy to answer further questions.
Follow-up Comments
Dear authors, Thanks for the responses. I have read the rebuttal and comments from other reviewers. The additional comparisons indeed strengthen the experiments. However, my major concern about the novelty remains. The authors clarified the differences between the proposed method and previous works. But from my point of view, the novelty of these modifications is limited, making this paper hard to meet the bar of a NeurIPS paper. From the additional results provided by the authors, it seems that the hand and body box detection has a great impact on the final results. The robustness of previous whole-body solutions can be also improved with a better hand/body detector. As for the proposed method, the major improvements come from the body subnetwork, which is trained additionally and provides better hand/face detection for the re-cropping of hand/face. As pointed out in the first-round review, similar localization modules have been explored in previous work. These localization modules are indeed improved by this work but the contribution is not significant. Given the additional results provided in the rebuttal, I would only slightly raise my score but keep it negative.
Follow-up 2
Dear reviewer, We appreciate the adjustment in your scoring. Regarding your concerns about the limited novelty of our approach, we provide additional clarifications below. While the body subnetwork indeed provides better hand/face detection, it isn’t the primary advancement. Our hand and face subnetworks are more robust even when the inputs are imprecise. This is demonstrated in Tables 4 and 5, and in Figures 14 and 15, where in instances of inaccurate crops, our method consistently outperforms existing techniques. Furthermore, It's crucial to highlight that the localization module is not the sole contribution for the improved robustness of each network. We have two other modules - contrastive feature extraction module and pixel alignment module, both of which significantly contribute to the overall robustness of the network. We hope that you can also consider the merits of the other two modules when evaluating our approach. We are happy to answer further questions.
Follow-up Comments
Dear authors, Thank you for providing more clarifications. Despite the novelty, the evaluation of the proposed method remains insufficient from my point of view. Actually, I have pointed out the insufficient evaluation of Tables 3,4,5, and Table 7 in the first-round review. These tables are not updated in the rebuttal due to the lack of the pre-trained hand and face models of previous methods or the unfair settings. I do not agree with that as the subnetwork is typically contained in the whole-body network for whole-body methods. One can still evaluate the subnetworks and report results by feeding face/hand images to the subnetworks (please correct me if it is impossible). As for the unfair settings of whole-body evaluation, I would recommend reporting comprehensive results to Tables as possible as we can and explaining the performance gap and in-depth reasons in the paper. In this way, it could be clearer for readers to have more comprehensive comparisons regarding the performances of the proposed and recent solutions. Insufficient evaluations in benchmark Tables would indicate that the results are selective and not comprehensive/state-of-the-art in my opinion, especially when there is a lack of recent state-of-the-art methods.
Follow-up 3
Dear Reviewer, We appreciate the valuable feedback. Regarding your concerns, we provide additional clarifications below. **Q1: I have pointed out the insufficient evaluation of Tables 3,4,5, and Table 7 in the first-round review. These tables are not updated in the rebuttal due to the lack of the pre-trained hand and face models of previous methods or the unfair settings.”** **A1:** Regarding Tables 3, 4, 5, and 7 Evaluation: 1. **Table 3:** As previously highlighted, the VGG Face2 training dataset for face is no longer available to the public. While we can incorporate PyMAF’s face results into Table 3, we have to make a caveat that these results cannot be replicated. 2. **Tables 4 & 5**: These tables specifically evaluate the hand and face subnetwork under different positional augmentation on hand and face specific datasets. As we rely on pre-trained subnetworks, we will update the results once the hand and face subnetworks of PyMAF-X, or their respective training codes become publicly available. 3. **Table 7**: As mentioned in Follow-up 1, PyMAF-X was excluded from the whole body evaluation under different scale and alignment in Table 7 **as they used ground-truth hand and face bounding boxes from the EHF test set**. Beyond the fairness concern, we believe it is not meaningful to include them as the errors will be invariant across positional alignments. **Q2: I do not agree with that as the subnetwork is typically contained in the whole-body network for whole-body methods. One can still evaluate the subnetworks and report results by feeding face/hand images to the subnetworks (please correct me if it is impossible).** **A2:** Indeed, extracting the hand and face subnetworks directly for comparison would be ideal. However, there are some practical challenges: 1. In methods including ExPose, PIXIE, Hand4Whole, OS-X, and our proposed model, **crucial layers for shape and global orientation are discarded in the hand and face subnetworks during the whole-body training**. This absence means that without all required parameters—pose, shape, global orientation, and facial expression—we cannot generate a hand or face mesh using MANO or FLAME layers. 2. Feeding hand or face images directly into the whole-body pipeline and then trying to crop the hand and face mesh from the SMPL-X mesh is also ambiguous. The output mesh relies on parameters such as body shape, wrist rotation, and neck rotation, all of which are predicted by the main body subnetwork, and dependent on the input body image. It is not so straightforward to disentangle the hand and face subnetworks from the body subnetwork. For example, in PyMAF-X, the wrist's rotation (global orientation of the hand subnetwork) is tied to the body's overall pose, as shown [here](https://github.com/HongwenZhang/PyMAF-X/blob/95e60745c68445cac4ba7c6a1d02071493297ddd/models/pymaf_net.py#L453-L454). **Q3: As for the unfair settings of whole-body evaluation, I would recommend reporting comprehensive results to Tables as possible as we can and explaining the performance gap and in-depth reasons in the paper. In this way, it could be clearer for readers to have more comprehensive comparisons regarding the performances of the proposed and recent solutions. Insufficient evaluations in benchmark Tables would indicate that the results are selective and not comprehensive/state-of-the-art in my opinion, especially when there is a lack of recent state-of-the-art methods** **A3:** We understand and are open to the inclusion of PyMAF-X in Tables 3 and 6, accompanied by an in-depth comparative analysis. For Tables 4 and 5, updates will be made once PyMAF-X's respective subnetworks or training codes are publicly available. We are happy to answer further questions.
Dear authors, Thank you for more clarifications. After reading the follow-up responses, I acknowledge the challenges of reporting the results of other methods in Tables 4 and 5. However, my primary concerns regarding the limited contributions and insufficient evaluation remain. I would like to summarize my thoughts as follows: - The contributions of two (of three) modules proposed in this paper are limited. I have pointed out the incremental modification of the localization module in previous comments. As for the Pixel Alignment Module, the differentiable rendering used in the proposed method is very similar to [DSR](https://dsr.is.tue.mpg.de). The authors highlight the usage of the part segmentation obtained from the projected ground-truth mesh. However, such operations can be dated back to [SURREAL](https://github.com/gulvarol/surreal) and are common practices in the task of 3D HPS Estimation. The code of [SPIN](https://github.com/nkolot/SPIN/blob/master/eval.py#L183) also contains similar operations . - As pointed out in my previous comments, the benchmark tables reported in the main paper are misleading due to the lack of comprehensive comparisons. For instance, the results shown in Table 6 indicate the performance of the proposed method is top-two among existing methods. But it is actually not true when considering the results of the recent method PyMAF-X. As the EHF and AGORA datasets are the only two benchmark datasets used in whole-body methods. A comprehensive comparison with recent methods is crucial as potential readers only read the paper but not the rebuttal. The authors blame this issue on unfairness. However, most of the benchmark comparisons in recent papers are unfair due to the usage of different training data and backbones. Including recent state-of-the-art methods is required for more in-depth discussions of the performance gaps and reasons. - Some claims/presentations/statements in the paper and rebuttal are overclaimed, misleading, and even wrong. - Line 52: "This module applies differentiable rendering to ensure precise pixel alignment", I do not agree that the differentiable rendering used in this module can ensure alignment as it is not an optimization-based method. The qualitative results shown in the paper are also far from precise alignment. - Line 88: "dense predictions and differentiable rendering have not been employed in whole-body pose and shape estimation", dense predictions are also used in PyMAF-X. - As I am familiar with the EHF dataset, I know that this dataset does not provide ground-truth hand and face bounding boxes. When downloading the EHF dataset, we can obtain the images, SMPL-X models, and 2D whole-body joints. As explained in the file EHF_README.txt, the 2D joints are estimated with OpenPose from monocular RGB images, which are not considered as ground-truth annotations. Hence, saying PyMAF-X used ground-truth hand and face bounding boxes from the EHF test set is wrong. Actually, the comparisons in Table 7 basically show the performances of different solutions under different detection results. It is confusing to me why the authors refuse to include PyMAF-X in Table 7 as it can be viewed as a two-stage solution, which relies on off-the-shelf face/hand detection modules. The two-stage solutions are common in practices as the detection modules have been investigated extensively for decades. It remains unclear to readers regarding the performance gaps between the proposed method and the combination of strong detection modules and whole-body pose and shape estimators. Overall, after in-depth reading of the original submissions and the rebuttal, I hold the opinion that the current paper is not suitable to be accepted as a NeurIPS paper. I have rollbacked my rating to 3: Reject.
Follow-up 4 (1/2)
Dear Reviewer, Regarding your concerns, we provide additional clarifications below. **Q1: The contributions of two (of three) modules proposed in this paper are limited. I have pointed out the incremental modification of the localization module in previous comments. As for the Pixel Alignment Module, the differentiable rendering used in the proposed method is very similar to DSR. The authors highlight the usage of the part segmentation obtained from the projected ground-truth mesh. However, such operations can be dated back to SURREAL and are common practices in the task of 3D HPS Estimation. The code of SPIN also contains similar operations.** **A1:** In Follow-up 1, we have delineated how our method differs from DSR. For SURREAL, they tackle the part segmentation task. This is fundamentally different from the use of differentiable rendering for part segmentation masks to facilitate mesh recovery. As for SPIN, it utilizes an optimization technique to iteratively refine its predictions. Our method is a direct regression approach. **Q2a: As pointed out in my previous comments, the benchmark tables reported in the main paper are misleading due to the lack of comprehensive comparisons. For instance, the results shown in Table 6 indicate the performance of the proposed method is top-two among existing methods. But it is actually not true when considering the results of the recent method PyMAF-X. As the EHF and AGORA datasets are the only two benchmark datasets used in whole-body methods. A comprehensive comparison with recent methods is crucial as potential readers only read the paper but not the rebuttal. The authors blame this issue on unfairness. However, most of the benchmark comparisons in recent papers are unfair due to the usage of different training data and backbones. Including recent state-of-the-art methods is required for more in-depth discussions of the performance gaps and reasons.** **A2a:** Our intent was never to strategically exclude any method, let alone because of its performance. When raised in the rebuttal, our initial reasoning for not comparing with PyMAF-X is due to its distinct paradigm. While our approach and many existing methods operate on a one-stage mechanism, PyMAF-X employs a two-stage method where the hand and face bounding box localizations are dependent on a separate pose estimation network. By evaluating PyMAF-X, we would also be indirectly evaluating the off-the-shelf pose estimation network. Our apprehension, as voiced earlier, was rooted in the perceived 'unfairness' of comparing the detection ability inherent in one-stage methods to that of the borrowed pose estimation network. Upon your strong recommendation, we recognize the value of including PyMAF-X in our comparisons. As mentioned in Follow-up 3, and emphasized here again, **we are entirely open to integrating PyMAF-X results in Table 3, 4, 5, 6, 7 along with a thorough comparative analysis.** **Q2b: A comprehensive comparison with recent methods is crucial as potential readers only read the paper but not the rebuttal.** **A2b:** We would like to highlight that, during this current discussion period, **we are not allowed to make edits to the main paper**. However, if given an opportunity in future revisions or updates, we will include the necessary comparisons for a complete representation. **Q3: Line 52: "This module applies differentiable rendering to ensure precise pixel alignment", I do not agree that the differentiable rendering used in this module can ensure alignment as it is not an optimization-based method. The qualitative results shown in the paper are also far from precise alignment.** **A3:** We understand that the phrasing "ensure precise pixel alignment" might be seen as an absolute claim. Our intention was to highlight the improved pixel alignment provided by the differentiable rendering, not to suggest it guarantees perfect alignment. Based on your feedback, we will adopt a more nuanced wording that it “improves pixel alignment” or “ensures a more precise pixel alignment”. **Q4: Line 88: "dense predictions and differentiable rendering have not been employed in whole-body pose and shape estimation", dense predictions are also used in PyMAF-X.** **A4:** If the term "dense predictions" is a point of contention, we will rephrase Line 88 to: "The integration of differentiable rendering with dense part segmentation has not been previously employed in whole-body pose and shape estimation."
Follow-up 4 (2/2)
**Q5: As I am familiar with the EHF dataset, I know that this dataset does not provide ground-truth hand and face bounding boxes. When downloading the EHF dataset, we can obtain the images, SMPL-X models, and 2D whole-body joints. As explained in the file EHF_README.txt, the 2D joints are estimated with OpenPose from monocular RGB images, which are not considered as ground-truth annotations. Hence, saying PyMAF-X used ground-truth hand and face bounding boxes from the EHF test set is wrong.** **A5:** The OpenPose keypoints are provided alongside the EHF test set upon its release, and therefore we consider it as ground-truth. We'll ensure to make this distinction clear that PyMAF-X employs part-bounding boxes derived from ground-truth keypoints provided by the EHF test set, and not ground-truth part bounding boxes provided by the EHF test set. **Q6: Actually, the comparisons in Table 7 basically show the performances of different solutions under different detection results. It is confusing to me why the authors refuse to include PyMAF-X in Table 7 as it can be viewed as a two-stage solution, which relies on off-the-shelf face/hand detection modules. The two-stage solutions are common in practices as the detection modules have been investigated extensively for decades. It remains unclear to readers regarding the performance gaps between the proposed method and the combination of strong detection modules and whole-body pose and shape estimators.** **A6:** The primary purpose of Table 7 was to evaluate the robustness of different whole-body methods under varying scale and alignment. Our method and other works (ExPose [5], PIXIE [9], Hand4Whole [32], OS-X [25]) encompass a self-integrated module designed to extract hand and face bounding boxes directly from the image. Nevertheless, we are open to accommodating it. By adding PyMAF-X, we could highlight the performance gaps between existing one-stage methods and two-stage methods that combine whole-body pose and shape estimators and strong whole-body keypoint estimators. **In summary**, we have no intentions to exclude the results of any method, including PyMAF-X, whether its performance was superior or otherwise. At the time of our paper's submission, we were not apprised of PyMAF-X's recent acceptance into a peer-reviewed journal, TPAMI 2023. As mentioned in Follow-up 3, we recognize the value of including PyMAF-X in our comparisons. We are fully prepared to incorporate PyMAF-X outcomes in Tables 3 through 7, complemented by a detailed comparative review. It's also important to note that, during this ongoing discussion phase, we are not permitted to modify the main manuscript. Do let us know if you have further concerns.
Follow-up Comments
Dear authors, I acknowledge the efforts the authors devoted to the rebuttal. As the authors are open to discussions, I would like to post additional comments regarding the follow-up responses. - "our approach and many existing methods operate on a one-stage mechanism". I disagree with this statement. i) Most existing methods are two-stage solutions, including PIXIE, H4W, and PyMAF-X. All these methods have separate detection networks and SMPL-X regression networks. PIXIE and H4W integrate the detection networks in their code, while PyMAF-X resorts to the usage of another detection module. ii) The detection networks may not be considered as a part of the whole-body methods as they are not claimed as contributions or are not even mentioned in previous papers. iii) Whole-body methods focus on the final performances of the whole-body pose and shape estimation, and most of the previous methods can be integrated with a stronger off-the-shelf whole-body detector. Excluding the solutions using stronger whole-body detectors from comparisons makes the experiments less convincing. iv) Besides, when evaluating a method on AGORA, most recent methods rely on strong off-the-shelf detectors to crop person images. I do not see there is any reason to exclude the solutions using off-the-shelf detectors. - "The OpenPose keypoints are provided alongside the EHF test set upon its release, and therefore we consider it as ground-truth." I disagree that the joints estimated by OpenPose can be considered as the ground truth. The EHF dataset has never claimed the 2D joints as the ground truth. Actually, assuming that the EHF did not provide 2D joints, one can use OpenPose to generate 2D joints and crop body/hand/face images accordingly. The bounding boxes used in PyMAF-X can be viewed as the outputs of a strong detector based on OpenPose but are not related to ground-truth joints or bounding boxes. - From my point of view, it is necessary to include the comparisons of the proposed method and the combined solutions of strong whole-body detectors and whole-body pose and pose estimators. As the authors are willing to revise the main paper, I would like to see the comparisons of the proposed method and PyMAF-X in Tables 3, 6, and 7. The revision of Tables 4 and 5 is optional as I acknowledge the challenges of reporting the results of other methods in these two tables. Please post the updated Tables (3, 6, and 7) in the following days, and I will adjust the rating accordingly.
**Q11: How are the joints for the body, hand and face determined? How does the 137 joints body skeleton look like?"** **A11:** We follow the same setting used by H4W and OSX. The whole-body skeleton comprises 137 joints, broken down as follows: 25 body joints, 40 hand joints, and 72 face joints. The 25 body joints adhere to the SMPL body convention. The joints determined by the MANO model consist of 21 for each hand. However, we omitted the wrist joint as it is already accounted for in the body, resulting in a total of 40 hand joints (calculated as (21-1) x 2). The 72 face joints are derived from the FLAME convention, with the exclusion of the neck joint, which is encompassed within the body convention, adjusting the count from 73 to 72. **Q12: More failure cases could be shown in the paper. Also in comparisons to the competitors.** **A12:** Thank you for the suggestion. Please refer to Figure 3 in Rebuttal document.
Follow-up
Dear reviewer, We would like to follow up to check if your concerns have been addressed. In the previous response, we have made the following updates/clarification: - Regarding the evaluation inconsistency and completeness (Q1), we provided explanations of why certain methods are compared against others, given the underlying frameworks. Regarding your advice to show the effectiveness of the localization module (Q2), we have evaluated the accuracy of the predicted bounding boxes in `Rebuttal.pdf`. - Regarding the use of data augmentation instead of contrastive setting (Q4), we demonstrated that the Contrastive Module can help to avoid domain shift, results in improved performance and is only used during training. - Regarding your concern for the quantitative evaluation of pixel alignment (Q5), we have added projected vertex error as an additional metric. - Following your advice, we plan to refine the experiment section (Q6), correct table formatting errors (Q7), clarify the meaning of estimation error (Q8) in the main text. - We have also provided extra clarification regarding why the ablation study was not conducted with the wholebody (Q2), the consistency for representations across varied augmentations (Q9), and have detailed the training and inference specifics (Q10). We are happy to answer further questions.
Dear authors, thanks for the extensive rebuttal! I especially appreciate the additional experiments regarding the bounding box prediction accuracy and pixel alignment. My questions were also addressed satisfactorily. However, my main concern remains the wholebody performance on in-the-wild scenarios. Experiments on AGORA (Table 6) show that the proposed method is quite heavily outperformed by OS-X (and PyMAF-X). Additionally, with the release of more diverse and accurate SMPL-X datasets, e.g. RICH (CVPR'22) and BEDLAM (CVPR'23), SMPL-X prediction methods that focus on in-the-wild robustness should use such datasets for evaluation, instead of the extremely limited mocap dataset EHF. Furthermore, I disagree that OS-X cannot be compared with your method on Body-only accuracy (Table 2). In essence, the goal of both methods is to reconstruct whole-body meshes. Both methods utilize the SMPL-X body model and can be evaluated, among others, on body-only 3D reconstruction (as is done in both papers). It's rather a selling point that OS-X is a one-stage pipeline instead of a multi-stage pipeline with different expert models for face, hands and body, and that OS-X doesn't need or use hand/face/body-specific training data. Since I appreciate the rebuttal but some of my main concerns couldn't be resolved, I will only slightly raise my score.
Follow-up 2
Dear reviewer, We deeply appreciate your thoughtful feedback and the adjustment in your scoring. We are glad to know that our additional experiments and explanations addressed many of your questions. **Q1: Additionally, with the release of more diverse and accurate SMPL-X datasets, e.g. RICH (CVPR'22) and BEDLAM (CVPR'23), SMPL-X prediction methods that focus on in-the-wild robustness should use such datasets for evaluation, instead of the extremely limited mocap dataset EHF.** **A1:** We chose EHF as it served as the primary benchmark for SMPL-X methodologies, which has been commonly used in previous relevant works [5, 9, 32, 35]. However, we acknowledge your valid observation regarding the increased availability of diverse SMPL-X datasets, specifically RICH and BEDLAM. Thanks for your suggestion, we are working on the experiments. Due to the time constraint, we are not sure if we can get the evaluation results before the discussion period closes. If so, we will update you with the results. **Q2: Furthermore, I disagree that OS-X cannot be compared with your method on Body-only accuracy (Table 2). In essence, the goal of both methods is to reconstruct whole-body meshes. Both methods utilize the SMPL-X body model and can be evaluated, among others, on body-only 3D reconstruction (as is done in both papers). It's rather a selling point that OS-X is a one-stage pipeline instead of a multi-stage pipeline with different expert models for face, hands and body, and that OS-X doesn't need or use hand/face/body-specific training data.** Table 2. 3D reconstruction error trained without additional hand-only and face-only training datasets | | | MPVPE | | | PA-MPVPE | | | |------------------|--------|:--------:|:--------:|:--------:|:--------:|:--------:|:-------:| | | Params | All | Hands | Face | All | Hands | Face | | H4W (1st stage) | 77.84 | 79.2 | 43.2 | 25.0 | 53.1 | 12.1 | 5.8 | | OSX | 422.52 | **70.8** | 53.7 | 26.4 | **48.7** | 15.9 | 6.0 | | Ours (1st stage) | 120.68 | 76.7 | **39.9** | **21.2** | 51.79 | **11.2** | **5.5** | **A2:** We recognize the merit of OS-X as a streamlined, one-stage pipeline, and acknowledge its superior performance in certain body metrics. However, the hands and face results are not as competitive. Table 2 displays the results of Hand4Whole, OSX and our approach when trained in a one-stage pipeline using the same SMPL-X datasets (COCO, MPII, Human36M from NeuralAnnot). Both our method and Hand4Whole outperforms OSX in terms of hand and face metrics. We would also like to point out that OS-X uses a much larger backbone (Vit-L), which might have some advantage in the performance. Moreover, the limitations of a one-stage pipeline like OS-X become evident when one considers the vast potential of specialized datasets. By relying solely on SMPL-X data, the approach misses the opportunity to leverage on widely available hand and face-specific datasets. These datasets encapsulate nuanced scenarios, such as interacting hands (InterHand2.6M), hands with objects (GRAB, 100DOH), emotional facial expressions (AffectNet), and racially diverse faces (BUPT). Such rich datasets can significantly enhance the robustness of hand and face subnetworks. In addition, our multi-stage pipeline allows for modularity. Each subnetwork can be independently upgraded, leveraging new, more specialized datasets as they become available. We are happy to answer further questions.
Follow-up
Dear reviewer, We would like to follow up to check if your concerns have been addressed. In the previous response, we have made the following updates/clarification: - Following your advice on code release (Q1), we have created a GitHub for the project and plan to release the code there when the paper is accepted.. - Regarding the total parameters, FLOPs, and fps of the various methods, we have added it to General Concerns 2. - For your valuable suggestions to improve upon this project (Q3 and Q4), we intend to explore it in the future. We are happy to answer further questions.
I appreciate this nice rebuttal, I believe many concerns have been solved effectively. Please further refine the paper based on all valuable feedback from reviewers. I will keep my rating positive as I explained in the general response.
Follow-up 2
Dear Reviewer, We genuinely appreciate your acknowledgement and support of our work. We will continue to refine the paper based on the feedback provided by you and other reviewers.
Follow-up
Dear reviewer, We would like to follow up to check if your concerns have been addressed. In the previous response, we have made the following updates/clarification: - Following your precious advice to include more literature on use of contrastive setting (Q1), we have added extra discussions to the Related Works sections and provided clarification on how our module differs from previous work. - Regarding the difference in pixel alignment compared to ICON (Q2), we have outlined the key distinctions. - We have also added visual examples of the method’s failure cases (Q3) in Figure 3 of `Rebuttal.pdf`. We are happy to answer further questions.
I have read the rebuttal from the authors and my queries were properly answered. Therefore I will keep my ratings positive for this paper.
Dear Reviewer, We sincerely appreciate your acknowledgement of our work, and we are grateful for the emphasis you placed on our method's motivation, execution, and supporting experiments.
Dear AC, I appreciate the authors' effort in this whole-body human pose and shape estimation task, which is more challenging than any single human/face/hand task. I stand by my previous comments: 1. The paper demonstrates high quality, featuring clear motivation and well-organized sections. 2. The empirical study on the impact of three modules is crucial in breaking through barriers. The essence of this paper's contribution, as reiterated by the authors in their general response, is closely tied to the task of whole-body human pose and shape estimation. Assessing the contributions separately might undervalue their impact. While certain reviewers have raised concerns about insufficient comparisons, I agree with the authors that variations in settings across methods (e.g. training data combinations, strategies, and bounding box acquisition…) make fair comparisons challenging. I'm pleased to see substantial experiments in the rebuttal that showcase the proposed method's advantages over some mentioned methods. Indeed, the paper does have some issues as identified by reviewers. However, considering the paper's quality and contribution, my vote remains positive. I believe the authors can further enhance this paper based on the valuable feedback.
I concur with reviewer 4HYU. Having reviewed the feedback from other reviewers and the authors' responses, I find no compelling reason to reject the manuscript. My assessment hinges on three pivotal points: a) the method's clear motivation, b) the sound execution of the method, and c) comprehensive experiments supporting the proposal. Although there were initial reservations regarding the novelty of the implementation, as shared by several reviewers, including myself, these concerns were adequately addressed in the authors' rebuttal. Even though parts of it might appear incremental, the effectiveness of the method, especially considering the intricate nature of the task, is evident. Moreover, it doesn't perturb me that the presented method hasn't topped every benchmark in the pose estimation arena. Such occurrences can be attributed to varying training conditions, like data sets, methodologies, training duration, backbone, and so on. A prevailing issue in the human pose estimation domain is that many studies solely present a singular accuracy figure in the MPJPE table, possibly from their best-performing network iteration. Ideally, it would be more scientifically rigorous to also disclose accuracy variances, rather than only highlighting the top-performing run. Hence, my lack of concern regarding the method not securing the top spot in every table persists. I maintain my positive stance on the paper.
Decision
Accept (poster)