CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators

Thank You. Please see author rebuttal below. (1/3)

We really appreciate Reviewer 83JW for providing their insightful comments and questions. We address them below. > If the authors could show that the CP strategy also performs well in single-RGB input cases or demonstrates its effectiveness in single-image mesh estimation, it would significantly enhance the paper's contribution. The reviewer has raised a very valid point: When demonstrating the effectiveness of CHAMP, one problem is that the ambiguity in video 3D pose estimation where temporal information is abundant is not as significant as in the single-frame case. Thus, we test whether the proposed CP pipeline could still work for single-frame image-based 3D pose estimation. We emphasize that this requires minimal change to our method pipeline as we can adjust the number of frames ($N$) to 1 from 243. We train our method as well as the baseline methods with $N=1$ and test on the same dataset. We document the performance in **Table 5 of Appendix J**. It should be expected that with fewer frames available, all tested baselines should result in worse performance than with multiple frames. Interestingly, as we can see in Table 5, CHAMP-Agg is able to achieve a more significant edge over the previous SOTA (DiffPose) in single-frame setting vs in multi-frame (3.4mm vs 2.1mm). Moreover, CHAMP achieves more improvement over the backbone D3DP in the single-frame setting than in the multi-frame setting. **Results suggest that with less temporal information available (i.e. when the ambiguity is more severe), CHAMP achieves even more competitive performance compared to baselines.** > The improvement on 3DHP is quite marginal, making it unclear whether the improvement is due to random seed variation. These results do not convincingly support the method's effectiveness. While the improvement over current SOTA is not huge, we did add an important component on top of current SOTA methods, namely, the uncertainty quantification part. Moreover, while most improvement from prior work comes from fundamental changes to the architectures of the network backbones, **CHAMP achieves a noticeable performance boost over its backbone (D3DP) solely from the use of an end-to-end nonconformity score (uncertainty) function and the use of CP *after* training without any substantial changes to the backbone itself.** As one can see in Table 1, CHAMP did have a substantial improvement over D3DP, which is the human reconstruction backbone of CHAMP. Thus, as CHAMP outperforms its base architecture (D3DP) and improves even more after using more sophisticated aggregation methods, we argue that the uncertainty quantification component is beneficial. Another note here is that the D3DP performance we put in the tables are obtained after the J-Agg procedure, so it is expected to be slightly better than taking a simple mean as aggregation, which is what CHAMP-Naive did). CHAMP-Naive does not use CP at all during test time and thus serves as the "lower bound" of our method and should be comparable to D3DP with mean aggregation (which is indeed the case if one refers to Table 1 of D3DP P-Agg rows). As shown in Fig. 7 of the Appendix, our method did achieve reliable uncertainty quantification (empirical coverage), which is a trait that no other baseline method could achieve.

Thank You. Please see author rebuttal below. (2/3)

> The quantitative results are not very solid. The improvement on 3DHP is quite marginal, making it unclear whether the improvement is due to random seed variation. These results do not convincingly support the method's effectiveness. We also evaluate CHAMP’s performance on a much harder dataset, 3DPW, in **Appendix L**. To ensure fairness, we compare single-frame models CHAMP ($N=1$), D3DP ($N=1$), DiffPose ($N=1$), and CameraPose [1] trained jointly on Human 3.6M and MPII without using 3DPW during training and test on 3DPW using the PA-MPJPE. CHAMP outperforms other baselines by a noticeable margin. **Even though 3DPW was not seen during training, our method achieves competitive performance.** > It’s important to demonstrate that CP can filter out incorrect hypotheses in such challenging cases, where multi-hypothesis generation is truly necessary. We clarify that CHAMP assumes 2D keypoints are provided and improving the quality of 2D keypoints is not in the scope of our work. However, it would still be interesting to see how CHAMP behaves under ambiguous 2D keypoints (i.e., 2D keypoints with large uncertainty from truncation and occlusion). To address the reviewer’s comments, we evaluated CHAMP ($N=1$) performance in **harder settings in Appendix K**. The main failure mode/difficulty of CHAMP comes from wrong/uncertain 2D keypoints predictions. Thus, we categorize such large input uncertainty as follows: - Truncation in 2D RGB input. - Wrong 2D keypoints prediction. - Self-occlusion in 2D RGB input. While first two types large uncertainty, truncation in 2D RGB input (which gives inaccurate 2D keypoints) and wrong 2D keypoints, are not the focus of our work (since we assume 2D keypoints from some off-the-shelf method), it would be interesting to see how CHAMP behaves when such uncertainty occurs. To test this, and to make the task even more challenging, we use the single-frame model suggested by Reviewer 83JW ---this allows us to isolate the temporal effects and focus on uncertainty in 2D RGB inputs only, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. **Interestingly, CHAMP performs well in these cases and learns to discern occluded/wrong 2D keypoints from the good ones. Some interesting behaviors are observed —CHAMP tends to output more diverse proposals for ambiguous keypoints** (see below). - In **Fig. 13**, when the input is truncated, a decent 2D keypoint detector (such as HRNet and CPN) is still able to detect the truncated joints. As long as the joints’ 2D keypoints are available, CHAMP can run as usual. - Next, when the 2D keypoints are wrong to begin with, CHAMP does make mistakes and this is indeed one key failure mode of CHAMP. However, we observe interesting behavior that CHAMP tends to correct “unnatural” 2D keypoints output. Specifically, in **Fig. 14**, we select two images where the 2D keypoint detector makes mistakes. In the first image, the left leg keypoints are completely off. Interestingly, the proposed 3D keypoints are a lot more diverse for the left leg than other joints. After the CP refinement step, the left leg joints are still more diverse, covering a wider region than other joints’ outputs do. In the second image, the two hands keypoints are off. We again see a much noisier proposal set for the two joints. After the refinement, the two hands' outputs are noticeably more diverse. One possible reason for this interesting phenomenon could be that the 2D keypoints input itself is seen as ``unnatural'' compared to the training data, forcing the network to output more diverse 3D poses. - Under self-occlusion, where the human subject itself occludes the joints, CHAMP’s behaviors are more interesting. Note that this is different from truncation in that truncation is caused by external occlusion rather than self-occlusion. In **Fig. 15**, while the 2D detector is able to differentiate the two joints and output two different 2D keypoints, they are very close and the 3D locations are ambiguous. CHAMP outputs noisier proposals for the occluded arm joints (elbow and hand). Interestingly, after the CP refinement, the elbow joint pose outputs are almost ``bimodal'' and the hand pose outputs are still a lot noisier than other joints. For the second image, the dancer's right leg is occluded by the left leg. Similarly, CHAMP outputs noisier proposals for the right knee and right foot both before and after the CP refinement. One possible reason for this interesting phenomenon is that the network intrinsically learned to output different poses when the 2D keypoints are close to each other, suggesting possible self-occlusion.

Thank You. Please see author rebuttal below. (3/3)

> Therefore, the authors should explore more varied tasks to prove CP's efficacy, such as monocular pose estimation. We emphasize that our method is indeed monocular, as we only take as input 2D keypoints estimated from monocular videos or images.

Thanks for the response

Thank you for providing additional results. I believe these results significantly enhance the quality of the paper, and I have therefore raised my score to 5. However, my primary concern remains the **novelty and contribution** of the work, particularly given that the current approach seems to involve directly applying an existing CP method to a 3D Human Pose Estimation (3D HPE) task. The authors have overlooked a considerable number of state-of-the-art studies in 3D HPE, particularly in the realm of probabilistic methods, which have been extensively explored in recent years. Since the primary contribution lies in its application to 3D HPE, the authors must provide a more comprehensive discussion and comparison with prior works in this area to better situate their contribution within the existing body of literature. The following works should be thoroughly discussed and compared against, including but not limited to: 1. 3D Human Pose Estimation with Spatial and Temporal Transformers (ICCV 2021) 2. GFPose: Learning 3D Human Pose Prior With Gradient Fields (CVPR 2023) 3. PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation (CVPR 2023) 4. MotionBERT: A Unified Perspective on Learning Human Motion Representations (ICCV 2023) 5. MotionAGFormer: Enhancing 3D Human Pose Estimation With a Transformer-GCNFormer Network (WACV 2024) 6. ScoreHypo: Probabilistic Human Mesh Estimation with Hypothesis Scoring (CVPR 2024)

Thank You. Please see author rebuttal below. (1/2)

We really appreciate Reviewer aAY1 for providing their insightful comments and suggestions. We address them below. > However, one important experiment seems to be missing here, CHAMP-Naive + -Agg. How well it performs if the proposed comformal prediction is not used? This experiment will clearly show how important the proposed comformal prediction is. Adding **CHAMP-Naive-Agg** is indeed very insightful and we now have added and highlighted the new experiments in **Table 1 and 2**. Specifically, the CHAMP-Naive-Agg baseline makes sure the output is aggregated with the J-Agg method from D3DP instead of taking a simple average. To compare the effectiveness of using conformal predictions, we compare the improvement from using CHAMP-Naive to CHAMP-Naive-Agg vs. from CHAMP-Naive to CHAMP. Going from CHAMP-Naive to CHAMP-Naive-Agg, the MPJPE gets reduced by 0.5mm on Human 3.6M and 0.3mm on MPII 3DHP. From CHAMP-Naive to CHAMP, the MPJPE is reduced by 2.3mm on Human 3.6M and 0.8mm on MPII 3DHP. **Thus, adding CP refinement indeed brings more improvements, indicating the importance of conformal prediction here**. > The qualitative results are relatively limited. We show more qualitative results in the new contents in the Appendix, which exhibit really interesting behaviors. Specifically, it would be interesting to see how CHAMP behaves under ambiguous 2D keypoints (i.e., 2D keypoints with large uncertainty from truncation and occlusion). To address the reviewer’s comments, we evaluated CHAMP’s performance qualitatively in **harder settings** in Appendix K. The main failure mode/difficulty of CHAMP comes from wrong/uncertain 2D keypoints predictions. Thus, we categorize such large input uncertainty as follows: - Truncation in 2D RGB input. - Wrong 2D keypoints prediction. - Self-occlusion in 2D RGB input. While the first two types of large uncertainty, truncation in 2D RGB input (which gives inaccurate 2D keypoints) and wrong 2D keypoints, are not the focus of our work (since we assume 2D keypoints from some off-the-shelf method), it would be interesting to see how CHAMP behaves when such uncertainty occurs. To test this, and to make the task even more challenging, we use the single-frame model suggested by Reviewer 83JW ---this allows us to isolate the temporal effects and focus on uncertainty in 2D RGB inputs only, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. **Interestingly, CHAMP performs well in these cases and learns to discern occluded/wrong 2D keypoints from the good ones. Some interesting behaviors are observed —CHAMP tends to output more diverse proposals for ambiguous keypoints** (see below). - In **Fig. 13**, when the input is truncated, a decent 2D keypoint detector (such as HRNet and CPN) is still able to detect the truncated joints. As long as the joints’ 2D keypoints are available, CHAMP can run as usual. - Next, when the 2D keypoints are wrong to begin with, CHAMP does make mistakes and this is indeed one key failure mode of CHAMP. However, we observe an interesting behavior where CHAMP tends to correct “unnatural” 2D keypoints output. Specifically, in **Fig. 14**, we select two images where the 2D keypoint detector makes mistakes. In the first image, the left leg keypoints are completely off. Interestingly, the proposed 3D keypoints are a lot more diverse for the left leg than other joints. After the CP refinement step, the left leg joints are still more diverse, covering a wider region than other joints outputs do. In the second image, the two hands keypoints are off. We again see a much noisier proposal set for the two joints. After the refinement, the two hands outputs are noticeably more diverse. One possible reason for this interesting phenomenon could be that the 2D keypoints input itself is seen as ``unnatural'' compared to the training data, forcing the network to output more diverse 3D poses. - Under self-occlusion, where the human subject itself occludes the joints, CHAMP’s behaviors are more interesting. Note that this is different from truncation in that truncation is caused by external occlusion rather than self-occlusion. In **Fig. 15**, while the 2D detector is able to differentiate the two joints and output two different 2D keypoints, they are very close and the 3D locations are ambiguous. CHAMP outputs noisier proposals for the occluded arm joints (elbow and hand). Interestingly, after the CP refinement, the elbow joint pose outputs are almost ``bimodal'' and the hand pose outputs are still a lot noisier than other joints. For the second image, the dancer's right leg is occluded by the left leg. Similarly, CHAMP outputs noisier proposals for the right knee and right foot both before and after the CP refinement. One possible reason for this interesting phenomenon is that the network intrinsically learned to output different poses when the 2D keypoints are close to each other, suggesting possible self-occlusion.

Thank You. Please see author rebuttal below. (2/2)

> Performance on in-the-wild benchmarks, such as 3DPW, is not clear. To address the reviewer’s comment, we evaluate CHAMP’s performance on 3DPW in **Appendix L**. We compare CHAMP with other baselines. To ensure fairness, we compare single-frame models CHAMP ($N=1$), D3DP ($N=1$), DiffPose ($N=1$), and CameraPose [1] trained jointly on Human 3.6M and MPII without using 3DPW during training and test on 3DPW using the PA-MPJPE. CHAMP outperforms other baselines by a noticeable margin. **Even though 3DPW was not seen during training, our method achieves competitive performance.** > The performance improvement is not significant compared with the baseline SOTAs, which might effect the value of this work. While the improvement over current SOTA is not huge, we did add an important component on top of current SOTA methods, namely, the uncertainty quantification part ---by checking the set membership using the learned uncertainty score, one could decide if the results can be trusted with probabilistic guarantees. Moreover, while most improvement from prior work comes from fundamental changes to the architectures of the network backbones, **CHAMP achieves a noticeable performance boost over its backbone (D3DP) solely from the use of an end-to-end nonconformity score (uncertainty) function and the use of CP after training without any substantial changes to the backbone itself**. As one can see in Table 1, CHAMP did have a substantial improvement over D3DP, which is the human reconstruction backbone of CHAMP. Thus, as CHAMP outperforms its base architecture (D3DP) and improves even more after using more sophisticated aggregation methods, we argue that the uncertainty quantification component is beneficial. Another note here is that the D3DP performance we put in the tables are obtained after the J-Agg procedure, so it is expected to be slightly better than taking a simple mean as aggregation, which is what CHAMP-Naive did). CHAMP-Naive does not use CP at all during test time and thus serves as the "lower bound" of our method and should be comparable to D3DP with mean aggregation (which is indeed the case if one refers to Table 1 of D3DP P-Agg rows). As shown in Fig. 7 of the Appendix, our method did achieve reliable **uncertainty quantification (empirical coverage)**, which is a trait that no other baseline method could achieve. [1] Yang, Cheng-Yen, et al. "Camerapose: Weakly-supervised monocular 3d human pose estimation by leveraging in-the-wild 2d annotations." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.

Thanks for making the experiments clear.

Via adding the results of CHAMP-Naive-Agg, the effects of introducing conformal predictions become more obvious. Besides, results on 3DPW furthur reveal how well the model perform on an in-the-wild benchmark. With all these efforts, the influence / improvement brought by the proposed conformal prediction has been well validated. Therefore, I upgrade the rate.

Thank you for your feedback and raising the score.

We sincerely appreciate your invaluable feedback on our work and for raising the score. We believe the 3D HPE task with CP holds significant potential, and still has considerable room for further development. We will continue to explore this direction in our future work.

Thank You. Please see author rebuttal below.

We sincerely appreciate Reviewer rtwB’s comments and suggestions. We address their comments below. > The method is only evaluated on studio datasets, but not on outdoor ones, such as 3DPW and EMDB. Evaluation on such less restricted videos would be more convincing To address the reviewer’s comments, we evaluated CHAMP’s performance on 3DPW in **Appendix L**. We compare CHAMP ($N=1$), D3DP ($N=1$), DiffPose ($N=1$), and CameraPose [1] trained jointly on Human 3.6M and MPII without using 3DPW during training and test on 3DPW using the PA-MPJPE. CHAMP outperforms other baselines by a noticeable margin. Even though 3DPW was not seen during training, our method achieves competitive performance. > I don't fully understand the CHAMP-Agg method. Is this using J-Agg on the full set of hypotheses, or does it also use the learned scoring function in some way? How should we interpret the fact that "CHAMP-Agg" is better than "CHAMP"? Does it mean that the proposed learned conformity scoring does not perform better than the prior method of aggregation proposed by Shan et al. (2023)? We apologize for the confusion about the approach used in CHAMP-Agg. CHAMP-Agg is using J-Agg aggregation method on the refined hypotheses set after running CP. Instead of using weighted averages to aggregate all refined proposals, we use J-Agg to aggregate them based on joint locations. It is expected that CHAMP-Agg is better than CHAMP because —**on the same refined hypotheses set**— CHAMP simply takes an average while CHAMP-Agg uses more sophisticated joint-level aggregation. > To the paragraph after Eq. 5: How can we interpret the "true value to be included in" the conformal set in case of continuous values like 3D coordinates? In case of classification this makes more sense, but perhaps the language has to adjusted here. We have changed the wording after Eq. 5 and for regression-based tasks, we say “set membership”. This is a subtle wording difference in that the set is defined via checking the conformity score with respect to the threshold, instead of explicitly constructing a set from discrete values. Moreover, the set here, **whose membership is easily checked using the conformity score**, can be constructed, and continuous values can also be included by calculating their conformity scores to check the set membership. > What 2D pose estimator is used in the main experiments, or is the ground truth used? In all of our main quantitative experiments we follow previous work and use 2D keypoints detected by CPN [2]. > Why are values missing from Table 3? We apologize for the confusion in Table 3. Some authors did not report those metrics and we did not have the time to rerun the experiments. We now have run the experiments after submission and have filled in the numbers. > Smaller issues... We appreciate the reviewer for pointing out the notation and citation errors. We have fixed them and highlighted the changes. Similarly, we have changed the wording for Human 3.6M description, clarified E2E definition, and highlighted the changes. [1] Yang, Cheng-Yen, et al. "Camerapose: Weakly-supervised monocular 3d human pose estimation by leveraging in-the-wild 2d annotations." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023. [2] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multiperson pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7103–7112, 2018.

Thank You. Please see author rebuttal below.

We sincerely appreciate Reviewer dk3Y’s comments and suggestions. We address their questions below. > However, the results are mainly shown with easy cases where occlusions barely present. I am wondering how CP performs on hard cases where occlusions, truncations exist? We clarify that CHAMP assumes 2D keypoints are provided and improving the quality of 2D keypoints is not in the scope of our work. However, it would still be interesting to see how CHAMP behaves under ambiguous 2D keypoints (i.e., 2D keypoints with large uncertainty from truncation and occlusion). To address the reviewer’s comments, we evaluated CHAMP’s performance in **harder settings** in Appendix K. The main failure mode/difficulty of CHAMP comes from wrong/uncertain 2D keypoints predictions. Thus, we categorize such large input uncertainty as follows: - Truncation in 2D RGB input. - Wrong 2D keypoints prediction. - Self-occlusion in 2D RGB input. While the first two types of large uncertainty, truncation in 2D RGB input (which gives inaccurate 2D keypoints) and wrong 2D keypoints, are not the focus of our work (since we assume 2D keypoints from some off-the-shelf method), it would be interesting to see how CHAMP behaves when such uncertainty occurs. To test this, and to make the task even more challenging, we use the single-frame model suggested by Reviewer 83JW ---this allows us to isolate the temporal effects and focus on uncertainty in 2D RGB inputs only, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. **Interestingly, CHAMP performs well in these cases and learns to discern occluded/wrong 2D keypoints from the good ones. Some interesting behaviors are observed —CHAMP tends to output more diverse proposals for ambiguous keypoints** (see below). - In **Fig. 13**, when the input is truncated, a decent 2D keypoint detector (such as HRNet and CPN) is still able to detect the truncated joints. As long as the joints’ 2D keypoints are available, CHAMP can run as usual. - Next, when the 2D keypoints are wrong to begin with, CHAMP does make mistakes and this is indeed one key failure mode of CHAMP. However, we observe an interesting behavior where CHAMP tends to correct “unnatural” 2D keypoints output. Specifically, in **Fig. 14**, we select two images where the 2D keypoint detector makes mistakes. In the first image, the left leg keypoints are completely off. Interestingly, the proposed 3D keypoints are a lot more diverse for the left leg than other joints. After the CP refinement step, the left leg joints are still more diverse, covering a wider region than other joints outputs do. In the second image, the two hands keypoints are off. We again see a much noisier proposal set for the two joints. After the refinement, the two hands outputs are noticeably more diverse. One possible reason for this interesting phenomenon could be that the 2D keypoints input itself is seen as ``unnatural'' compared to the training data, forcing the network to output more diverse 3D poses. - Under self-occlusion, where the human subject itself occludes the joints, CHAMP’s behaviors are more interesting. Note that this is different from truncation in that truncation is caused by external occlusion rather than self-occlusion. In **Fig. 15**, while the 2D detector is able to differentiate the two joints and output two different 2D keypoints, they are very close and the 3D locations are ambiguous. CHAMP outputs noisier proposals for the occluded arm joints (elbow and hand). Interestingly, after the CP refinement, the elbow joint pose outputs are almost ``bimodal'' and the hand pose outputs are still a lot noisier than other joints. For the second image, the dancer's right leg is occluded by the left leg. Similarly, CHAMP outputs noisier proposals for the right knee and right foot both before and after the CP refinement. One possible reason for this interesting phenomenon is that the network intrinsically learned to output different poses when the 2D keypoints are close to each other, suggesting possible self-occlusion. > It will be better if this method can be extended to image-to-3d pipelines. We appreciate the reviewer for pointing out an interesting future direction. We are indeed working on the next version of CHAMP that works on full human mesh reconstruction based on SMPL learning. The same idea of using an end-to-end uncertainty score can be extended to both shapes and poses. Another direction we are working on is conformalized 3D object pose estimation. Again, the same idea can be applied.