The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues (i.e., using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose a straightforward yet powerful solution: leveraging the readily available intermediate visual representations produced by off-the-shelf (pre-trained) 2D pose detectors -- no finetuning on the 3D task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations (e.g., feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design a simple baseline named Context-Aware PoseFormer to showcase its effectiveness. Without access to any temporal information, the proposed method significantly outperforms its context-agnostic counterpart, PoseFormer, and other state-of-the-art methods using up to hundreds of video frames regarding both speed and precision. Project page: https://qitaozhao.github.io/ContextAware-PoseFormer
Paper
Similar papers
Peer review
Summary
This paper proposes a context-aware method that combines the 2D image feature and the detected 2D key points for 3D human pose estimation. The joint-centric spatial context represented by the intermediate image feature is able to reduce the amguity in 3D lifting. Under this motivation, a novel framework that consists of a defrmable context extraction module, a pose-context feature fusion module and a spatial inter-joint modeling module is proposed. Experiment validates the effectiveness of the proposed modules in reducing calculation time and achieving better performance. On two standard benchmarks, the proposed method shows its effectiveness and stability without using any temporal information compared to other single-frame methods and multi-frame methods.
Strengths
1. This paper is well presented and organized. The idea is clear and the method is somewhat novel. 2. New SOTA results are achieved with a significante improvement even compared to most multi-frame methods. 3. Experiments are well organized and the ablation study is insightful to support the claim of contributions.
Weaknesses
1. Leveraging 2D intermediate feature is not a new idea. Some previous works also tried this route, such as [a][b][c]. Compared to other skeleton-based methods, either single-frame-based or multi-frame-based, utilizting 2D image feature will lead to more memory consumption. Such a kind of limitation needs to be considered. [a] Yin, Binyi, et al. "Context-Aware Network for 3D Human Pose Estimation from Monocular RGB Image." 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019. [b] Habibie, Ikhsanul, et al. "In the wild human pose estimation using explicit 2d features and intermediate 3d representations." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019. [c] Sun, Xiao, et al. "Integral human pose regression." Proceedings of the European conference on computer vision (ECCV). 2018.
Questions
1. According to the study in Sec. 3 in the supplementary material, temporal information brings less improvement in the proposed method. Why the result when using 81 frames as PoseFormer is not compared? Could the authors give more explaination about this?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have discussed their limitation of motion jitter without temporal information in the supplementary material. But the limitation of memory consumption should also be discussed as mentioned in the weakness.
Summary
This paper proposes Context-Aware PoseFormer, the key idea is to extract multi-scale spatial features for 3D pose lifting from 2D pose. The authors claim that the single-frame beats hundreds of frames and demonstrate that their single-frame methods achieve comparable or better results than multi-frame methods, such as poseformer.
Strengths
The motivation of this article is clear, and the writing is simple and easy to understand. The experiments in this paper verify the effect of the proposed method, verify that the spatial context information is very important, and the effect of the proposed context aware poseformer. The ablation study in this article is detailed and verifies the effect of each module in the method.
Weaknesses
I am skeptical of the claim in this paper that the single frame is better than the multiple frame method. This article only verifies that his single-frame method is better than some multi-frame methods such as poseformer, but there are 3 problems below： 1) The comparison of the methods in table 1 is unfair. For example, other methods use the 2D pose result of CPN as the output of the lifting network, while the method in this paper uses HRNet. It is well known that HRNet can perform better 2D detection than CPN. 2) In addition, the method in this paper also inputs more visual features from images, so this paper should also be compared with other methods that use visual features, such as IVT. 3) If the single-frame method in this paper works well under the same 2D network, it cannot be verified that it is definitely better than the multi-frame method. In the method of this article, is it possible to achieve better results by adding timing information? This is the consensus in the field, and this paper should also verify this. In addition, the core contribution of the method in this paper is the multi-scale visual features of the image. The author uses a deformable method to extract multi-scale features to improve the performance of the network. In the field of 3D posture, it is a consensus that extracting better spatial context features can improve performance. In terms of context information extraction, this article has no additional contribution, only using other people's methods. Therefore, in general, the method in this paper is a bit incremental in terms of innovation, and the contribution is not enough to be accepted by NeurIPS。
Questions
see weakness
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
No
Summary
The paper targets the challenging 3D pose estimation problem based on a new context-aware lifting algorithm. The proposed approach is simple to implement and reproduce. Attractive experimental results have been reported on the challenging benchmarks. Also, the detailed ablations well validate the design of the proposed algorithm.
Strengths
1. The idea of lifting 2d image features for 3D human pose estimation is interesing. 2. The proposed algorithm report state-of-art performance on the 3d pose estimation benchmark. 3. It also provides sufficient ablation studies to validate the algorithm design of the proposed approach.
Weaknesses
1. In stage 2, it involves a Deformable context extraction module to extract the context features (F1, F2, F3, P). How is the feature difference against H1-H3? Also, is it possible to add H1-H3 with positional embeding to the pose-context feature fusion module? 2. It would be reasonable that 2D pose information can be helpful for the 3d pose estimation. But lifting from 2d to 3d is not-trivial. How could the proposed approach guarantee the consistency with the 3d location of the pose? 3. If there are more temporal frames available, is it possible to further boost the performance of the proposed approach?
Questions
Please address the questions in the weakness section.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The paper does not discuss the potential limitations of the proposed algorithm. I would suggest the authors to have a discussion of the limitations (like the cases with high self-occlusion which temporal information may be helpful).
Summary
This paper leverages the readily available intermediate visual representations for 3d human pose estimation. The method discards temporal information to solve the time-intensive issue of existing lifting-based methods. The authors design a simple pipline, named Context-Aware PoseFormer, to extract informative context features and fuse these features to learn more positional clues. After reading authors' rebuttal, my previous concerns were all addressed.
Strengths
1.The paper is generally well written. The problem to address and the shortcomings of the existing approaches are discussed well. 2. The motivation and observation are clear and meaningful, and the network is well designed to solve these problems. 3.The experiments demonstrate well the superiority of the proposed method over the prior art. The authors also provide adequate ablation studies.
Weaknesses
More STOA methods in 2023 should be chosen for comparison.
Questions
1. Does the resolution of input images influence the performance? 2. It would be interesting to apply the proposed method to other related tasks like hand pose estimation to verify its generalization capability.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
The authors should describe the limitation of the proposed method.
Summary
This paper introduces Context-Aware PoseFormer, a new approach that leverages intermediate visual representations from pre-trained 2D pose detectors to implicitly encode spatial context for 3D human pose estimation. Despite the simple network structure, the proposed method outperforms existing state-of-the-art single- and multi-frame methods by a large margin without relying on temporal information.
Strengths
-This paper provides an interesting insight that leverages the spatial contextual information of pre-trained 2D pose detectors, referred to as "context-aware" information, to improve the accuracy of 3D human pose estimation. - The proposed approach provides considerable benefits by elevating the accuracy of single-frame methods to a level on par with state-of-the-art multi-frame methods
Weaknesses
-Incremental novelty in the idea of context aware features and lack of discussion on the existing context aware features. - Modeling context-aware operations or networks has been studied in many other computer vision topics. For example, pixel-aligned features [1, 2] have been extensively used in 3D human/hand reconstruction from single/multi images. This pixel-aligned feature is demonstrated to be the most useful module in Table 3 while other modules only improve the performance marginally. - Incomplete evaluation regarding the pretrained context-aware feature network. - For the pretraining, what's the impact of different pre-training datasets (e.g. number of training data) on the estimation? - For the pretraining backbone, what's the impact of latest backbone on the final results? e.g. ViT-based [3] pre-trained backbones MAE [4]. Reference - [1] Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. ICCV 2021 - [2] Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV2019 - [3] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ICLR - [4] Masked autoencoders are scalable vision learners. CVPR 2022
Questions
Please see the weakness
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Please see the weakness
We kindly wish to remind the reviewer to consider our response and additional experiments. We are more than willing to provide further clarifications if there are any lingering questions or concerns.
The rebuttal well addressed most of my concerns.
Thanks for authors' feedback. First, the additional experiments solve the concerns on fair comparison and temporal information. From the experiments, temporal information is also useful. Thus, I don't think the title of this paper is suitable, although their method outperforms other temporal methods. Second, the most important thing, as authors claimed that ''1. We Identify and Tackle A New Research Problem', ...'. They claimed that they want to reduce the temporal information, but the additional experiments verified that temporal information could improve performance. Besides, if we do not use temporal information, the detected 3D poses are not smooth in temporal dimension, it's no practical in the application. So I don't agree with the authors' claim about this. I can agree that it's a good baseline compared with single-image methods. But I strongly disagree with the author's thesis on temporal motivation and contribution, which should be in single-frame approach track. Third, authors claimed that ''2. We Provide Fresh Insights regarding the Problem Cause ....''. I also do not agree with this claim. Actually, there are many methods use visual features to improve the performance of 3D pose lifting. It's overclaim. I will provide some works later. I am looking forward your response.
### Response to Reviewer jUmb [1/2] We thank the reviewer for the detailed feedback and are glad to address the concerns further! > First, the additional experiments solve the concerns on fair comparison and temporal information. We appreciate the reviewer for acknowledging that our performance comparison is fair and that our experiments on temporal modeling are solid. > From the experiments, temporal information is also useful. Thus, I don't think the title of this paper is suitable, although their method outperforms other temporal methods. > > Second, the most important thing, as authors claimed that ''1. We Identify and Tackle A New Research Problem', ...'. They claimed that they want to reduce the temporal information, but the additional experiments verified that temporal information could improve performance. We also appreciate the reviewer for acknowledging the superiority of our single-frame method over other multi-frame methods. We wish to emphasize that the assertion **"temporal information still helps improve performance"** does *not* contradict the fact that **"our approach achieves strong performance without using temporal modeling."** It is important to clarify that we have *never* negated the significance of temporal information. In fact, the enhancements derived from temporal information underscore the **versatility** and **scalability** of our method. Our title endeavors to emphasize a new **alternative** to the conventional approach of incorporating an increased number of video frames, consequently enhancing the accuracy of 3D human pose estimation. While we acknowledge that our title might sound overly assertive, we are receptive to refining it. What is the reviewer's perspective on the title **"Single 2D Pose with Context is Worth Hundreds of Frames for 3D Human Pose Estimation"**? We welcome the reviewer's insights and recommendations. > Besides, if we do not use temporal information, the detected 3D poses are not smooth in temporal dimension, it's no practical in the application. So I don't agree with the authors' claim about this. We agree that temporal smoothness is a limitation of our method. **It's important to note that temporal smoothness is a general challenge faced by all single-frame methods in contrast to multi-frame methods, as non-temporal approaches inherently struggle to ensure such smoothness.** Thus, it's reasonable to assert that our single-frame method should not be critiqued on this particular dimension. Nevertheless, it's worth highlighting that in `Figure 5, Section 4.4`, our approach demonstrates improved temporal smoothness when compared against the context-agnostic baseline, despite not having direct access to temporal information. In `Tab. 1 of the supplementary`, we also provide a solution to solve such limitation by extending our method to model **short-term** temporal dependencies. In terms of the MPJVE metric (where lower values indicate better smoothness), our 9-frame temporal model achieves comparable results with 81-frame PoseFormer (3.4 v.s. 3.1mm); our 27-frame model even significantly outperforms 81-frame PoseFormer (2.1 v.s. 3.1mm). These experimental findings underscore that **our approach reduces the necessity for extensive long-term temporal modeling to achieve satisfactory temporal smoothness, distinguishing it from conventional multi-frame methods**. Consequently, the introduction of short-term temporal extensions to our model (e.g., utilizing 9 video frames) holds real-world applicability in mitigating smoothness concerns.
### Response to Reviewer jUmb [2/2] > I can agree that it's a good baseline compared with single-image methods. But I strongly disagree with the author's thesis on temporal motivation and contribution, which should be in single-frame approach track. We further thank the reviewer for appreciating that our work establishes a good single-frame baseline. It seems there could be a contradiction within the reviewer's feedback. Reviewer **jUmb** initially praised our work for its transparent motivation and accessible writing ("The motivation of this article is clear, and the writing is simple and easy to understand"). Similarly, reviewer **ez57** also offered positive feedback on our motivation ("The motivation and observation are clear and meaningful"). However, the subsequent comments by reviewer **jUmb** appear to raise doubts about our motivation. Our work is motivated by several challenges posed by the adoption of extensive long-term temporal modeling, a default setting in 3D HPE: high computation, performance saturation, and the non-causal problem. This work presents an alternative approach to long-term temporal modeling while showing encouraging performance. Specifically, our CA-PF-CPN model surpasses the prior SOTA single-frame method [1] by an impressive **13.2%**, and it achieves comparable, if not superior, results in comparison to leading multi-frame methods that leverage up to 351 video frames. Consequently, **we believe our contribution would be significantly underrated if our work is simply discussed in the context of non-temporal methods**. [1] Learning skeletal graph neural networks for hard 3d pose estimation. ICCV 2021. > Third, authors claimed that ''2. We Provide Fresh Insights regarding the Problem Cause ....''. I also do not agree with this claim. Actually, there are many methods use visual features to improve the performance of 3D pose lifting. It's overclaim. I will provide some works later. I am looking forward your response. Based on our observation: the majority of current SOTA lifting-based methods discard image features to accommodate extremely long input 2D pose sequences (e.g., 243 frames for MixSTE [2] and P-STMO [3], 351 frames for MHFormer [4]). In contrast, we present an alternative approach that harnesses discarded features to enhance performance. [2] Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. CVPR 2022. [3] P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. ECCV 2022. [4] Mhformer: Multi-hypothesis transformer for 3d human pose estimation. CVPR 2022. While some methods also incorporate image features [5,6,7], our work stands out due to the following distinctions: **1) Different Motivation:** We leverage image features to remove the dependency of existing (two-stage) lifting-based methods on temporal information; None of these studies [5,6,7] have employed image features for this specific purpose. **2) Different Pipeline:** Our method follows the 2D-to-3D lifting pipeline (generating 2D pose by the 2D pose detector first); Importantly, the 2D pose detector is frozen to ensure consistent 2D pose outputs. **We do not finetune the 2D pose detector during training!** We **reuse** the discarded feature maps, which are **inherently** part of a lifting-based 3D HPE pipeline. In contrast, existing methods [5,6,7] introduce **extra** **learnable** image backbones and **no longer explicitly use 2D poses**. Consequently, their usage of image features becomes part of an end-to-end process (estimating the 3D pose directly from images), deviating from the strict 2D-to-3D lifting pipeline since the 2D pose is no longer present. **3) Different Ways to Use Image Features:** In our work, we reuse feature maps from 2D pose detectors, **employing a sparse strategy**, i.e., we extract and aggregate point-wise feature vectors based on the estimated 2D joints (to make the features extracted more informative). These works [5,6,7] use image feature maps **as a whole** instead of in a sparse manner. This distinction makes our method lightweight yet capable of achieving superior performance compared to those methods. [5] Context-Aware Network for 3D Human Pose Estimation from Monocular RGB Image. IJCNN 2019. [6] In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representations. CVPR 2019. [7] Integral Human Pose Regression. ECCV 2018. We are glad and open to discussions if the reviewer could share more such works with us!
Thanks for the response. The memory issue I metioned should be compared to the single-frame-based methods if this method is presented as a single-frame pose estimation baseline. Although some points are overclaimed in the general response, I will keep my rating based on the strengths listed in my original review.
We express our gratitude to the reviewer for the feedback! We will discuss the memory issue in our `Limitation` section following the suggestions from the reviewer. Additionally, we are committed to refining our paper to avoid potential overstatements.
We thank the reviewer again for the considerate feedback and valuable insights! We will integrate the experimental results and analysis to enhance the comprehensiveness of our paper.
Decision
Accept (poster)