We study the problem of estimating the body movements of a camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, we propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop a two-stage approach that decomposes the problem into temporal completion and spatial completion. First, our method employs masked autoencoders to impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full-body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our method was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego-Exo4D datasets.
Paper
Similar papers
Peer review
Summary
This paper presents a framework to estimate full-body human poses from egocentric head mounted display. The input of the system mainly contains 2 parts: head tracking signal given by HMD, and sparse hand pose signal estimated from egocentric video. The algorithm, DSPoser, is composed of two stages: temporal completion and spatial completion. In the temporal completion stage, DSPoser uses an encoder-decoder to generate the gaussian distribution of hand pose states. In spatial completion stage, DSPoser generates fullbody poses from hand trajectories and head tracking signals. Experiments were performed on Ego-Exo4D and AMASS dataset.
Strengths
(1) The quantitative results are excellent, especially for doubly sparse data. Table.1 and Table.2 supports the effectiveness of the proposed method. (2) The overall paper writing is clear. (3) Limitations are well-explained. (4) The ablation study about aleatoric and epistemic uncertainties is interesting to me.
Weaknesses
(1) The novelty of “newly introduced task (L.205)” is a bit limited. Both motion imputation and pose completion are widely investigated problems, and the combination of both problems seems not difficult to solve. (2) The paper only proves that the proposed hand trajectory imputation is better than linear interpolation for the doubly sparse task. As the motion imputation/interpolation/inbetweening is a long-standing task, the contribution of the proposed hand imputation method (Uncertainty-aware MAE) is not clear.
Questions
(1) For hand pose estimation, why use FrankMocap instead of recently introduced ACR (ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction [CVPR 2023]) or IntagHand (Interacting Attention Graph for Single Image Two-Hand Reconstruction [CVPR 2022]), which are designed for hand mesh recovery only? Does the performance of hand mesh recovery affect the final full-body pose estimation? (2) May the training on AMASS benefit Ego-Exo4D performance? (3) In Figure 3(b), why not demonstrate hand pose here? (4) Sec.3.3 is titled with "imputed hand trajectories and head tracking signal". However, L.146 said "from imputed hand trajectories" and did not mention "head tracking signal" in this section at all. Minor questions: In table.1, “x” should also be briefly explained like “y”. At L.81, V_1 … V_T_w should be defined (RGB images?) Definition of T_w (L. 114) is better moved to its first appearance (L.81)
Rating
5
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
Limitations are clearly addressed.
Summary
This paper proposes a system to estimate full-body pose from forward-facing egocentric videos. Dubbed “doubly sparse video data,” such data streams have the distinct characteristic that only the headset pose is persistent, while the hand pose estimation is only occasionally available. The proposed method first infills the hand motion from estimated hand motion from video information, and then uses the infilled motion (with estimated uncertainty) to estimate the full-body pose. The full-body pose estimation is built upon VQ-VAE representation and VQ-diffusion. Experiments show that the proposed pipeline outperforms SOTA methods.
Strengths
- This work is very well-motivated; estimating the full-body pose from sparse egocentric views could have many applications in AR/VR and animation. The task is also very challenging, as hands are only visible in very few frames. - The proposed system is a complete solution to estimate full-body pose (including fingers) from egocentric videos and head tracking. It leverages the persistent signals (headset tracking) and occasional signals (hand pose) well by formulating it as a probabilistic infilling problem. The infilled hand motion then serves as input to a diffusion-based full-body pose estimator. - I find the uncertainty formulation a great addition to the current literature. While most methods just use diffusion-based pose estimation plus masking, the proposed MAE solution seems to be a principled way of obtaining a better hand pose trajectory based on sparse input. - Experimental results on the Ego-Exo-4D dataset and AMASS show that the proposed method outperforms SOTA methods. The evaluation is extensive and shows the results from the method well. Showing the results of using dense information (Table 3) also demonstrates the strength of the proposed method.
Weaknesses
- For pose estimation, it would be very beneficial to provide estimated motion as videos to better judge the quality of the estimated pose and motion. Not providing videos weakens this work. - Since a complex system is proposed, it would be great to see some ablation about VQ-VAE. - Looking at Figure 7 in the appendix, it appears to me that the proposed method could be overfitting. There is no information on kicking the feet up for the human, but the estimated pose is kicking the feet up.
Questions
If possible, some metric on the velocity/acceleration error of the estimated motion would help indicate the smoothness of the motion.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations are addressed.
Summary
The paper introduces the task of full-body pose estimation from temporally and spatially sparse tracking inputs. It differs from the prior work in assuming only the partial availability of hand tracking, which is a common scenario for head-mounted displays (HMD) without hand controllers. To address this problem, a two-stage approach is proposed. First, a masked autoencoder (MAE) infills the missing hand joints along with an uncertainty prediction using only the available frames and the head tracking. Then, the imputed tracking data (hands and head) are passed to a VQ-Diffusion model to predict the remaining body. The proposed model, namely DSPoser, is evaluated on the AMASS and Ego-Exo4D datasets where it performs better than the baselines in this new problem setting.
Strengths
Originality: The paper introduces a new challenge to the full body tracking domain, which has become an active research area due to the increasing number of HMD devices in the market. The proposed solution combines techniques from various works effectively. Quality: The proposed two-stage approach is practical. The masked autoencoder with uncertainty estimation addresses the temporal sparsity problem and decouples the formerly known spatially sparse body tracking (i.e., hands and head are always available) from the temporal sparse setting. Clarity: The paper is well-organized and easy to follow. There is enough background information to understand the proposed method and make connections to the prior works. The authors also provide experimental details thoroughly which seem to be sufficient for reproducibility. Significance: The new problem setting is novel and I expect it to be more commonly addressed in the future. Hence, this paper could be a reference for future works.
Weaknesses
1- I think the evaluations in Tables 1 and 2 could be better structured and also more fair. To make an apples-to-apples comparison, it would be better to group methods using a particular type of input data. For example, EgoEgo should be compared against the DSPoser with only the head tracking inputs. Similarly, the Bodiffusion could also use the MAE imputation. Considering that the VQ-Diffusion and the motion tokenizer are taken from the prior work, MAE as being the main contribution could be better highlighted in this way. The “naive Bodiffusion extension” is simply too naive. Stronger baselines could be introduced. 2- The runtime performance analysis is missing. Considering that the proposed problem setting aims for real-time applications, a masked autoencoder with additional uncertainty computations followed by a diffusion inference is not the optimal candidate. I acknowledge that this is covered in the limitations section. What could be done about it? 3- After reading the “Uncertainty-aware MAE” section, I assumed the total uncertainty (Eq. 6) is proposed. The ablation study, however, reveals that the aleatoric uncertainty gives better performance. Is it the one used in the experiments? 4- A supplementary video with qualitative comparisons would be very helpful. 5- [1*] and [2*] (as concurrent work) could also be covered in the related work section. These are not weaknesses but suggestions: - I think mixture density networks [3*] could be applied in this setting which would also simplify the story around the uncertainty. The network predicts parameters of a Gaussian Mixture Model, not very different from the current aleatoric uncertainty. - Line #243: “The dropout strategy achieves...” -> “The sampling strategy achieves...” ``` [1*] Du, Yuming, et al. "Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [2*] Dai, Peng, et al. "HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3*] Bishop, Christopher M. "Mixture density networks." (1994). ```
Questions
Please see my questions in the previous section and address my main concern on the evaluations.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes.
> Clarification on inference setting in the evaluations. Thank you for your feedback on our evaluation protocol. We would like to clarify that our method follows the BoDiffusion protocol, which uses a sliding step of 20. This approach averages the overlapping frames, producing a 40-frame output for each sliding step of 20. In contrast, AvatarPoser and AvatarJLM use a 1-step sliding window, focusing only on the final frame for real-time applications. > Is the inference time per frame or total time for 40 frames? Our inference time is measured based on generating 40 frames. However, unlike AvatarPoser and AvatarJLM, our method's input and output are fixed (40 frames). Therefore, we can answer your question with a 'yes,' but considering the question may be assumed in an online-inference setup, we need to clarify that the inference time listed in the table B also represents the per-frame time in an online setup. > Additional experiments under real-time inference setup. Recognizing that our initial comparison in previous rebuttal was not conducted under fair conditions, we performed additional experiments to ensure a fair comparison with the baselines. \begin{array}{l|c|c|c|cc} \hline \textbf{Methods} & \text{imputation} & sliding step & averaging & \textbf{MPJPE} & \textbf{MPJVE} & \textbf{MPJRE} & \textbf{Jitter} \\\\\hline \text{AvatarPoser [10]} & \text{MAE} & \text{1} & & 9.88 & 62.31 & 5.98 & 37.89 \\\\ \text{BoDiffusion [3]} & \text{MAE} & \text{20} & \text{temporal avg} & 7.35 & 31.33 & 5.47 & 1254.84 \\\\ \text{AvatarJLM [34]} & \text{MAE} & \text{1} & & 7.12 & 37.60 & 5.24 & 16.95 \\\\\hline \textbf{DSPoser (Paper)} & \text{MAE} & \text{20} & \text{temporal avg} & 5.51 & 24.19 & 4.09 & 4.27 \\\\ \text{DSPoser (\\#1) } & \text{MAE} & \text{1} & & 5.87 & 52.38 & 4.31 & 34.12 \\\\ \text{DSPoser (\\#2) } & \text{MAE} & \text{1} & \text{temporal avg} & 5.23 & 21.73 & 3.83 & 5.94 \\\\ \text{DSPoser (\\#3) } & \text{MAE} & \text{1} & \text{4 samples} & 5.68 & 29.48 & 4.23 & 12.98 \\\\\hline \end{array} Before discussing the results, we want to clarify that our primary focus is on introducing two key aspects: (1) the underexplored problem of doubly sparse video data, and (2) a generic multi-stage framework to address such problems. The specific choices within our framework (e.g. VQ-Diffusion, MAE, and the sliding window) were deliberately kept simple to demonstrate the efficacy of the intermittent tracking signal and our multi-stage framework. As reviewers jBAY and 2EAc noted, the computational demands of diffusion models pose concerns for real-time applications. While we provided baseline results to justify our design choices, we want to emphasize that alternatives such as AvatarJLM and AvatarPoser (instead of VQ-Diffusion style pose estimation algorithms used in the submission) are also viable within our multi-stage framework. These alternatives can balance time complexity and accuracy, making them suitable for real-time applications. > Exp. \#1, fair comparison. We first tested our method **using the evaluation protocol of AvatarPoser and AvatarJLM**, with a sliding window step of 1 where only the final output frame (current frame) is used for each step. Our method performed worse in Jitter and MPJVE compared to AvatarJLM but showed better performance in MPJPE and MPJRE. We believe this drop is due to the probabilistic nature of our method, unlike the deterministic approach of AvatarJLM and AvatarPoser. > Exp. \#2, temporal averaging Next, we modified Exp #1 to better utilize the diversity from our uncertainty modeling by **averaging overlapped frames** while advancing the sliding window. This significantly reduced errors across all metrics. However, when it comes to real-time infernece, the improvement doesn't affect to current frames but only applies to **historical frames**, which are not useful for real-time inference. > Exp. \#3, multiple sampling Finally, we implemented a multiple sampling approach, **averaging four samples** for each step using the end frames (current frames). This method outperformed AvatarJLM across all metrics and can be efficiently implemented with parallel processing, resulting in minimal time overhead from #1. In conclusion, we agree with the reviewer 2EAc that the better performance in MPJVE and Jitter of our method is due to the difference in sliding steps. However, based on Exp #2 and Exp #3, the performance drop appears to stem from our framework's ability to generate diverse motions from the same input. This issue can be mitigated by sampling multiple times using parallel processing and averaging the results.
Dear Reviewer jBAY, We want to express our sincere gratitude for your insightful feedback on our manuscript. As the author-reviewer discussion period is coming to an end, we wonder whether our responses have addressed your concerns? We're looking forward to the further discussion regarding any more questions you might have! Thank you once again for your time and valuable input. Your comments have significantly contributed to improving our work. Best regards, Authors
Summary
This paper presents a new method for ego-body pose estimation from egocentric videos. Compared to previous methods that assume hand tracking signals are always available, this paper focused on the case that hand poses captured intermittently from egocentric videos. To solve this, this paper proposes a two-stage method that firstly does temporal completion for hand trajectory and then spatial completion for full-body poses. Experiments show better performances than selected baseline methods.
Strengths
1. The paper is well-written and very easy to follow. The figures and tables are well presented. 2. I like the point that the authors use the hand poses captured intermittently from egocentric video data instead of assuming dense tracking signals. The two-stage method also sounds reasonable for this case.
Weaknesses
1. It would be interesting to see how the proposed method would compare to the FoV modeling in [1], which focused on the same task setting when the hand tracking signals are intermittent. However, [1] is not discussed in the submission. [1] Jiang et al. EgoPoser: Robust Real-Time Ego-Body Pose Estimation in Large Scenes, arXiv 2023 2. In Table 3, when trained only on dense data, the proposed methods performed worse than previous methods. So, I was wondering whether the model generalizes well to different settings. 3. Metrics related to computational complexity, such as the number of parameters, FLOPs, and inference time, are not provided. This is important to ensure a fair comparison with previous methods, and to see the potential in real-world applications. 4. Sota methods like AvatarPoser and AvatarJLM are only compared in Table 3 but not in Tables 1 and 2. It would be better to have consistency when comparing methods. 5. Some technical details seem missing. For example, what are the lengths of input and output frames? How is an evaluation made during the evaluation (e.g., the step size of the sliding window)? 6. There are no video comparisons provided, which are important for this task.
Questions
1. The authors acknowledged that using diffusion models could limit the usage of real-time applications. What is the motivation for using diffusion models for this task? Some recent papers like [2] even show AvatarPoser performed better than Diffusion models. In Table 3, the previous method, AvatarJLM, also shows better performance than the proposed method. [2] Ma et al. Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild, arXiv 2024 2. IMU measurements are mentioned multiple times in this paper. Are they used in the paper? If yes, how? 3. Will the code be published?
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitation has been discussed in Section 7.
Summary
The paper introduces an approach for estimating full-body pose from egocentric videos combined with sparse head and hand positions. The key contribution lies in utilizing sparse temporal annotations of hand positions to achieve a complete representation. The method is evaluated on the publicly available Ego-Exo4D and AMASS datasets, demonstrating performance that surpasses current state-of-the-art approaches.
Strengths
- **Relevance of the task/scope of the paper:** - The task of estimating full-body pose from egocentric video and sparse positional signals is relevant for the NeurIPS community, and the paper presents this relevance adequately. - **Technical novelty of the approach:** - The approach utilizes both images and sparse information from the hands and head, which is a novel idea. The use of a masked autoencoder to complete the information from the hands is also innovative. - **Technical correctness of the paper:** - The methods section describes the proposed approach clearly. - **Related Work:** - The review of related work is comprehensive, covering all relevant literature. - **Experimental validation:** - The experiments are evaluated on the AMASS and Ego-Exo4D datasets, with a thorough comparison against current state-of-the-art approaches. The quantitative results show improvement over previous methods. - **Writing and presentation:** - The paper is mostly well-written, and the ideas are conveyed clearly despite a few typographical errors.
Weaknesses
**Weaknesses:** - **Technical contributions:** - The key contributions could be consolidated into one. The first and second contributions are similar, and the third discusses the potential for AR experiences without providing experiments involving real-life AR devices. - **Experimental validation:** - The paper lacks qualitative comparisons against state-of-the-art approaches and video visualizations of the results. Given the low MPJPE and MPJRE values in current methods, visual assessments are important to ensure the improvements are not just quantitative artifacts.
Questions
**Justification:** Overall, the paper has clear contributions and thorough quantitative comparisons against previous state-of-the-art approaches. However, the claims about the method's use in AR and VR applications could be substantiated better, and there is a lack of qualitative comparisons. **Additional comments:** - There are missing articles (e.g., "the") in multiple instances (L84, L124). - The phrase "using the transformer architecture" in L155 is redundant. - Correct "L161 strategies" and "L45 signals".
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The paper adresses the limitations. One of the main limitations is the lack of publicly available code, which will hinder progress in the community.
The authors' responses are adequate to solve most of my concerns. I would like to keep my initial rating because I still think the proposed task is a bit incremental. Both technical descriptions and evaluations are satisfied.
Thank you for your thorough review and for acknowledging that our responses addressed most of your concerns. We appreciate your thoughtful consideration and respect your decision to maintain your initial rating. Once again, thank you for your valuable feedback and for taking the time to review our work.
The authors responses appropriately addressed my concerns, and after reading the other comments from the reviewers I would like to maintain my initial rating.
Thank you for your thoughtful review and for confirming that our responses addressed your concerns. We appreciate your decision to maintain your initial rating and value the feedback you have provided. Thank you again for your time and consideration.
We've completed the baseline experiments on the Ego-Exo4D dataset and would like to share the results. \begin{array}{l|ccc} \hline \textbf{Methods} & \textbf{Imputation} & \textbf{MPJPE} & \textbf{MPJVE} & \textbf{Jitter} \\\\\hline \text{AvatarPoser [10]} & \text{Interpolation} & 47.28 & 89.34 & 65.39 \\\\ \text{Bodiffusion [3]} & \text{Interpolation} & 59.81 & 120.12 & 142.32 \\\\ \text{AvatarJLM [34]} & \text{Interpolation} & 43.01 & 61.98 & 54.23 \\\\ \text{AvatarPoser [10]} & \text{MAE} & 24.54 & 62.34 & 44.24 \\\\ \text{Bodiffusion [3]} & \text{MAE} & 22.12 & 53.30 & 93.80 \\\\ \text{AvatarJLM [34]} & \text{MAE} & 21.08 & 45.77 & 39.04 \\\\\hline \textbf{DSPoser (Ours)} & \text{MAE} & \textbf{16.84}\pm0.04 & \textbf{39.86}\pm0.05 & \textbf{19.21}\pm0.04\\\\\hline \end{array}
We've completed the baseline experiments on the Ego-Exo4D dataset and would like to share the results. \begin{array}{l|ccc} \hline \textbf{Methods} & \textbf{Imputation} & \textbf{MPJPE} & \textbf{MPJVE} & \textbf{Jitter} \\\\\hline \text{AvatarPoser [10]} & \text{Interpolation} & 47.28 & 89.34 & 65.39 \\\\ \text{Bodiffusion [3]} & \text{Interpolation} & 59.81 & 120.12 & 142.32 \\\\ \text{AvatarJLM [34]} & \text{Interpolation} & 43.01 & 61.98 & 54.23 \\\\ \text{AvatarPoser [10]} & \text{MAE} & 24.54 & 62.34 & 44.24 \\\\ \text{Bodiffusion [3]} & \text{MAE} & 22.12 & 53.30 & 93.80 \\\\ \text{AvatarJLM [34]} & \text{MAE} & 21.08 & 45.77 & 39.04 \\\\\hline \textbf{DSPoser (Ours)} & \text{MAE} & \textbf{16.84}\pm0.04 & \textbf{39.86}\pm0.05 & \textbf{19.21}\pm0.04\\\\\hline \end{array}
I thank the authors for their rebuttal and also providing new experiments! I read other reviews as well. All my concerns except one is addressed. As the reviewer **2EAc** pointed out, I'd appreciate if the authors clarify the inference setting in the evaluations. The proposed DSPoser's jitter performance seems significantly better than the baselines (pity there is no video illustrating this performance). How is this achieved? Ideally, the model should be evaluated in an online setting where only a single frame is available at a time, with no constraints on the number of past frames. In other words, to make a 40-frame prediction, the model must be run 40 times by sliding the input window 1 frame at a time. Is this the case? Is the inference time per frame or total time for 40 frames?
Reviewer Response
I thank the authors for the detailed response and additional experiments. My concerns are addressed and I would like to raise my score to Accept.
Thank you for your thoughtful feedback and for taking the time to review our additional experiments. We appreciate your support and are pleased that our response addressed your concerns. We're grateful for your decision to raise the score.
Thank you for your detailed response! While some of my concerns have been addressed, there remains an issue regarding the sliding window size. Previous methods like AvatarPoser and AvatarJLM set the sliding window size to one frame to simulate real-time inference. However, according to the rebuttal, the sliding window size is 20, which could explain the improved smoothness metrics (i.e., MPJVE and Jitter, as mentioned in Q3 of the rebuttal). To ensure a fair comparison with previous methods, I would recommend adjusting the sliding window size accordingly.
Thank you for your valuable feedback on ensuring a fair comparison. We agree with your concern and have conducted additional experiments on the AMASS dataset to provide a more accurate and fair comparison with the baseline methods. \begin{array}{l|c|c|c|cc} \hline \textbf{Methods} & \text{imputation} & sliding step & averaging & \textbf{MPJPE} & \textbf{MPJVE} & \textbf{MPJRE} & \textbf{Jitter} \\\\\hline \text{AvatarPoser [10]} & \text{MAE} & \text{1} & & 9.88 & 62.31 & 5.98 & 37.89 \\\\ \text{BoDiffusion [3]} & \text{MAE} & \text{20} & \text{temporal avg} & 7.35 & 31.33 & 5.47 & 1254.84 \\\\ \text{AvatarJLM [34]} & \text{MAE} & \text{1} & & 7.12 & 37.60 & 5.24 & 16.95 \\\\\hline \textbf{DSPoser (Paper)} & \text{MAE} & \text{20} & \text{temporal avg} & 5.51 & 24.19 & 4.09 & 4.27 \\\\ \text{DSPoser (\\#1) } & \text{MAE} & \text{1} & & 5.87 & 52.38 & 4.31 & 34.12 \\\\ \text{DSPoser (\\#2) } & \text{MAE} & \text{1} & \text{temporal avg} & 5.23 & 21.73 & 3.83 & 5.94 \\\\ \text{DSPoser (\\#3) } & \text{MAE} & \text{1} & \text{4 samples} & 5.68 & 29.48 & 4.23 & 12.98 \\\\\hline \end{array} Before discussing the results, we want to clarify that our primary focus is on introducing two key aspects: (1) the underexplored problem of doubly sparse video data, and (2) a generic multi-stage framework to address such problems. The specific choices within our framework (e.g. VQ-Diffusion, MAE, and the sliding window) were deliberately kept simple to demonstrate the efficacy of the intermittent tracking signal and our multi-stage framework. As reviewers jBAY and 2EAc noted, the computational demands of diffusion models pose concerns for real-time applications. While we provided baseline results to justify our design choices, we want to emphasize that alternatives such as AvatarJLM and AvatarPoser (instead of VQ-Diffusion style pose estimation algorithms used in the submission) are also viable within our multi-stage framework. These alternatives can balance time complexity and accuracy, making them suitable for real-time applications. > Exp. \#1, fair comparison. We first tested our method **using the evaluation protocol of AvatarPoser and AvatarJLM**, with a sliding window step of 1 where only the final output frame (current frame) is used for each step. Our method performed worse in Jitter and MPJVE compared to AvatarJLM but showed better performance in MPJPE and MPJRE. We believe this drop is due to the probabilistic nature of our method, unlike the deterministic approach of AvatarJLM and AvatarPoser. > Exp. \#2, temporal averaging Next, we modified Exp #1 to better utilize the diversity from our uncertainty modeling by **averaging overlapped frames** while advancing the sliding window. This significantly reduced errors across all metrics. However, when it comes to real-time infernece, the improvement (averaging overlapped frames) doesn't affect to current frames but only applies to **historical frames**, which are not useful for real-time inference. > Exp. \#3, multiple sampling Finally, we implemented a multiple sampling approach, **averaging four samples** for each step using the end frames (current frames). This method outperformed AvatarJLM across all metrics and can be efficiently implemented with parallel processing, resulting in minimal time overhead from #1. In conclusion, we agree with the reviewers that our method's better performance in MPJVE and Jitter is due to the difference in sliding steps. However, as shown in Exp #1, #2, and #3, the performance drop seems to result from our framework's ability to generate diverse motions from the same input. This issue can be mitigated by sampling multiple times using parallel processing and averaging the results.
Dear Reviewer 2EAc, We want to express our sincere gratitude for your insightful feedback on our manuscript. As the author-reviewer discussion period is coming to an end, we wonder whether our responses have addressed your concerns? We're looking forward to the further discussion regarding any more questions you might have! Thank you once again for your time and valuable input. Your comments have significantly contributed to improving our work. Best regards, Authors
Thanks for your honest response. While I don't see the multiple sampling approach as a contribution (because such a moving average filter is quite common in signal processing and can also improve other methods), I find the overall paper interesting and thus decide to increase my score to weak accept. I hope the authors can improve the paper in the revision, particularly by comprehensively comparing it with existing work, and clearly describing the sliding window setup.
Thank you for your thoughtful feedback and for taking the time to review our paper and rebuttal. Without your comments, we might have overlooked the differences in sliding window sizes in the additional experiments included in the rebuttal. We are committed to improving the paper by adding more baselines to our experiments and providing a clearer explanation of the evaluation setup. We are grateful for your decision to raise the score.
Decision
Accept (poster)