PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition

Response to Reviewer eQ8c

We sincerely appreciate your detailed and insightful reviews. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping our response can address your concerns. >**Q1: The conclusion is too short, lack of analysis of network limitations and discussion of application scenarios.** Thanks for your advice! It is very necessary to discuss the limitations and potential application scenarios of our model. 1. Limitations: Our one-shot query strategy, designed for high efficiency, sometimes compromises the depth of geometric detail captured compared to dense querying methods. This trade-off, while beneficial for processing speed and resource utilization, can affect the model's ability to handle complex geometric structures in point cloud data. 2. Application scenarios: Efficient point cloud video perception algorithms have immense potential in various real-world applications, especially where dynamic 3D environmental understanding is crucial. Here's an analysis of their potential applications: * Robotics: Enable real-time navigation, precise object manipulation, and coordinated swarm operations. * AR/VR: Enable immersive experiences through precise motion tracking and seamless integration of virtual content with real-world environments. * Autonomous Vehicles: Facilitate real-time mapping, dynamic object detection, and predictive trajectory modeling to enhance safety and decision-making in complex traffic scenarios. We have updated conclusion as suggested. We hope this extended discussion provides a more comprehensive perspective and addresses your concerns. Thank you for your constructive feedback! >**Q2: Can the method proposed in the paper still effectively solve the problem of longer time-series data or very fast high-speed motion ?** Thanks for your insightful question! It is very necessary to discuss the performance of our model on longer time-series data or very fast high-speed motion. As you suggested, we further evaluate the performance of our approach on the additional **HOI4D**[1] dataset, a large-scale 4D egocentric dataset that captures diverse and category-level human-object interactions, it comprises 2.4M RGB-D egocentric video frames over 4000 sequences. We evaluated our method on the dataset’s action segmentation task, which includes 2971 training scenes and 892 test scenes. Each sequence contains **150** frames, with each frame represented by 2048 points. This dataset is particularly suitable for assessing the model’s ability to handle extended time-series data. The task requires the model to predict an action label for each frame within a point cloud sequence. As presented in the table below, our approach achieves an accuracy of 78.5%, outperforming the advanced method, PPTr, which achieves an accuracy of 77.4%. Additionally, our approach demonstrates superior performance in metrics like Edit Distance and F1 scores, indicating its robustness in handling longer time-series data (150 frames). All these results highlight the model’s ability to effectively capture temporal dependencies and motion dynamics over longer sequences. While the HOI4D dataset includes diverse motion dynamics, it does not explicitly focus on scenarios with very fast high-speed motion. To address this, future evaluations could involve synthetic datasets or benchmarks specifically designed to simulate rapid temporal changes, such as high-frame-rate point cloud sequences or scenarios with significant motion blur. We appreciate your suggestion and recognize the importance of addressing these challenges. Future work will focus on expanding our evaluations to include high-speed motion datasets and further refining the model for such scenarios. Thank you for your valuable feedback! | Method | Length | Accuracy | Edit| F1@10 | F1@25 | F1@50 | |--------------|---------| ---------| --- |------ |------ |------ | |P4Transformer | 150 | 71.2 | 73.1 | 73.8 | 69.2 | 58.2 | | PPTr | 150 | 77.4 | 80.1 | 81.7 | 78.5 | 69.5 | | Ours | 150 | **78.5** | 84.5 | 85.6 | 82.4 | 73.0 | Reference: [1] HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction CVPR 2022.

Response to Reviewer Gk9h (Part 1)

We sincerely appreciate your detailed and insightful reviews. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping our response can address your concerns. >**Q1: Evaluation of the learning performance of the motion imitator.** Thank you for your advice! It is very necessary to clarify the evaluation of the learning performance of the motion imitator. To further evaluate the learning performance of the motion imitator, we compute the chamfer distance between the predicted point cloud and the ground truth (GT) point cloud. We compare the distance between the predicted point cloud (generated by our method) and the ground truth (GT) point cloud with the distance between the predicted point cloud (generated by nearest-neighbor sampling) and the GT point cloud. Nearest-neighbor sampling strategy, the paradigm commonly adopted in prior approaches[1,2,3,4], serves as the baseline for this comparison. As shown in the table below, the chamfer distance produced by our method is significantly lower, than that of the baseline, indicating that the motion imitator learns correlations more effectively. This reduction indicates that our motion imitator effectively predicts point cloud displacements and aligns them closely with ground truth motion. The superior learning of motion dynamics facilitates better temporal consistency across frames, which improves downstream performance in tasks like action recognition. We hope this explanation provides clarity on the evaluation of the motion imitator's learning performance. Thanks for your valuable feedback! | Method | Chamfer Distance($\downarrow$) | |-------------------|----------| | Nearest-neighbor sampling | 5.23e-3 | | Ours | **1.35e-3**| Reference: [1] PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences. ICLR 2021. [2] Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos. CVPR 2021. [3] Deep Hierarchical Representation of Point Cloud Videos via Spatio-Temporal Decomposition. TPAMI 2022. [4] Point Spatio-Temporal Transformer Networks for Point Cloud Video Modeling. TPAMI 2023. >**Q2: Whether point tracking is still easy to learn in other scenarios, such as occlusion problems.** Thanks for your advice! Evaluating the performance of our model in occlusion scenarios is indeed crucial for understanding its practical applicability in real-world environments. As you suggested, we further evaluate the performance of our approach in occlusion scenarios. While we could not access a dedicated point cloud video dataset explicitly designed to include occlusion scenarios, we adopted a simulation-based approach to validate the effectiveness of our method. Specifically, we simulated the occlusion effect encountered by LiDAR sensors, which often leads to partial data loss. Inspired by the Drop-Local corruption operation from the ModelNet-C[5], we applied a similar strategy to the MSR-Action3D dataset. This approach artificially introduced local point cloud occlusions to simulate real-world challenges and constructed a simplified occlusion benchmark for evaluating the performance of our approach in such scenarios. As shown in the table below, our approach achieves an accuracy of 88.22% under occlusions, outperforming the advanced PST-Transformer by 1.69%. This result highlights the ability of our method to maintain robust point tracking and effectively capture spatial-temporal dynamics even in the presence of partial data loss. These results demonstrate that the Motion Imitator and Single-Step Motion Encoder components of our framework can effectively model motion dynamics, even when some data points are missing. This robustness suggests that point tracking with our method remains learnable and effective in scenarios involving occlusions. While the simulation provides valuable insights, real-world occlusions may introduce additional complexities, such as dynamic occlusion patterns or dense clutter. We see this as an exciting direction for future work, including experiments on more diverse datasets or real-world scenarios explicitly designed for occlusion handling. We hope this analysis addresses your concerns and showcases the robustness of our approach. Thank you for your valuable feedback! | Method | Accuracy |Accuracy_occ | |-------------------|----------|-------------| | PST-Transformer | 91.98 | 86.53 | | Ours | **93.93** | **88.22** | Reference: [5] Benchmarking and Analyzing Point Cloud Classification under Corruptions. ICML 2022.

Response to Reviewer wjsW

We sincerely appreciate your detailed and insightful reviews. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping our response can address your concerns. >**Q1: Clarify the practical significance of using point cloud video for action recognition, especially considering that RGB, depth, and pose data have demonstrated better performance.** Thanks for your advice! The practical significance of point cloud videos recognition lies in their unique ability to capture the geometric and spatial-temporal structure of 3D environments directly. This capability is particularly critical in applications such as robotics, augmented reality (AR), autonomous navigation, and other tasks requiring an understanding of real-world 3D spatial relationships. 1) Point cloud videos inherently encode both spatial (3D geometry) and temporal (motion) data, enabling a more accurate depiction of real-world depth, scale, and dynamics. Unlike RGB or depth images, point clouds avoid the distortions or loss of information caused by projections or conversions. 2) Sensor Compatibility and Real-World Data Trends: With the increasing use of LiDAR and depth sensors, point cloud data is becoming more accessible and practical for real-world applications. Methods leveraging raw point cloud data can directly benefit from this growing trend, bypassing the need for conversion into less efficient formats. While RGB, depth, and pose-based methods currently show superior performance in some contexts, they also have limitations, such as sensitivity to lighting (RGB) or reliance on predefined skeletal models (pose). Point cloud methods, by contrast, offer robustness and adaptability to unstructured environments. Our goal with this work is not to surpass other modalities in action recognition but to explore and develop methods tailored specifically for point cloud video recognition. By addressing challenges such as computational efficiency and temporal modeling, we aim to contribute to the broader understanding and practical handling of point cloud data, paving the way for its use in future 3D video research and applications. >**Q2: Limited Applicability and Potential Difficulty in Capturing Fine-Grained Motion.** Thanks for your advice! It is very necessary to discuss the generalizability of our model. As you suggested, we extended our evaluation to the HOI4D dataset [1], a large-scale 4D egocentric dataset focusing on category-level human-object interactions. This dataset is particularly challenging due to its diverse action categories, extensive sequences, and detailed human-object interaction dynamics, making it an excellent benchmark for assessing fine-grained motion recognition. In the action segmentation task of HOI4D, our approach achieved an accuracy of 78.5%, outperforming the advanced method, PPTr, which achieved 77.4%. Notably, our method also demonstrated improvements in additional metrics like Edit Distance and F1 scores (e.g., F1@10 increased from 81.7% to 85.6%, as shown in the table below). These results underline our model's capacity to accurately capture fine-grained motion patterns within egocentric 4D sequences. We hope this evaluation and explanation address the reviewer’s concerns about the generalizability and fine-grained motion capture capabilities of our model. Thank you for your valuable feedback! | Method | Length | Accuracy | Edit| F1@10 | F1@25 | F1@50 | |--------------|---------| ---------| --- |------ |------ |------ | |P4Transformer | 150 | 71.2 | 73.1 | 73.8 | 69.2 | 58.2 | | PPTr | 150 | 77.4 | 80.1 | 81.7 | 78.5 | 69.5 | | Ours | 150 | **78.5** | 84.5 | 85.6 | 82.4 | 73.0 | Reference: [1] HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction CVPR 2022. >**Q3: Please update Table 10 to include comparisons with more recent methods, and add efficiency evaluation metrics such as FLOPs and Params.** Thanks for your advice! We have updated Table. 10 as suggested.