Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba

Summary

The paper introduces Hamba, a novel technique for reconstructing 3D hand models from a single RGB image. This technique addresses the limitations of previous transformer-based methods, which struggle with occlusion, truncation, and capturing the intricate spatial relationships between hand joints. Hamba combines graph learning with Mamba state space modeling to create a Graph-guided State Space (GSS) block. This block effectively learns the structured relationships among hand joints and leverages both local and global features through a fusion module. The proposed framework utilizes significantly fewer tokens than traditional methods, improving both efficiency and precision. Hamba has demonstrated its effectiveness through extensive benchmarking, outperforming state-of-the-art methods on metrics such as PA-MPVPE and F@15mm on the FreiHAND benchmark. Additionally, the approach promises scalability and adaptability, as the GSS block can be integrated into other tasks, indicating potential applications beyond hand modeling.

Strengths

1. The proposed network structure effectively improves prediction accuracy, as evidenced by results on the FreiHAND and HO3D leaderboards. 2. The experiments related to accuracy in hand pose estimation are thorough and detailed. 3. The method is easy to follow. 4. The implementation code is provided in the supplementary materials.

Weaknesses

1. The authors emphasize the efficiency of their method, but the paper includes only accuracy-related experiments. There are no ablation studies to demonstrate the claimed efficiency. 2. The authors assert that their method can serve as a “plug-and-play module for other tasks.” However, the paper only includes experiments related to hand pose estimation. The authors should at least attempt to apply their module to full-body pose estimation. 3. The paper contains several typographical errors, such as the citation error on Line 92.

Questions

My questions based on the weaknesses are: 1. Can the authors provide evidence of their method’s efficiency in terms of inference time and GPU memory usage? 2. Can the authors attempt to transfer their module to full-body pose estimation or other scenarios to validate its plug-and-play capability?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss limitations and neglect the social impacts in the paper.

Summary

This paper proposes Hamba, a Mamba-based framework for single-view 3D hand reconstruction. Its main contribution is to introduce a graph-guided bidirectional scanning mechanism to fully exploit the joint relations and spatial sequences for accurate hand reconstruction. It additionally fuses global spatial tokens with local graph-based features to further improve the performance. The proposed state space block uses 88.5% less tokens than the attention-based methods, while achieving new state-of-the-art hand reconstruction accuracy on various benchmarks for single-image 3D hand reconstruction.

Strengths

**(1) Technical novelty.** This paper proposes a quite novel image feature extractor that is particularly effective for hand reconstruction; it modifies the Mamba framework to further exploit the graph-based joint relations. To the best of my knowledge, this is the first work to demonstrate that visual Mamba can be effective for image-based articulated shape reconstruction task. I wonder if this method works well for, e.g., human body reconstruction as well. **(2) Strong experimental results.** The proposed method achieves strong experimental results across various widely-used hand reconstruction benchmarks. The comparisons are also done against the competitive baselines that are very recently proposed (e.g., HaMeR [57]). **(3) Good presentation.** Overall, the paper is well organized and easy to read. The figures are also clearly presented. Additionally, I appreciate the authors for also submitting the code for the reproducibility of the proposed method.

Weaknesses

**(1) Ablation study with Transformer + GCN.** I’ve found that the motivation for using Mamba and GCN - to capture both long-range dependencies and graph-based local dependencies - is very similar to the motivation for Graformer [R1, R2], where Transformer (instead of Mamba) and GCN are used together. Additional ablation study with Graformer-like architecture (e.g., replacing the state space block with self-attention block in the proposed architecture) would be informative. [R1] Gong *et al.*, DiffPose: Toward More Reliable 3D Pose Estimation, In CVPR, 2023. [R2] Zhao *et al.*, Graformer: Graph-oriented transformer for 3d pose estimation, In CVPR, 2022.

Questions

**(1) Justification for the intermediate 3D reconstruction.** To obtain the 2D joint positions used for image feature extraction, the model intermediately performs the full 3D reconstruction via regressing MANO parameters (Equation 4). I am not sure why this is less overhead compared to employing off-the-shelf 2D joint detectors (lines 151-152).

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Authors have discussed the limitations.

Summary

This paper presents an approach for 3D hand reconstruction from a single view. The main idea is to introduce a graph-guided Mamba framework in the model for hand reconstruction, by bridging graph learning and state space modeling. Building on top of the recent Hamer approach, the final proposed model, Hamba is evaluated on various datasets where it demonstrates strong performance.

Strengths

+ The quantitative results show improvements over the baselines.

Weaknesses

- I am not sure I follow the motivation of the paper. We read in the paper that "existing [..] methods fail to capture the semantic relations between different joints" (Ln 4) which is mentioned again later as "lack of understanding the semantic relations between hand joints" (Ln 38) or "applying attention to all tokens does not fully utilize the joint spatial sequences, resulting in an inaccurate 3D hand mesh in real-world scenarios" (Ln 41). I am not sure how the proposed approach improves over these observed weaknesses. I can see some quantitative improvements, but I do not think that we get enough support for these arguments, i.e., how Hamba improves on these issues, more specifically over the baseline approach (Hamer [57]). - The improvements over the baseline are consistent, but relatively minor, particularly on the HInt dataset. - Based on the training details of the supplementary, it looks like the method is starting from the Hamer checkpoint and finetunes it for 170k iterations. What is the performance of the Hamer model if it also finetuned for 170k iterations? It would be interesting and fair to see this comparison as well. - There are very few qualitative comparisons with the baseline Hamer model.

Questions

- Why there are no results on the Ego4D subset of HInt? - The paper mentions that the GSS block uses 88.5% less tokens. How does that affect the final system in terms of runtime, number of parameters and FLOPs, etc. - Could you clarify their motivation that I discuss in the weaknesses? - Is it possible to see the additional results of the Hamer model when finetuned the same way that Hamba is finetuned?

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

I think the discussion on limitations could be longer. The paper shows a few failure cases, but the limitation section reads like an afterthought. It could be extended with more of the observed failure cases, as well as other limitation/weaknesses (potentially runtime? number of parameters? reliance on a hand detector? etc).