FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow

Summary

The paper proposes a method to reconstruct 4D hand (3D hand sequence) from a short RGB sequence with two types of Fourier Query Flow (pose flow and shape flow). In the Fourier Query Flow, the 3D trajectory of each point is transformed into 3 Fourier series along the time dimension and represented with the first few coefficients (3*(2*6+1) = 39 in this paper). Pose flow is generated with joint flow via LBS. The geometry is represented in the canonical space with a pretrained occupancy network and warped into the real space with the pose flow. Then shape flow adds small displacements to it. Experiments demonstrate that the proposed method outperforms existing methods and can produce continuous and smooth results.

Strengths

Originality: The idea of represent 3D flow with Fourier series is novel and interesting. Quality: The presented results is of high-quality. Clarity: The paper is well-written and easy to follow. Significance: It seems that this method prevents the results from temporal jitters or abrupt motions, which I’ll discuss in Questions. And it’s very computational efficient.

Weaknesses

I do not see some major weakness. But I still have some questions about this method. I’ll leave them to the Question part. And it would be great if authors can add these discussions to the paper. A typo: line 228 t()^2 ?

Questions

1. It seems that using the low frequency representation can prevent temporal jitters or abrupt motions. However, lack of high frequency parts may lead to Gibbs phenomenon. So, I want to see some discussion about this. Does your method still work in the swift movement situation? 2. It seems that the shape flow is working in the real space (not the canonical space), which is different from some methods, like MANO. So, I wonder if it works for different hand shapes, like longer finger or wider palm. If it works, I’d like to see some examples with different hand shapes. 3. How about applying Fourier Query Flow to MANO model directly but not a pretrained occupancy network? I don’t see why using occupancy network would be better in this scenario. I’d like to see more discussion on this.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

I appreciate the authors’ discussions about limitations of the proposed approach in Sec. 5, which helps the understanding of the suitable scenarios.

Summary

This paper introduces FourierHandFlow - an implicit 4d representation for learning spatio-temporal hand shape deformations. The core idea is to introduce a coarse-to-fine implicit deformation model parameterized with a fixed Fourier basis to ensure smoothness and efficient inference. The coarse (pose / joint flow) part models the dynamics of the joints conditioned on images and pose predictions, and the fine part models the dynamics of the per-query point deformations on top of the joint flow, conditioned on image features. Experimental evaluation is conducted on InterHands2.6M and indicates that the proposed method outperforms recent baselines in terms of quality of shape reconstruction.

Strengths

- Overall coarse-to-fine flow formulation generally makes sense, as most of the local per-point deformations would be strongly dependent on the joints. - Proposed formulation uses a fixed Fourier basis parameterisation of the flows is interesting, and bears multiple benefits: a) it allows for more efficient inference, since re-sampling points over temporal domain can be computed in closed form for different timesteps. b) it is guaranteed to lead to smooth trajectories by construction. - Implicit 4D formulation leads to automatic learning of correspondences, which allows texture transfer. - Quantitative results indicate that the proposed method outperforms multiple recent baselines (although there is not enough clarity on the evaluation protocol, see below).

Weaknesses

1. The paper is at times hard to follow. - For example, when introducing the method (L136), authors do not really specify an exact form of the underlying representation, and vaguely refer to it taking a sequence of RGB frames as input. Does the overall method take any other sources of supervision or conditioning? - Similarly, from Section 3.1 it is not really possible to understand what is the ground truth used to pre-train the occupancy and LBS functions. Do you have geometry ground truth here? What is the resolution of this ground truth (a short look on InterHand2.6 suggests that geometry that is provided is coming from MANO)? - After going through the experimental section I am still not sure I understand what is the ground truth geometry that this paper is comparing against. 2. Motivation / quality. - The quality of the resulting geometry seems to be of low resolution and does not contain any high-frequency details. - The main argument for using more complex machinery of implicit 4D representations over explicit mesh-based representations (L108) is the ability of those to capture high-resolution geometry details. - Yet, the resulting meshes seem to contain no high-resolution details, and examples e.g. in Figure 3 show that all the considered methods struggle at rough alignment with the images - which should be possible to solve with more robust keypoint / segmentation constraints. - The question thus arises if there is enough motivation for the development of the complex implicit machinery. If the resolution is the only limiting factor, why not upsample the mesh of MANO and fit it to ground truth scans, followed by a simple image2shape regression? 3. Evaluation / baselines. - There is no comparison to a parametric model MANO fitted to sparse constraints such as keypoints / segmentation.

Questions

- Could you please clarify what exactly is the geometry you are using for training the occupancy/lbs models and for evaluation? If it is MANO fitted shapes, then, if your model does not have any appearance, I am confused how can it learn any more detailed reconstruction than the underlying low-resolution parametric model? - Why is there no comparison to a simple MANO model fitted to whatever InterHand2.6m constraints are available?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

Authors discuss limitations and broader societal impact.

Summary

This paper introduces an implicit spatio-temporally continuous hand representation for RGB videos. Firstly, based on LEAP [25], the occupancy function and LBS weights are pretrained as priors for query points. Then two query flow representations are introduced to model the skeleton and the shape, respectively. The query flow representations are Fourier coefficients, which can be treated as a low-pass filter for 4D representations. Experiments on InterHand2.6M and RGB2Hands show the proposed method achieves accurate and efficient 4D predictions.

Strengths

- The proposed 4D implicit representation is highly efficient during both training and testing. - The Fourier coefficients are effective to get smooth and continuous temporal dynamics.

Weaknesses

- [Balance] This paper proposes using Fourier coefficients to model 4D hand and emphasizes this will guarantee smooth and continuous temporal dynamics. Especially, it learns coefficients for N=6 basis functions (Line 175). I am wondering how Fourier coefficients and N balance the accuracy, the efficiency and the smoothness. For example, will N=6 lead to over smooth? why not use a larger N? is it because of efficiency? - [Optimisation] Unlike existing representations, the proposed representation is coefficients. I am wondering if this will raise any training or optimation issues. For example, will this representation require more epochs to converge or be harder to fit during training? - [$\Psi$] For pose flow, there is an off-the-shelf pose estimator $\Psi$. I'm curious about the role this pose estimator plays. Is it like a condition that directly determine the final performance or an initial values that only reduce search space (Line 209) or a refinement (Tab.3)? Would it possible to use different $\Psi$ with varied performance and see the impact on the final results during training? Also, what will happen if $\Psi$ provides bad predictions during testing? What if we do not have $\Psi$? - [Metric] It would be better to compare SOTA methods like [19,44,45] wrt other metrics like MPJPE, even if they focus on a single frame or are based on MANO.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yeah.

Summary

This paper proposes FourierHandFlow, a 4d hand pose and shape representation that inherently uses Fourier series as query flow representation. Given RGB sequence, a fixed number of Fourier series are learned to represent hand pose and shape. The authors use two types of flows to decompose pose and shape: pose flow (joint flow) and shape flow. Such decomposition makes it more efficient to reconstruct 4d hands. Experiments on Interhand2.6M and RGB2Hands datasets demonstrate its superiority over existing two hand estimation methods.

Strengths

This paper is clearly written and well-motivated. The compact Fourier series representation is proved to be effective for two hand 4d reconstruction in a smoother way. The video results are impressive. The quantitative results show large improvements.

Weaknesses

The main weakness is that, I am not very sure whether such representation could correctly model the surface details (or pose dependent deformation) of the hand because Fourier series based representation seem to generate over smoothed results due to its low dimension nature of shape flow. For example, at video 04:06 (frame 7101), the little finger of the left hand in “Alt. View 1” is too thin.

Questions

1. How to obtain the gt 3d shape of InterHand2.6M for training? Is that simply the MANO labeling? If true, what’s the advantage of using implicit representation instead of MANO for hand shape learning? 2. L. 228, what does ^2 mean? Is that a mistake?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Summary

This paper introduces FOURIERHANDFLOW, which is a spatio-temporal continuous representation for the human hands. It combines a continuous 3D hand occupancy field with articulation-aware query flows represented as Fourier series along the temporal axis. These query flows are parameterized by coefficients learned from an input RGB sequence. Specifically, two types of Fourier query flows, namely pose flow and shape flow, are used to address the challenges of continuous and smooth 4D reconstruction, computational efficiency, and articulated shape modeling with correspondences.

Strengths

- This paper provides a well-articulated description of the problems with existing implicit methods. It presents a clear logical flow and addresses specific challenges effectively. The results show particularly noticeable improvements for abrupt or jittery motions correction. - Although Fourier series is not a novel concept, this paper extends it to the temporal dimension.

Weaknesses

- It is undeniable that this work is closely related to the Fourier Occupancy Field [9]. The authors should consider adding comparative experiments with this work. - This paper focuses on describing the acquisition of Fourier Query Flow in the method section but lacks a direct description of how the final pipeline for generating the 4D hand is constructed (e.g., how the pre-trained canonical field occupancy is utilized). A more comprehensive overview of this aspect should ideally be included in the first paragraph of the methodology section for better clarity. - The pose flow and the shape flow are two important branches proposed in this paper, and the final flow is the sum of both. It would be helpful to provide clearer illustrations that show the trajectories generated by each branch separately, as well as the combined trajectory. This would enable readers to gain a clearer understanding of the method.

Questions

- In the prediction of pose flow, I think that the essence of being able to perform articulated shape modeling is to represent the skeleton of the hand in the form of a graph, which can maintain the topological structure of the hand. Therefore, in this process, the overall skeleton of the hand is predicted frame by frame. But the subsequent prediction of Fourier coefficients is based on the trajectory of each joint, so how is the temporal smoothness of each joint constrained here? - For different input sequences, is it necessary to normalize them to the same sampling space? If so, what are the specific implementation details? - In the pose flow, the paper estimates the joint flow of the keypoints and then propagates it to arbitrary points based on LBS. Therefore, the estimation of the shape flow after arbitrary points heavily will rely on the accuracy of LBS. Does the paper have any special handling or explanation for this situation? - Recommendation: For the ODE based method, although its computational complexity is relatively high, it can maintain the property of Diffeomorphism. This aspect can be considered later.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors have partially addressed the limitations of their work, though there is space for improvement (see the section Strengths And Weaknesses).

reply

I have read the rebuttal. The authors solved my concerns.

Comment to the rebuttal

I thank authors' time and effort to answer my questions. The authors' replies address most of my concerns. However, there is still one thing I want to point out. In my Question 2, what I really concern is why p = LBS(p') + \delta p (this is a simple version of Equation 5 in your paper)，but not p = LBS(p' + \delta p). Why is it designed this way? Theoretically, the former one won't work well on shape variations. And I think Fourier-->inputCond in your experiments is not the latter one ( p = LBS(p' + \delta p) ), which is what I mean. So, I maintain my score for this work.