HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

3D hand shape and pose estimation from a single depth map is a new and\nchallenging computer vision problem with many applications. Existing methods\naddressing it directly regress hand meshes via 2D convolutional neural\nnetworks, which leads to artefacts due to perspective distortions in the\nimages. To address the limitations of the existing methods, we develop\nHandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions\ntrained in a fully supervised manner. The input to our network is a 3D\nvoxelized-depth-map-based on the truncated signed distance function (TSDF).\nHandVoxNet++ relies on two hand shape representations. The first one is the 3D\nvoxelized grid of hand shape, which does not preserve the mesh topology and\nwhich is the most accurate representation. The second representation is the\nhand surface that preserves the mesh topology. We combine the advantages of\nboth representations by aligning the hand surface to the voxelized hand shape\neither with a new neural Graph-Convolutions-based Mesh Registration\n(GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach\n(NRGA++) which does not rely on training data. In extensive evaluations on\nthree public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and\nHO-3D, the proposed HandVoxNet++ achieves state-of-the-art performance. In this\njournal extension of our previous approach presented at CVPR 2020, we gain\n41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19\ndatasets, respectively. Our method is ranked first on the HANDS19 challenge\ndataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the\nsubmission of our results to the portal in August 2020.\n