Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics

We propose a novel learned deep prior of body motion for 3D hand shape\nsynthesis and estimation in the domain of conversational gestures. Our model\nbuilds upon the insight that body motion and hand gestures are strongly\ncorrelated in non-verbal communication settings. We formulate the learning of\nthis prior as a prediction task of 3D hand shape over time given body motion\ninput alone. Trained with 3D pose estimations obtained from a large-scale\ndataset of internet videos, our hand prediction model produces convincing 3D\nhand gestures given only the 3D motion of the speaker's arms as input. We\ndemonstrate the efficacy of our method on hand gesture synthesis from body\nmotion input, and as a strong body prior for single-view image-based 3D hand\npose estimation. We demonstrate that our method outperforms previous\nstate-of-the-art approaches and can generalize beyond the monologue-based\ntraining data to multi-person conversations. Video results are available at\nhttp://people.eecs.berkeley.edu/~evonne_ng/projects/body2hands/.\n