GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction

Creating realistic characters that can react to the users' or another\ncharacter's movement can benefit computer graphics, games and virtual reality\nhugely. However, synthesizing such reactive motions in human-human interactions\nis a challenging task due to the many different ways two humans can interact.\nWhile there are a number of successful researches in adapting the generative\nadversarial network (GAN) in synthesizing single human actions, there are very\nfew on modelling human-human interactions. In this paper, we propose a\nsemi-supervised GAN system that synthesizes the reactive motion of a character\ngiven the active motion from another character. Our key insights are two-fold.\nFirst, to effectively encode the complicated spatial-temporal information of a\nhuman motion, we empower the generator with a part-based long short-term memory\n(LSTM) module, such that the temporal movement of different limbs can be\neffectively modelled. We further include an attention module such that the\ntemporal significance of the interaction can be learned, which enhances the\ntemporal alignment of the active-reactive motion pair. Second, as the reactive\nmotion of different types of interactions can be significantly different, we\nintroduce a discriminator that not only tells if the generated movement is\nrealistic or not, but also tells the class label of the interaction. This\nallows the use of such labels in supervising the training of the generator. We\nexperiment with the SBU and the HHOI datasets. The high quality of the\nsynthetic motion demonstrates the effective design of our generator, and the\ndiscriminability of the synthesis also demonstrates the strength of our\ndiscriminator.\n