Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects

When we humans look at a video of human-object interaction, we can not only\ninfer what is happening but we can even extract actionable information and\nimitate those interactions. On the other hand, current recognition or geometric\napproaches lack the physicality of action representation. In this paper, we\ntake a step towards a more physical understanding of actions. We address the\nproblem of inferring contact points and the physical forces from videos of\nhumans interacting with objects. One of the main challenges in tackling this\nproblem is obtaining ground-truth labels for forces. We sidestep this problem\nby instead using a physics simulator for supervision. Specifically, we use a\nsimulator to predict effects and enforce that estimated forces must lead to the\nsame effect as depicted in the video. Our quantitative and qualitative results\nshow that (a) we can predict meaningful forces from videos whose effects lead\nto accurate imitation of the motions observed, (b) by jointly optimizing for\ncontact point and force prediction, we can improve the performance on both\ntasks in comparison to independent training, and (c) we can learn a\nrepresentation from this model that generalizes to novel objects using few shot\nexamples.\n