Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors

We introduce (HPS) Human POSEitioning System, a method to recover the full 3D\npose of a human registered with a 3D scan of the surrounding environment using\nwearable sensors. Using IMUs attached at the body limbs and a head mounted\ncamera looking outwards, HPS fuses camera based self-localization with\nIMU-based human body tracking. The former provides drift-free but noisy\nposition and orientation estimates while the latter is accurate in the\nshort-term but subject to drift over longer periods of time. We show that our\noptimization-based integration exploits the benefits of the two, resulting in\npose accuracy free of drift. Furthermore, we integrate 3D scene constraints\ninto our optimization, such as foot contact with the ground, resulting in\nphysically plausible motion. HPS complements more common third-person-based 3D\npose estimation methods. It allows capturing larger recording volumes and\nlonger periods of motion, and could be used for VR/AR applications where humans\ninteract with the scene without requiring direct line of sight with an external\ncamera, or to train agents that navigate and interact with the environment\nbased on first-person visual input, like real humans. With HPS, we recorded a\ndataset of humans interacting with large 3D scenes (300-1000 sq.m) consisting\nof 7 subjects and more than 3 hours of diverse motion. The dataset, code and\nvideo will be available on the project page:\nhttp://virtualhumans.mpi-inf.mpg.de/hps/ .\n