3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans

We present a unified representation for actionable spatial perception: 3D\nDynamic Scene Graphs. Scene graphs are directed graphs where nodes represent\nentities in the scene (e.g. objects, walls, rooms), and edges represent\nrelations (e.g. inclusion, adjacency) among nodes. Dynamic scene graphs (DSGs)\nextend this notion to represent dynamic scenes with moving agents (e.g. humans,\nrobots), and to include actionable information that supports planning and\ndecision-making (e.g. spatio-temporal relations, topology at different levels\nof abstraction). Our second contribution is to provide the first fully\nautomatic Spatial PerceptIon eNgine(SPIN) to build a DSG from visual-inertial\ndata. We integrate state-of-the-art techniques for object and human detection\nand pose estimation, and we describe how to robustly infer object, robot, and\nhuman nodes in crowded scenes. To the best of our knowledge, this is the first\npaper that reconciles visual-inertial SLAM and dense human mesh tracking.\nMoreover, we provide algorithms to obtain hierarchical representations of\nindoor environments (e.g. places, structures, rooms) and their relations. Our\nthird contribution is to demonstrate the proposed spatial perception engine in\na photo-realistic Unity-based simulator, where we assess its robustness and\nexpressiveness. Finally, we discuss the implications of our proposal on modern\nrobotics applications. 3D Dynamic Scene Graphs can have a profound impact on\nplanning and decision-making, human-robot interaction, long-term autonomy, and\nscene prediction. A video abstract is available at https://youtu.be/SWbofjhyPzI\n