Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

In this paper, we propose a deep convolutional recurrent neural network that\npredicts action sequences for task and motion planning (TAMP) from an initial\nscene image. Typical TAMP problems are formalized by combining reasoning on a\nsymbolic, discrete level (e.g. first-order logic) with continuous motion\nplanning such as nonlinear trajectory optimization. Due to the great\ncombinatorial complexity of possible discrete action sequences, a large number\nof optimization/motion planning problems have to be solved to find a solution,\nwhich limits the scalability of these approaches.\n To circumvent this combinatorial complexity, we develop a neural network\nwhich, based on an initial image of the scene, directly predicts promising\ndiscrete action sequences such that ideally only one motion planning problem\nhas to be solved to find a solution to the overall TAMP problem. A key aspect\nis that our method generalizes to scenes with many and varying number of\nobjects, although being trained on only two objects at a time. This is possible\nby encoding the objects of the scene in images as input to the neural network,\ninstead of a fixed feature vector. Results show runtime improvements of several\nmagnitudes. Video: https://youtu.be/i8yyEbbvoEk\n