Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data

This paper introduces Action Image, a new grasp proposal representation that\nallows learning an end-to-end deep-grasping policy. Our model achieves $84\\%$\ngrasp success on $172$ real world objects while being trained only in\nsimulation on $48$ objects with just naive domain randomization. Similar to\ncomputer vision problems, such as object detection, Action Image builds on the\nidea that object features are invariant to translation in image space.\nTherefore, grasp quality is invariant when evaluating the object-gripper\nrelationship; a successful grasp for an object depends on its local context,\nbut is independent of the surrounding environment. Action Image represents a\ngrasp proposal as an image and uses a deep convolutional network to infer grasp\nquality. We show that by using an Action Image representation, trained networks\nare able to extract local, salient features of grasping tasks that generalize\nacross different objects and environments. We show that this representation\nworks on a variety of inputs, including color images (RGB), depth images (D),\nand combined color-depth (RGB-D). Our experimental results demonstrate that\nnetworks utilizing an Action Image representation exhibit strong domain\ntransfer between training on simulated data and inference on real-world sensor\nstreams. Finally, our experiments show that a network trained with Action Image\nimproves grasp success ($84\\%$ vs. $53\\%$) over a baseline model with the same\nstructure, but using actions encoded as vectors.\n