Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointclouds

There is a large variety of objects and appliances in human environments,\nsuch as stoves, coffee dispensers, juice extractors, and so on. It is\nchallenging for a roboticist to program a robot for each of these object types\nand for each of their instantiations. In this work, we present a novel approach\nto manipulation planning based on the idea that many household objects share\nsimilarly-operated object parts. We formulate the manipulation planning as a\nstructured prediction problem and design a deep learning model that can handle\nlarge noise in the manipulation demonstrations and learns features from three\ndifferent modalities: point-clouds, language and trajectory. In order to\ncollect a large number of manipulation demonstrations for different objects, we\ndeveloped a new crowd-sourcing platform called Robobarista. We test our model\non our dataset consisting of 116 objects with 249 parts along with 250 language\ninstructions, for which there are 1225 crowd-sourced manipulation\ndemonstrations. We further show that our robot can even manipulate objects it\nhas never seen before.\n