Semantic Mapping of Environments for Autonomous Devices

Methods, systems, and apparatus for receiving a reference to an object located in an environment of a robot, accessing mapping data that indicates, for each of a plurality of object instances, respective probabilities of the object instance being located at one or more locations in the environment, wherein the respective probabilities are based at least on an amount of time that has passed since a prior observation of the object instance was made, identifying one or more particular object instances that correspond to the referenced object, determining, based at least on the mapping data, the respective probabilities of the one or more particular object instances being located at the one or more locations in the environment, selecting, based at least on the respective probabilities, a particular location in the environment where the referenced object is most likely located, and directing the robot to navigate to the particular location.