Today's robotic fleets are increasingly measuring high-volume video and LIDAR\nsensory streams, which can be mined for valuable training data, such as rare\nscenes of road construction sites, to steadily improve robotic perception\nmodels. However, re-training perception models on growing volumes of rich\nsensory data in central compute servers (or the "cloud") places an enormous\ntime and cost burden on network transfer, cloud storage, human annotation, and\ncloud computing resources. Hence, we introduce HarvestNet, an intelligent\nsampling algorithm that resides on-board a robot and reduces system bottlenecks\nby only storing rare, useful events to steadily improve perception models\nre-trained in the cloud. HarvestNet significantly improves the accuracy of\nmachine-learning models on our novel dataset of road construction sites, field\ntesting of self-driving cars, and streaming face recognition, while reducing\ncloud storage, dataset annotation time, and cloud compute time by between\n65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline\nalgorithms and scalably runs on embedded deep learning hardware. We provide a\nsuite of compute-efficient perception models for the Google Edge Tensor\nProcessing Unit (TPU), an extended technical report, and a novel video dataset\nto the research community at https://sites.google.com/view/harvestnet.\n