Reinforcement learning is a powerful framework for robots to acquire skills\nfrom experience, but often requires a substantial amount of online data\ncollection. As a result, it is difficult to collect sufficiently diverse\nexperiences that are needed for robots to generalize broadly. Videos of humans,\non the other hand, are a readily available source of broad and interesting\nexperiences. In this paper, we consider the question: can we perform\nreinforcement learning directly on experience collected by humans? This problem\nis particularly difficult, as such videos are not annotated with actions and\nexhibit substantial visual domain shift relative to the robot's embodiment. To\naddress these challenges, we propose a framework for reinforcement learning\nwith videos (RLV). RLV learns a policy and value function using experience\ncollected by humans in combination with data collected by robots. In our\nexperiments, we find that RLV is able to leverage such videos to learn\nchallenging vision-based skills with less than half as many samples as RL\nmethods that learn from scratch.\n