We are motivated by the goal of generalist robots that can complete a wide\nrange of tasks across many environments. Critical to this is the robot's\nability to acquire some metric of task success or reward, which is necessary\nfor reinforcement learning, planning, or knowing when to ask for help. For a\ngeneral-purpose robot operating in the real world, this reward function must\nalso be able to generalize broadly across environments, tasks, and objects,\nwhile depending only on on-board sensor observations (e.g. RGB images). While\ndeep learning on large and diverse datasets has shown promise as a path towards\nsuch generalization in computer vision and natural language, collecting high\nquality datasets of robotic interaction at scale remains an open challenge. In\ncontrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive\ncollection of people doing interesting tasks across a diverse range of\nsettings. In this work, we propose a simple approach, Domain-agnostic Video\nDiscriminator (DVD), that learns multitask reward functions by training a\ndiscriminator to classify whether two videos are performing the same task, and\ncan generalize by virtue of learning from a small amount of robot data with a\nbroad dataset of human videos. We find that by leveraging diverse human\ndatasets, this reward function (a) can generalize zero shot to unseen\nenvironments, (b) generalize zero shot to unseen tasks, and (c) can be combined\nwith visual model predictive control to solve robotic manipulation tasks on a\nreal WidowX200 robot in an unseen environment from a single human demo.\n