Assistive robots have the potential to help people perform everyday tasks.\nHowever, these robots first need to learn what it is their user wants them to\ndo. Teaching assistive robots is hard for inexperienced users, elderly users,\nand users living with physical disabilities, since often these individuals are\nunable to show the robot their desired behavior. We know that inclusive\nlearners should give human teachers credit for what they cannot demonstrate.\nBut today's robots do the opposite: they assume every user is capable of\nproviding any demonstration. As a result, these robots learn to mimic the\ndemonstrated behavior, even when that behavior is not what the human really\nmeant! Here we propose a different approach to reward learning: robots that\nreason about the user's demonstrations in the context of similar or simpler\nalternatives. Unlike prior works -- which err towards overestimating the\nhuman's capabilities -- here we err towards underestimating what the human can\ninput (i.e., their choice set). Our theoretical analysis proves that\nunderestimating the human's choice set is risk-averse, with better worst-case\nperformance than overestimating. We formalize three properties to generate\nsimilar and simpler alternatives. Across simulations and a user study, our\nresulting algorithm better extrapolates the human's objective. See the user\nstudy here: https://youtu.be/RgbH2YULVRo\n