Generalization is a central challenge for the deployment of reinforcement\nlearning (RL) systems in the real world. In this paper, we show that the\nsequential structure of the RL problem necessitates new approaches to\ngeneralization beyond the well-studied techniques used in supervised learning.\nWhile supervised learning methods can generalize effectively without explicitly\naccounting for epistemic uncertainty, we show that, perhaps surprisingly, this\nis not the case in RL. We show that generalization to unseen test conditions\nfrom a limited number of training conditions induces implicit partial\nobservability, effectively turning even fully-observed MDPs into POMDPs.\nInformed by this observation, we recast the problem of generalization in RL as\nsolving the induced partially observed Markov decision process, which we call\nthe epistemic POMDP. We demonstrate the failure modes of algorithms that do not\nappropriately handle this partial observability, and suggest a simple\nensemble-based technique for approximately solving the partially observed\nproblem. Empirically, we demonstrate that our simple algorithm derived from the\nepistemic POMDP achieves significant gains in generalization over current\nmethods on the Procgen benchmark suite.\n