In order to deploy autonomous agents to domains such as autonomous driving,\ninfrastructure management, health care, and finance, they must be able to adapt\nsafely to unseen situations. The current approach in constructing such agents\nis to try to include as much variation into training as possible, and then\ngeneralize within the possible variations. This paper proposes a principled\napproach where a context module is coevolved with a skill module. The context\nmodule recognizes the variation and modulates the skill module so that the\nentire system performs well in unseen situations. The approach is evaluated in\na challenging version of the Flappy Bird game where the effects of the actions\nvary over time. The Context+Skill approach leads to significantly more robust\nbehavior in environments with previously unseen effects. Such a principled\ngeneralization ability is essential in deploying autonomous agents in real\nworld tasks, and can serve as a foundation for continual learning as well.\n