Many real-world problems are compositional - solving them requires completing\ninterdependent sub-tasks, either in series or in parallel, that can be\nrepresented as a dependency graph. Deep reinforcement learning (RL) agents\noften struggle to learn such complex tasks due to the long time horizons and\nsparse rewards. To address this problem, we present Compositional Design of\nEnvironments (CoDE), which trains a Generator agent to automatically build a\nseries of compositional tasks tailored to the RL agent's current skill level.\nThis automatic curriculum not only enables the agent to learn more complex\ntasks than it could have otherwise, but also selects tasks where the agent's\nperformance is weak, enhancing its robustness and ability to generalize\nzero-shot to unseen tasks at test-time. We analyze why current environment\ngeneration techniques are insufficient for the problem of generating\ncompositional tasks, and propose a new algorithm that addresses these issues.\nOur results assess learning and generalization across multiple compositional\ntasks, including the real-world problem of learning to navigate and interact\nwith web pages. We learn to generate environments composed of multiple pages or\nrooms, and train RL agents capable of completing wide-range of complex tasks in\nthose environments. We contribute two new benchmark frameworks for generating\ncompositional tasks, compositional MiniGrid and gMiniWoB for web\nnavigation.CoDE yields 4x higher success rate than the strongest baseline, and\ndemonstrates strong performance of real websites learned on 3500 primitive\ntasks.\n