Deep Reinforcement Learning (RL) is mainly studied in a setting where the\ntraining and the testing environments are similar. But in many practical\napplications, these environments may differ. For instance, in control systems,\nthe robot(s) on which a policy is learned might differ from the robot(s) on\nwhich a policy will run. It can be caused by different internal factors (e.g.,\ncalibration issues, system attrition, defective modules) or also by external\nchanges (e.g., weather conditions). There is a need to develop RL methods that\ngeneralize well to variations of the training conditions. In this article, we\nconsider the simplest yet hard to tackle generalization setting where the test\nenvironment is unknown at train time, forcing the agent to adapt to the\nsystem's new dynamics. This online adaptation process can be computationally\nexpensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there\nis just a single train environment. To do so, we propose an approach where we\nlearn a subspace of policies within the parameter space. This subspace contains\nan infinite number of policies that are trained to solve the training\nenvironment while having different parameter values. As a consequence, two\npolicies in that subspace process information differently and exhibit different\nbehaviors when facing variations of the train environment. Our experiments\ncarried out over a large variety of benchmarks compare our approach with\nbaselines, including diversity-based methods. In comparison, our approach is\nsimple to tune, does not need any extra component (e.g., discriminator) and\nlearns policies able to gather a high reward on unseen environments.\n