Accuracy and generalization of dynamics models is key to the success of\nmodel-based reinforcement learning (MBRL). As the complexity of tasks\nincreases, so does the sample inefficiency of learning accurate dynamics\nmodels. However, many complex tasks also exhibit sparsity in the dynamics,\ni.e., actions have only a local effect on the system dynamics. In this paper,\nwe exploit this property with a causal invariance perspective in the\nsingle-task setting, introducing a new type of state abstraction called\n\\textit{model-invariance}. Unlike previous forms of state abstractions, a\nmodel-invariance state abstraction leverages causal sparsity over state\nvariables. This allows for compositional generalization to unseen states,\nsomething that non-factored forms of state abstractions cannot do. We prove\nthat an optimal policy can be learned over this model-invariance state\nabstraction and show improved generalization in a simple toy domain. Next, we\npropose a practical method to approximately learn a model-invariant\nrepresentation for complex domains and validate our approach by showing\nimproved modelling performance over standard maximum likelihood approaches on\nchallenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL\nsetting we show strong performance gains with respect to sample efficiency\nacross a host of other continuous control tasks.\n