A core challenge in Machine Learning is to learn to disentangle natural\nfactors of variation in data (e.g. object shape vs. pose). A popular approach\nto disentanglement consists in learning to map each of these factors to\ndistinct subspaces of a model's latent representation. However, this approach\nhas shown limited empirical success to date. Here, we show that, for a broad\nfamily of transformations acting on images--encompassing simple affine\ntransformations such as rotations and translations--this approach to\ndisentanglement introduces topological defects (i.e. discontinuities in the\nencoder). Motivated by classical results from group representation theory, we\nstudy an alternative, more flexible approach to disentanglement which relies on\ndistributed latent operators, potentially acting on the entire latent space. We\ntheoretically and empirically demonstrate the effectiveness of this approach to\ndisentangle affine transformations. Our work lays a theoretical foundation for\nthe recent success of a new generation of models using distributed operators\nfor disentanglement.\n