A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations

Learning disentangled representations of textual data is essential for many\nnatural language tasks such as fair classification, style transfer and sentence\ngeneration, among others. The existent dominant approaches in the context of\ntext data {either rely} on training an adversary (discriminator) that aims at\nmaking attribute values difficult to be inferred from the latent code {or rely\non minimising variational bounds of the mutual information between latent code\nand the value attribute}. {However, the available methods suffer of the\nimpossibility to provide a fine-grained control of the degree (or force) of\ndisentanglement.} {In contrast to} {adversarial methods}, which are remarkably\nsimple, although the adversary seems to be performing perfectly well during the\ntraining phase, after it is completed a fair amount of information about the\nundesired attribute still remains. This paper introduces a novel variational\nupper bound to the mutual information between an attribute and the latent code\nof an encoder. Our bound aims at controlling the approximation error via the\nRenyi's divergence, leading to both better disentangled representations and in\nparticular, a precise control of the desirable degree of disentanglement {than\nstate-of-the-art methods proposed for textual data}. Furthermore, it does not\nsuffer from the degeneracy of other losses in multi-class scenarios. We show\nthe superiority of this method on fair classification and on textual style\ntransfer tasks. Additionally, we provide new insights illustrating various\ntrade-offs in style transfer when attempting to learn disentangled\nrepresentations and quality of the generated sentence.\n