Summary
The basis of the proposed work is a practical means to decompose the information contained in two random variables into common and unique components. With a couple tweaks to a vanilla VAE setup, data that has been paired such that certain factors of variation have a constant value within the pair can be used to train latent spaces that separate the components of the information. The common component of the information -- the Gacs-Korner common information -- is the primary goal here, and it’s generally difficult to acquire because it requires finding a random variable that is simultaneously a deterministic function of both input variables. Except for highly constrained joint distributions, such a variable generally does not exist (beyond the vacuous solution). The current work proposes a relaxation where the common component can be a stochastic function of the inputs, and this facilitates optimization as well as makes the method applicable to cases without cleanly shared information (at least in theory; it’s not shown here).
As a route to Gacs-Korner common information, the paper offers a principled and pragmatic methodology that (as far as I can tell) offers something original and of value. The example scenarios and applied metrics, however, paint the work more as a route to partially disentangled latent spaces given grouped data, and in that regard it’s lacking novelty and missing relevant comparisons. The following papers leverage the same weak supervision (paired/grouped data with a subset of factors held constant) to learn latent spaces that separate common from unique information:
- “Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations”, Bouchacourt 2018
- “Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders”, Jha 2018
- “Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis”, Esser 2019
- “NestedVAE: Isolating Common Factors via Weak Supervision” Vowels 2020
And one that does so without a generative model:
- “Learning Disentangled Representations via Mutual Information Estimation”, Sanchez 2020
In summary, the method is an interesting contribution but the experimental results are not very effective.
Strengths
The principled route to extracting the common information in a pragmatic methodology is great. Though it is not demonstrated in this work, the authors motivate the work in terms of multi-modality data, which could be a rich area of application of the method.
Weaknesses
See the summary for the primary weakness. I can imagine a couple routes to make the experimental results more strongly support the method. First, the current experiments could include direct comparisons to some of the attached methods with a demonstration of what GKVAE brings. However, if GKVAE is not more effective in the scenarios currently in the manuscript, qualitatively different experiments could help show its merit. Paired data streams from different modalities could highlight the strengths of GKVAE, or perhaps an example where the content of the common information allows for insight about the relationship between the variables (ie, through inspection of the learned latent variable). Or some example where GK information is desirable.
On a much more minor scale: I doubt the majority of readers will have a better intuition for “usable information” than for standard mutual information; it seems a weakness to me to rely on that rather than any of a number of mutual information estimators or bounds that should work fine on the low-entropy examples of this work.
Questions
- As common information is a lower bound to mutual information (L73), how can a lower bound to MI be used as a lower bound for common information (L194)?
- Was $\lambda_c$ shown to be necessary given the random swapping of the common representation (L225)?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.