Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Summary

This work is mainly concerned with the methodological contribution of estimators for partial information theory statistics, to quantify the levels of redundancy, uniqueness and synergy, which are expressed in information theoretic terms, through the use of mutual information. Depending on dataset size and dimensionality, the authors propose two variants of estimators: one that is exact but can only be used for small, low-dimensional data, and one that is approximate, but that scales to large, high-dimensional data. PID statistics can be computer for models as well, which enables the extension of to the proposed framework to deal with model selection. A very large series of experiments are discussed to support 1) the usefulness of PID statistics, 2) the quality of the proposed estimators, 3) the applicability of the proposed methods to real-world dataset. In addition, several experiments on model selection are carried out as well. Finally, three real world applications are studied through the lenses of partial information decomposition, further substantiating the impact of such statistics. When possible, evaluation is exact, given the the ground truth PID statistics value could be computed analytically. Otherwise, the authors resort to human feedback.

Strengths

* Deepening our understanding of multimodal data, and models built to address tasks that rely on multimodal data is of fundamental importance. The use of partial information decomposition is, in my opinion, a good approach to achieve such goals, and this paper is convincing in both showcasing that PID statistics are indeed representative for multimodal data and modelling tasks, as well as supporting the proposed methodology to compute such PID statistics. * The experimental campaign is very extensive and overall (some comments in the questions section should be addressed though) very convincing and clear. * The proposed methodology (especially the approximate one) for computing PID statistics is simple, and easy to implement

Weaknesses

* Multimodal VAEs [1,2,3] are missing. To the best of my knowledge, a large body of research on multimodal representation learning is missing from the study of PID statistics for model predictions. On the one hand, I understand that in this work a model is used to address a given downstream task, whereas multimodal VAEs are used mostly to learn latent representations of all modalities, for generative purposes, but also to solve downstream tasks. Since the literature on Multimodal VAEs clearly addresses relevant questions on coherency, and synergy, I think this work could greatly benefit from an extension to such models. * Alternatives to compute MI are not considered as “competitor” methods for the approximate scheme. Approaches such as [4, 5] have received a lot of attention as means to compute MI. Would it make sense to consider those as building blocks for alternative approximate PID estimation methods? * Some details in the experimental section need further clarification. For example, in section 4.3 on the robustness, e.g. missing modalities, it is not clear if this is considered at training or a test time. [1] Wu, et al.,”Multimodal Generative Models for Scalable Weakly-Supervised Learning”, https://arxiv.org/abs/1802.05335 [2] Shi, et al., “Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models”, https://arxiv.org/abs/1911.03393 [3] Palumbo, et al., “MMVAE+: Enhancing the Generative Quality of Multimodal VAEs without Compromises”, https://openreview.net/forum?id=B42UJTVdDZ5 [4] Belghazi, et al., “MINE: Mutual Information Neural Estimation”, https://arxiv.org/abs/1801.04062 [5] Hijelm, et al., “Learning deep representations by mutual information estimation and maximization”, https://arxiv.org/abs/1808.06670

Questions

* I understand that both the methodology and experiments tackle two modalities. Now, even if datasets with more than two modalities, and models capable of handling more than two modalities are not extremely common, a natural question that arises both for the proposed methodology, its foundations on PID, and the experiments is how well does the schemes proposed in this work scale with the number of modalities. * In the appendix, it is mentioned that the BATCH estimator, in its current form, relies on pre-trained classifiers, including multimodal ones. As explained, this can have the unintended effect of perturbing the estimation of PID statistics, resulting in errors on synergy. While a theoretical analysis of approximation guarantees seem to be rather difficult, could you discuss about means to keep approximation errors under control? * For the BATCH estimator, a key aspect to consider is the dimensionality of the hidden representation that is used to compute matrix A (before sinkhorn is applied). Do you have any insights concerning the effects of this hyper-parameter? * On the values obtained for R, U and S. As an overview on the various result tables, it is somehow hard to justify the choice for bold numbers. In particular, if we consider results on synthetic data for which a ground truth exists, we can see that the values for PID statistics obtained by either the exact or approximate schemes is very small. In some cases the differences between the values obtained is small, and bold numbers seem to be chosen to “comply” with group truth: see for example Table 2 $y=f(z_2^*, z_c^*)$ for the CVX scheme. More generally, the question I would like to ask is related to the “degree” aspects of PID statistics, and how to interpret correctly numerical values, especially when such values are very low, say what is the difference between R=0.07 and R=0.06 in practical terms? I understand that the first statistics indicate more redundancy than the second, but is this going to be discernible in practice? * Finally, if time allows, would it be possible to address the points mentioned in the "weaknesses" part of the review?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

To the best of my knowledge, there is no explicit discussion on limitations nor societal impact. However, in the appendix, it is possible to find information about caveats, assumptions, which help understanding some limitations of the proposed methods.

Summary

This paper proposes an information-theoretic approach to quantify the degree of three PID(Partial Information Decomposition) statistics of a multimodal distribution: redundancy, uniqueness and synergy. To solve the difficulties in computing PID over high-dimensional continuous distributions, two scalable estimators for PID statistics are introduced, enabling the estimation of PID statistics over large multimodal datasets using small subsampled batches. To validate PID estimation, experiments are conducted on synthetic and real-world multimodal datasets. The usefulness of PID is also demonstrated in model selection and real-world case studies.

Strengths

+ This paper proposes an useful approach to quantify the nature and degree of modality interactions based on information theory. The proposed PID estimators can scale to high-dimensional multimodal datasets and models. + The definitions of redundancy, uniqueness, and synergy are intuitive and could help to better understand multimodal datasets and models. + The proposed approach could generalize to real-world applications and provide reasonable interpretation for dataset properties and model selection.

Weaknesses

- The results in Table 2 and 3 are unable to prove the accuracy of the PID estimators. Estimated results only show a rough trend. There is an order of magnitude difference between the approximation and the ground truth in most cases. Besides, it seems that the approximation becomes more inaccurate when the value distribution of R, U, S is more uniform (i.e. The second main row in Table 3). Is this method only applicable to simple and distinctive datasets? - Lack of comparison and discussion with existing methods. For example, Shapley Value could measure the contribution of each modality in multimodal learning. What are the similarities and differences between them? Why is PID better than them?

Questions

Major concerns have been mentioned in Weakness. Line 163: missing ')'

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors did not address the limitation.

Summary

This paper addresses a fundamental question in multimodal modeling: given multiple input modalities, how can we quantify the ways they must interact (as part of a model) to solve a particular task. Furthermore, can we use knowledge to learn how to design models that better leverage these interactions. The authors propose an information-theoretic approach to this problem by quantifying three properties (redundancy, uniqueness, and synergy) using mutual information across modalities. They solve for these properties using two estimators (one using convex optimization and the other sampling). The first estimator uses convex optimization and the second uses a sampling-baed approximation. They evaluate their approaches for quantifying these properties using synthetic and real-world datasets. For the latter, annotators manually went through each dataset and quantified their intuition for overlap between modalities. They show that their models are able to do a reasonable job estimating these quantities.

Strengths

* Overall this is a well written and carried out paper with a lot of depth. The authors approach the problem in a principled manner and have thought deeply about underlying concepts of multi-modality. * While I did not go through the derivations, the authors introduce a nice framework for thinking about multi-modal interactions. * I appreciate the combination of synthetic experiments, real-world experiments, snapshot "case studies," and various ablation experiments. * The appendix includes a large amount of interesting additions.

Weaknesses

* I found the results hard to interpret and -- given synthetic results -- am wary of the validity of the claims. For example, in the second set of synthetic experiments (Table 2) the recovered values are MUCH lower than ideal (e.g., D_R set has R(CVX)=0.16, R(Batch)=0.29, and R(Truth)=1). This is especially true in the "mixed interaction" synthetic datasets. Is there a good reason that the estimated values are so far from the truth? Are the proposed estimators good in the absence of noise (i.e., the bit-wise experiment) but do they quickly fail once there is uncertainty? Is the issue with the estimators, the underlying generative model, or something else? * The core idea of the paper is solid, but I'm wary of the use of human judgements for creating ground truth. The models are using features/representations extracted from each input modality, and a human's perception/judgement here may differ significantly from signal in the data. I acknowledge there may not be a better solution for annotations. (aside: Thanks for putting info about the annotators' backgrounds and instructions in the appendix). * Similarly, in "Results on multimodal fusion," given my experience with these kinds of models I'm hesitant to believe the rational/intuition such as "since sarcasm is often due to a contradiction between what is expressed in language an speech." It seems like a natural conclusion, but my understanding is that speech emotion (or sarcasm) detectors use a more limited context to compute their predictions, which may not have enough context to encode this contradiction (but maybe I'm wrong?)? The rational here "make sense" but seem cherry picked and might not be "true." Couldn't there be other (potentially spurious) correlations that cause the results to look nice? * I imagine there are alternative correlations or measures one could use for evaluating redundancy/uniqueness/synergy. It would be useful to see more discussion (or comparison) with this.

Questions

* Sec 3.1 (preprocessing via feature binning): Is estimating this using binning a reasonable idea here? I suppose this depends on the bin size, dimensionality, size of the dataset, and the distribution of latents. * The "Batch" estimator is interesting. I immediately wondered if VQ-VAEs or other discrete state latent variable models could be used as an alternative here. This would give you (discrete) tokens instead of having to optimize over a continuous space. Does this seem like a reasonable idea? This idea seems simpler to me and means that you don't have to subsample batches as done here.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, more so in the appendix.

Summary

This paper attempts to quantify the interactions between different input modalities using Principle Information Decomposition (PID) from Information Theory. Using this approach, one can quantify not only interactions within multimodal datasets but as well as interactions captured by multimodal models. The paper provided a comprehensive evaluation of their approach and provided examples of its application in real-world case studies.

Strengths

Provided a detailed explanation of its theoretical background, and thorough comprehensive evaluation. They also provided a list of real-world applications which are useful for casual users.

Weaknesses

The presented work, would benefit from more visuals. As this paper is quite detailed, it would benefit from a closing summary listing the main points of the paper.

Questions

Are there any potential use cases of this approach outside multimodal models? Like, e.g. measuring the domain gap between synthetic and real-world data?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There is no dedicated section on its Limitations, nor is it focused on in the paper as far as I could find. It would be useful so that the readers would know in which cases this could potentially not be effective.

Summary

This paper proposes an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. The authors introduce two new estimators for these PID statistics that scale to high-dimensional distributions. They validate the PID statistics on synthetic and real-world datasets, demonstrating their ability to identify meaningful interactions and select appropriate models. The authors also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives. They provide examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. The authors discuss the limitations of their approach, including the need for large amounts of data and the difficulty of interpreting the results. They suggest future directions for research, such as extending the approach to continuous-valued data and incorporating causal relationships between modalities. Overall, this paper provides a useful framework for quantifying and modeling multimodal interactions that has potential applications in a wide range of fields.

Strengths

1. Novel approach: The paper proposes a novel information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. This approach is different from other methods for quantifying multimodal interactions and has the potential to provide new insights into how different modalities interact. 2. Extensive validation: The authors validate their approach on both synthetic and real-world datasets, demonstrating its ability to identify meaningful interactions and select appropriate models. They also compare their approach to other methods for quantifying multimodal interactions, showing that their method outperforms these alternatives. 3. Real-world applications: The paper provides examples of real-world applications where the PID statistics were useful in quantifying interactions within multimodal datasets or selecting appropriate multimodal models. This demonstrates the practical usefulness of the approach in a wide range of fields. 4. Clear presentation: The paper is well-written and clearly presents the approach, the validation experiments, and the real-world applications. The authors provide detailed explanations of the methods used and the results obtained, making it easy for readers to understand and replicate their work

Weaknesses

1. Limited scope: The paper focuses on a specific information-theoretic approach to quantifying multimodal interactions and does not consider other approaches or methods. This may limit the generalizability of the results and the usefulness of the approach in other contexts. 2. Lack of comparison to ground truth: The paper validates the approach on synthetic and real-world datasets, but does not compare the results to a ground truth or gold standard. This makes it difficult to assess the accuracy of the approach and the validity of the results. 3. Limited interpretability: The paper acknowledges that the results of the approach can be difficult to interpret, particularly in high-dimensional datasets. This may limit the usefulness of the approach in practice, as it may be difficult to draw meaningful conclusions from the results. 4. Lack of implementation details: The paper provides a high-level overview of the approach and the methods used, but does not provide detailed implementation instructions or code. This may make it difficult for other researchers to replicate the work or apply the approach in their own research.

Questions

1. How does the proposed approach compare to other methods for quantifying multimodal interactions, such as correlation analysis or principal component analysis? 2. Can the approach be applied to continuous-valued data, or is it limited to discrete data? 3. How does the approach handle missing data or incomplete modalities? 4. How does the approach handle interactions between more than two modalities? 5. In Table 1, what is Exact task? Why all task and measures have the same numbers? 6. Table 3 is hard to understand, maybe better to use the same scale?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Thanks for your detailed rebuttal

Dear Authors, thanks for a detailed rebuttal, both for the general comments, and precise answers to the questions. I think this is a solid piece of work, and that it will be worth discussing it in the community. There are surely a number of issues to investigate more, but these issues are related to the general interpretability of MI, and PID statistics, as well as ways of designing experiments (including those with human feedback) that are strong and conclusive, rather than on the techniques, experiments and results presented in this paper. For these reasons, I will keep my score of 7: Accept, which I think this paper well deserves.

Thanks for the response

Many Thanks for the authors' response. For the raised concerns, most of them are addressed to a certain degree. Hence, I decide to raise my score to 6-Weak Accept. Meanwhile, I strongly encourage the authors further polish the submission, especially fully considering the discussion and limitation.