A Theory of Multimodal Learning

Summary

This paper aims to provide a theoretical framework for multimodal learning, that aims to explain why multimodal learning is sometimes better than unimodal learning even when the model is only applied on unimodal tasks. To do this, the authors measure the generalization bounds of multimodal learning on the excess population with Gaussian average, and showed multimodal has a superior generalization bound. They also try to formulate the problem with connection and heterogeneity, and show when multimodal is better than unimodal under their evaluation metrics.

Strengths

1. Motivation is strong. Theoretical understanding of multimodal learning is an important yet less investigated problem. 2. The theoretical investigation is interesting and relatively novel to me. It is particularly interesting to characterize/formulate the multimodal problem into connection and heterogeneity.

Weaknesses

1. The authors nicely concluded their weakness in their limitations section, I think the below points are the major weakness that are non-neglectable (although pointed out in the final section): The work needs more natural assumptions, it currently requires too many hypothesis, and it needs more realistic examples. 2. Related to the above point -- I think the results would be much stronger if it can be backed up by some toy experiments with synthetic datasets to demonstrate how the proof are useful in real-world cases. The current theoretical results are nice, but that content is not sufficient enough to be published on its own for NeurIPS. 3. The presentation/writing of the paper could be further improved (by a lot). e.g. Line 17 ’Diamond Sutra’; line 224 "The story is not yet complete"; line 245 "Let’s". Overall, I think this is an interesting work that needs further iterations/improvements to be able to be published at NeurIPS.

Questions

NA

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors adequately addressed the limitations.

Summary

This paper introduces a theoretical framework aimed at elucidating the phenomenon wherein a multimodal neural network, trained on multimodal data, can exhibit strong performance on unimodal data. The framework incorporates Gaussian averaging and operates within a semi-supervised context. The findings demonstrate that under the conditions of heterogeneity and connectivity, it is possible to identify a bridging function $\hat{g}$ and a projection function $\hat{f}$ that enable comparable generalization error on unimodal data, as if the network were provided with multimodal data.

Strengths

Clear presentation, nice illustrative sinusoidal function, solid technical contribution.

Weaknesses

Some clarifications are needed.

Questions

Overall, I believe the paper is of good quality and deserves acceptance. The questions I'm about to raise are intended for discussion purposes and should be considered by the authors. Lastly, I have some suggestions for improving the writing, particularly in areas that could benefit from greater clarity, especially for readers encountering the content for the first time. 1) I recommend that the authors rephrase the explanations regarding Equation (1). Based on Section 3 with T=1, my understanding is that we solve for $\hat{f_1}$ and $\hat{g}$ using ERM, substituting them back into $L(\hat{g},\hat{f}_1)$ as shown between lines 192 and 193, which is identical to Equation (1). In other words, we learn $\hat{f}_1$ and $\hat{g}$ on multimodal data. However, the current wording in line 137 states that we are attempting to minimize Equation (1) with respect to $\hat{f}$ and $\hat{g}$, which may confuse readers. When I first encountered Equation (1), I assumed that the second term had no impact because it does not depend on $\hat{f}$ and $\hat{g}$, leading me to believe that Equation (1) simplifies to solving for $\hat{f}$ and $\hat{g}$ based solely on $x$. 2) If my understanding is correct, I suggest explicitly writing out $s_{t1}=(x_{t1},y_{t1},z_{t1})$ in line 186. 3) The learning framework presented in this section assumes learning the bridging function $\hat{g}$ on a set of unlabeled data and the projection functions $\hat{f_t}$ on the labeled dataset. While this is certainly a valid learning framework, it seems to me that the most common one might be a fully supervised setting where paired data $(x_i, y_i, z_i)$ are given. 4) In line 8, the authors state that a multimodal network trained on a multimodal dataset can outperform a unimodal network even on unimodal tasks, and this paper aims to demonstrate that. However, in my understanding, this paper actually shows that a multimodal network can perform comparably on a unimodal dataset **as if** multimodal data is provided, as indicated by Equation (1). These two statements differ. 5) The model considered by the authors takes the form of $f(x,y\approx g(x))$. It seems to deviate from what is typically used in the literature. A common form found in the literature is $f(g(x),h(y))$, where $g$ and $h$ are two encoders, and $f$ could represent early/middle/late fusion. 6) I would like to bring an important topic to the attention of the authors. There have been several explorations on multimodal knowledge distillation. Under this topic, it would be highly interesting to theoretically analyze the performance of multimodal (or unimodal) teachers (or students). For further reference, please see the following paper: https://arxiv.org/abs/2206.06487. It would be fascinating to extend this work to knowledge distillation settings as a future area of research. 7) I would suggest the authors to include literature on multi-view learning as Ref [16] did. Because that is closely related to multimodal learning, or some might argue their theory are essentially the same. 8) It will be good to include numerical experiments to support the theorems (or at least the claim in line 258-260).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors clearly stated the limitation of the work in the last section.

Summary

This study proposes a new theoretical foundation for multimodal learning. In particular, regarding the phenomenon that models trained in multiple modalities perform better on single-modality tasks than fine-tuned single-modality models, this paper proposes that multimodal learning is a composition of connection and label predictor, and shows that generalization performance is better than single-modality learning when both connection and heterogeneity are present.

Strengths

- This paper is well-organized and very easy to understand. - Although I am not an expert in learning theory, I have checked throughout and found the theory to be valid. In particular, it is interesting and novel to compare the generalization bound with the usual unimodal learning case by setting multimodal learning as a composition of connection and label predictor.

Weaknesses

- In this study, multimodal learning is considered to be prediction from unimodal data, consisting of prediction by connections from one modality to another and label prediction from them. However, this setting may appear somewhat unnatural when viewed from the perspective of normal multimodal learning. It would be valuable to have a more in-depth discussion on this aspect. - In multimodal learning, it is common to consider more than two modalities. It would be even more insightful to explore the potential implications and extensions of this theory when applied to settings involving multiple modalities.

Questions

I would appreciate it if the authors could respond to the above-mentioned weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors explain the limitations of this theory very clearly.

Summary

The paper establishes theoretical bounds for generalization in multimodal learning, where functions mapping between two modalities and to the label are learned. The authors demonstrate that multimodal learning achieves better generalization by decoupling the learning of hypotheses and provide insights regarding the connection and heterogeneity between modalities.

Strengths

1. The paper aims at addressing an important topic and offers theoretical insights into the superiority of multimodal learning over unimodal learning, particularly regarding the decoupling of hypothesis learning. 2. The use of simple examples at the beginning enhances the readability of the paper and helps illustrate certain aspects of the main ideas.

Weaknesses

1. The motivating example appears to be restrictive as it relies on F containing only the ground truth mapping from modality Y to label Z, and by the construction the map from Y to Z is exatly the only `hard part' of the learning. It might be slightly better to use the harder example in Remark 2. How would the comparison between multimodal and unimodal learning be in this case? 2. Theorem 7 only shows the existence of cases where multimodal learning surpasses unimodal learning, without delving into a deeper discussion on the required conditions for such cases. As a result, it remains unclear whether these instances solely rely on trivial constructions, such as letting F be a class containing a single function, which is exactly the true map between Y and Z, with learning class G (map between X and Y) constructed as a trivial task. More importantly, from proof of theorem 7, this is exactly how the existence is proved, i.e., by taking the motivating example in section 2.1. Such constructions imply that (1) learning the mapping between modalities is exceedingly simple, and (2) one can directly determine the relationship between a modality and the labels without the need for learning, both of which are impossible in practice. This greatly limits the paper's ability to capture the underlying mechanisms behind the success of multimodal learning. On direction for improvement could be to seek for more realistic instances. 3. The authors claim that heterogeneity (e.g., lines 167, 255) is one of the two factors leading to the superiority of multimodal learning. However, heterogeneity appears to be more of a consequence rather than the underlying reason. The statement in line 167, explaining heterogeneity as 'multimodal data being easier to learn than unimodal'. Furthermore, in definition 6, heterogeneity is directly defined as the difference between population risks of learning with single and multimodal approaches. This raises my concerns about circular reasoning within the argument. As a result, section 4 and the arguments regarding heterogeneity currently lack clarity and informativeness.

Questions

In addition to the questions raised in Weaknesses, I am also curious if the framework presented in the paper can offer insights or potentially be extended to provide analysis for other methods, such as CLIP, where the objective is to learn shared representations for both modalities instead of learning a map from one modality to another.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have adequately discussed limitations in the paper.

Summary

The authors presents an interesting theoretical framework that allow them to estimate a generalisation bound for multimodal learning. The main result consist in proving the the bound for the multimodal case is superior to the unimodal one up to a factor $O(\sqrt{n})$ that depends on the sample size of the dataset considered.

Strengths

The paper is pleasant to read and the work is refreshingly original. Moreover, even if at an abstract level, it builds a theoretical framework for multimodal learning that was long missing in the community, especially given the importance and the relevance that multimodal models are acquiring in recent times.

Weaknesses

As also the author points out the results presented are rather abstract and more fine-grained analysis of specific multi-modal learning algorithms would be of broad interest. Another note is the notation usage, some key terms and notations should be briefly explained (it is sufficient in the appendix) to allow less experienced readers to follow the paper more straightforwardly.

Questions

Some questions that I would like to have feedback from the others on are: - How would you proceed to relax the condition of having a predictor class containing not only Lipschitz functions? - How would you generalise the results presented to the case of considering more than two modalities? Namely having $m$ modalities with multiple mapping functions linking single and combination of modalities. Do you see any specific limitation in the current framework?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

First of all, the authors did an amazing job in highlighting the limitations of their work in their manuscript. Here I simply highlight the major ones: - Lipschitz assumption on $\mathcal{F}$ (predictor class) are restrictive as it prevents a direct extension to DNN - Lack of a concrete analysis for specific multimodal learning algorithms

Thanks for the clarification

I value the clarifications provided by the authors and find no more questions. To recap, this paper adds valuable contributions to the theoretical realm of multimodal learning. Considering potential subjective preferences, I am inclined to believe that the mathematical rigor alone warrants its publication.

Thanks for your response

Thanks for your response! I'd like to better grasp the logic behind the role of heterogeneity. From my understanding so far, the notion of the "heterogeneity gap" assumes that the "population risk of learning a single modality is notably higher than that of learning both modalities, for any algorithm". In your argument, it seems you wanted to establish that this would lead to the conclusion that "population risk of learning a single modality is notably higher than that of learning both modalities, for a particular algorithm". I'm curious why the transition from the heterogeneity gap to this superiority isn't trivial, as the conclusion appears to be inherently contained within the definition of the heterogeneity gap. I'm also interested in understanding how the joint impact of heterogeneity and connection contributes to this superiority. In Theorem 4, it appears that heterogeneity alone is sufficient.

Thanks for the clarifications and the response

The authors properly addressed my questions and I confirm my evaluation of a strong accept.

Response to rebuttal

Thank you for the detailed response. I read the other reviewers' comments as well and agreed that the theory, as it is rigorously proven, might be sufficient by itself. The experimental setup that I would like to see as a reader and as a reviewer is also only toy experiments, which might be compensated by the simple examples at the beginning as pointed out by other reviewer. However, as I personally am not familiar with the other works assumptions well enough, I will leave it for the AC to decide (1) if the assumptions used in the paper are natural/general enough such that the theoretical results are of sufficient value; (2) As the paper considers multimodal generalization properties instead of multimodal optimization, if the corresponding results are of sufficient value. Thus I will increase my score from 4 to 5 and decrease my confidence score from 3 to 2.