Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this modality-complete assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with a simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during training, we further improve the model learning with pseudo-supervision indicating the reliability of a modality's prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaobai1217.github.io/Unseen-Modality-Interaction/.
Paper
Similar papers
Peer review
Summary
This work studies the problem of learning interactions of unseen modality combinations. Specifically, all training data is modality-incomplete, and the model must learn to perform inference on modality-complete data. The paper claims to be the first to study inference under such settings, and proposed two novel improvements to tackle this challenge: (1) feature projection layer that projects all encoded modalities into the same dimensionality and constrained by an alignment loss, and (2) a dual-branch prediction layer that predicts a pseudo-label in addition to the real label. The paper performed experiments on 3 datasets that contains a diverse set of modalities, domains and tasks (including classification, retrieval and regression). The paper included thorough ablation studies on their methods to justify all of their design choices, and they showed that under the their new setting, their method significantly outperforms previous modality-complete and modality-incomplete approaches.
Strengths
1. This paper proposed a new setting: learning to infer from unseen combinations of modalities, and proposed a new method to tackle this challenge. 2. The paper's experiments and evaluations are very comprehensive. The experiments involved 3 datasets that contains a very diverse set of different domains (kitchen videos, robotics, Youtube videos), modalities (14 total modalities), and task (classification, regression, retrieval), and the diversity shows that their method generalizes well. There is a comprehensive ablation study that justifies the design choice of each model component as well as loss function. Since this is a new setting, the authors re-implemented or re-ran several existing approaches on the new setting and showed that the new method outperforms all of them.
Weaknesses
The presentation/clarity of the paper needs improvement, especially in some parts of the methods section. For example, the use of different variable names in the feature projection are inconsistent (e.g. F'm in line 96 and line 102 have different dimensions); the notation of the alignment process is confusing, and the intuition behind the whole alignment process is unclear; and the description of how exactly the psudo-labels are obtained is very vague and unclear. See Questions section for more details.
Questions
(1) I think figure 1 can be a bit misleading, since in this paper, we focused on the setting where the test set is always the union of all modalities. (2) The dimension of F'm on line 96 and 102 are inconsistent. (3) For feature projection, is it correct that we simply do a linear combination to change the sequence dimension? When we change dm to d*, do we have a fully connected layer that maps each dm-dimensional vector to a d* dimensional vector individually, or does it map the entire k*x dm matrix to a k* x d* matrix? (4) On line 111, why do we average the features within each modality? It seems to me that the alignment loss is sort of like a vector-qualitization process, and it is difficult for me to understand why quantizing each modality separately (as opposed to, for example, quantizing each d-dimensional vector in the sequence, or the average of the vectors in the same corresponding locations across different modalities) helps features from different modalities occupy a common space. Wouldn't the current approach prevent the projected features from occupying a common space? For example, we could have modality 1 always close to the first few u in the dictionary, and modality 2 always close to the next few u, etc. (5) In equation 1, the notation of u_m is a bit confusing. I think it is supposed to mean the closest vector in [u1, u2,... ] to fm, but it could be confused with the mth vector in [u1,u2,...] (6) What exactly does "average across training epochs" mean? Since we need the pseudo-labels during training, do we just average the first-branch prediction on this data point from each previous epoch? How do we obtain this for the first epoch? Also, is it correct that the pseudo-labels are probability-distributions across all labels in the classification case? I also have a hard time understanding the intuition behind dual-branch prediction helping with overfitting problem. Perhaps a more clear and detailed description on how the pseudo-labels are obtained could make it more clear. (7) In line 167, since there are 3 modalities in this dataset, how exactly are they partitioned into the two training set partitions?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
2 fair
Contribution
3 good
Limitations
Limitation discussion is adequate.
Summary
The paper is about multimodal learning and deals in particular with the mismatch between modality combinations at training and inference time. The authors propose to project the multimodal features in a shared space, apply an alignment, and enforce the discriminative ability of the method with a dual branch prediction with full and pseudo supervision. The method is evaluated on three scenarios, ablation studies and comparative analysis are reported.
Strengths
Overall, the paper is well-written, with some originality with respect to previous approaches. What is reported in the document is clear, figures are appropriate and descriptive of the concepts.
Weaknesses
My main concerns on the paper are the following. On the main goal: while I understand the benefit of learning from multiple modalities while being able to use a single modality at inference time, I am not sure I fully understand the benefit of learning according to the set defined in Sect. 2, where at training time the method learns from a group of modalities which is a subset of what is seen at inference time. I expected to see in the empirical evaluation a quantitative justification that using at inference time a superset of the training modalities helps to improve the results. For instance, what if at inference time I use the combination used at training time, with no additional modalities? Is it really helpful to have these extra modalities at inference time? This is not clear to me. On the objectives: in my opinion, some somehow strong statements of the authors on the abilities of their method are not appropriately justified with an empirical evaluation. For instance, the authors state that their method is more robust to scenarios where some modalities are corrupted by severe noise. More robust with respect to who and what? There is only one experiment tackling this challenge where only the proposed method is used. Moreover, the authors state that one of their challenges "...is reducing the overfitting to the specific modality combinations from the modality-incomplete training data." Again, I think this would deserve more attention on the comments to the experimental analysis I am not sure I fully understood how the architecture I structured: I assume the architecture is accommodating a maximum number of input modalities while disregarding (or having learnable tokens for) the ones that are not present. From what I understand this requires the input modalities to be provided always in the same order. Is this interpretation correct?
Questions
- I find it confusing having a separation of the considerations on the existing literature at the beginning and at the end of the manuscript, I would group them - On the same topic, the authors mention existing approaches and how their method is different. However (at least in the first part) only for some of them ([25, 30]) they explicitly mention limitations, what about the others? - As reported above: the statement "Our approach enables effective unseen modality interaction and is more robust to scenarios where some modalities are corrupted by severe noise." would require a deeper investigation I think - What about the ability to generalize to new tasks/settings, different from the ones of the training set? What I mean is: if you have in training data collected with a certain combination of sensors from a certain platform to solve an action classification problem for instance, could the method be able to generalize to test data for the very same problem and possibly different sensors combinations acquired with a different platform? - The section on method lacks details in the description that are probably then reported at the beginning of the section on experimental analysis. For instance, it is said in Section 3 that the attention matrix Om is obtained through several transformer layers. Be sure all the details are properly reported. This is very important for the reproducibility of the results - I don’t understand when the alignment is applied, before or after the integration via the sum. - Eq. (2) appears without the appropriate context, I find - “… we propose to generate pseudo-supervision which reflects the discriminability a modality combination” Sentence to be rephased - Are the splittings in the training-validation-test used for the experiments provided with the datasets? If yes it should be stated, if not it should be justified - How many epochs for the training stage? Learning rate? Again, this is important to share as many implementation details as possible to favour the reproducibility - References to tables 7 and 8 should be 2 and 3 instead - “While our model learns from modality-incomplete data, previous experiments use modality-complete data in testing” It would be better to cite the methods - In the section “Benefit for Modality-Incomplete Testing” it is not quite clear how the experiment has been designed. Are the combinations in Figure 3 used at inference time? What was the training? I think some details are missing. The same considerations can be done for other sections, for instance “Benefit for Noisy Modalities“ - In the experiments: “We use publicly available implementations where available, otherwise re-implementing ourselves.” Please be more specific: for which ones you used public implementations?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors explicitly mention some limitations of their method in the last section.
Summary
This paper introduces a method that can enhance the performance of multimodal models in scenarios involving unseen modality interactions.
Strengths
1, The issue of "unseen modality interaction" explored in this paper is quite novel. 2, This paper maps the features of different modalities into the same space and merges different modalities by simple addition. As the number of modalities increases, the number of parameters required for fusion does not significantly increase.
Weaknesses
1, The performance of the baseline in this paper is so low that it makes the entire method proposed by the paper unconvincing. Specifically, although the method of this paper surpasses its own set baseline methods on EPIC-Kitchens, it only achieves an accuracy of 23.7%. Meanwhile, the most naive baseline in the paper [1] also has an accuracy of 23.7%. Therefore, it's hard for me to be convinced by the experimental conclusions of this paper. 2, In Table 2, the authors compare the performance of different methods on different datasets, but why can't many of these methods outperform a simple late-fusion? Is it because there are significant differences in certain settings? If the author cannot clarify the situation here, I would consider the experimental comparison to be very unfair. 3, Also in Table 2, the performance of DEQ Fusion in MM-IMDB is 61.52/53.38, while the performance of MMBTl[2] is 66.8/61.6 (Micro F1/Macro F1). Why choose not to report this method? It's a well-known classic approach that has already one hundred of citations. [1] What Makes Training Multi-modal Classification Networks Hard? [2] Supervised Multimodal Bitransformers for Classifying Images and Text
Questions
See Weaknesses
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
I don't think the experimental results of this paper convince me, so I tend to reject this paper.
Summary
This paper tackles the issue of unseen modality interaction, which challenges the conventional assumption of modality completion during training. The approach taken in this study involves formulating a training setting that accounts for modality incompleteness. Subsequently, the proposed method focuses on projecting multiple modalities into a shared feature space through an alignment objective and leveraging a pseudo-labeling strategy to alleviate the model's tendency to overfit to unreliable modalities. The experimental results on multiple benchmarks demonstrate the efficacy of the proposed framework.
Strengths
- The paper addresses a practical problem, since incompleteness of modality occurs often in reality. - The authors conduct their experiments on the datasets with various modalities.
Weaknesses
- There are some ambiguous parts in the manuscript. - The authors mention that L_{align} ensures the projection of features from different modality spaces into the same feature space. However, based on Eq (1), it appears that the loss is computed as the summation of the difference between modality-specific features and modality-specific learnable tokens. In light of this, how can we ensure that the features from all modalities are projected into the same feature space? - Regarding the dual branch prediction part, please provide further elaboration on the intuition behind the pseudo-labeling strategy to address less discriminative modalities. Can we consider less discriminative modalities as unreliable modalities? Even if a modality is less discriminative than others, it may still contain important information for the model. I am curious about the purpose of using pseudo labels to suppress such less discriminative modalities or encourage them. - The numbering of tables is incorrect in the experimental section. For instance, the table referenced in line 160 should be Table 1, not Table 6. Most of the tables are misnumbered. - Line 273 needs to be ended with ‘.’
Questions
- In a similar context, is there any reason to split $\hat{F}$ into exactly half? Since the contribution of $L_{pseudo}$ is extremely far from $L_{supervised}$, may be dividing $\hat{F}$ into half is not the optimal solution. - In line 146, the paper mentions selecting the modality-specific pseudo-label that is closest to the ground truth annotation. How is the distance measured between the label and ground truth annotation when using a one-hot vector label? - Why the number of video classification accuracy in Table 2, 3 (23.7%) and Table 4 (23.8%)?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The paper sufficiently deals with the limitation of the paper in the conclusion.
Thank you to the author for responding to my questions and pointing out my mistakes. I have raised my score to borderline accept.
Still on Table 2
I thank the authors for the detailed responses to my concerns. They clarify a number of points raised in my first review. However, I still do not fully understand Table 2 and how I should interpret it. Considering the proposed approach focuses on settings where some of the modalities at inference time might not be available at training time, I do not quite understand how I should read Table 2. In particular, is each column referring to a specific combination at inference time? If yes, what happens in the Multimodal approach? What modalities are employed at training time in such cases? Considering the way the setting is defined, I would have said only one, but then where is the multimodality? Thanks in advance for your clarifications
Clarification on Table 2
***We thank the reviewer for engagement and encouragement***. The reviewer is correct that in this ablation we only have one modality for each training sample (e.g., either RGB only or audio). However, we have two modalities (the unseen modality combination) during inference. For each column, we consider two different modalities to study the unseen modality combination at inference. While each video sample in EPIC-Kitchens contains all the three modalities, we divide the original training set into two splits and let only one modality be available in each split during training. We leave the test set as is. For example, for the RGB & Audio column, one training split has RGB available only and the other training split has audio available only. During inference, both of the two modalities are available. We train an unimodal encoder on each split for the video classification task. The performance of these unimodal encoders are reported in the three rows of the `unimodal’ part in Table 2. For the multimodal part in Table 2, the late fusion indicates we directly average the predictions from the unimodal encoders of the two modalities during inference. For the rest of the multimodal approaches, we send a single modality into each variant of our multimodal model during training while we send both modalities into the model at inference time. We will expand the setting description to make this clearer.
Thanks again for the feedback
I thank the authors again for their detailed response. I will increase my rating
Thanks for the response.
Thanks for the responses. I still want to discuss the following points. Regarding $L_{align}$, is it correct to say that $u_{n_{m}}$ is not uniquely determined for each modality? In other words, within the set $[u_{1}, …, u_{n_{u}}]$, is it possible for features from different modalities $(f_i, f_j)$ to be assigned to the same $u_{k}$? This scenario would occur if $u_{k}$ is the nearest token for both modalities. Consequently, does the proposed objective facilitate the projection of features from distinct modalities into a shared space? It’s the authors’ claim, right? Please kindly correct me if there are any misconceptions in my understanding. For the pseudo-labeling, I now understand what the authors tried to do after reading the response. Within this context, I believe that the paper could benefit from a more comprehensive exploration of the pseudo-labeling process. Specifically, the inclusion of analysis or qualitative results of the pseudo-labels generated for various modalities would support the authors' claim and enhance the overall robustness of the paper. One more thing to point out is that I think the sentence “the RGB modality is more discriminative than optical flow or audio” in line 124 should be revised. For video classification [A, B], optical flow often performs better than RGB, so the statement is not always correct. [A] : Carreira et. al, Quo vadis, action recognition? a new model and the kinetics dataset, CVPR 2017, [B] : Wang et. al, Temporal segment networks: Towards good practices for deep action recognition, ECCV 2016
Further Clarifications
**We thank the reviewer for engagement and the opportunity for further clarification**. **$L_{align}$**. The reviewer is correct that $u_{n_{m}}$ is not uniquely determined for each modality. It is possible for features from different modalities $f_i, f_j$ to be assigned to the same $u_k$ when $u_k$ is the nearest token for both modalities. The proposed objective does facilitate the projection of features from distinct modalities into a shared space. We further verify this claim by computing the feature distance between modalities on EPIC-Kitchens with the variants of our model used in Table 2 of the main paper (we provide more explanations for Table 2 in the response to reviewer 2qSh). Specifically, we first obtain the average feature before the multimodal transformer per class for each modality and then compute the average Euclidean distance between modalities across classes. For RGB & Audio, after adding the feature projection to the vanilla transformer, the average Euclidean distance reduces from 84.1 to 75.4. For RGB & Flow, the distance reduces from 81.3 to 72.5 and for Audio & Flow, it drops from 83.9 to 76.2. We will add this into the paper. **Comprehensive Exploration of Pseudo-Labeling**. In the second row of Table 3, we show that when using the same supervised loss with groundtruth for both branches instead of the pseudo-labeling, our multimodal model suffers a 2.3% accuracy decrease. This is because the pseudo-labeling eliminates the overfitting to groundtruth labels, which can be harmful when a particular modality combination cannot give a reliable prediction. We also observe that when using the pseudo-labeling, the validation accuracy becomes higher than the groundtruth-supervision only, with the same number of epochs. Since we cannot provide figures in the discussion phase, we describe several examples for various modalities here and will provide the qualitative examples in the appendix. For the RGB modality, given a video sample of taking bowl, the pseudo-label has a probability of 0.60 for *take bowl* while 0.30 for *take cup* since the activity happens out of view and it is hard to judge whether there is a bowl or cup. For the audio modality, given an audio track of opening cupboard, the pseudo-label has a probability of 0.50 for *open cupboard* and 0.45 for *open fridge* as the sounds are similar. For the optical flow modality, given a video sample of taking soy milk, the pseudo-label has a probability of 0.35 for taking soy milk while 0.65 for milk since the objects have a similar motion appearance. As a result, forcing our multimodal model to be far away from any of the similar activity classes would result in overfitting. We will add the clarifications into the paper. **Inappropriate Statement**. The reviewer is quite right that our statement on RGB discriminability is far too general. We will modify the sentence “the RGB modality is more discriminative than optical flow or audio” into “for activity recognition in EPIC-Kitchen videos, the RGB modality is often more discriminative than audio”. Thank you.
Finalize rating
Thanks to the author for responding to my questions. Although I believe the framework is not super-effective or fancy, and there's room for enhancement in the manuscript, this submission is valuable as a pioneer of a novel and practical problem setup. Therefore, I raised my rating as borderline accept.
Decision
Accept (poster)