Summary
This paper addresses the challenge of generalized missing modalities in multimodal learning, where a modality can be absent during any learning phase (e.g., training, testing, or both). he authors investigate prompt learning with missing modalities and propose deep correlated prompts designed to capture various types of correlations between prompts and input features across different modalities. Specifically, the proposed prompts include mechanisms for perceiving beneficial information from preceding layers, dynamically generating prompts based on input characteristics, and leveraging the complementary information from multimodal inputs. These designs improve the robustness of large multimodal models (e.g., CLIP) to missing modalities. Extensive experiments and ablation studies demonstrate consistently superior performance and verify the effectiveness of the proposed method.
Strengths
1. This paper addresses a more challenging missing modality setting, where modalities may be absent during both training and testing phases, making it highly practical and essential for real-world applications.
2. The paper is well-motivated. The authors highlight the weaknesses of prior work and propose several designs (e.g., deep correlated prompts, dynamic prompts, common prompts) to improve robustness.
3. The paper explores various types of correlations between prompts and input features across different modalities, and the proposed designs for each are technically sound.
4. Extensive experiments show great improvement on the baseline and consistently superior performance compared to other methods across all benchmarks.
5. Comprehensive ablation studies are conducted to validate the effectiveness of each proposed component.
Weaknesses
1. The paper lacks a detailed explanation or discussion on the efficacy of different prompt designs. In Figure 2, it shows that sequentially adding different designs improves the baselines, but it does not discuss the individual improvement gains for each design. Additional discussion on each design could help validate whether the increasing gains from sequentially adding designs are not merely due to more learnable parameters.
2. The paper lacks visualization of each learnable prompt (e.g., deep correlated prompts, dynamic prompts, and common prompts). Visualizations could help validate whether the different components work as expected. For example, do dynamic prompts genuinely capture the different characteristics of inputs, or do they merely distinguish between different missing cases, which might be easier to learn due to the obvious absence of a modality?
3. For each available modality, it seems there are a total of $(3*(2^M-1))$ prompts for each missing modality case. This could lead to an exponential increase and redundant prompts as more modalities are considered (i.e., M>2). For example, in a vision-and-language task, in the case of complete and missing-image, the text modality is available for both cases. However, it requires two separate prompt sets for the text encoder, which may actually learn the prompts for the same “text-available” case.
Questions
1. In Table 4, I noticed that some values of the related work MMP are the same as the figures recorded in the paper. For example, the settings with:
- missing rate = 70% (100% image and 30% text) in MM-IMDb,
- missing rate = 70% (30% image and 100% text) in Hateful Memes,
- missing rate = 70% (65% image and 65% text) in Hateful Memes.
As far as I know, the MMP backbone model is the multimodal transformer ViLT. The authors state they re-implemented MMP on their setting (i.e., CLIP) for a fair comparison. It seems that the numbers should not be the same since they use different backbone models. Can the authors clarify why the values are identical despite using different backbone models?
2. According to the design of prompts, it seems that the proposed method is not limited to two-stream models (e.g., it could be applied to single-stream models without using Eq. (5)). Generalizing the method to single-stream models and comparing it with related works could be helpful in verifying the generalizability of the proposed method. Have the authors tried it for single-stream models? If so, what were the results?
3. I am willing to revise my rating if the authors also address the concerns mentioned in the weaknesses.
Limitations
One limitation is that the proposed method requires modality-specific deep correlated prompts for each available modality, which could be challenging to extend to more modalities (e.g., five or more modalities).