Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts

Summary

The paper aims to deal with the task of AD patient classification with missing modalities, and proposes a framework, Flex-MoE. Flex-MoE also utilizes a learnable missing modality combination bank to complete the missing embeddings of missing modalities. The training strategy and the gating mechanism of Flex-MoE are also well-designed to handle this task. The experiments verify that Flex-MoE is able to achieve SoTA performance in this task on the shown modality combinations.

Strengths

- The task explored by this study is practical in real-world scenarios. Addressing the issue of missing modalities in the AD domain is worth the attention of the community. - The introduction section is well-written and easy to follow. (suggestion: using .pdf or .svg format for the figures is recommended.) - The motivations for the proposed techniques in Flex-MoE, which you have claimed in your paper, are rational and decent. - The ablation and sensitivity study is sufficient, showcasing the effectiveness of the components in Flex-MoE.

Weaknesses

- The expression from Line 159 to Line 167 is a little bit confusing. More explanation (or even illustration) is needed. - Is the usage of the missing modality bank just a simple look-up operation? I am concerned that it seems to be naive and there may be a more smart approach to achieve this. - There is not $R_{(x_j)}$ of Line 199 in your equations. You need to correct this. - Why the S-router can be trained to activate the corresponding expert index by Eq. (3)? - It would be better if you could use some equations to assist to express the content of Line 204-215. - This paper also leaks sufficient discussion of some related works, such as other tasks of biomedical images [1,2], in Sec.2 or appendix. - The modality combinations of the experiments in Table 2 is few. It would be better to provide the experiments based on more combinations to showcase the performance of Flex-MoE. [1] Zhou, D., Gu, C., Xu, J., Liu, F., Wang, Q., Chen, G., & Heng, P. A. (2023). RepMode: learning to re-parameterize diverse experts for subcellular structure prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3312-3322). [2] Jiang, Y., & Shen, Y. (2024). M$^4$oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Experts. arXiv preprint arXiv:2405.09446.

Questions

Please see the section of weaknesses. I may raise or lower my score after the rebuttal.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have claimed the limitations and broader impacts of the proposed method.

Summary

The paper presents a multimodal learning framework, Flex-MoE (Flexible Mixture-of-Experts), designed to integrate diverse modalities in Alzheimer's Disease (AD) research using a Sparse Mixture-of-Experts design. Flex-MoE sorts samples based on the number of available modalities and processes them through modality-specific encoders. The framework trains experts with full modality samples and uses an S-Router to adapt the knowledge for fewer modality combinations.

Strengths

1. The research topic addressing missing modalities in Alzheimer’s disease is crucial and highly relevant. 2. Source code is provided, facilitating the replication of experiments.

Weaknesses

1. The model's specificity to Alzheimer's disease is unclear, as it appears to be a general approach for handling missing modalities. 2. The experimental setup lacks clarity. Details on how labels (e.g., dementia, CN, MCI) were decided, the prediction timeline, and the handling of patients at different stages are not provided. Additionally, the use of clinical data and whether it is time series data is not explained. 3. The evaluation is unconvincing. The imbalance rate of the datasets is not discussed. Only accuracy and F1 score are used for evaluation, but additional metrics like AUC could provide a more comprehensive understanding of the model’s performance. The process for selecting the threshold for classification to obtain the F1 score is also unclear. 4. The computational resources required for training and inference using Flex-MoE are not specified, especially given the need to train the model for various modality combinations. 5. No external dataset is included to validate the model's performance.

Questions

1. Why is the model specifically designed for Alzheimer's disease, and how does it differ from a general model for missing modalities? 2. How were the labels (dementia, CN, MCI) decided? What is the prediction timeline, and were patients at different stages treated as separate samples? What time period of data was used, and was the clinical data time series data? 3. What is the imbalance rate of the datasets? Including metrics like AUC can provide a more complete understanding of the model’s performance. How was the threshold chosen for determining the classification boundary to obtain the F1 score? 4. What specific computational resources are required for training and inference using Flex-MoE? 5. How would the model perform on an additional external dataset?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

1. The evaluation and practical application of the model are challenging to assess. The necessity of training different models for each modality combination is not well justified. 2. The lack of external dataset validation limits the generalizability of the model's performance.

Summary

This paper introduces Flex-MoE, a novel multimodal learning framework for Alzheimer's Disease that handles missing modalities using a Sparse Mixture-of-Experts design and demonstrates its efficacy on the ADNI dataset.

Strengths

1. The idea of Flex-MoE is clear and straightforward, effectively addressing the challenge of integrating and handling missing modalities in Alzheimer's Disease research. 2. The expression is smooth and clear, making the complex concepts and methodologies easily understandable.

Weaknesses

1. The paper emphasizes that the Sparse Mixture-of-Experts (SMoE) selectively activates only the most relevant experts to improve scalability, but does not provide experimental results demonstrating computational efficiency or scalability. 2. The interpretation of \( B_{M \setminus m, m} \) in Equation (2) is unclear; it should be better explained in details. Whether it involves a learnable module that outputs missing modality features based on available modality features, or via other method? 3. The approach to handling missing modalities differs from the method in the paper "Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling," which uses averaging for missing modalities. A comparison with this approach would be beneficial. The paper provides code https://github.com/billhhh/ShaSpec. 4. The paper overlooks some existing missing modality baselines, such as mmFormer and ShaSpec. Some minor weaknesses: 1. The abstract states that "few recent studies attempt to integrate multiple modalities," but there are actually many methods for handling missing modalities (as noted in point 4 above), which are not compared experimentally in this paper. 2. The font size in Table 1 is too small and needs to be redrawn for better readability. 3. In line 180, the bold formatting for "It" is incorrect. 4. In Table 2(b), the value 61.08 in the first row is not the best result, which should be corrected. 5. Table 2 should ideally use a visualization approach similar to that in the ShaSpec paper to represent different modality combinations more clearly.

Questions

The most biggest question for me is pointed in weaknesses Q2: The interpretation of \( B_{M \setminus m, m} \) in Equation (2) is unclear; it should be better explained in details. Whether it involves a learnable module that outputs missing modality features based on available modality features, or via other method?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Please refer to the weaknesses section

The authors have addressed my concerns well

Dear authors and AC, Sorry for my late reply. The authors have addressed my concerns, with additional experiments, explanations, and discussion for some unclear expression in the paper. I would like to raise my score to 7, if the authors can add these contents to the main paper or the appendix. Thanks for your time and effort. Best, Reviewer 6yys

Adjusted the score to 7

I'd like to adjust my score to 7 according to the efforts of rebuttal by authors