FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

Summary

The paper “FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion” introduces a novel framework called FuseMoE, designed to tackle challenges associated with multimodal data in machine learning. This framework addresses key issues such as missing elements, temporal irregularity, and sparsity in data, which are prevalent in fields like healthcare where data is often incomplete and irregularly sampled. The core innovation in FuseMoE is its gating function that integrates various modalities efficiently. The framework enhances predictive performance through: 1. Sparse MoE Layers: Incorporating sparsely gated MoE layers to manage tasks and learn optimal modality partitioning. 2. Laplace Gating Function: A novel gating function theoretically proven to ensure better convergence rates than traditional Softmax functions. 3. FlexiModal Data Handling: Efficiently handling scenarios with missing modalities and irregularly sampled data trajectories. The framework is validated through empirical evaluations on diverse prediction tasks, demonstrating its effectiveness in real-world applications.

Strengths

1. Existing demand of a more flexible gating function in the field of multimodal MoE makes the idea of this paper promising. Also, the possible implementations of this new MoE system is promising and may lead to some practical medical tools. 2. The theoretical proof is sufficient and rigorous. 3. The experimental design is comprehensive from the perspectives of scenarios and data, to a certain extent corresponding with the theory. The appendix contains a large amount of supplementary information, which helps verify the completeness and correctness of the article.

Weaknesses

1. The comparison works are all very outdated (the latest one UTDE[1] is publicized in 2023, with over half of the compared works predating 2020). Why not compare with more recent works? 2. Fig.1 lacks clear description, as the data processing workflow is not well explained. 3. Gaussian gating and your proposed form are very similar in format, with formula $h(x)=\operatorname{Top} \mathrm{K}\left(-\|W-x\|_{2}^{2}\right)$. Thus, theoretical comparison with original softmax may be unfair as the form is derived from and similar to gaussian gating. Therefore, although theoretical results may be valid, their importance diminishes due to this aspect. [1] Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling

Questions

Please refer to the weaknesses. The biggest question at my side is that the similarity with gaussian gating may lead to some theoretical misunderstanding. Any further explanation on this topic could lead to higher score.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors adequately addressed the limitations, there is some redundancy in the encoder part that heavily draws inspiration from UTDE's work. From the appendix it can be observed that encoder might have a greater impact on overall performance than what gate function alone contributes to improvement; hence further discussion is needed on whether this gate function must necessarily be tied to methods related to mTAND and imputation for obtaining higher-quality features as inputs for downstream MoE layers from an implementation perspective. Of course, the discussion on this topic is optional.

Summary

This paper proposes an MOE-based model to handle multimodal data fusion. It addresses two challenges: missing modalities and irregularly sampled data trajectories. A Laplace gating function is applied to the MoE Backbone. An entropy regularization loss is proposed to ensure balanced and stable expert utilization. The author validates the method in diverse datasets.

Strengths

1. Strong motivation. Two challenges presented in this paper, missing modalities and irregularly sampled data trajectories, are important in the multi-modal data fusion area. It is reasonable to address them through sparsing encoding and gating networks in the MOE layer. 2. Good theoretical analysis. This paper presents a detailed theoretical analysis of the Laplace gating over the standard Softmax gating in MoE. Other mathematical proofs seem to extensively illustrate the characteristics of the proposed MOE model. 3. Good experimental results. Comprehensive evaluations of FuseMoE on image and video datasets seem to validate its effectiveness.

Weaknesses

Addressing the following weaknesses may improve the paper: 1. The author should clearly distinguish the proposed method and others’ modules. From the paper, the Laplace gating function is proposed as a new one. I am not sure the author made some contributions to the encoder design, router design, and loss design. The author should make more illustrations about the contributions, not just combine other people’s work together. 2. The experimental results are not extensive. The author should clearly demonstrate the data modality of the chosen benchmarks. It seems that other modalities, such as text, are not included. The author should explain this. 3. Please explain more concisely how the gating functions can stabilize the imbalance and sparse multi-modal data. The paper should point this out more concisely and better with experimental results. Too many mathematical proofs and theorems in Sec.3 seem not helpful in illustrating the advantages of the proposed method.

Questions

Please see the weakness part to answer the questions.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

The Paper introduces “FuseMoE”, a novel mixture-of-experts (MoE) framework that can handle multi-modal data even in scenarios with missing elements, sparsity of samples and temporal irregularity. It proposes an innovative Laplace gating function with theoretical proof to enhance convergence rates and predictive performance across various tasks. The Laplace gating function allows sparser distribution of weights among MoE and encourages more balanced utilization of experts with sharper peaks and heavier tails preventing certain experts to be over dominant. FuseMoE includes modality and irregularity encoder using a discretized multi-time attention (mTAND) module to support generic fleximodal data ingestion with unlimited input modalities instead of other popular pairwise setup that exist in the literature. Paper also outlines various router designs for processing multimodal inputs and discusses the tradeoff between them and demonstrates . Key: 1. Laplace gating function showed superior performance compared to Softmax gating across multiple tasks 2. Unlike baseline models, FuseMoE demonstrated improved scalability with increasing number of modalities 3. Handles missing modalities with tAND and per modality routers with entropy loss function 4. theoretical analysis of convergence rates for parameter estimation 5. FuseMoe outperformed baseline methods on several fleximodal datasets.

Strengths

1. Sparse MoE backbone with novel Laplace gating function that ensures better convergence rates compared to traditional Softmax with theoretical guarantees 2. Modality and Irregularity Encoder to use mTAND module to descretize irregularly sampled observations to mix with continuous features, Effective management of missing data and irregular sampling 3. MoE fusion layer to integrate embedding from different modalities with different router designs. Flexibility to handle variable number of modalities because of per modal experts routing 4. Novel methods, strong theoretical contributions and comprehensive emperical results and evaluation

Weaknesses

1. Potential over-parameterization when input is small 2. limited discussion of computational needs and scalability to huge datasets 3. Most of the items discussed were already established and proved. Paper seems to be combining existing literature for these datasets.

Questions

1. The paper claims to work with unlimited input modalities. is it really so? how do you anticipate the fuseMoE to handle extremely sparse data across modalities? Any thoughts/estimation on how far we can go here? How does the complexity of setup scale with number of modalities? 2. Is Laplace only tested with CIFAR-10 dataset? Why was it not tested with other bigger datasets? 3. Have you thought about or explored the interpretability side of expert assignement process? 4. Few other paper discusses and introduces Laplace instead of gaussian for MoE. Can you expand more on how FuseMoE is differnt from these? - Nguyen, Hien D., and Geoffrey J. McLachlan. "Laplace mixture of linear experts." *Computational Statistics & Data Analysis* 93 (2016): 177-191. - Wu, Lc., Zhang, Sy. & Li, Ss. Heteroscedastic Laplace mixture of experts regression models and applications. *Appl. Math. J. Chin. Univ.* **36**, 60–69 (2021). https://doi.org/10.1007/s11766-021-3591-2

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Potential limitation of over-parameterization of networks especially when input data/size is small is adequately discussed in the paper.

Summary

This papers proposes a novel MoE arcchitecture for handling and fusing multiple modalities, along with two core contributions: - a novel router design that can handle missing modalities; - a laplace gating function that is theoretically proven to ensure better convergence.

Strengths

- 1 This paper is generally well written and easy to follow; tables and figures are neat and informative; - 2 The motivations of core designs are well elaborated; - 3 Theoretical proofs are providied to further justify its effectiveness; - 4 Extensive experiments, along with diverse modalities. are conducted to validate the effectiveness of the proposed approach. - 5 The capabilty of handling missing inputs is interesting (Fig. 4 c).

Weaknesses

- 1 The proposed method achieves promising results on the tested benchmark, which are still comparably small scale datasets, e.g, CMU-MOSI is with 2000+ samples. I am wondering if this work can be applicable to training/fine-tuning with larger scale data. - 2 Again, for vision task, only CIFAR-10 is used. This cannot fully justify its effectiveness for more general vision or multi-modal tasks; - 3 The capability in handling missing modalities is promising. The auhor is encouraged to explain why, in some scenarios, mising modalities + the proposed method surposses the variant with full modalities (Fig 4 c). I am still wondering its generalizability under more scenarios. - 4 The visualization and quanlitative analysis of the learned gaing weight would help the reader better understand the method.

Questions

- 1 In table 5, Laplace gaiintg shows limited superiority against other variants, any insight for it? - 2 In table 4c, why the method achieves better results with less modaltieis? Any further insight or ablations?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N

Official Comment from reviewer 6Ay7

I thank the authors for the rebuttal and for taking the time to conduct additional experiments and comparisons. The rebuttal and the additional clarification address most of my concerns so I am happy to increase my score from 6 to 7. Cheers!

Summary After Author-Reviewer Discussion

Dear Area Chairs and Reviewers, As the Author-Reviewer Discussion period comes to an end, we would like to provide a brief summary of our paper at this point for your reference. We have carefully reviewed all comments and have provided detailed responses to each. These will be comprehensively addressed in the final version of the paper. We extend our sincere gratitude to all reviewers for their insightful feedback and suggestions. ## **Motivation**: We introduced FuseMoE, a novel MoE framework specifically designed to enhance multimodal fusion for inputs with arbitrary missingness or irregularity. Multiple reviewers (`P6x6`, `6SUH`, `sZSN`) have **agreed that the problem we are addressing is well-motivated and represents a promising direction for investigation**. ## **Method**: FuseMoE incorporates sparsely gated MoE layers in its fusion component, which routes each modality to designated experts specializing in those specific data types while mitigating the influence of missing modalities. **This capability is critical to our method, as it allows for leveraging a vast amount of patient data without relying solely on complete modalities.** As we explained to Reviewer `P6x6`, most other methods that only leverage full modality data result in the exclusion of patient records with incomplete modalities. FuseMoE, by contrast, greatly improves generalization by accessing a broader array of samples. Additionally, FuseMoE employs various router design mechanisms tailored for multimodal inputs with distinct combinations or interactions, giving users the flexibility to choose based on their specific applications. ## **Theory**: FuseMoE integrates a novel Laplace gating function, which is particularly well-suited for multimodal input data, especially in large-scale experiments, compared to the commonly used Softmax gating function. Theoretically, it ensures better convergence rates than Softmax, as discussed in detail with high-level intuitive explanations in our rebuttal to Reviewer `6SUH`. Additionally, we have illustrated the superiority of our gating function over the Gaussian gating function in our response to Reviewer `sZSN`. ## **Experiments**: We demonstrate that FuseMoE significantly outperforms existing well-known and representative methodologies within specific categories, in integrating diverse input modality types: - `Text`: MIMIC-III, MIMIC-IV, CMU-MOSI, CMU-MOSEI - `Images`: MIMIC-IV, CIFAR-10, ImageNet (new) - `Time series`: MIMIC-III, MIMIC-IV, PAM - `ECG waveform`: MIMIC-IV - `Video frames`: CMU-MOSI, CMU-MOSEI - `Audio signals`: CMU-MOSI, CMU-MOSEI with varying missingness and irregular sampling on challenging tasks: - `Mortality prediction`: MIMIC-III, MIMIC-IV - `Length-of-stay`: MIMIC-III, MIMIC-IV - `Phenotyping`: MIMIC-IV - `Sentiment analysis`: CMU-MOSI, CMU-MOSEI - `Activity recognition`: PAM - `Image classification`: CIFAR-10, ImageNet (new) **The majority of the multimodal datasets we tested are considered large-scale**, including MIMIC-III with 36,212 samples, MIMIC-IV with 73,173 samples, and CMU-MOSEI with 22,777 samples. These datasets cover a sufficient number of benchmarks and research areas, particularly in a context where benchmarks are not abundant. **We also conducted comprehensive ablation studies and visualizations of FuseMoE** to assess the effectiveness of each building block and enhance the interpretability of our proposed method. These studies revealed that our model architecture consistently outperforms other alternatives and insights into how much each input modality contributes to our tasks of interest. ## **Reproducibility**: We have submitted (1) implementations of FuseMoE and its baselines, (2) scripts to process and generate multimodal input data as part of our supplementary material. Thank you again for your efforts in reviewing this submission. We appreciate all the valuable feedback, which has significantly contributed to improving our manuscript. **We look forward to continuing the discussion with any reviewers who still have questions during the remaining discussion period.** Sincerely, Authors

Kindly Request for Reviewer's Feedback

Dear Reviewer `sZSN`, We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the **Author-Reviewer Discussion period ends in just around 12 hours from this comment, i.e., 11:59 pm AoE on August 13.** We are happy to answer any further questions you may have before then, but we will be unable to respond after that time. If you agree that our responses to your reviews have addressed the questions you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments! Sincerely, Authors