HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data

Summary

The paper proposes an attention based early-fusion method for multi-modal learning. Specifically the paper uses a perceiver style iterative updates to a latent vector using a fusion layer. The fusion layer sequentially updates the latent vector using cross-attention with each of the modalities. The approach supports cross-modal interactions and provides a way to handle missing modalities at inference time. The authors compare the approach against existing baselines for fusing biomedical modalities and show promising results on the survival prediction task.

Strengths

The paper is well written and provides good context on existing methods. The proposed approach is simple and explained clearly. The iterative style fusion is useful as it can support cross-modal interactions without exploding the embedding dimension and can also handle missing modalities at inference time.

Weaknesses

Its unclear how much the iterative fusion is helpful as the parameters are shared. Can the authors provide more some information about when the iterative fusion is useful and how many iterations are typically needed. The proposed iterative style updates to the latent vector seems similar to an LSTM style aggregation. In many VLM and multimodal papers (https://arxiv.org/pdf/2405.09818, https://arxiv.org/pdf/2304.08485), the modality specific tokens are projected to a shared latent space and are aggregated through self-attention across the shared space. This provides more opportunity for cross modal interactions than iterative cross-attention. Have the authors compared with such approaches? The results are shown for a specific task (survival prediction) and specific modalities where the datasets are small and models are prone to overfitting. Its unclear if this approach works well for problems which need more complex interactions and where dataset sizes are larger.

Questions

Its unclear how the features from different patches in the WSI aggregated? What is the embedding dimension? Do the early-fusion models have more parameters? If so are they more prone to overfitting on these datasets? Since the latent array is updated sequentially with information from different modes, how much does the order of fusing modalities impact the result?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Given the weaknesses and open questions, I'm basing my decision but would be open to changing the decision post rebuttal once the questions and weaknesses are addressed.

Summary

The authors present HEALNet as an early fusion approach for integrating different data modalities. HEALNet utilizes an end-to-end training process with an additive method for combining modalities, rather than handling them in parallel. This strategy enables HEALNet to scale and adapt effectively to datasets of varying sizes and characteristics without requiring explicit pre-training.

Strengths

- The idea of the authors to implement an additive approach to combine modalities instread of processing them in parallel is promising.

Weaknesses

- The methodology is not clearly written and is difficult to understand. - The presentation also falls short. For instance, Figure 3 features a bar plot that is unreadable. - Additionally, the color scheme in the attention map seems unusual; typically, a blue-red gradient is used as it is more intuitive. - The visualization analysis is lacking. There is only one figure displaying the model's inspection capabilities, which features a WSI attention map. - The iterative attention mechanism, with multiple steps and updates, can be computationally expensive. This could be particularly challenging when dealing with very high-dimensional data or when scaling up to larger datasets. - There is no ablation study regarding the number of the fusion layers. The number of fusion layers is a critical parameter that can significantly affect the computational efficiency of the model.

Questions

- Could you include metrics such as as Floating Point Operations (FLOPs) and the number of trainable parameters to provide a more comprehensive understanding of the computational cost and efficiency of the HEALNet approach? - We have seen in recent papers [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://arxiv.org/html/2405.11618v1) that in cases where representations come from different sources or modalities the different repressentations are aligned in a meaningful way to facilitate effective learning and integration. How does your model handle discrepancies in feature scales or semantic differences between modalities?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The methodology needs to be clearer. The dimensions of the cross-attention layers are not clearly defined, which complicates understanding of how these layers align and integrate features from modalities of differing dimensions. Additionally, the visualizations and illustrations require significant improvement, as current figures, such as bar plots and attention maps, are difficult to interpret and hard to read. Enhancing these visual elements and providing clearer explanations of the methodology will improve the quality of the paper.

Summary

This paper presents HEALNet, a method for end-to-end multimodal fusion of mixed-type biomedical data. In contrast to methods like feature concatenation or Kronecker product fusion, HEALNet employs an iterative cross-attention structure that operates on the raw input modalities, representing a hybrid between early and intermediate-fusion approaches. This allows for the necessary cross-modal interaction, while natively handling missing modalities at test time and aiding interpretability (by operating on raw data rather than embeddings). HEALNet outperforms existing state-of-the-art multimodal fusion methods on survival analysis from histopathology and multi-omic data.

Strengths

- The problem motivation is very clear, and the explanation of related prior work is thorough. Care is taken to properly inform the reader of relevant background so that the authors can demonstrate why the proposed method is unique. - The structure and logical flow of the paper is excellent. Tables and illustrations are of high quality throughout. - The proposed method is interesting and is designed to solve domain-specific problems in biomedical multimodal fusion such as handling missing modalities and interpretability, while still enabling powerful cross-modal learning. - The experiments appear to be soundly conducted, and results are convincing against competitive relevant baselines.

Weaknesses

- Treatment of hyperparameters is somewhat confusing. It is unclear if Bayesian hyperparameter optimization (HPO) was applied to all methods or just to HEALNet. If HPO was applied to all methods, then why do their hyperparameters not appear in Table 5? These are important details to ensure a fair comparison between methods. - The latent update in Equation 2 is unclear, and this seems to be a critical component of the proposed method. What *exactly* is this function $\psi(\cdot)$? Also, there is an additional parameter $\rho$ to $\psi(\cdot)$ in the pseudocode that is not present in Equation 2. This needs to be clarified.

Questions

- Was Bayesian HPO applied to all methods or only HEALNet? Why does only HEALNet appear in Table 5? Please clarify this and which exact hyperparameters underwent optimization; for instance, it is confusing that all shared hyperparameters appear to be identical except the L1 regularization term. - What exactly is the update function $\psi(\cdot)$? What is the parameter $\rho$ in the pseudocode, and why is it not present in Equation 2? Minor comments: - L18: Unnecessary to use MMML acronym if never used again - L83: “additive approach to combining modalities (rather than handling them in parallel).” It is unclear what is meant by “additive” – sequential? - L118: “fusing operators” -> “fusion operators” - L125: Remove semicolon - L145: I would write “…Learning Network and its key component, (b) the early fusion layer (as given in Equation 3).” - Eq 4: Use \left[ and \right] to make square brackets larger - L186: Can omit “contains” - L187: Consider citing https://ieeexplore.ieee.org/abstract/document/10230356/, which addresses multimodal fusion methods to prevent overfitting in multimodal fusion. This is particularly relevant since this paper also considered fusion of histopathology imaging with tabular data, and forms results in contrast to the parameter-intensive Kronecker product as well.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Several limitations are adequately addressed in Section 6.

Summary

The authors present a multi-modal fusion architecture, named HEALNet, which aims to preserve modality-specific structural information and capture cross-modal interactions in a shared latent space. HEALNet enables intuitive model inspection by learning directly from raw data inputs instead of opaque embeddings. The iterative paradigm can skip modality update steps to handle missing modalities. The authors validate the framework through multi-modal survival analysis on four cancer datasets.

Strengths

1. The paper is well-organized and clearly written, with a logical flow of information that is easy to follow. 2. The iterative paradigm's ability to skip modality update steps to handle missing modalities is a practical approach.

Weaknesses

1. HEALNet shares a very similar structure with Perceiver. The authors should highlight the innovative parts more clearly to help readers understand which aspects are novel and which are based on related work. 2. The evaluation is limited to survival analysis. Given that this is a multi-modal fusion framework, it would be beneficial to perform additional tasks, such as classification or regression, to better evaluate its performance. 3. The sample sizes for the four datasets used are relatively small. Instead of 5-fold repeated random subsampling, more repetitions should be conducted, and confidence intervals should be reported to ensure the reliability of the results. 4. In addition to the concordance index, other metrics or plots such as cumulative proportion surviving plots should be reported to better visualize the results.

Questions

1. Higher performance was reported in the compared paper MOTCat [1]. Why did this happen? 2. Since the modalities are different, what is the rationale for sharing weights across all fusion layers in HEALNet? How is modality-specific structural information preserved during the fusion process?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. The evaluation of HEALNet is limited to survival analysis, which may not fully capture its potential across different biomedical tasks. 2. The small sample sizes and limited number of datasets may affect the generalizability of the results. 3. The absence of a comprehensive set of performance metrics limits the understanding of the model's strengths and weaknesses.