Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

Clinical decision-making in oncology involves multimodal data such as\nradiology scans, molecular profiling, histopathology slides, and clinical\nfactors. Despite the importance of these modalities individually, no deep\nlearning framework to date has combined them all to predict patient prognosis.\nHere, we predict the overall survival (OS) of glioma patients from diverse\nmultimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to\ncombine information from multiparametric MRI exams, biopsy-based modalities\n(such as H&E slide images and/or DNA sequencing), and clinical variables into a\ncomprehensive multimodal risk score. Prognostic embeddings from each modality\nare learned and combined via attention-gated tensor fusion. To maximize the\ninformation gleaned from each modality, we introduce a multimodal\northogonalization (MMO) loss term that increases model performance by\nincentivizing constituent embeddings to be more complementary. DOF predicts OS\nin glioma patients with a median C-index of 0.788 +/- 0.067, significantly\noutperforming (p=0.023) the best performing unimodal model with a median\nC-index of 0.718 +/- 0.064. The prognostic model significantly stratifies\nglioma patients by OS within clinical subsets, adding further granularity to\nprognostic clinical grading and molecular subtyping.\n