Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation

In large studies involving multi protocol Magnetic Resonance Imaging (MRI),\nit can occur to miss one or more sub-modalities for a given patient owing to\npoor quality (e.g. imaging artifacts), failed acquisitions, or hallway\ninterrupted imaging examinations. In some cases, certain protocols are\nunavailable due to limited scan time or to retrospectively harmonise the\nimaging protocols of two independent studies. Missing image modalities pose a\nchallenge to segmentation frameworks as complementary information contributed\nby the missing scans is then lost. In this paper, we propose a novel model,\nMulti-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute\none or more missing sub-modalities for a patient scan. MGP-VAE can leverage the\nGaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the\nsubjects/patients and sub-modalities correlations. Instead of designing one\nnetwork for each possible subset of present sub-modalities or using frameworks\nto mix feature maps, missing data can be generated from a single model based on\nall the available samples. We show the applicability of MGP-VAE on brain tumor\nsegmentation where either, two, or three of four sub-modalities may be missing.\nOur experiments against competitive segmentation baselines with missing\nsub-modality on BraTS'19 dataset indicate the effectiveness of the MGP-VAE\nmodel for segmentation tasks.\n