Understanding and quantifying uncertainty in black box Neural Networks (NNs)\nis critical when deployed in real-world settings such as healthcare. Recent\nworks using Bayesian and non-Bayesian methods have shown how a unified\npredictive uncertainty can be modelled for NNs. Decomposing this uncertainty to\ndisentangle the granular sources of heteroscedasticity in data provides rich\ninformation about its underlying causes. We propose a conceptually simple\nnon-Bayesian approach, deep split ensemble, to disentangle the predictive\nuncertainties using a multivariate Gaussian mixture model. The NNs are trained\nwith clusters of input features, for uncertainty estimates per cluster. We\nevaluate our approach on a series of benchmark regression datasets, while also\ncomparing with unified uncertainty methods. Extensive analyses using dataset\nshits and empirical rule highlight our inherently well-calibrated models. Our\nwork further demonstrates its applicability in a multi-modal setting using a\nbenchmark Alzheimer's dataset and also shows how deep split ensembles can\nhighlight hidden modality-specific biases. The minimal changes required to NNs\nand the training procedure, and the high flexibility to group features into\nclusters makes it readily deployable and useful. The source code is available\nat https://github.com/wazeerzulfikar/deep-split-ensembles\n