Automatic quality control framework for more reliable integration of machine learning-based image segmentation into medical workflows

Machine learning algorithms underpin modern computer aided diagnosis software, which has proved valuable in clinical practice, particularly in radiology. However, inaccuracies, mainly due to the limited availability of clinical samples for training these algorithms, hamper their wider applicability, acceptance, and recognition amongst clinicians. We present a framework to evaluate state-of-the-art automatic quality control (QC) methods that can be implemented within these algorithms to estimate the certainty of their outputs and to sort out quantitatively insufficient predictions based on established thresholds. We demonstrate that such a framework reduces the number of segmentations which need manual evaluation by 84.5%. We validated the framework with two different QC approaches on a brain image segmentation task identifying white matter hyperintensities (WMH) which are particularly challenging to segment due to their varied size, and distributional patterns in magnetic resonance imaging data. Our work reveals how the evaluation framework for machine learning algorithms can help to detect failed segmentation cases and guide in the selection of a QC method, thereby making machine learning-based automatic segmentation more reliable and suitable for clinical practice.