In high-stakes applications of machine learning models, interpretability\nmethods provide guarantees that models are right for the right reasons. In\nmedical imaging, saliency maps have become the standard tool for determining\nwhether a neural model has learned relevant robust features, rather than\nartefactual noise. However, saliency maps are limited to local model\nexplanation because they interpret predictions on an image-by-image basis. We\npropose aggregating saliency globally, using semantic segmentation masks, to\nprovide quantitative measures of model bias across a dataset. To evaluate\nglobal saliency methods, we propose two metrics for quantifying the validity of\nsaliency explanations. We apply the global saliency method to skin lesion\ndiagnosis to determine the effect of artefacts, such as ink, on model bias.\n