We successfully demonstrate the use of explainable artificial intelligence (XAI) techniques on astronomical datasets in the context of measuring galactic bar lengths. The method consists of training convolutional neural networks on human classified data from Galaxy Zoo in order to predict general galaxy morphologies, and then using SmoothGrad (a saliency mapping technique) to extract the bar for measurement by a bespoke algorithm. We contrast this to another method of using a convolutional neural network to directly predict galaxy bar lengths. These methods achieved correlation coefficients of 0.76 and 0.59, and root mean squared errors of 1.69 and 2.10 respective to human measurements. We conclude that XAI methods outperform conventional deep learning in this case, which could be reasonably explained by the larger datasets available when training the models. We suggest that our XAI method can be used to extract other galactic features (such as the bulge-to-disk ratio) without needing to collect new datasets or train new models. We also suggest that these techniques can be used to refine deep learning models as well as identify and eliminate bias within training datasets.