Due to the nature of opaque machine learning (ML) models, software engineers and data scientists struggle to understand how ML models make decisions [1]. Explainability research aims to provide transparency for these models [2] through two types of explanations. Global explanations describe how a model works generally and provide insight into its accuracy, biases, and fairness. Local explanations describe individual predictions made by the model in specific use cases.