Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

Measuring bias is key for better understanding and addressing unfairness in\nNLP/ML models. This is often done via fairness metrics which quantify the\ndifferences in a model's behaviour across a range of demographic groups. In\nthis work, we shed more light on the differences and similarities between the\nfairness metrics used in NLP. First, we unify a broad range of existing metrics\nunder three generalized fairness metrics, revealing the connections between\nthem. Next, we carry out an extensive empirical comparison of existing metrics\nand demonstrate that the observed differences in bias measurement can be\nsystematically explained via differences in parameter choices for our\ngeneralized metrics.\n