Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds

Most NLP datasets are not annotated with protected attributes such as gender,\nmaking it difficult to measure classification bias using standard measures of\nfairness (e.g., equal opportunity). However, manually annotating a large\ndataset with a protected attribute is slow and expensive. Instead of annotating\nall the examples, can we annotate a subset of them and use that sample to\nestimate the bias? While it is possible to do so, the smaller this annotated\nsample is, the less certain we are that the estimate is close to the true bias.\nIn this work, we propose using Bernstein bounds to represent this uncertainty\nabout the bias estimate as a confidence interval. We provide empirical evidence\nthat a 95% confidence interval derived this way consistently bounds the true\nbias. In quantifying this uncertainty, our method, which we call\nBernstein-bounded unfairness, helps prevent classifiers from being deemed\nbiased or unbiased when there is insufficient evidence to make either claim.\nOur findings suggest that the datasets currently used to measure specific\nbiases are too small to conclusively identify bias except in the most egregious\ncases. For example, consider a co-reference resolution system that is 5% more\naccurate on gender-stereotypical sentences -- to claim it is biased with 95%\nconfidence, we need a bias-specific dataset that is 3.8 times larger than\nWinoBias, the largest available.\n