Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms

Understanding and removing bias from the decisions made by machine learning\nmodels is essential to avoid discrimination against unprivileged groups.\nDespite recent progress in algorithmic fairness, there is still no clear answer\nas to which bias-mitigation approaches are most effective. Evaluation\nstrategies are typically use-case specific, rely on data with unclear bias, and\nemploy a fixed policy to convert model outputs to decision outcomes. To address\nthese problems, we performed a systematic comparison of a number of popular\nfairness algorithms applicable to supervised classification. Our study is the\nmost comprehensive of its kind. It utilizes three real and four synthetic\ndatasets, and two different ways of converting model outputs to decisions. It\nconsiders fairness, predictive-performance, calibration quality, and speed of\n28 different modelling pipelines, corresponding to both fairness-unaware and\nfairness-aware algorithms. We found that fairness-unaware algorithms typically\nfail to produce adequately fair models and that the simplest algorithms are not\nnecessarily the fairest ones. We also found that fairness-aware algorithms can\ninduce fairness without material drops in predictive power. Finally, we found\nthat dataset idiosyncracies (e.g., degree of intrinsic unfairness, nature of\ncorrelations) do affect the performance of fairness-aware approaches. Our\nresults allow the practitioner to narrow down the approach(es) they would like\nto adopt without having to know in advance their fairness requirements.\n