FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods

Response to Reviewer 8vvg [key points: explain the scope of our work]

Dear Reviewer 8vvg, We sincerely thank you for your time and effort in reviewing our paper and your positive support of our paper. We address your concerns point by point in the following: **Q: I think that authors should elaborate on the limited amount of of state of the art (un)fairness mitigation methods** Thank you for your valuable feedback on our paper. We appreciate your suggestion to include a broader array of methods for mitigating unfairness. Indeed, diversifying the techniques we explore could add depth to our research. In the following, we would like to clearfy the decision to focus on the most commonly used baseline fairness mitigation methods. 1. **Benchmarking most commonly used methods:** Our choice to concentrate on most commonly used methods was driven by a clear intent: to establish a solid foundation for fairness. By employing these well-accepted baseline methods, we not only adhere to established standards but also provide a reliable benchmark. This approach ensures that our findings are grounded in the field’s current understanding and opens avenues for direct comparison with existing research. 2. **In-Depth Analysis:** Our experimental design was not a mere surface-level examination on numerous methods. We conducted extensive experiments with these chosen methods to ensure a comprehensive and nuanced understanding. Our investigation delved into the intricacies and outcomes of these established techniques, offering a detailed and rigorous examination of their effectiveness and limitations across various scenarios. This depth of analysis contributes significantly to the body of knowledge. We thank you for your support of our paper again and hope that our response adequately addresses your concerns. Thanks,\ Authors

[1/2] Repsonse to Reviewer Tj8b [key points: explain the scope of our work; adjust the observation accordingly]

Dear Reviewer Tj8b, We sincerely thank you for your time and effort in reviewing our paper. Your comments that 'a good time to make a comprehensive comparison' and 'reasonable comparison of in-processing methods' are highly appreciated and encouraging. Most importantly, we value your suggestions and address them point by point in the following: **Q1: scope of this study seems a bit limited because it only considers in-processing methods..... Nowadays, performing in-processing training without sensitive attributes is also actively studied, and a comparison with these methods would be interesting. ......One of the future works is to include a wider range of in-processing group fairness methods. This direction should not be a future work as the current paper claims to be a complete benchmark for such methods.** *---we combine your three concerns about the number of benchmarking methods in this question.---* We thank you for your insightful and valuable suggestion. We agree that incorporating more settings would result in a more comprehensive fairness benchmark. This includes integrating more fairness algorithms at different stages (pre-processing, post-processing), expanding the definitions of fairness (individual fairness, counterfactual fairness), and performing in-processing training without sensitive attributes. In the following, we would like to clarify the decision to focus on the most commonly used baseline fairness mitigation methods. 1. **Benchmarking most commonly used methods:** Our choice to concentrate on the most commonly used methods was driven by a clear intent: to establish a solid foundation for fairness. By employing these well-accepted baseline methods, we not only adhere to established standards but also provide a reliable benchmark. This approach ensures that our findings are grounded in the field’s current understanding and opens avenues for direct comparison with existing research. 2. **In-Depth Analysis:** Our experimental design was not a mere surface-level examination of numerous methods. We conducted extensive experiments with these chosen methods to ensure a comprehensive and nuanced understanding. Our investigation delved into the intricacies and outcomes of these established techniques, offering a detailed and rigorous examination of their effectiveness and limitations across various scenarios. This depth of analysis contributes significantly to the body of knowledge. **Q2: Concluding that HSIC is the best approach seems misleading because not all in-processing methods were compared as explained above.** We thank you for your insightful suggestion. We have modified this observation to "The HSIC achieves the best utility-fairness trade-off among the tested methods". **Q3: Some of the key observations are already known in the fairness community.** In our paper, we acknowledge that several observations made might not be entirely new. We have carefully reviewed and removed some observations that are well-established in the field. Instead, our focus has shifted to highlighting those findings that have not been explicitly detailed in previous work as follows: - ➀ Not all widely used fairness datasets stably exhibit fairness issues. - ➁ Utility-fairness trade-offs are generally controllable by the hyperparameter. - ➂ The HSIC achieves the best utility-fairness trade-off among the tested methods. - ➃ Stopping training while learning rate is lower enough is effective. We also modified our paper about the findings from our experiments on Page 2.

Response to Reviewer dSax [key points: provide details about Table 4 and Figure 1]

Dear Reviewer dSax, We are genuinely grateful for the time and effort you devoted to reviewing our paper. Your acknowledgment of our work addressing a prominent problem in the Fairness community is highly appreciated. We are also pleased to hear that our writing is clear and our contribution is clear. Additionally, we are glad that our open-source code meets your standards and is easily adaptable for fellow researchers. Most importantly, we highly value your suggestions and have addressed each of them in detail in the following: **Q1: More detailed information about Table 4 this would be appreciated.** In Table 4, we offer recommendations on whether to use specific datasets for fairness, based on qualitative assessments rather than quantifiable measurements. Although we recognize the importance of the measurements, our current methodology focuses mainly on evaluating the numerical values of fairness metrics to form our recommendations. Regarding the experimental procedure, we conducted ten trials using the same neural network architecture but with varying data splits. **Q2: Figure 1: Are the results different because of different random seeds or did you change the gamma values of the loss functions to obtain different acc-fairness trade-offs?** In our experiments, we varied both the random seeds and the gamma values of the loss functions, which contributed to different accuracy-fairness trade-offs. Using all the raw results with various random seeds and gamma values ensures a comprehensive and realistic presentation of the results. The substantial number of runs (27,568) represented in these figures guarantees the reliability of our findings. We thank you for your support of our paper and hope that our response adequately addresses your concerns. Thanks,\ Authors

Response to Reviewer 5ZmP [key points: minimalistic code advances research objectives, adjust the observation accordingly]

Dear Reviewer 5ZmP, We sincerely thank you for your time and effort in reviewing our paper. We are encouraged by your comments that it 'solves a very important problem in the fairness literature', 'well states the challenges', and 'includes extensive experiments and summarizes their observations'. Most importantly, we value your suggestions and address them point by point in the following: **Q1: it would be better if the paper could provide more explanations on the strengths of FFB itself. It seems some explained characteristics (like minimalistic aspect) are not supported by enough convincing explanations** *---Since Q2 is based on the answer to this question, we will address it first.---* We thank you for these great and valuable comments. The term 'minimalistic' exemplifies our philosophy in this work: that code for research should be both simple and free from redundant elements. In this work, we firmly adhere to and follow this philosophy. This guiding principle also significantly shapes and differentiates FFB. - **Emphasis on Code Simplicity**: The FFB embodies our philosophy that simplicity is paramount in research code. Its design is deliberately straightforward, including the dataset loading function and metric functions, which can be easily understood without documentation. This simplicity not only facilitates modifications and extensions but also promotes seamless integration into one's own research. - **Facilitation of Research Objectives**: Adhering to these principles, FFB effectively furthers our goal of making research more accessible and focused. Researchers can concentrate on their primary objectives, avoiding complex or redundant code. Following the above two, we achieve our research goal that - **Comprehensive Evaluation of Commonly Used Fairness Methods**: We provide extensive experimental evidence to support previous observations, such as the instability of adversarial training and the utility-fairness trade-off. In addition, we also make new findings, including the presence of fairness in widely-used datasets and strategies for determining training stop. **Q2: The paper does not clearly explain how to use FFB for testing new algorithms or new datasets.** We thank you for your valuable suggestion. Our code follows our philosophy that simplicity is paramount in research code, making it easy to use FFB for testing new algorithms or datasets as follows: 1. **Testing New Algorithms:** The FFB is designed with a straightforward code structure, making it easy and accessible for integrating new algorithms. To test a new algorithm, you should follow the existing method style and use the dataset function and metric function. 2. **Incorporating New Datasets:** The FFB features a user-friendly Dataset loading API that offers flexibility to accommodate more datasets. To integrate new data, the function *load_XXXXX_data* serves as an initial step for incorporating new datasets. Below is the basic structure of the data loading function: ```python def load_XXXXX_data(path="../datasets", sensitive_attribute="sex"): # Code to load and process new dataset # X represents the features, y is the target variable, and s is the sensitive attribute return X, y, s ``` 3. **Comprehensive Fairness Metrics:** The FFB includes a wide range of fairness metrics, which, to the best of our knowledge, are the most comprehensive. They provide a thorough assessment of algorithm performance across different dimensions of fairness. These metrics can be easily used to evaluate both existing and new algorithms on various datasets and adapted to other research projects. **Q3: the paper explains their observations on several algorithms and datasets by using FFB, but many of the observations are not very surprising and already discussed in the literature..** In our paper, we acknowledge that several observations made might not be entirely new. We have carefully reviewed and removed some observations that are well-established in the field. Instead, our focus has shifted to highlighting those findings that have not been explicitly detailed in previous works, as follows: - ➀ Not all widely used fairness datasets stably exhibit fairness issues. - ➁ Utility-fairness trade-offs are generally controllable by the hyperparameter. - ➂ The HSIC achieves the best utility-fairness trade-off among the tested methods. - ➃ Stopping training while the learning rate is lower enough is effective. We also modified our paper about the findings from our experiments on Page 2. We hope our response adequately addresses your concerns. If so, we kindly request the reviewer to re-evaluate our work and consider raising their score. Thanks,\ Authors

Hope you recover soon

Dear Reviewer dSax, I'm sorry to hear you had an emergency medical intervention on your eyes. I hope the intervention was successful and you have a full recovery. Take good care of yourself and I look forward to hearing from you whenever you have fully recovered. Sincerely,\ Authors