Summary
The paper focuses on Bias Mitigation for face identification, i.e., ensuring that face identification works “well” for different identities: gender, race, etc. Unlike prior work, which focuses on model backbone agnostic methods to mitigate bias, this work explores the relevance of the inductive bias encoded in different Deep Learning model's backbones to the bias issue. In other words, are there model backbones better at learning robust features and ensuring good performance on samples from different identities? The authors uncover model configurations that significantly improve performance over standard model backbones through an extensive empirical analysis based on Neural Architecture Search on two face identification datasets. More surprisingly, the authors prove that these backbones' performance is better or more competitive than when paired with standard bias mitigation methods. Moreover, the authors confirm the generalizability of the configurations uncovered using CelebA and VGGFace2 by testing them on other datasets, further confirming their competitive performance. Finally, the authors analyze the new model configurations and ensure that the features learned by the models are less likely to be discriminative between the biased groups confirming that they learn more diverse and robust features.
Strengths
1- The work is well motivated; prior work needs to include an analysis of model architecture relevance to the bias issue.
2-The results are interesting and relevant to fairness/bias community practitioners.
3-The advantages of newly discovered model configurations are explored by a well-designed empirical analysis that confirms the configurations' more robust learned features.
Weaknesses
1- Some choices in the experimental design would benefit from further motivation. For example:
Why was the “multi-fidelity Bayesian optimization method SMAC3” chosen in particular? Are there other methods that could also work?
2- SMAC_301 was the architecture that works well across datasets. I understand that 301 denotes the operations that constitute the novel architecture. However, the authors do not discuss the details of these operations or why they think they are meaningful choices compared to other choices ruled out by the NPS. Some discussion here would be helpful. Why do the authors think this configuration is better able to learn non-linearly separable bias features?
3- In Section 3, why are the models trained on a gender-balanced subset of the dataset? Wouldn’t one want to train on a gender-imbalanced split to see which architectures are less likely to be influenced by the imbalance? Is this the same in Section 4? Please clarify.
4- In the analysis in Section 4.2, were the hyper-parameters (learning rate, optimizer) of the timm models tuned too? Or were they the ones used by the original papers? If it is the latter, then it is an unfair comparison to the SMAC models since those hyper-parameters were tuned as outlined in Section 4.1.
5- From reading the paper in detail, I understand now that one of the motivations of Section 3 analysis was to limit the number of architectures explored in Section 4 (only DPN was considered since it achieved Pareto optimal performance on both datasets). However, I did not get that from the first read, so further clarification in the text about that would be helpful.
6- In Section 3, the hyper-parameters of the different models were not optimized. It is likely very computationally expensive. Nevertheless, it does undermine some elements of the analysis, so I would make that clear as limitations.
Questions
I am overall positive about this work. However, I need further clarifications as outlined in the Weakness sections, particularly questions: (2,3,4). I am happy to increase my score upon adequate further clarification.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I commend the authors for explicitly discussing their work's technical limitations and that while it improves the notion of technical fairness, the advancement could still be harmful in downstream applications.