We address the challenge of generating fair and unbiased image retrieval results given neutral textual queries (with no explicit gender or race connotations), while maintaining the utility (performance) of the underlying vision-language (VL) model. Previous methods aim to disentangle learned representations of images and text queries from gender and racial characteristics. However, we show these are inadequate at alleviating bias for the desired equal representation result, as there usually exists test-time bias in the target retrieval set. So motivated, we introduce a straightforward technique, Post-hoc Bias Mitigation (PBM), that post-processes the outputs from the pre-trained vision-language model. We evaluate our algorithm on real-world image search datasets, Occupation 1 and 2, as well as two large-scale image-text datasets, MS-COCO and Flickr30k. Our approach achieves the lowest bias, compared with various existing bias-mitigation methods, in text-based image retrieval result while maintaining satisfactory retrieval performance. The source code is publicly available at \url{https://anonymous.4open.science/r/Fair_Text_based_Image_Retrieval-D8B2}.
Paper
Similar papers
Peer review
Summary
The paper considers fair text-based image retrieval, in particular when the target retrieval set is biased. The chosen fairness metric is _equal representation_, e.g., the number of retrieved male and female images should be identical. The authors conjecture that if the target retrieval set is biased, debiasing the (multimodal vision-language) model during training is not sufficient to guarantee fairness of the retrieved results in test-time. Therefore, they propose Post-hoc Bias Mitigation (PBM): first, they sort the most relevant candidate images from each group (e.g., male and female) separately and then take equal amount of images from the two groups. In their experiments they consider different ways of estimating the (unknown at test time) sensitive attributes: zero-shot embedding, prompting and supervised classification. Experiments are executed on 4 standard datasets: Occupation 1 and 2, MS-COCO and Flickr30k.
Strengths
* The paper is well-written and easy to follow. * The prior work is presented fairly and it is somewhat clear how this paper is positioned with respect to it. It touches on a relevant topic with potentially high social impact. * The presented solution to the proposed problem formulation is intuitive, straightforward and seems to work well in practice.
Weaknesses
1. The idea of post-processing is not novel and the technical contribution of the proposed solution is limited and rather incremental, in my opinion. Hardt et al. (2016) and Calders and Verwer (2010) propose probability thresholding, while Karako and Manggala (2018) consider a similar procedure of iterative selection of retrieved images based on their relevance and diversity. I acknowledge that the problem settings in these works are sufficiently different from the current paper, but the idea that we can take an equal amount of images from the two demographic groups (sorted by relevance) if we want to equalize the number of images from both groups, is not particularly exciting or insightful, even if the demographic attributes are inferred in a zero-shot manner during test-time. Calders and Verwer, Three naive Bayes approaches for discrimination-free classification, 2010\ Karako and Manggala, Using Image Fairness Representations in Diversity-Based Re-ranking for Recommendations, 2018 2. It seems to me that the key factors for the performance (in terms of fairness; for a given, fixed query) of the method are the distribution of male/female among the ground-truth samples in the target retrieval set as well as the accuracy of $\hat{g}(\cdot)$ - the sensitive attribute predictor. I think it may be informative to further explore what happens when the distribution of samples from each demographic group is varied or when $\hat{g}(\cdot)$ is not very accurate. For example, according to Ghosh et al. (2021), "developers should not use inferred demographic data as input to fair ranking algorithms, unless the inferences are extremely accurate". Ghosh et al., When Fair Ranking Meets Uncertain Inference, SIGIR 2021 3. The details how to control the fairness/utility trade-off seem to be missing from the supplementary material (L350). I am unable to find other details from the SM as well (e.g., adversarial learning results, L187).
Questions
Q1: I think it will be prudent to show the distribution of the sensitive attributes for the target ground-truth retrieval set(s) ($V_{c}^{\*}$) as well as the whole test sets ($\mathcal{V}$), e.g., the male:female ratio per occupation in the Occupation datasets. I would be interested to see the relationship between bias/recall and the distribution ratio (between male/female in $V_{c}^{\*}$). For example, if the target retrieval set is relatively balanced, are the other baseline methods still biased? Q2: How do you control the bias-recall trade-off? Do you vary the probability (L348) from 0.1 to 0.9 on Figure 4? \ Minor: Are the results in Figure 4. for Occupation 1 and 2 datasets and gender as the sensitive attribute? Q3: How would you combine multiple sensitive attributes (e.g., if we want to be fair w.r.t. both gender and race in Occupation 2)? Q4: Is the problem formulation and the metrics standard/adapted from prior work or novel/custom? I see that you closely follow (Wang et al., 2021a), so might be good to mention this. Q5: Formal metric definitions. For Recall@K why don't you consider $\frac{1}{|C|}\sum_{c \in C} \frac{| V_{c,K} \cap V_{c}^* |}{|V_{c}^{*}|}$ ? Or it does not matter as $ | V_{c}^{\*} | = K$ is the same for all queries $c$? When computing the bias, why do you take the absolute value, instead of considering Bias@K, following (Wang et al., 2021a)? If you take the absolute values, how do you get negative gender bias in Figs. 2 and 3? Q6: When instantiating PBM, do you use the original CLIP as image/text encoders, or any of the other baselines MI-clip, Adversarial training, Debias Prompt? What is the effect if you instantiate PBM with any of these "debiased" VL models? Q7: What is the underlying reason for the recall to still be low? Are the VL models not good enough? What happens if we lower K and consider the precision, esp. for the Occupation datasets? Unless male:female ratio is 50:50, it is impossible to obtain 100% recall for the given query anyway, as we have to retrieve other images from the underrepresented group that do not correspond to the occupation for which we query. L320: should it be 61:39?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
In general, I suggest trying to (partially) include the Limitations section in the main paper or at least refer to it. Lim1: Fairness metric definition. Translated in the context of supervised classification, it would look like $p(g = 0 | \hat{y} = 1) = p(g = 1 | \hat{y} = 1)$, where the value of the sensitive attribute $g$ is unknown at test time and "prediction" of $\hat{y} = 1$ corresponds to the image being retrieved. This does not look like any other "standard" (supervised classification) fairness metric. I appreciate that the authors have included motivation about this design choice, but maybe some further discussion is needed. I would like to point out that in some cases or other scenarios the chosen metric may lead to unfairness, e.g., for the content providers (if a sample with lower utility is preferred over another solely due to having the "correct"/"desirable" sensitive attribute). Lim (comment) 2 [out of scope]: Depending on the setting, if there are insufficient number of samples with a particular sensitive attribute, synthetic samples might be generated (Tanjim et al., 2022). Tanjim et al., Generating and Controlling Diversity in Image Search, WACV 2022 Lim (comment) 3 [out of scope]: An end-to-end pipeline would need to decide if the input query is gender-neutral or not. E.g., querying for a famous actor/actress is clearly not gender-neutral. Then, is gender neutrality a 0/1 binary property, or it can be scored?
Summary
The paper tackles the problem of bias in text-image retrieval. The goal is to generate fair and unbiased image retrieval results given neutral textual queries. As opposed to previous approaches that aim to separate the learned representations of images and text from characteristics such as gender and race, the paper claims that these methods are insufficient for achieving the objective of equal representation, primarily because bias often persists during the evaluation phase within the target retrieval set. In order to address this, the authors propose PBM, a straightforward and efficient test-time post-processing debiasing algorithm that generates fair retrieval subsets, guided by predicted gender/race information obtained from an off-the-shelf classifier or inferred via zero-shot using the VL model itself and evaluate their approach on various benchmarks.
Strengths
The paper tackles an important problem. I think that addressing bias is a very important direction and this paper can provide useful insights to the research community.
Weaknesses
My main concern with the paper is that I don't understand if there is a link between the performance of the gender/race classifier and the bias. Since the performance of these classifiers directly affect the performance, I would have wished to see more ablations on this front. Some parts seem to be unpolished. For example Fig 4 is very hard to read, especially the last plot. Also, the colors are quite similar and it's very hard to assess which method is which.
Questions
1. What architecture do you use for the text encoder and video encode? I think this information should be added in sec 3.2 2. What is the performance of the gender prediction? (both zero-shot and supervised) Does the zero-shot prompt works better than supervised classifier? The results from Tab 1 seem to me to be in contradiction with Fig 4 (if I am reading it correctly) since AbsBias is always lower for supervised vs zero-shot, but in Tab 1 the results are different. 3. Any insights on why the performance drop is so large in Tab 2? Minor - Can the method be applied to other types of biases except gender and race biases? Are there any other datasets that have other bias information? What is needed to adapt the method to other biases?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations are discussed and a brief societal impact is also presented.
Summary
The paper proposes a method for mitigating test-time bias in image retrieval of vision-language models. More concretely, the authors demonstrate how prior work debiasing methods fail to ensure equal representation of different sensitive groups in the retrieval bag at test-time when the test set is imbalanced across different sensitive groups (for example, there are more male engineers at the test set than female engineers). To mitigate this issue, they propose a simple method where they train a sensitive group classifier (for example, a gender classifer) to split the test set across the sensitive groups and then perform the retrieval over each subset independently (if the task is to retrieve K samples, and the sensitive groups are: Male/Female then they would retrieve K/2 from the predicted Male group and K/2 from the predicted Female group). They show through experiments that their method reduces the disparity between the number of retrieved Male and Female samples at test time while maintaining “satisfactory” retrieval performance.
Strengths
1- The problem is well motivated; indeed, when the test set is imbalanced, it is clear that the prior work debiasing method, while not amplifying the bias of the test set, will nevertheless exhibit the proportional bias of the test set. 2- The method is simple and easy to implement. 3- The writing is clear, and the results are well explained.
Weaknesses
This work's main weakness is using a sensitive group classifier (for example, gender classifier) to split the test set into different subgroups. An apparent problem with this approach is that the sensitive group classifier will likely be biased. This is because, similar to how VL models correlate class (for example, professions) with gender, it will also correlate gender with class. So, for example, if the dataset has more Male engineers, then a gender classifier based on the VL model features is more likely to correctly predict engineers as male than predicting engineers as Females. This apparent problem is not discussed in the paper not are solutions proposed to address it. Furthermore, the experimental results could explain why this might be an issue. More concretely, the proposed method consistently reduces the recall performance, often by significant margins. For example, in Table 2, Recall 1 is reduced from 67.2 to 41.2 on Flickr30k. This pattern might be directly related to the bias of the gender classifier. Indeed, if the classifier, for example, is not good at predicting Female Engineers as Female, then the test set predicted Female split might contain very few engineers, and thus, the K/2 extracted samples from the predicted split might contain very few relevant engineers and thus reducing the recall performance.
Questions
Given the methodology’s weakness of relearning the bias through the sensitive group classifier, I am not confident in the validity of this method. In order to prove me otherwise, I encourage the authors to provide the following: 1- Report the difference between the recall performance on different sensitive groups in the test set. For example, for each profession (doctor, for example), I would encourage the author to report the absolute difference between the recall of the Male samples versus Female samples annotated in the test set and then average the metric over all professions. Their method will have a high absolute difference compared to other methods, thus exacerbating the bias. 2- Even if the results above are satisfactory, I would like to see a theoretical or methodological argument proving that the gender classifier will not demonstrate the abovementioned problem. Furthermore, I encourage the authors to include an experiment where prior work methods like MI-CLIP and FairSample but with explicit bias text in the prompt (e.g., “Male Engineer”) and then perform K/2 retrieval for each gender-based prompt. I expect this will result in a balanced retrieval bag at test time with minimal additional cost.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
1 poor
Presentation
2 fair
Contribution
3 good
Limitations
The issue above should be discussed in their limitations sections. I encourage the authors to expand their limitations section and include a more critical discussion of their use of sensitive group classification systems. Overall, I need more evidence about the validity of the contribution. However, I am happy to update my score in case I missed any central point that could be pointed out in the authors’ replies.
Summary
This paper proposes a post-hoc fairness method for text-to-image retrieval tasks. Instead of pre-processing the dataset or in-processing via fairness-derived regularization, this paper focuses on the post-processing of the retrieved items. The proposed post-hoc bias mitigation (PBM) utilizes a pre-trained bias estimator, e.g., a gender classifier. Using the gender classifier, PBM first classifies the gallery set (or test set) into male / female / NA sets. Then, it samples the final retrieval items from each set by balancing each gender for the final retrieval results. The experimental results show that the proposed PBM can show great bias mitigation while showing small performance drops for the target task.
Strengths
- This paper addresses an important problem in vision-language tasks, especially for text-to-image retrieval tasks. - The analysis of the source of the bias ("Two Types of Bias") is insightful and well-described. - The proposed method is easy-to-implement and practical. Moreover, as this method is a post-processing method that does not require a re-training of the model, this method can be readily applied to any text-to-image retrieval method. - Although the proposed method highly relies on the performance of the pre-trained gender classifier, this paper extensively studied many possible alternatives of the gender classifier, e.g., zero-shot embedding or zero-shot prompt. It is also pleasurable to see the "oracle", or the upper bound of the proposed method, by employing ground-truth gender labels. It shows the advantages and disadvantages of the proposed method more clearly. - This paper is easy-to-follow
Weaknesses
- The proposed method needs a pre-trained gender (bias) classifier. As shown in the experiments, the performances of PBM heavily rely on the quality of the gender (bias) classifier. Although this paper shows a generalizable approach using zero-shot prompts, I think it is an unavoidable weakness of this paper. - Compared to in-processing methods, PBM shows worse target task performances. I think it depends on the target objective of the actual user: if one needs a highly performed text-to-image retrieval system (i.e., with high recall) while allowing somewhat moderate discrimination, in-processing methods would be better. On the other hand, if one needs a strictly fair text-to-image retrieval system while sacrificing recall performances, the proposed PBM would be better. I would expect related discussions in the future revision, if possible. - As another fundamental limitation, PBM cannot improve the original target task (e.g., Recall@K) because it is a post-processing method. - The notion of fairness can be written in multiple ways. For example, the target fairness problem in this paper is similar to "demographic parity (DP)" problem, rather than "equalized odds (EO)". As far as the reviewer understood, mutual information minimization (MI) is more focusing on EO, while the proposed method is focusing on DP. In my opinion, both EO and DP are valuable problems, while a user has to choose a better fairness notion by considering the user's application. I think the problem raised by this paper is very important, especially considering that it is a retrieval task, not a classification task, but in my opinion, it would be helpful for the readers to discuss the target fairness scenario and the benefit of the target fairness scenario.
Questions
As my previous comment, I would like to expect more discussions related to the notion of fairness chosen in this paper. In my opinion, the notion of fairness can vary by situation.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
I don't think this paper has a potential negative societal impact. As my previous comment, the proposed method has fundamental performance limitations caused by (1) the need of an accurate gender classifier (2) the nature of the post-processing method (PBM does not update the original model, therefore, PBM cannot improve R@K). I think this paper is a generally okay paper, while its strengths slightly outweigh its flaws. In my opinion, this paper would become stronger if discussions of the target fairness scenario and other fairness scenarios when the listed limitation would be / not be critical.
Thank you for the detailed response. Indeed, while the gender classifier performance seems to have an impact on the recall per Figure 1 (rebuttal), the performance on the occupation dataset seems adequate to ensure good performance (Table 3). I encourage the authors to update the paper writing and include these new results in order to ensure no confusion for future readers. Eitherway, given these results, I am more confident about the ability of a reasonably accurate gender classifier to ensure adequate recall performance for PBM. It is also nice to see that combining PBM with prior work Fairness methods further improves these methods and sometimes outperforms PBM; this makes sense, given that the features are more robust. Overall, given the novelty of the problem definition and the adequate initial proposed solution, I upgraded my review to Weak Accept.
Thank you for your feedback and the decision to upgrade the review score for our manuscript. We genuinely appreciate your recognition and trust in our work. We will incorporate your suggestions in the revision of our paper.
I thank the authors for their clarifications and for providing further data and details. I still believe that the technical novelty is limited (and more rigorous datasets and setups are required to drive this problem further) but the major questions raised by me and most of the reviewers were addressed during the rebuttal, so I increase my score (to 5). I still hope that the authors can answer the following question (Q1): if the retrieval set is balanced, do the existing methods still result in more biased retrievals than PBM? Moreover, I also hope that the authors will acknowledge some of the limitations brought by me. In particular, (Limitation 1) some fairness decisions can still lead to unfairness (e.g., preferring a sample with lower utility over another solely due to having the "correct"/"desirable" sensitive attribute might be unfair and discriminatory to the content providers [1]). I find this particularly relevant to the discussion of the bias/recall tradeoff and the lower recall results on COCO and Flickr30k. [1] Wang et al., A survey on the fairness of recommender systems, ACM Transactions on Information Systems, 2023
Thank you for acknowledging our efforts in our rebuttal. We acknolwedge the limitations(1,2,3) highlighted in your comments and include them into our limitation section. Below is our response to your follow-up questions: #### *"(Q1): if the retrieval set is balanced, do the existing methods still result in more biased retrievals than PBM?"* Ideally, when the retrieval set is balanced, the retrieval bias of existing in-processing methods should be no more than our PBM's result, if the retrieval bag size is sufficiently large. To justify this idea, we conduct an additional debiasing experiment on an artificially balanced subset of Occupation 1 and Occupation 2. Here, we focus on the binary gender scenerio for simplicity. To ensure that the retrieval set corresponding to each query is balanced, we select images related to occupations that have relatively balanced gender representation in the original dataset. These occupations include 'chemist', 'pharmacist', 'insurance sales agent', 'biologist', 'author', 'cook', 'real estate agent' and 'doctor' . The resulting subset comprises 322 male and 326 female. Below, we show the debias results using in-processing methods such as MI-clip, Adversarial Learning and Debias Prompt, comparing to our PBM - Supervised Classifier method: | | AbsBias@100 | Recall@100 | |-----------------------------|-------------|------------| | Original CLIP | .0950 | 39.0 | | MI - Clip | .0550 | 34.9 | | Adversarial Learning | .0675 | 33.8 | | Debias Prompt | .1025 | 38.4 | | PBM - Supervised Classifier | *.0525* | *39.0* | From the results, we observe that the resulting bias of MI-clip and Adversarial Learning is very close to our PBM, thereby justifying the idea that if there is no dataset bias, in-processing methods should work as well as PBM. Debias Prompt does not perform well. We hypothesize that it is due to its debiasing effect failing to generalize to Occupation 1 and Occupation 2 dataset, consistent with the results shown in Table 1(paper). Interestingly, it is should be noted that in a balanced dataset, the PBM does not exhibit a drop in recall performance compared to the original CLIP. This is because the ground truth retrieval set aligns with the equal representation that PBM pursues. In contrast, MI-clip and Adversarial Learning experience performance drops due to their adverse effects on the feature space of the retrieval model. Thereby, when a balanced retrieval set is presented, employing PBM is also a promising choice, as it is less likely to have a negative impact on the retrieval performance. #### *"Moreover, I also hope that the authors will acknowledge some of the limitations ..."* Yes, we sincerely acknowledge the limitations (1, 2, 3) that you have highlighted in your comments. Specifically, for limitation 1, as we prioritize the "equal representation", our retrieval results sacrifice recall performance to ensure a retrieval bag that contains equal representations of each demographic group. From the standpoint of content providers, fairness should imply that similar samples are treated similarly, regardless of the demographic group membership of the sample. The former one(ours) is pertinant to group fairness[1], while the latter one(content providers' perspective) is more closely aligned with individual fairness[2]. The intrinsic conflict between group fairness and individual fairness[3] presents a significant challenge, making it difficult for our work to optimally address both. Determining the appropriate trade-offs between these two aspects of fairness is an important direction for future work. For limitation 2, we acknowledge that our work does not consider any techniques, such as using synthetic sampels, to mitigate the issues arising from insufficient representations of certain demographic groups. Addressing the bias in insufficient retrieval pools is a direction we plan to explore further. For limitation 3, we acknowledge that currently our work does not incorperate any techniques to identify if a query is neutral or not. Also, we acknowledge our assumption of gender neutrality being binary is limited. Whether the neutrality of a text query should be considered as binary or as continuous is an intriguing question to explore. Again, thank you for your insightful comments and suggestions. We are committed to continually refining our work. [1] Chouldechova, Alexandra, and Aaron Roth. "A snapshot of the frontiers of fairness in machine learning." Communications of the ACM 63.5 (2020): 82-89. [2] Dwork, Cynthia, et al. "Fairness through awareness." Proceedings of the 3rd innovations in theoretical computer science conference. 2012. [3] Binns, Reuben. "On the apparent conflict between individual and group fairness." Proceedings of the 2020 conference on fairness, accountability, and transparency. 2020.
Thank you. The extra details make me more confident in my score (increasing to 6). I hope the authors will be able to incorporate these clarifications as well as the suggestions from the discussions with the other reviewers in their final revision.
Thanks for the authors answering my questions. As my initial concerns are centered around its presentations that can be enhanced by minor modification, I will keep my rating as "weak accept". The listed revision plan looks great to me. Also, thanks for pointing out that DP and EO are defined for classification tasks. I totally agree with the response, but still, there are many readers who are not unfamiliar with algorithmic fairness. I would like to encourage the authors to emphasize the significance of the proposed fairness scenario in the revision.
Thank you for acknowledging our response and the proposed revision plan. In our revision, we will place greater emphasis on the proposed fairness scenario. We appreciate your constructive feedback and we are committed to enhancing the quality of our work based on your input.
Rebuttal
I acknowledge that I have read the rebuttal and the author response. I appreciate the detailed response that the authors made. While I think there still are things that need to be changed, the authors promised to address these in the final version, hence I raise my score to Weak Accept.
Thank you for your time and efforts in carefully reading our rebuttal and reconsidering the score of our paper. We are committed to making the necessary changes in the revision as promised.
Decision
Accept (poster)