Biases with respect to socially-salient attributes of individuals have been well documented in evaluation processes used in settings such as admissions and hiring. We view such an evaluation process as a transformation of a distribution of the true utility of an individual for a task to an observed distribution and model it as a solution to a loss minimization problem subject to an information constraint. Our model has two parameters that have been identified as factors leading to biases: the resource-information trade-off parameter in the information constraint and the risk-averseness parameter in the loss function. We characterize the distributions that arise from our model and study the effect of the parameters on the observed distribution. The outputs of our model enrich the class of distributions that can be used to capture variation across groups in the observed evaluations. We empirically validate our model by fitting real-world datasets and use it to study the effect of interventions in a downstream selection task. These results contribute to an understanding of the emergence of bias in evaluation processes and provide tools to guide the deployment of interventions to mitigate biases.
Paper
Similar papers
Peer review
Summary
This paper studies the issue of biases present in evaluation processes like hiring and school admissions. The authors propose a model that estimates the distribution of utility that incorporates two main features: resource constraints for information, and risk-averseness of the decision-maker. They formulate an optimization problem to estimate the utility distribution with two parameters that represent the aforementioned two features. They study the effect of these two parameters on the solution of the optimization problem, and they conduct a numerical study to study the effect of interventions in a downstream selection task.
Strengths
The problem of bias in evaluation processes is a significant problem, and I like the approach of understanding how bias can emerge in this process via a stylized model. The authors model two important phenomena, resource constraints and risk aversion, that have been shown to arise in many settings and can contribute to bias. The authors use real-world datasets to validate their study.
Weaknesses
The exposition of the paper was poor, making it challenging to comprehend the main message of the work. I understood the paper as positing a stylized model of an evaluation process that incorporates resource-information and risk averseness, and the main question that is studied is to understand how these two features contribute to the emergence of bias. However, I did not get a satisfactory understanding of this question from reading this work. 1. The main point of confusion was the lack of a formal model of bias, and an interpretation of what the main framework, OptProg, represents in reality. OptProg outputs a density of scores, $f_{\epsilon}$, where $f_{D}$ is a “true” distribution of scores. I interpret this via the following example: the distribution of SAT scores across all students ($f_{\epsilon}$) is not equal to the distribution of “true ability” of same students ($f_{D}$). However, only looking at the distribution seems insufficient for understanding “bias”, since it does not specify how _each_ true score gets mapped to each “biased” score. This is the source of the issue in some of my next comments. 2. Section 3.2 studies how tau and alpha influence the solution to OptProg, which seems to be the main contribution of this work. However, this section relies on specific distribution examples and numerical investigations, with long discussions and alot of notation that was difficult to follow. Moreover, it fails to explain why mean and variance are the relevant statistics of interest and how they capture the notion of "bias". Providing explicit theorem statements would have greatly improved this section. 3. The numerical study had several parts that I found confusing: - in Section 4.1, fitting the OptProg model to real-world datasets and evaluating the TV distance does not demonstrate the model's ability to capture "Ability to capture biases in data" (section title). Why does a small TV distance imply bias? - The model fits the best tau and alpha, but wouldn’t tau = -\infty always be the best tau? Also, is it the case that alpha > 1 is a better fit than alpha <= 1? - The interventions aimed at changing tau and alpha from the best fit would increase the error in estimation (increase TV). Why would these interventions improve utility? The confusion arises from the lack of a formal model of utility and bias. - There was no clear conclusion from the numerical results: each intervention was the best in some regime. What should the reader take away from this, and how should a DM use these results? - To strengthen the study, I suggest including numerical analyses that (a) validate the model and (b) rigorously estimate alpha and tau for different groups to demonstrate the existence of this type of bias.
Questions
Please let me know if I have any misinterpretations in my review.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Yes.
Summary
The paper proposes an optimization-based framework for modeling bias in evaluations. The perspective of the paper is to provide a well-founded and interpretable model of evaluations that can replicate biases observed in real settings without invoking an intrinsic utility for producing biased evaluations. This model is a generalization of previous models for evaluation bias which assume a particular parameterized model for the output density (e.g. Gaussian or Pareto). The primary utility of this model is to help policy-makers study the effectiveness of different interventions aimed to reduce bias, such as requiring proportional fairness of evaluations or decreasing the informational cost of evaluations. The authors apply their model to data such as IIT JEE-2009 scores, obtaining a closer fit with realized scores than previous models, and test out different interventions. Specifically, the paper models the density of evaluations f_{\mathcal{E}} as the output of an optimization problem, in which the evaluator seeks to produce an evaluation as close to the true value v of the individual as possible, which is drawn from a density f_{D}. Closeness is measured by a loss function \ell(x,v) between evaluation x and quality v. This optimization problem is subject to constraints and adjustments which introduce bias, which are well-founded in the literature. The first is information constraints. Evaluations are noisy in practice because it is difficult and costly to obtain a clear evaluation signal, and this cost can vary across different groups (e.g. groups that speak different languages). This is modeled as a constraint requiring that the entropy of f_{\mathcal{E}} be lower-bounded by some \tau. The second source of bias is risk-aversion: the evaluator's loss may not be symmetric between over and under-estimation of the true value, and may penalize over-estimation more. This is modeled in the loss function between the evaluation and the true value v, in which loss aversion (parameterized by \alpha\geq0) causes the evaluator to penalize over-estimation of v more than under-estimation by a factor of \alpha. They derive an exponential functional form for the solution of this optimization problem, which is standard by the maximum-entropy principle. They also perform sensitivity analysis of how the solution changes with \tau and \alpha. This modeling framework generalizes the implicit variance and multiplicative bias models. The only inputs to the model are the information constraint \tau, risk aversion parameter \alpha and loss function \ell. For empirical validation, they consider two real-life datasets and one synthetic example of scores in different contexts. They fix a group G_{1} of individuals to be a baseline group and use the distribution of scores from that group to represent the true distribution of values. They consider another group G_{2} whose scores are potentially biased (and thus arising from f_{\mathcal{E}}), and they compare how well different models of evaluation bias reproduce the distribution of G_{2} using scores from G_{1} as the true values. They find that their more generalized model can improves the fit of the score distribution of G_{2} across all datasets. They then use their model to assess various interventions for subset selection tasks, based on whether they reduce bias and allow the evaluator to pick a subset of individuals with higher expected value. These interventions include requiring proportional or equal representation, increasing \tau, and lowering \alpha, which all are related to interventions considered in practice. They find that optimal interventions depend case by case based on different parameters of the subset selection task (e.g. how many individuals to hire, etc.), and their model can guide policy-makers to understand when certain interventions will be more effective than others.
Strengths
The greatest strength of the paper is in its conception. Invoking the maximum entropy principle obtains a natural generalization of previous models in the literature, without involving an excess of extra parameters. Including risk aversion makes sense too, given that it is a well-studied source of bias and better enables the framework to model skewed distributions. The method is computationally feasible since evaluation scores are 1-D and ultimately discrete, and produces realistic evaluation distributions. The optimization-based formulation retains much of the interpretability of simpler models, while crucially allowing for greater modeling capacity through computation. In contrast to previous papers, which are primarily concerned with producing simple models that illustrate a particular source of bias, this paper provides a computational framework designed to be applied to data (I believe this aspect should be emphasized more in the paper). The paper is clearly written and the empirical benchmarking is solid with compelling examples (JEE-2009 and Semantic Scholar). The evaluation of different interventions is also a nice illustration of the usefulness and interpretability of the model.
Weaknesses
Overall the paper could benefit from more exposition. While the paper references key sources in the literature, it does not clearly explain the mechanism by which information constraints and risk aversion lead to bias. This could be clarified by 1 or 2 specific examples. Without this, it is difficult to understand why these are the particular sources of bias incorporated in the framework and why others are not. It would also be better to spend more time discussing concrete interventions, e.g. moving some of the material from Supplemental Material H to the main text. To aid in this, would it be possible to evaluate an intervention like 'change score from out of 100 to out of 10' or 'truncate the range of scores'? If so, this could help in demonstrating the applicability of the model for helping with realistic policy decisions. The theoretical results on the evaluator's optimization problem seem to be fairly standard and straight-forward implications of the maximum entropy principle. This is not a bad thing, but at least this section could be trimmed down and streamlined. The proof of Theorem 1 seems fairly involved, and it would be useful to explain how this setting departs from standard maximum entropy settings. The results on sensitivity analysis with respect to \tau and \alpha are also not surprising and this section could be shortened and crystalized to better capture how \tau and \alpha affect the output density (with greater focus on how the mean changes). Figure 1 is difficult to interpret and could either be removed entirely or replaced. I think the biggest need for improvement is in showing the strength of the contribution compared to previous models. I think a strong case can be made using supplemental material, but from the current draft alone the strength of the contribution is not very clear. First, it would help to use some supplemental material (e.g. Figure 6 from Supplemental Material H) into the paper to show that a wider class of models is actually necessary for modeling real life score distributions (comparing the best fits from other models). One could also look at a synthetic example with exponential or laplacian tails, but this is not necessary. And since the framework a generalization of previous models, it is perhaps unsurprising that it achieves greater fit to data. More than fitting the data, ultimately what matters are implications for intervention evaluation/sensitivity analysis for policy-making. If the conclusions are identical to what one obtains from less sophisticated but simpler models, it's unclear whether a more powerful modeling class actually helps. What would have been more compelling is if there are instances where the current model gives different answers for what interventions work well or not compared to simpler models, and justifying why the current model's conclusions are more sensible. In that light, it would be useful to evaluate how well the framework assesses interventions, which can be done through synthetic examples (e.g. intervening on the synthetic example and seeing how well the model predicts the effect of the intervention).
Questions
When fitting \alpha and \tau to minimize total variation distance with the G_{2} distribution, is there any train-test splitting, or are they fit on all the data? Otherwise there might be a concern about overfitting. In the empirical evaluation, the underlying assumption is the distribution of true values for the G_{2} individuals is exactly the same as the distribution for G_{1}. While this assumption makes sense for convenience since we don't observe the true values of G_{2}, if this assumption were false in practice, how could this affect the validity when assessing different interventions?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The paper primarily mentions the limitation of the work as being that it only concerns with scalar values, and mentions that the framework can be extended to multivariate values. The paper also mentions that there may be other models out there that achieve the same fit as the maximum entropy one. This is all reasonable. It does not include the limitation that the distribution of true values for G_{2} is assumed to be identical for G_{1}, in which case one cannot disentangle evaluation bias from natural differences in these distributions.
Summary
This work presents a theoretical model to quantify bias in the task of evaluating candidates (ie minimizing loss while subject to an information constraint). It presents a formula/representation of the problem, parametrized by roughly "real-world" factors of 1) resource-information tradeoffs; and 2) risk-aversion. After presenting some properties of this model, the work loosely applies it to quantify types of bias in real-world datasets (eg standardized testing by class/gender; citations; etc). Finally, it explores how different real-world-inspired interventions (eg Rooney Rule for representational constraint, structured interviews for standardization) could impact the state of bias in the parametrized models. edit: I have had the rebuttal and feel better about the derivations. I remain borderline accept
Strengths
- I quite like the design decision to choose model parameters and intervention types that are inspired by plausible tradeoffs and concerns in evaluation bias. - This paper would not be as strong if it were just the formulas/models without trying to measure any grounding in real-world datasets.
Weaknesses
- Although engaging with real-world datasets empirically is commendable, the analysis conducted was rather light. It didn't offer particularly novel insights or alternative ways of thinking about bias in the data (eg could try to demonstrate how the model can help policymakers with actionable interventions; I don't think a policymaker would be able to gain such insight with the current presentation of results/discussion)
Questions
- Overall this paper seems very interesting, though I couldn't follow all of the derivations/formulas. The empirical part, alone, doesn't have enough insight, but I do like applying the theoretical parametrized model to the real-world datasets to discuss how to interpret them. I'm inclined to accept, though I would feel more comfortable about this paper if one of the other reviewers were able to speak to the derivations' validity and contribution.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
- Some of the modeling choices were noticeably reductive. For instance, on page 8, Section 4.2 uses college admissions as an example of a subset selection task. However, it suggests modeling the task is the sum of each accepted applicant's individual values, rather than considering network effects (eg Scott Page's "The Diversity Bonus" where a team of different people can contribute different kinds of knowledge to solution resulting in a better result than if the top-2 individual values had high overlap & therefore didnt offer complementary strengths). Of course, that is just nitpicking one example, however my general concern is that this work may or may not ultimately prove useful enough for policymakers. It might end up being the case that any tasks that are tractable enough to model mathematically are poor fits for social science dynamics in practice)
Summary
The paper studies how to examine the group distributional difference using loss minimization. The authors propose a loss with a max-entropy constraint.
Strengths
The paper studies an important problem of how to examine the evaluation bias in many applications such as hiring and school admissions. The authors nicely motivate the problem and have a detailed related work on how such biases arise in practice.
Weaknesses
The paper became quite hard to read after related work. I had a hard time understand what is the formulated problem and the reasons for many design choices are not clear to me. I do not follow the loss formulation, since it seems that the authors are trying to do a density estimation task with certain constraints on the density function. Why not minimize classic metrics like f-divergence, IPM, or use methods like GMM or kernel density estimation? There is also little explanations on why we should use max-entropy as constraint. Evaluation: It seems the experiments are a density estimation task for two groups, why usual density estimation methods cannot be applied here? I am also not convinced the proposed method can be used to examine effect of interventions, since they are dynamic and more information is needed to see the effect of interventions in the long run. Typo: “risk averseness:”, incomplete sentence L121-122
Questions
See weaknessses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
1. The paper only considers binary sensitive groups, while in practice, sensitive groups are often overlapping with multiple attributes. 2. How would the estimation error and model misspecification lead to negative impact?
Summary
In this paper, the authors model evaluation processes that estimate the density of value for an individual (on a task) as a loss minimization problem subject to constraints. The authors proceed to derive various properties of the output densities of their model and evaluate it on two real world datasets.
Strengths
- This is a good solution that seems to provide clarity to a difficult and important problem - Strong (though limited) empirical section, and ablation studies on the effects of $\tau$ and $\alpha$
Weaknesses
- I found it really hard to read this paper as someone not too familiar with the field, in particular I thought the intro could use some more clarity. There's a bit of measure theory too that I wonder is necessary - - Id like to see a theorem about the performance of this model relative to others, it seems like its fairly to compare it to some of the Related Work mentioned. Though this might be fixed by adding more clarifications --
Questions
- So how would a policy maker use this model? It isnt immediately clear to me as someone unfamiliar with the literature
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The authors note some limitations to their work - though I didn't see much discussion of the misuse of their model and it's negative societal impact.
Thank authors for the responses. I will keep my score.
Thank you to the authors for their response. I don't quite understand the authors’ first response, which was my main concern that there was no stated mechanism that maps "true ability” to “biased ability”. Could the authors spell this out further? Next, if there is such a mapping, then the goal of these interventions should be to _recover_ the density of true abilities, from the biased abilities (i.e. "correct" the bias). But it doesn’t seem like the interventions are aiming to measure "recovery". For example, increasing tau increases both the mean and variance (Section 3.2), but what does that mean in terms of correcting bias? Let me know if this is not the goal of the interventions being studied.
A concrete example of how "true ability" gets mapped to "biased ability"
Thanks for reading our rebuttal and responding. We answer your question below and will be happy to provide further clarifications. **1)"...mechanism that maps true ability to biased ability. Could the authors spell this out further?"** Certainly. Consider the setting where a single individual with "true ability" $u$ is being evaluated. One concrete way in which our framework can model this is by setting the input density $f_{\mathcal{D}}$ to be concentrated on $u$ and letting the loss function to be $\ell_2^2$-loss (see Lines 204-205 for a definition) over the domain $\mathbb{R}$. For fixed parameters $\tau$ and $\alpha$, the output distribution $f^\star$ (over the "biased ability") can be shown to have a mean of $u - \sqrt{\frac{\gamma^\star}{2}}\cdot \frac{\sqrt{\alpha}-1}{\sqrt{\alpha}+1}$. Here $\gamma^\star > 0$ is as in Theorem 3.1 and is a function of $\tau$ and $\alpha$. Thus, our framework allows one to derive a mapping from "true ability" to the (mean of the) "biased ability". This mapping can be used to understand how the parameters $\tau$ and $\alpha$ in the evaluation process transform the true ability. For instance, using the fact that $\gamma^\star$ increases as $\tau$ increases, and that $\frac{\sqrt{\alpha}-1}{\sqrt{\alpha}+1}$ is positive for $\alpha > 1$, it follows that decreasing $\tau$ pushes the expected biased ability of the individual up towards $u$, their true ability. Similar mappings can be derived for other loss functions as well and, if one is interested in the distribution of biased abilities, Theorem 3.1 gives a characterization. **2)"...then the goal of these interventions should be to recover the density of true abilities, from the biased abilities (i.e. "correct" the bias)."** Yes, that could be one goal, however, it may neither be necessary nor directly achievable. Obtaining estimates of abilities is not an end to themselves: they are used in downstream tasks such as selection. The interventions (in our paper and in prior works) try to ensure that the outcomes of the downstream tasks with biased abilities and interventions are (approximately) the same as the outcomes with true abilities. **3)"For example, increasing tau increases both the mean and variance (Section 3.2), but what does that mean in terms of correcting bias?"** Concretely, as discussed in the example point (1) above, decreasing $\tau$ moves (mean of) the biased ability of the individual up toward their true utility. Moreover, as can be seen by the expression of the mean of the biased utility in point (1) (i.e., $u - \sqrt{\frac{\gamma^\star}{2}}\cdot \frac{\sqrt{\alpha}-1}{\sqrt{\alpha}+1}$), moving $\alpha$ towards $1$ also ensures that the mean of the biased ability of the individual approaches their true ability.
Thanks to the authors for their response. I have raised my score to a 4. I am still concerned about the presentation of the paper (I still find the paper to be very confusing, even with the author’s clarifications), which I don’t think can be addressed in a small revision.
Plan for revision
Thank you for raising your score. Our discussion with the reviewers has been invaluable, and because the revisions are a matter of clarifications (which we have already outlined in the rebuttal) and not of developing new material or empirical results, we are confident that we can complete them in a timely manner. We provide a detailed summary of changes below and believe that we will be able to incorporate these changes within a week. In addition, we will correct typos and address any minor comments omitted in the list below. Note that one extra page is allowed for the final version, which allows sufficient space for these changes. If the AC allows, we would be happy to upload our revision by early next week. **Section 2** * We will explain the mechanism that maps true ability to biased ability in Section 2. (Included in responses to Aa19.) * We will further explain the use of max-entropy constraint and contrast our approach with density estimation in Section 2. (See the response to 7ABm.) * In Section 2, we will add a remark on how our model extends to multiple sensitive groups with overlapping attributes (We present the idea in the response to 7ABm.) **Section 3** * We will add an overview of the proof of Theorem 3.1 in Section 3.1. (Presented in the "Author Rebuttal by Authors" above.) * We will add informal versions of key results in Sections D and E (on how changing $\alpha$ and $\tau$ affect the mean and the variance of the output density) in Section 3.2. * We will add two theorems that show how our framework captures the implicit variance and the beta-bias models in Section 3.2 (the statements appear in the response to G1nQ). The proofs follow from the results in Sections F and G in the supplementary material and will be added as a separate section after Section G in the supplementary material. (Presented in the response to G1nQ.) * In Section 3.2, we will replace the plots in Figure 1 with 2-dimensional plots which show the effect of $\alpha$ or $\tau$ on the mean of the distributions. These plots are also shown in the attached 1-page pdf file. If space is a constraint, we will move them to the Supplementary material. **Section 4** * We will add a version of Figure 6 that compares the best fits of our model and the best fits of previous models with the distribution of biased utilities in Section 4. * In Section 4 (and corresponding sections in Supplementary Material), we will update the experimental validation with the train-test split. (See the response to gzsA and the table in the one-page pdf attached with the rebuttal.) * Time permitting, we will incorporate the suggestion of gzsA to include further empirical evaluation to give qualitatively better insights to compare different policies. **New section between Section 4 and Section 5** * We will add a section between Sections 4 and 5 to include specific examples of mechanisms by which information constraints and risk aversion lead to bias. Here we will briefly discuss examples of concrete interventions (details are given in Section H) and how policymakers can use our model. (Presented in the "Author Rebuttal by Authors" and in the response to G1nQ.) **Section 5** * We will add a short discussion on the limitations of our model when it comes to long-term applicability in Section 5. * We will add a comment on the potential misuse of our model and its negative societal impact in Section 5. (See the response to G1nQ.) * We will mention the limitation of our work in understanding the effect of the interventions considered in our paper in the long term in Section 5. * We will add a discussion on the limitations of our model when there are differences in the distribution of true utility among the two groups at the population level in Section 5. (See the response to gzsA.) * We will add a comment in Section 5 on how the output of our model can be used by policymakers to assess the impact of interventions in this supermodular setting as well. (See the response to 2dLt.)
Thank you for the thoughtful response. I appreciate the explanation for the proof of Theorem 1. I also really appreciate the explanations for why \alpha and \tau would affect bias, they help to illustrate why these parameters are invoked in the model and why other parameters are not. All of my concerns and questions have been addressed by the above response. I do still wish there were a more extensive empirical evaluation that could clearly show that this more computationally powerful method gives _qualitatively_ better insights into which policies would work better than others. And also if there is any way to concretely interpret the fitted values of \tau and \alpha. I think this is a mostly a matter of rewriting the paper so that it's usefulness to potential policymakers (or even possibly empirical social scientists) is more apparent. I will stick to my rating since I believe it is appropriate. Once again, I like the core idea of the paper, my comments are only on the presentation.
Decision
Accept (poster)