CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias

Summary

This paper introduces a new benchmark to assess the presence of gender and race bias in LMs. The main novelty lies in the high number and diversity of templates for these two bias categories. This is achieved by selecting templates from three existing datasets covering the tasks of QA, sentiment analysis, and NLI, respectively. The benchmark is used to quantify bias in 20 open-source LLMs.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The new benchmark can be useful to complement the existing ones, and contribute to a more reliable evaluation of gender and race bias in LMs thanks to its wide range of templates.

Reasons to reject

Reproducibility can be improved, especially w.r.t. the template selection process, the template modifications to assess robustness, and the section on the template subset) While the benchmark can be clearly useful, the analysis conducted on top of it leaves space for improvement. Specifically: - Statistical significance of bias score differences is never mentioned - Many patterns are drawn from many bias score differences that could just be due to noise. For instance, many observations are made on the effect of model size, but as it stands these differences could be due to different contents of the datasets on which different-sized models were trained. Similarly, it is suggested that T0's lowest reported bias score could be due to its training procedure, but here too dataset content seems to be a more natural explanation [Not per se a reason to reject, but reflects the general lack of substance in the paper]: - much space is dedicated to repeating how this benchmark is better than others (e.g. Figure 2 and much of section Discussion), instead of letting the readers judge this by themselves based on the content of the paper. Some points of strengths are overstated (e.g. yes there are more templates in this work, but other works had a larger variety of bias categories which is also important). In my opinion it's more important to convincingly show complementarity wrt previous work, rather than overstating superiority.

Questions to authors

OTHER COMMENTS: "If a model is unbiased, its score for each group should match the baseline, resulting in a bias score of 0%." => I'm not convinced this is the case. This is because each group accuracy is computed on a different, smaller set of instances than the baseline score which is computed on all instances. It would be more reasonable to expect that, in a less biased model, the group-specific scores will approach a baseline score averaged over several similar-size, randomly sampled subsets of the entire dataset. In any case, expecting a difference of 0 seems unrealistic. Some testing of statistical significance is advised especially when differences among models' bias scores are small. The comparison to the BBQ dataset (the largest of existing bias benchmarks cited in the paper) could be more specific. Examples of observations on the results that seem a bit shaky: - for OPT series, larger model means more bias: This is only true if you ignore the smallest OPT which has by far the largest bias in this series. The other differences may not even be statistically significant. - tradeoffs between gender and racial bias: Sometimes gender bias is higher in a larger model version, sometimes race is. There's a big chance this is a random effect, possibly based on idiosyncrasies of the datasets on which these models were trained. Why call it a "tradeoff"? - tradeoffs between bias in different tasks: Idem. The template error analysis focusing on a few cherry-picked words (e.g. "competitive" associated with male bias) doesn't seem very insightful. Typos: - Prior work on bias measurement are -> is - Construction of CALM -> The construction of CALM Clarity: - Many appendices are included, but all references from the main body point to a generic "Appendix" - some appendices (...) provide hardly more information than what's already in the main body - explain what HELM does

Summary

This paper proposes a bias evaluation method (and dataset) called CALM, which builds off the idea of existing template-based evaluations, but greatly increases the number and variety of templates, resulting in a broader evaluation. Furthermore, the authors find that their evaluation is markedly less sensitive to certain perturbations that have previously been shown to drastically affect the results of previous methods, indicating that those other methods are too unstable. In terms of experiments, the authors assemble their set of templates from 16 existing QA, sentiment analysis, and NLI datasets, identifying person mentions and pronouns where they appear in various instances in those datasets. They then consider various fill-ins for those names and pronouns along racial and gender axes; for race, they use names from US census data and the Harvard dataverse for Caucasian, African American, Hispanic, and Asian people, and for gender, they use male and female names sampled from the US Social Security dataset, and also gender-neutral names sampled from an article listing many such names. The authors evaluate 20 different existing language models for bias using CALM, present their findings, and close their experimental results with a discussion of how bias relates to sizes of different model series.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The way the authors operationalize bias is well-motivated and well connected to previous work. The authors did a great job of situating their work within the problem space and articulating the key issue with previous work that their work addresses (namely, a lack of robustness in bias evaluation methods). The experiments involved a wide variety of models, which is great. Section 4.1 in particular is a really nice experiment that provides a strong argument for CALM's robustness. Borrowing the methods for this analysis from Selvam et al. 2023 that found other methods to be highly sensitive is a nice touch. On the writing side, the paper's also (generally) very clear; I asked questions about the few parts I found unclear below.

Reasons to reject

Based on the source datasets used for constructing the templates, it's unclear to me how many of the produced templates are relatively anodyne/wouldn't be expected to correlate with a real-world bias anyway. Essentially, this concern is echoed at the end of section 6.1: "Out of all templates, we identified 8— less than 4%— that reveal these common bias trends" (although in section 6.1, that's framed as a strength). But I'm not too concerned about this, seeing as CALM *does* seem to surface biases in a variety of models. (That said, maybe weaken the description of "criteria directed at socioeconomic distinctions" at the top of page 4 to something like "names and pronouns, since those can carry information about socioeconomic distinctions"-- the current wording makes it sound more like every mention of a name or pronoun that's flagged by your method carries important socioeconomic distinctions, which in a sentence such as "Mary went for a walk," is maybe a bit of an exaggeration.) The paper's argument for how CALM relates to real-world bias is, as currently presented, pretty weak; in paragraph 2 of section 1, we have "We believe that models having lower CALM bias scores are likely to exhibit reduced biases in practical scenarios," and in section 7, "A higher LM bias score is associated with an increased potential for harmful real-world impacts from use of the model." The first of those sentences is very hedgy, the second of those sentences feels like it should either be citing something or referencing experimental results in order to make that claim, and the paper doesn't actually do any experiment with real-world usage scenarios. I still think the paper overall is valuable for providing a bias evaluation similar to previously proposed bias evaluations that lacks some of their issues, but the current connection to real-world behavior of the models feels too thin. I would recommend either adding some more justification, or removing the discussion of real-world bias as it relates to CALM altogether.

Questions to authors

(1) Just to check, the average bias score for a single model is computed as an average of the race and gender per-template bias scores, right? In other words, the average bias score for a model is computed by first concatenating the lists on the two following lines, and then averaging that concatenated list, right? $$\left[ \left( \max_{r \in \textrm{4 race categories}}bs(r, t) - \min_{r^* \in \textrm{4 race categories}}bs(r^*, t) \right) \mid t \in \textrm{templates}\right]$$ $$\left[ \left( \max_{g \in \textrm{3 gender categories}}bs(g, t) - \min_{g^* \in \textrm{3 gender categories}}bs(g^*, t) \right) \mid t \in \textrm{templates}\right]$$ If so, great-- that was my reading of the paragraph directly below equation 1. But if it's calculated some other way, please let me know! And regardless, it would be helpful if you could reword the paragraph under equation 1 (or add a formal mathematical expression for the model-level bias score) to clarify. Thanks! (2) I'm not sure exactly what implication you're driving towards with the point communicated in table 5 (and in the first paragraph of 4.4) about consistent averaged accuracy scores. Is your reading of the results essentially a warning not to just compare aggregated per-category accuracies, since those can obscure important inter-category differences? If so, it would be nice to clarify that. (3) What were the "targeted experiments" described in the first part of section 6.1? How did you end up deciding to remove those 68 templates in particular? (4) Which figure is being referenced in the last paragraph of section A.5 in the appendix? Currently the first sentence there ends with "Figure ??."

Summary

The paper introduces the Comprehensive Assessment of Language Models (CALM), a benchmark designed to measure gender and racial biases across 3 tasks -- Question Answering, Sentiment Analysis, and NLI. The benchmark creation process involves selecting popular datasets for each task, generating templates by replacing names and pronouns with a <PERSON> tag, and subsequently replacing the <PERSON> tag with names sampled from different gender and racial groups, resulting in 78,400 examples. To ensure the robustness of the benchmark, the authors also created four alternative versions of CALM by perturbing the original templates through synonym substitution, addition of clauses, and inclusion of adjectives. The authors also perform a battery of experiments on 6 LLM families (Llama2, Bloom, Falcon, OPT, T0, GPT-Neo) to validate the robustness of the dataset.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

This benchmark covers a variety of tasks and could be useful for measuring the gender and racial bias of LLMs.

Reasons to reject

While the main contribution lies in introducing this benchmark, the paper lacks a thorough qualitative analysis of the resulting dataset examples. I’m not convinced that simply replacing names in datasets such as SNLI or SST would be sufficient to obtain a dataset to benchmark social biases of strong LLMs like ChatGPT reliably, and it would be good to show experiments on this. It is unclear whether the prompts mentioned in Appendix A.8 are fit for accessing the model’s performance on some of the chosen datasets. Example: - Is the template ”Passage:<context>\n. Question: <question>\n. True or False?\n Answer: ”. fit for evaluating SNLI performance?). - How was the QA template (”Passage: <context>.\n Question: <question>.\n Answer:”) applied to Multiple Choice QA datasets such as MCScript? This needs clarification.

Summary

The authors construct a template-based bias evaluation dataset based on existing NLI, QA and sentiment task datasets. The dataset focuses on gender and racial bias.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

The authors repurpose existing task datasets in a unique way to capture the variability of LLM responses across demographic groups.

Reasons to reject

- If I understand correctly, the proposed bias score primarily measures variability of LLM responses across gender/race-associated names, without capturing any given direction of bias. This is a meaningful distinction from existing bias benchmarks, which often seek to capture positive/negative biases associated with particular demographic groups. While ideally models should respond uniformly across all names, the construction of the current dataset means that higher variability across names, even if not systematically biased, directly translates to a higher bias score. - The authors make several comparisons to existing bias benchmark in the paper, but based on the above distinction, I do not think that comparisons on e.g. the number of templates is meaningful.

Questions to authors

- It looks like the evaluation is on a mix of instruction-tuned and pretrained-only models. Should they be treated differently?

Thanks for your response!

Thanks for responding to the points I raised, your answers help to clarify some points I was wondering about. Based on your responses, I'm inclined to leave my score as-is; I still believe the demonstration that the proposed evaluation method is more stable merits acceptance of the paper.

Thanks for your answers. I still find some of the conclusions stretched.

Thanks for your answers. I still find some of the conclusions stretched. For instance, I explicitly mentioned this one: “for OPT series, larger model means more bias: This is only true if you ignore the smallest OPT which has by far the largest bias in this series. The other differences may not even be statistically significant.” but the authors did not comment on it. To be clear the smallest OPT model has a bias score of 24.5 versus all the other models have scores between 11.6 and 15.0. This strongly undermines the conclusion that larger parameters lead to larger bias. The author response also does not address several of my other comments, for instance: - "If a model is unbiased, its score for each group should match the baseline, resulting in a bias score of 0%." => I'm not convinced this is the case.  -  it is suggested that T0's lowest reported bias score could be due to its training procedure, but here too dataset content seems to be a more natural explanation