Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

The perceived toxicity of language can vary based on someone's identity and\nbeliefs, but this variation is often ignored when collecting toxic language\ndatasets, resulting in dataset and model biases. We seek to understand the who,\nwhy, and what behind biases in toxicity annotations. In two online studies with\ndemographically and politically diverse participants, we investigate the effect\nof annotator identities (who) and beliefs (why), drawing from social psychology\nresearch about hate speech, free speech, racist beliefs, political leaning, and\nmore. We disentangle what is annotated as toxic by considering posts with three\ncharacteristics: anti-Black language, African American English (AAE) dialect,\nand vulgarity. Our results show strong associations between annotator identity\nand beliefs and their ratings of toxicity. Notably, more conservative\nannotators and those who scored highly on our scale for racist beliefs were\nless likely to rate anti-Black language as toxic, but more likely to rate AAE\nas toxic. We additionally present a case study illustrating how a popular\ntoxicity detection system's ratings inherently reflect only specific beliefs\nand perspectives. Our findings call for contextualizing toxicity labels in\nsocial variables, which raises immense implications for toxic language\nannotation and detection.\n