ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

Toxic language detection systems often falsely flag text that contains\nminority group mentions as toxic, as those groups are often the targets of\nonline hate. Such over-reliance on spurious correlations also causes systems to\nstruggle with detecting implicitly toxic language. To help mitigate these\nissues, we create ToxiGen, a new large-scale and machine-generated dataset of\n274k toxic and benign statements about 13 minority groups. We develop a\ndemonstration-based prompting framework and an adversarial\nclassifier-in-the-loop decoding method to generate subtly toxic and benign text\nwith a massive pretrained language model. Controlling machine generation in\nthis way allows ToxiGen to cover implicitly toxic text at a larger scale, and\nabout more demographic groups, than previous resources of human-written text.\nWe conduct a human evaluation on a challenging subset of ToxiGen and find that\nannotators struggle to distinguish machine-generated text from human-written\nlanguage. We also find that 94.5% of toxic examples are labeled as hate speech\nby human annotators. Using three publicly-available datasets, we show that\nfinetuning a toxicity classifier on our data improves its performance on\nhuman-written data substantially. We also demonstrate that ToxiGen can be used\nto fight machine-generated toxicity as finetuning improves the classifier\nsignificantly on our evaluation subset. Our code and data can be found at\nhttps://github.com/microsoft/ToxiGen.\n