RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Pretrained neural language models (LMs) are prone to generating racist,\nsexist, or otherwise toxic language which hinders their safe deployment. We\ninvestigate the extent to which pretrained LMs can be prompted to generate\ntoxic language, and the effectiveness of controllable text generation\nalgorithms at preventing such toxic degeneration. We create and release\nRealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level\nprompts derived from a large corpus of English web text, paired with toxicity\nscores from a widely-used toxicity classifier. Using RealToxicityPrompts, we\nfind that pretrained LMs can degenerate into toxic text even from seemingly\ninnocuous prompts. We empirically assess several controllable generation\nmethods, and find that while data- or compute-intensive methods (e.g., adaptive\npretraining on non-toxic data) are more effective at steering away from\ntoxicity than simpler solutions (e.g., banning "bad" words), no current method\nis failsafe against neural toxic degeneration. To pinpoint the potential cause\nof such persistent toxic degeneration, we analyze two web text corpora used to\npretrain several LMs (including GPT-2; Radford et. al, 2019), and find a\nsignificant amount of offensive, factually unreliable, and otherwise toxic\ncontent. Our work provides a test bed for evaluating toxic generations by LMs\nand stresses the need for better data selection processes for pretraining.\n