Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

The detection of offensive language in the context of a dialogue has become\nan increasingly important application of natural language processing. The\ndetection of trolls in public forums (Gal\\'an-Garc\\'ia et al., 2016), and the\ndeployment of chatbots in the public domain (Wolf et al., 2017) are two\nexamples that show the necessity of guarding against adversarially offensive\nbehavior on the part of humans. In this work, we develop a training scheme for\na model to become robust to such human attacks by an iterative build it, break\nit, fix it strategy with humans and models in the loop. In detailed experiments\nwe show this approach is considerably more robust than previous systems.\nFurther, we show that offensive language used within a conversation critically\ndepends on the dialogue context, and cannot be viewed as a single sentence\noffensive detection task as in most previous work. Our newly collected tasks\nand methods will be made open source and publicly available.\n