Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data

Neural text generation (data- or text-to-text) demonstrates remarkable\nperformance when training data is abundant which for many applications is not\nthe case. To collect a large corpus of parallel data, heuristic rules are often\nused but they inevitably let noise into the data, such as phrases in the output\nwhich cannot be explained by the input. Consequently, models pick up on the\nnoise and may hallucinate--generate fluent but unsupported text. Our\ncontribution is a simple but powerful technique to treat such hallucinations as\na controllable aspect of the generated text, without dismissing any input and\nwithout modifying the model architecture. On the WikiBio corpus (Lebret et al.,\n2016), a particularly noisy dataset, we demonstrate the efficacy of the\ntechnique both in an automatic and in a human evaluation.\n