How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

Current language models can generate high-quality text. Are they simply\ncopying text they have seen before, or have they learned generalizable\nlinguistic abstractions? To tease apart these possibilities, we introduce\nRAVEN, a suite of analyses for assessing the novelty of generated text,\nfocusing on sequential structure (n-grams) and syntactic structure. We apply\nthese analyses to four neural language models (an LSTM, a Transformer,\nTransformer-XL, and GPT-2). For local structure - e.g., individual dependencies\n- model-generated text is substantially less novel than our baseline of\nhuman-generated text from each model's test set. For larger-scale structure -\ne.g., overall sentence structure - model-generated text is as novel or even\nmore novel than the human-generated baseline, but models still sometimes copy\nsubstantially, in some cases duplicating passages over 1,000 words long from\nthe training set. We also perform extensive manual analysis showing that\nGPT-2's novel text is usually well-formed morphologically and syntactically but\nhas reasonably frequent semantic issues (e.g., being self-contradictory).\n