Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Retrieval-based methods have been shown to be effective in NLP tasks via\nintroducing external knowledge. However, the indexing and retrieving of\nlarge-scale corpora bring considerable computational cost. Surprisingly, we\nfound that REtrieving from the traINing datA (REINA) only can lead to\nsignificant gains on multiple NLG and NLU tasks. We retrieve the labeled\ntraining instances most similar to the input text and then concatenate them\nwith the input to feed into the model to generate the output. Experimental\nresults show that this simple method can achieve significantly better\nperformance on a variety of NLU and NLG tasks, including summarization, machine\ntranslation, language modeling, and question answering tasks. For instance, our\nproposed method achieved state-of-the-art results on XSum, BigPatent, and\nCommonsenseQA. Our code is released, https://github.com/microsoft/REINA .\n