Summary
This paper presents infini-gram, an n-gram model with unbounded n-gram order trained on very sized corpora containing 5T tokens. The paper shows that infini-gram can substantially outperform vanilla 5-gram models in terms of next token prediction (62% relative improvement) and can yield improvement in perplexity upon interpolating with neural lms. The paper shows that infini-gram can be implemented efficiently using suffix arrays, a technique that has been proposed in the past but not used for scaling the LM to trillions of tokens.
Strengths:
* Shows that an n-gram model can be scaled to substantially large corpora with trillions of tokens.
* Shows that an n-gram model can yield perplexity improvements when combined with neural LMs
* Demonstrates how the infini-gram can be used to conduct analysis of effective n (length of longest suffix with non-zero count) in neural LMs.
Weaknesses:
* There are some critical metrics which are not reported in the paper
e.g. next token prediction for neural LM, perplexity of the infini-gram by itself
* Though the infini-gram outperforms a 5-gram LM, it is still substantially weaker than a neural LM. While it is good to see that the infini-gram improves the perplexity of the neural LM, this result has been reported in prior literature
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/rnn_ctxt.pdf
* The paper does not report comparison with other non-parametric approaches such as kNN LMs (Khandelwal et al. 2020)
Reasons to accept
* Shows that an n-gram models can be scaled to very large corpora with trillions of tokens - which are used currently for training neural LMs
* Shows that the n-gram model can be used to perform interesting text analysis of neural LMs using metrics such as 'effective n' - which is the length of the longest suffix with non-zero count in training data
* Demonstrates that the infini-gram model can be implemented efficiently using suffix arrays. Although this technique has been used in the past, this paper may further popularize its application in the research community
Reasons to reject
* The paper does not report key metrics such as next-token prediction accuracy of neural LM, the perplexity of infini-gram LM by itself.
* The paper does not compare the proposed approach to other non-parametric LM approaches such as the kNN LM (Khandelwal et al)
* The paper does not present an error analysis showing where the infini-gram either outperforms/underperforms a neural LM. e.g. is a neural LM better at generalization?
Questions to authors
* "When plotting the agreement level with respect to the suffix length, text generated by neural LMs with nucleus sampling is most similar to human-written text, among other decoding methods like greedy decoding and temperature sampling. For greedy decoding, the agreement plot suffers from significant fluctuation, which may be rooted in deficiencies in neural LM pretraining and the positional embeddings of Transformer"
It is unclear how this finding is related to infini-gram.
* Figure 3: what is the next token prediction accuracy for neural LM?