Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence

Summary

This paper provides extensions to RWKV-4 architecture, which is an linear attention based architecture for faster training and inference with limited memory, overcoming the limitations of RNN-like architectures. The contributions of the paper are as follows: (1) Eagle architecture - improves the earlier architecture with multi-headed valued states, and additional gating mechanism. (2) Finch architecture - which extends eagle architecture with Lora layers to allow for trainable weights to effectively learn data decay vectors. (3) Release of multilingual datasets and pre-trained models can benefit the community.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. Paper clearly highlights the proposed extension and they improve upon the original RWKV architecture. 2. Extensive empirical results on various benchmarks (english and multilingual) show the efficacy of the proposed changes. 3. Multimodal experiments in speech and vision showcase the utility of such architectures in modelling long sequences.

Reasons to reject

1. While there are computational advantage in using extend RWKV architecture, their performance on long context tasks still need more investigations. 2. Exploration of RWKV architectures with instruction tuning needs to be explored. 3. Performance on coding benchmarks leaves room for improvement.

Questions to authors

1. How well does the proposed changes work for needle in the haystack like long context tasks? 2. What are the reasons for poor performance on MTEB like information aggregation task? 3. What is the performance on code benchmarks? What are the reasons for shortcomings?

Summary

This paper introduces Eagle (RWKV-5) and Finch (RWKV-6), novel RNN architectures building upon the RWKV framework. These architectures utilize multi-headed matrix-valued states and dynamic recurrence mechanisms for improved expressivity while maintaining RNN-like efficiency. The authors also present a new multilingual dataset, RWKV World v2, and a fast tokenizer to enhance performance on multilingual and code data. Their experiments demonstrate that Eagle and Finch achieve competitive performance on a wide variety of benchmarks, including language modeling, associative recall, and even multimodal tasks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

This paper presents an advancement in RNN-based language models with the introduction of Eagle and Finch (RWKV-5 and RWKV-6). The authors demonstrate that these architectures, incorporating multiheaded matrix-valued states and dynamic data-driven recurrence, achieve strong performance on a diverse set of benchmarks, including MQAR and various linguistic tasks. Notably, their performance rivals, and in some cases surpasses, traditional transformer models of similar size, highlighting the potential for creating more efficient models without sacrificing accuracy. The authors also introduce a new multilingual corpus, RWKV World v2, and a fast tokenizer based on greedy matching, helping to address the need for diverse language representation in language models. They also publicly release six pre-trained models under open source, trained on this expansive 1.12 trillion token dataset.

Reasons to reject

The primary concern lies in the lack of detailed analysis and justification for the architectural choices made. The paper states that Eagle and Finch improve upon RWKV-4, introducing features like matrix-valued attention states and data-dependent token shifting. However, the specific benefits of these design decisions are not thoroughly explored. The authors do not provide sufficient evidence or ablation studies to support these claims. Additionally, the paper lacks a comparative analysis of Eagle and Finch against other recently proposed efficient transformer variants and state-space models in an apples-to-apples comparison as far as I can tell. Complicating comparisons, the paper introduces a new dataset and tokenizer specifically designed for the proposed models, which makes a direct comparison with other models more hazy.

Questions to authors

In the results tables, were the comparisons apples-to-apples for any of the comparisons? It is hard to tell what has the same training data, number of steps, etc. to make a sound judgment on improvements over RWKV 4 and other baselines.

Summary

The paper presents RWKV-5 and RWKV-6, RNN sequence models improved upon the RWKV-4 model of Peng et. al., 2023. Along with the paper, the code for training and inference, as well as a 1.12 trillion-token data set with an emphasis on multilinguality is released. The results include multilingual benchmark results, English-focused benchmark results, using the LM evaluation harness framework of Gao et. al., 2023. RWKV-5 and RWKV-6 appear to beat Llama-2-7b and a list of other competitive models in the 1B-7B range in the multilingual benchmarks. Both models are worse than Mistral-7B-v0.1 in the English-focused benchmarks. The paper also include additional multimodal experimental results in the Appendix.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper represents a substantial amount of work in the space of RNN-based large language models. The results demonstrate RNN-based models can be as competitive as Transformer-based ones. More is to be seen in even larger scales. But the existing experimental results and insights are already valuable to the research community. The idea of opening sourcing everything should be encouraged.

Reasons to reject

Some of the claims about the supremacy of the presented models seem to be hyperbolic. For example, the two models are clearly worse than Mistral-7B-v0.1 in Table 4. The claim in the second paragraph of Section 8 is not valid. Since RWKV-5 and RWKV-6 are trained on the RWKV World v2 dataset, it is not clear how much of the improvement comes from the model and how much of it comes from the data. There are not results supporting the tokenizer presented in Section 5.

Questions to authors

Can you include Llama-3-7b in the results? Please put the main results of Section 9 in the main body of the paper. Additional results can stay in the Appendix.

Reply to Rebuttal

Thank you for answering my questions and all of the additional experiments. I am bumping my score up to 7. I hope these (and the results mentioned in the other rebuttals) are added to the next version of the paper as I think they really do strengthen it.