Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

Summary

This paper compares performances of the state of the art recurrent neural networks RWKX and Mamba, with that of a transformer Pythia on 9 from a human oriented perspective, through 9 datasets by use of 4 metrics of human language processing. The method is by computing the surprisal. The authors empirically show that Pythia does not necessarily outperform RWKX and Mamba, rather the contrary. The authors argue that RNNs perform better than Transformer, from a human-oriented perspective.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The problem setting is interesting that is to compare the capacity of RNN against Transformer how they perform differently for human cognitive tasks.

Reasons to reject

There are two large concerns of this paper. 1. According to Table 1, the authors compare three models under comparable settings. The number of parameters for every experiment is comparable by scale. Although I understand this is one "fair" way of setting, I wonder whether Transformer's capability is fully exploited this way. Transformer performs better because it is able to accomodate larger number of parameters than RNN, that is, O(n^2) vs. O(n), respectively, for n being the length of the sequence. Transformer architecture includes that of RNN. The present result shows that if Tranformer is trained on a system with equivalent number of parameters of RNN, then Transformer (could) perform worse than RNN. This is trivial to me, because Transformer uses parameters to extract relation among tokens that cannot be captured by RNN. Therefore for Transformer to perform fully, it must use much more parameters than RNN. 2. The authors focus only on human cognition. If comparing under the same setting of Table 1, the data does not have to be be restricted to human cognition. I guess even for a general evaluation dataset, the same comparison of Transformer vs. RNN can be made and presumably the same conclusion could be obtained. In other words, since the authors do not show the evaluation results for other evaluation dataset, the value of this work is not well highlighted.

Questions to authors

-Did you compare the three systems on the same data when using the maximum number of parameters available for each system? If results do you think you would get? -Did you compare the results for other data that are not oriented to human cognition? If you do that, what do you think you would get?

Summary

An empirical comparison of how different LM architectures (Transformers vs. two latest-generation recurrent models, RWKV and Mamba) predict various metrics of language processing difficulty by human subjects. The paper starts from the somewhat suprising observation that traditional RNNs have been superseded by a less cognitively plausible architecture (Transformers) not only in a range for NLP tasks but also as predictors of behavioral metrics of language comprehension by humans. It asks whether latest-generation RNNs would, in turn, fair better than Transformers in this context. The comparison is performed on a suite of 9 language processing datasets using models of different architectures and sizes, all trained on the same corpus. Results show a complex picture whereby different architectures are best across different datasets and model sizes. Also, an earlier finding is confirmed by which processing difficulty tends to be better predicted by models that have better language modeling performance (i.e. perplexity) in general.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- timely and thorough evaluation of post-Transformer recurrent architectures on the task of predicting human processing difficulty - no less than 9 datasets and four different metrics of processing difficulty are included in the study - models are compared across different sizes to give a more complete picture

Reasons to reject

- The results themselves are rather inconsistent and the overall conclusions a bit vague - No attempt is made at explaining why some architectures may be outperforming others in different conditions

Questions to authors

Clarity improvements: - AIC -> spell out acronym - For clarity, I'd give an acronym/ID to each dataset in Table 2 and use those acronyms/IDs for the discussion in Section 4. All those citations make it hard to follow the text - Sect. 4.1 could end with some cleared take-aways. Overall, what have you learned from this experiment? Typos: - identifies filters the input - initialized in based - In thus study - again see -> we again see (sect 4.2)

Summary

The paper evaluates a recent crop of recurrent language models on predicting the N400 response and reading times, finding that the modern recurrent models perform on par with Transformers on this task, updating previous old findings that Transformers outperform LSTMs.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

- The paper makes an important contribution to an ongoing research program about the match between neural network architectures and human language processing, based on the surprisal linking mechanism. - The experiments are straightforward and their main result (modern recurrent architectures can match Transformers or exceed them) is clear. - The analyses involving model size and perplexity are well-motivated and the results are interesting.

Reasons to reject

I ultimately felt dissatisfied by this paper because I wanted more, both in terms of results/experiments and in terms of cognitive upshots. The current paper feels like an incremental progress update, although the beginning of something better. I think that is probably enough for publication in CoLM, but I have concerns below: When comparing neural architectures on their ability to predict N400/RT, the question is always what we are learning about human language processing from the comparison. I don't feel like the current paper sheds much light on this yet. What's on my mind is: What is it exactly about Transformers that made them do better than LSTMs, and what has changed about the new recurrent models that make them match Transformers? Is it the better gating mechanism in RWKV? Or is it the restriction to linear dynamics as in Mamba? I think to get real cognitive insights, more models would have to be compared, including some custom models (like, various ablated versions of Mamba). In particular, Mamba could be compared to its predecessors like S4. It would also be necessary to give LSTMs a fair shot. The LLM field basically stopped training LSTMs: how would they perform if trained on the same data and with the same large number of parameters as the modern Transformers? A comparison with n-gram models trained on large datasets would also be informative. Basically, I feel like deriving real cognitive insights would require a much more thorough comparison than what is here. Even given the current results, the discussion about cognitive upshots is sparse. What are we to make of the relationship between scaling direction and RWKV vs. Mamba performance? If recurrent models really are just as good as Transformers, what are we to make of the arguments that Transformers act like cue-based retrieval models---was it just a spurious connection? There are rich cognitive implications that aren't explored. Other issues - Measuring performance by AIC is not as good as measuring performance by delta-log-likelihood on heldout data. AIC is just log likelihood with some complexity penalties. These complexity penalties would ideally be the same across different datasets, reflecting use of the same model; and if different models were used, the complexity penalties are only crude measures of the actual complexities of these models. I think pure delta-log-likelihood based on maximally similar regression models and heldout data would be the best evaluation. - Coverage of the literature is incomplete. The most important omission in the literature review is is Ryu & Lewis (2021) who argue that Transformers directly parallel the cue-based retrieval models that are common in psycholinguistics, in contravention of the general wisdom that recurrent models are more cognitively realistic. Two other omissions are Wilcox et al. (2023, TACL) and Xu et al. (2023, EMNLP) which look at the surprisal-reading time relationship cross linguistically.

Questions to authors

- What is the motivation for the datasets used? Why not use Natural Stories SPR in addition to Natural Stories Maze, or any other eyetracking corpus like Dundee?

Summary

This is a well written paper with a clear structure. The paper is well motivated with clear arguments. The paper also presents a good amount of empirical evidence to back the conclusions. The paper is quite original as it shows evidence that RNNs can outperform transformers in human language compression. However, in terms of technicality, the paper doesn't proposal noticeable novelty.

Rating

4

Confidence

3

Ethics flag

1

Reasons to accept

I enjoyed reading the paper. The paper presents well the background and the literature.

Reasons to reject

I'm skeptical about some points in the paper. Firstly, the authors argue that one reason why transformers are less cognitively plausible is their finite context window. I believe that the finite context window is only a limitation in practice because in theory, one can choose the window size as large as possible. Besides, isn't that human memory has some certain limitation? My second concern is the models chosen in the experiments. The authors specifically chose Mamba and RWKV because they have been shown to be very fast and perform on par with transformers. However, there are several variations of transformers yet the authors didn't give a good argument why Pythia was picked. Another related concern is about how the chosen models were trained. The paper seems lack that details. Besides, the authors draw conclusions without disentangling model architecture and model optimization. In fact, with reasonable optimization (e.g. some simple tricks), one can improve a model performance.

thanks for the clarifications

Thanks for your answers. I stand by my overall positive judgement of this paper. Even in the lack of a clear winner, the paper has enough contributions with its analysis of novel models on this task.

response

I would like to thank the authors for the response. My response is below: First, re model selection & optimization, one specific point unclear to me is how the models were trained. I couldn't find training details in the paper, and thus I can't tell if there're tricks to optimize the models. Second, the authors shape the contents under human cognition theme; and from cognitive plausibility point of view, the authors discuss transformers vs recurrent models. But I really don't think that's relevant because the models have several technical details that aren't proved to be cognitively plausible. How do we know that the used recurrent models outperform the transformer because their recurrent mechanism or because of their cognitively unplausible technical details? Third, it's unclear what the authors want to conclude here. The authors show performance results, but then what could we learn from them to help us understand human cognition?