Joint processing of linguistic properties in brains and language models

Summary

While several papers have recently shown that large language model embeddings are aligned with / predictive of fMRI reading data, this paper pushes these analyses forward by asking: what properties of the representations are responsible for this alignment? In particular, they compare LLM/fMRI alignment on reading data using BERT, but then compare this to BERT's representations _when certain linguistic properties have been removed_ (using both a novel and an existing removal method). The results show a significant decrease in alignment (crucially, not found when removing "random" information or when using random vectors) across all linguistic properties, suggesting that it really is BERT's encoding of linguistic information that's responsible for the alignment with fMRI reading data. This represents a finer-grained level of analysis than existing alignment works and should be of interest to many in the field.

Strengths

- Analyzes LLM representations' ability to predict fMRI data before and after "removing" linguistic information from said representations. - Conducts analyses at both macro (whole-model, and whole-brain) as well as micro (layer-wise, ROI and sub-ROI) levels.

Weaknesses

- The Pearson correlations in Figure 3 are quite low, on average less than 0.1. What do the authors make of this? The ROI-level analyses seem relevant, but are not done with the same metric, so direct comparison is hard. - Linguistic annotations for the text in the reading data come from an off-the-shelf NLP tool (stanza), so these are "silver" rather than "gold" annotations. - More baselines would have been useful.

Questions

- Table 1: what are baselines for each task? What would a most-frequent-class baseline score on each? This would be a helpful comparison for the post-removal numbers. - Word length probing: I'm a bit confused here. Probing for word length should be hard for BERT since it does not have character-level information but instead uses sub-word tokenization. I noticed in the Appendix that the task is referred to as "Sentence Length". Is the task length-of-individual-word (that's what "Word Length" sounds like to me) or is it number-of-words-in-sentence ("Sentence Length")? Consistency with naming, and clarity on the exact task, would be welcome here. - Why the choice of BERT as the model for comparison? In particular, I am curious about the choice of a "bidirectional" masked language model as opposed to a "left-to-right" language model like the GPT series. Some would argue that the latter more closely resemble the reading process, and so might also better predict data from reading. ### Typographic comments: - Line 125 needs a closing period. - Lines 185 and 186: "z-scored" -> "$z$-scored". Same for "t-test" and "p-value" throughout the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- Use of stanza corenlp for linguistic annotations; these are "silver", not "gold". - Only looked at one model, with relatively weak overall correlations with brain activity.

Summary

This paper aims to investigate the correspondence between HIPSs and models based on Neural Networks such as the transformers. The key innovation of this paper is to study this correspondence by elimiating specicific linguistic properties form BERT and to observe how this intervation affects the alignment with fMRI brain recording.

Strengths

- This is really an innovative standpoint

Weaknesses

- There is a major innovation of this paper that is the cornerstone of the theory. Yet, this is not listed as a major result: the removal of linguistic properties from Transformers. Indeed, I was expecting citations to other papers as this is not listed in the main results of the paper. Yet, there are not citations and it seems to be proposed in this paper. - Since the removal of linguistic properties from Transformers is a major cornerstone, all the theory could crumble if it is not investigated in a deep and correct way. - Experiments to demonstrate that the removal of linguistic properties from Transformers are inconclusive. Indeed, Table 1 is not sufficient. It only says that after the adpatation BERT weights are so mess up that the classification task cannot be performed. There should be a comparative part or a sort of proof that the model is not compromised in the other parts. For example, there is a missing column: BERT with random parameters. Indeed, if the "after" models behave as BERT with random inizialization, results are not relevant.

Questions

see weaknesses

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

1 poor

Limitations

not applicable

Summary

The paper investigates the relationship between specific linguistic properties and brain alignment across layers of a language model. The authors provide evidence for the importance of a linguistic property to the brain alignment. The focus of the study is to understand the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.

Strengths

**Originality:** The paper offers an interesting perspective on the link between language properties and brain alignment in language models. It builds on existing work but adds a new dimension by directly showing the importance of specific linguistic properties. **Quality:** The paper uses a large, public fMRI dataset and word features from the BERT model to get word-level representations. The results are interesting. **Clarity:** The paper is well-structured and easy to understand. It explains the methods and results clearly, making it accessible to a wide range of readers. **Significance:** The approach paves a path towards a way to understand how linguistic properties and brain alignment relate. This could help improve language models and give new insights in neuroscience and language processing.

Weaknesses

**Choice of Language Model:** The authors have chosen to focus on BERT, an encoder-only language model, for their study. Given that previous research has shown a deeper connection between autoregressive language models and the brain, the choice of BERT might limit the generalizability of the findings. The authors could consider including autoregressive language models in their analysis to provide a more comprehensive understanding of the relationship between linguistic properties and brain alignment. **Correlation vs. Causation:** The authors aim to identify the effect of linguistic properties on brain alignment, but they rely on correlational metrics, which do not provide evidence of causation. To address this, the authors could consider methods from the causality+NLP literature to produce causal model explanations. This would strengthen their findings and provide more definitive evidence of the impact of linguistic properties on brain alignment. As it currently stands, it’s unclear which conclusions can be drawn from this study. **Projection Method:** The authors use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). The authors could consider alternative methods for identifying the effect of linguistic information to ensure the accuracy and reliability of their findings. Some potential approaches include causal abstractions, which aim to provide a low-dimensional causal representation of LLMs.

Questions

1. **Choice of Language Model:** Could you elaborate on why you chose to focus only on BERT, an encoder-only language model, for your study? Given that previous research has shown a deeper connection between autoregressive language models and the brain, have you considered including autoregressive language models in your analysis? How do you think the inclusion of such models might affect your findings? 2. **Correlation vs. Causation:** Your study aims to identify the effect of linguistic properties on brain alignment, but it relies on correlational metrics. Could you consider methods from the Causality+NLP literature to produce causal model explanations? How do you think a causal analysis might change your findings or conclusions? 3. **Projection Method:** You use a projection method to identify the effect of linguistic information, which has been shown to be problematic and potentially misleading (Ravfogel et al., 2022). Could you consider alternative methods for identifying the effect of linguistic information? How do you think a different method might affect your findings?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors do recognize some important limitations: 1. **Removal Approach:** They note that their method might unintentionally remove information correlated with the linguistic property they're studying. They suggest using larger datasets to mitigate this. 2. **Unaccounted Linguistic Properties:** They admit that there's still significant brain alignment even after removing all studied linguistic properties, indicating that there might be other linguistic properties at play that they haven't considered. 3. **Single Model Use:** They acknowledge that they've only tested their findings on one language model (BERT) and suggest that future work should test other models.

Summary

The authors provide an analysis of correlations between BERT representations and fMRI data when abstracting away different linguistic properties from the BERT representations. They provide an in-depth analysis of the ways in which elimination of word length, tree depth, top constituents, tense, subject number and object number have on the overall and layer-wise alignment of BERT with fMRI data from 18 people. The authors find that removing any of these linguistic properties from the BERT representations reduces fMRI data alignment across all layers in the BERT model. Furthermore, syntactic properties are most relevant for the BERT-brain alignment globally, but semantic properties are highly locally aligned with brain regions generally associated with semantic processing.

Strengths

The overall question and method is relevant to the community. This paper contributes to the cognitive debate on the overall alignment that today's language models have with people's cognitive systems, here concretely their linguistic neural processing of stories. By abstracting away a number of linguistic properties from the BERT representations in a controlled way, the results speak for the type of information encoded in BERT that lead to the reasonably high alignment with fMRI data. The work therefore contributes to a better understanding of BERT representations and the usefulness of investigating them as cognitive systems. The authors integrate valuable controls into their study, such as random baselines and correlational analyses of tasks that nicely contextualize the findings. The paper is very well-written.

Weaknesses

To me, section 5.3 (Decoding task performance vs. brain alignment) has a number of interesting observations but I'm missing the overall motivation for this section. The authors write "While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers." (line 265ff). There seems to be a close connection between this question and Figure 3a from the previous section and it would be helpful to emphasize what the previous section lacks. To be clear, I think section 5.3 adds a lot, especially when it comes to the alignment of local instead of global brain regions but I currently can't find a motivation for whom this analysis is most relevant -- does this tell us something about BERT or are there any potential insights for neuroscience? The semantic alignment results seem to be backed up by prior findings in neuroscience. Is the same true for the rest, or if not, what does that mean? BERT's layer-wise analysis of the correlations with fMRI data are central in the overall framing and centered in the main paper. I think that's rightfully so but I would like to encourage the authors to consider a strategy of substantiating claims that "BERT embeds a rich hierarchy [with] surface information at the initial and middle layers, syntactic information in the middle to top layers [...]" (lines 218ff). From inspecting the corresponding results table (Table 1), surface information seem maybe slightly more represented in the first and least in the second layer, but all other layers seem to be fairly evenly distributed. For the syntactic information, it seems uniformly high, and only subject number and object number seem to have a fairly monotonously increasing and decreasing structure that might suggest that it can't simply be due to random variance. Is there any way to estimate the variance of these terms or find a definition for when something is robustly represented more in some than other layers?

Questions

I find Figure 3 very intriguing. Removing any linguistic property results in a correlation drop from 0.1 to about 0.75. The authors also previously added a correlational analysis indicating that most linguistic properties aren't correlated with each other. Consequently, it would make sense to me that when one removes all linguistic properties, the correlation needs to drop further since different sources of variance have been removed which don't correlate with each other but clearly correlate with the fMRI data. However, even this correlation is at about 0.75. Do you have an explanation why this might be? Picking up on a notion in the "Weaknesses" section: Why doesn't figure 3 already provide us with an answer to the central motivation of section 5.3 ("we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers.")? Or did you mean to motivate section 5.3 with "across brain regions"?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes