What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores

Thank you for your thoughtful response and for raising your score based on the clarifications we provided. We're glad to hear that our additional analyses and explanations addressed your concerns effectively. If there are any further questions or aspects you think could still benefit from more clarification, particularly regarding whether GPT2-XL is exploiting temporal auto-correlation rather than linguistic similarity, please let us know. We appreciate your feedback and the opportunity to improve our work.

I thank the authors for responding to my comments. W2: Why are the pink/yellow bars so low for "LR, LT, Untrained, Pearson"? There appears to be quite big difference to BR, as well as SP? Can you please plot the scores as scatters against the R2 scores you report in the paper? It is challenging to see how much scores align with looking at different bar plots. W3: Thanks, this should be much clearer throughout the paper, and statistics/plots should consistently be reported for language-responsive or not (or at least make it very apparent what is what). Q1: Thanks for noting the confusion with the legends. However, I don't think my original question was answered with respect to visual regions? And for the dots below the diagonal in Figure 2b, how was it determined that they were not significant? And the fact that they were in the language network (and not elsewhere) makes sense? Because again, then visual regions might be driving the statistics? (this relates a bit to W3 where it is hard to make sense of which scores are reported for which voxels). Q2: I am not sure my original question was answered (sorry if I was unclear). But ultimately, doesn't OASM correspond to a predictor that has block-like structure, which is also the case when you have contextualized LLM representations in a block? Q3: Can you please clarify this? Other comments on global rebuttal: 1: Please correct me if I am reading Table 1 wrong, but GPT2-XL contributes a solid amount of neural variance? 2: Regarding your comment that "Finally, we used sum pooling because we found that LLM brain scores were in general higher when using sum pooling as compared to last token pooling (particularly so for untrained models), and we aimed to use methods that allow the LLM to perform maximally to avoid making our goal artificially easy. Furthermore, we disagree with the notion that using the last token representation is inherently more principled than sum-pooling. For each fMRI TR, the corresponding sentence was read between the 8th and 4th seconds preceding the TR. If the participant reads each sentence one word at a time, and each word induces a neural response, then the recorded image would show an HRF convolved with the neural responses to each word, roughly equal to the sum of the neural responses to each word.". Yes, I agree, but that *only* holds for Blank. The Pereira dataset was presented with a sentence on the screen, one at a time (full sentence, not word as you mention), and hence the logic you argue for does not hold -- and last-token pooling could be argued to be more suitable. 3: Per the global rebuttal and other reviewers comments on reproducing Schrimpf et al 2021, are you intending on changing the paper title?

Thank you for the additional responses and questions. W2: Thank you for pointing this out. As we outlined in the detailed expansion section in the global response, we are working with “very ill-conditioned regressions due to high dimensional feature spaces, relatively few samples compared to the number of features, and noisy targets”. Vanilla linear regression (LR) is ill-suited in these cases, resulting in overfitting and leading to poor cross-validated correlations in contiguous train-test splits. This is additional evidence that BR is better suited for these cases. As for why LT doesn’t perform as well as SP, we speculate that this is because the sentence length (number of words/tokens) is not as linearly decodable from intermediate layers with LT as it is with SP. We explain why sentence length is linearly decodable from the intermediate layers when using SP in the first paragraph of *Section 3.2*, and provide a more formal proof of this in *Section A.9*. In summary, the fact that we are working with ill-conditioned regressions as well as the linear decodability of sentence length from intermediate layers with SP are explanations for why the pink/yellow bars are so low for “LR, LT, Untrained, Pearson”. We will make sure to include this explanation in our updated draft. We will include the scatter plot in the camera-ready version of the paper to facilitate easier comparison. Thank you for this feedback. W3: Thank you, we will make it more clear which statistics/plots correspond to which functional networks in our updated draft. To clarify for now, aside from Figures 2d and 3d in the main paper (and Figures 5 and 6d in the Appendix), every plot corresponds exclusively to language network voxels/electrodes/fROIs (as defined on the basis of their language-responsiveness by the Fedorenko lab and described further in the *Functional Localization* paragraph of **Section A.1**). Q1: Thank you for these comments. Regarding performance for the visual regions, we neglected to mention that the glass brain plots for each experiment come from a single representative participant. We will make sure to include this information in our updated draft. The result you mention where SP+SL predicts visual regions well was not robust across participants, which can be seen in the EXP2 plots on the right (where there is much higher predictivity in language network voxels). We will make sure to include glass brain plots for all participants in the Appendix to emphasize this. Also to clarify, the main focus of our glass brain plots in Figure 2d was to show that SP+SL* and SP+SL+GPT2-XLU* perform virtually identically across functional networks, including the language network. We will make sure to include an expanded discussion of our results on the glass brain plots in the updated draft. Regarding significance testing, we state in the results section that “we performed a one-sided t-test between the squared error values obtained over sentences between SP+SL+GPT2-XLU and SP+SL”. We provide a justification for this statistical test in **Section A.4**. As we describe in **Section 3.2**, lines 223-227, we then perform FDR correction within each participant and functional network (e.g. language network). This corrects for multiple comparisons, while still minimizing the number of false negatives since the number of tests in a given FDR correction is only the number of voxels in a single participant’s functional network, rather than the total number of voxels across all functional networks and/or all participants. We would like to clarify that it is not possible for visual regions to be driving the statistics, because we perform voxel-wise significance tests. Specifically, we find that “across all functional networks, only 1.26% (EXP1) and 1.42% (EXP2) of voxels were significantly (α = 0.05) better explained by the GPT2-XLU model before false discovery rate (FDR) correction; these numbers dropped to 0.001% (EXP1) and 0.078% (EXP2) after performing FDR correction within each participant and network [31]. None of the significant voxels after FDR correction were inside the language network. Taken together, these results suggest GPT2-XLU does not enhance neural prediction performance over sentence length and position even at the voxel level.” Since FDR correction is done within the language network within each participant, there is no influence of visual area voxels on our finding that there were no significant voxels within the language network. Lastly, although there were voxels where SP+SL+GPT-XLU performs better than SP+SL in the language network as you mention, none of these were significant. In other words, GPT2-XLU does not explain significant neural variance over SP+SL specifically in the language network. Please let us know if we have not addressed any of your questions, we are happy to provide additional insight.

Q2: Yes, OASM does indeed correspond to a predictor that has block-like structure (specifically with temporally closer samples within a block being represented more similarly). It is true that such structure can also be present for contextualized (or decontextualized) LLM representations. However, such structure is also expected for the stimulus-independent (i.e. “noisy”) component of neural responses. The main message made in our rebuttal to this point is that, when using shuffled train-test splits, it is unclear whether the reported neural predictivity is due to predicting stimulus-dependent activity or stimulus-independent autocorrelated noise. Our additional variance partitioning analyses (Figure 3 in the new pdf) indicated that even when using decontextualized inputs for the LLM, the vast majority of the neural predictivity of the LLM could be accounted for by OASM. Importantly, because both stimulus-dependent and independent activity would be expected to have this OASM-like structure in Pereira, the fraction of neural responses predicted by GPT2-XL that can also be predicted by OASM cannot be disambiguated as stimulus-dependent or independent when using shuffled train-test splits. We will explain this clearly in our updated draft. Please let us know if we can clarify this point any further. Q3: Thank you for giving us the opportunity to clarify our voxel-wise correction approach. The core motivation is that we never want the incorporation of our simple features to bring down the reported performance of a model that includes an LLM as feature space. Hence, when using the voxel-wise correction for a model that includes an LLM as a feature space, the R2 taken for each voxel is that of the maximally performing sub-model for that voxel that includes the LLM, including the model that is just the LLM itself. To make the comparison fair, we do a similar procedure for the simple (non-LLM containing model), wherein we choose the best performing sub-model that does not include the LLM. For example, when comparing SP+SL* and SP+SL+GPT2-XLU*, we use the R2 of the best performing submodel in the set {SP, SL, SP+SL} for each voxel to compute SP+SL*, and we use the R2 of the best performing submodel in the set {GPT2-XLU, SP+GPT2-XLU, SL+GPT2-XLU, SP+SL+GPT2-XLU} for each voxel to compute SP+SL+GPT2-XLU*. We hope this clarifies any confusion, and are happy to answer any additional questions. Comment 1: It depends on how one defines a "solid amount". The main takeaway of Table 1 is that 81.2% of the variance explained by GPT2-XL can be explained by SP+SL+WORD in EXP1 and 90.1% of the variance explained by GPT2-XL can be explained by SP+SL+WORD in EXP2. Hence, the vast majority of variance explained by GPT2-XL here does not require contextual explanations. When further incorporating fairly limited contextual features (SENSE and SYNT), we explain 89.9% of variance explained by GPT2-XL in EXP1 and 97.8% in EXP2. Please let us know if we can make Table 1 more clear. Comment 2: We would like to clarify our argument regarding sum token pooling: we didn’t mean to imply that the sentence is presented one word at a time, rather that the participant “reads each sentence one word at a time,” and we apologize for the confusion. We assure you we appreciate your comments on the last token pooling, and we believe it is important to show that our results are consistent with both last token and sum pooling. To this end, we have shown that our results are consistent when using both last token and sum pooling methods in Pereira. We will ensure that our updated draft places a larger focus on the last token pooling method than the current draft. Comment 3: We are open to modifying the title of our paper. Based on our understanding, the original criticism of our title was that we did not use the design choices used in Schrimpf et. al 2021. In our response, we have shown that our key findings on Pereira hold when using these design choices. If there are additional concerns regarding the title, we are happy to understand them so we can be as thoughtful as possible in the title.

Thank you for the additional responses and questions. 1. We personally agree that this is not surprising, but this is not the prevailing practice in neural encoding studies with these specific, widely used datasets. The novelty of our finding is not that this is “surprising”, but rather that it questions the interpretation of several moderate to highly impactful papers in the field. For instance, according to Google Scholar, Schrimpf et. al 2021 has been cited over 500 times in the past three years. Furthermore, studies in the field are continuing to use shuffled train-test splits (AlKhamissi et al. 2024), and recent studies have argued that shuffled train-test splits can be appropriate (Kauf et al. 2023). To reiterate, we are not stating that the use of contiguous train-test splits is novel, we are calling into question results from several previous papers using these neural datasets that have used shuffled train-test splits. We believe it is important to bring light to this issue because many readers are not aware that these studies employ shuffled train-test splits. We believe the continued use of shuffled train-test splits is harmful for the field, and that it is critical to address. Our goal is that the rest of the field, like you, would also find this not surprising, but we find this must be done through rigorous analyses, especially as others argue that shuffled train-test splits can be valid (Kauf et al., 2023). 2. We are happy to include alternative statistical tests, such as the bootstrapping procedure outlined in Reddy et. al 2021, in our updated draft. As we clarify, our statistical testing design choices were made to be conservative in supporting our conclusions. Our guiding philosophy for the statistical tests was that we wanted to minimize the number of false negatives. In other words, we wanted to avoid cases where the statistical test falsely reported that a model containing GPT2-XLU as a feature space did not significantly outperform a model not containing GPT2-XLU as a feature space. This is because we wanted to be liberal when stating for how many voxels/electrodes/fROIs does GPT2-XLU explain significant variance over simple features (to avoid overstating the importance of simple features). In line with this, we accounted for multiple comparisons by performing FDR correction within each functional network for each participant separately in Pereira. This minimizes the number of false negatives since the number of tests in a given FDR correction is only the number of voxels/electrodes/fROIs in a single participant’s functional network, rather than the total number of tests across all functional networks and/or all participants. If by dependency between test conditions you mean the dependency between temporally adjacent samples in the test set, we address this concern in Section A.4. In brief, we do not account for this temporal dependency because it only increases the rate of false positives (i.e. makes it more likely for us to say that a model containing GPT2-XLU outperforms a sub-model that does not contain GPT2-XLU), and so the net impact is that we overestimate for how many voxels/electrodes/fROIs does GPT2-XLU explain significant additional variance over simple features. Since there are no voxels/electrodes/fROIs where GPT2-XLU explains significant variance over simple features, this turned out to not be a concern.

3. Thanks for raising these interesting points. We report the performance across all layers for GPT2-XLU when using the sum pooling method in Figure 5a, and find that while the lower layers of GPT2-XLU tend to perform better when sum pooling, it is not by a very large margin. Therefore, the fact that all variance is explained by trivial features alone does not imply that only lower layers of the untrained model predict brain signals well. If the reviewer is suggesting that later layers might in fact have more complex features driving the mapping and that this warrants more detailed analysis, we will repeat our analyses for all layers of GPT2-XLU in our updated draft. To clarify, we did not do this in our original draft because previous neural encoding studies, including Schrimpf et al. 2021, focus their analyses on the best-performing layer of the LLM and we wanted to remain consistent in this manner. As for predictions in sensory regions, no, we do not see similar levels of alignment when using untrained models to predict early sensory regions. This can be seen in Figure 5a for sum pooling by comparing the visual network (which we consider a sensory region since the participants are reading the sentence) performance to the language network performance (and also for last token pooling in Figure 5c). We observe that the alignment between GPT2-XLU and the brain is explained by simple features across all functional networks, including the visual network. We state in Section 3.2 that across all functional networks, 0.001% (EXP1) and 0.078% (EXP2) voxels were significantly better explained using SP+SL+GPT2-XLU than SP+SL, and we will report statistics specifically for each functional network in the updated draft. Furthermore, it is unclear whether this small fraction exists because we use a liberal statistical test, as we discuss in our response to Q2. We will update the manuscript to clarify that there is an extremely small fraction of voxels where GPT2-XLU explains additional neural variance over SP+SL across all functional networks, including sensory regions. If the reviewer still believes additional analyses are worth pursuing in light of this, please let us know. 4. To be clear, applying the noise ceiling does lead to an observable change for a single model. However, our focus here is on comparing the brain score values between two models (specifically between a model with an LLM as a feature space and a sub-model without an LLM as a feature space). In this case, computing a noise ceiling is redundant. Just to be perfectly unambiguous about what we exactly did, if the raw brain predictivity of model A is a, the raw brain predictivity of model B be b, and the noise ceiling is c, then the “normalized” brain predictivity for model A is a/c and for model B is b/c. When we quantify the performance of model A relative to model B, we compute a/b, which is unaffected by noise ceilings, (a/c) / (b/c) = a/b. This applies to all of our quantifications of relative performance, as well as to the variance partitioning metric described in the global response. We will clarify this in the manuscript. Also, as stated in our rebuttal to reviewer bjuX: We also note that noise ceilings, when calculated by predicting each participant’s voxels with all other participants’ voxels and then extrapolating the performance to infinitely many participants, typically underestimate how much variance can be explained in a dataset. For instance, in a more recent paper using the same analysis choices as Schrimpf et al. 2021, several LLMs exceed the “noise ceiling” in Pereira, often by a large margin (Aw et al. 2023). Lastly, computing the noise ceiling in the same style as in Schrimpf et al. 2021 is quite computationally expensive. For all of these reasons, we were unable to compute a noise ceiling prior to the end of the rebuttal period, but would be willing to include it in the camera ready version if desired. 5. Thank you for this visualization suggestion. We will add it to the manuscript.

Thank you, authors, for providing clarification on statistical tests. The early papers [Schrimpf et al. 2021], [Kauf et al. 2021] have introduced many novel findings. For instance, [Schrimpf et al. 2021] was the first study to reverse-engineer many language models as subjects to estimate brain alignment. This study explored 43 language models and introduced the concept of noise ceiling estimates. Similarly, [Kauf et al. 2021] focused on understanding the reasons for better alignment between language models and brain recordings, exploring various syntactic and semantic perturbations to verify these reasons. In contrast, the current study focuses on comparing shuffled train-test splits with contiguous train-test splits. What are the key takeaways for the computational cognitive neuroscience community? As it stands, the results are not particularly surprising. Therefore, the current version requires additional interpretation, particularly focusing on the analysis of simple features at different layers, as well as exploration using other narrative ecological datasets.

Q1: Thank you for bringing up these other studies. We would like to clarify our response that “our analyses showing that trained and untrained GPT brain scores can be explained by simpler features have not been done before” was made in the context of discussing the novelty of our analyses relative to Schrimpf et. al 2021, and was made in regards to the specific datasets used in Schrimpf et. al 2021 and our study. Throughout our paper and in our responses, we are very careful to state that our findings apply to the specific datasets used in our study. In fact, we state in our Limitations and Conclusion that “we did not analyze datasets with large amounts of neural data per participant, for instance Lebel et. al 2023 in which the gap between the neural predictivity of simple and complex features might be much larger.” We would like to also emphasize that the datasets we address have been used in many studies that have used LLM-to-brain mappings to motivate neuroscientific conclusions (e.g. Schrimpf et al. 2021, Hosseini et al. 2023, Kauf et al. 2023, Aw et al. 2023, AlKhamissi et al. 2024). We apologize if this was ambiguous, and thank the reviewer for giving us the opportunity to further emphasize this point. Regarding the studies cited by the reviewer, we would like to emphasize the following points. **1)** Notably, none of these studies use any of the three neural datasets used in our study, and therefore we make no claims about the degree to which simple features account for LLM to brain mappings in the neural datasets employed in these studies. **2)** Based on our understanding, the only paper cited which discusses untrained LLM findings is “Brains and algorithms partially converge in natural language processing”. However, the goal of that study was not to examine what features drive the above chance neural predictivity of untrained LLMs, and so they did not perform the same analyses we did using simple features. **3)** Based on our understanding, the studies cited do not show on the datasets they use that such simple, non-contextual features explain such a large fraction of trained LLM brain scores (which we showed on Pereira and Fedorenko), or that when performing train-test splits correctly trained LLM brain scores fall to chance levels (which we showed on Blank). Therefore, we believe our findings on trained LLMs with the specific datasets we use are also novel. We leave it to future work to examine whether simpler explanations account for as much of the variance predicted by LLMs in other datasets. We will reference these papers in a related works section in the updated draft. Q2: Thank you for bringing up this distinction in how noise ceiling values are computed. In our initial response, we were referring to the concept of noise ceiling generally rather than the specific methods used to compute them. We would like to emphasize that noise ceiling computations do not impact our findings (see point 4 two rebuttals ago).