Instruction-tuning Aligns LLMs to the Human Brain

Summary

This paper aims to study the difference between base and instruction-tuned models in terms of their alignment to both human brain and behaviours. 1) The brain alignment is measured by how well a linear regressor can predict fMRI activity from an LLM's hidden representation. 2) The behavioural alignment is measured by how well per-word perplexity correlates with per-word reading time. The work further studied the how well brain and behavioural alignment correlates with several test set and model properties.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- This paper is interesting to read and contributes knowledge to the field of HCI and LLM-humanalignment. I find this task/setting to be relatively novel. - Technically, the work presented a correlation analysis at two levels: - how base/instruction-tuning affects brain and behaviour alignment. - how test set (knowledge being tested) and model (size, capability) properties then correlate with the brain and behaviour alignment coefficients.

Reasons to reject

Major: - From the results in the brain alignment part, I agree that instruction tuning aligns "better" than base/pre-trained models, but the Pearson correlation coefficients are still in the "no correlation" to "weak positive" range. If I did not misunderstood: the average improvement (from base to instruction tuning) seems significant, but it is a % score derived from the difference between already weak correlation coefficients. I think that a quantitative definition of "alignment" should be provided or explained, e.g. how good the correlation needs to be in order to qualify for "alignment"; otherwise it is unsound to claim the title of this paper. - The introduction states that "to investigate why instruction-tuning increases alignment to human brain activity [...]" which formulates the problem as a causal analysis, but the presented methodology is a post-hoc correlation study. I fear that the conclusion might lack generalisability. Minor: - Slight overclaim in introduction "however, there has been no similar study on how instruction-tuning". [1] has done an analysis from the perspective of LLM and human attention. - The methodology used for behavioural alignment is "the similarity between LLM per-word perplexity and human per-word reading times", which is an accepted method but the perplexity and reading time should not be used to define behavioural alignment. ~~In fact the features used in [1] yielded different conclusions.~~ (the reviewer made an incorrect statement in the original review.) [1] Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (Gao et al., EMNLP Findings 2023)

Summary

This paper is an investigation into whether instruction-tuning of LLMs affects the alignment between intermediate LLM representations and brain activity as measured by fMRI. Alignment with brain representations is measured using the Brain-Score linear predictivity measure from prior work, and this measure is computed across a variety of language models both with and without instruction-tuning. The experiments find that instruction tuning increases brain alignment by 6.2% on average across different base and instruction-tuned model pairs. In addition, the brain alignment of the different LLMs is also compared with performance on benchmarks measuring world knowledge and reasoning, and a strong correlation is found with world knowledge benchmark performance but not with reasoning. Finally, there is also an experiment in the paper comparing "behavioral alignment", which looks at the correlation between per-word perplexity of LLMs and the human per-word reading times, but does not find any significant effect of instruction tuning on this metric.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

The finding that brain alignment correlates better with benchmarks measuring world knowledge than it does with benchmarks measuring reasoning is interesting. Though as the authors note, this finding could be different if the subjects whose brain activity was being measured were performing tasks requiring more reasoning rather than reading, and the correlation with world knowledge benchmarks was still less than with model size, which has a very strong correlation that has also been seen in prior work.

Reasons to reject

This paper does not introduce new methods of analysis, it runs existing implementations of LLM-brain alignment measurement across a different set of input LLMs. As the paper notes, much prior work has found that LLM-brain alignment is well correlated with improved LLM benchmark performance, and since instruction tuning is known to substantially improve benchmark performance it seems unsurprising that it would also follow this trend and improve alignment.

Summary

This paper explores the correlation between instruction-tuning and brain alignment as well as behavioral alignment, demonstrating that instruction-tuning can enhance scores for brain alignment without increasing behavioral alignment scores. Additionally, the paper analyzes the correlation between brain alignment, behavioral alignment, model size, world knowledge, and model capabilities.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The study evaluates 25 LLMs from the T5 and LLaMA families, as well as 8 models from the GPT2 family, including their original versions and various instruction-tuned versions, offering a comprehensive analysis of instruction tuning across these three model families. 2. The paper examines the correlation between brain alignment, behavioral alignment, and the characteristics of LLMs.

Reasons to reject

1. Figures 2 and 4 (B, C, D) are noted to be somewhat unclear, suggesting a need for refinement in their presentation for better clarity. 2. The paper would benefit from an expanded analysis that includes models such as LLaMA2 and Mistral to more effectively demonstrate the impact of instruction tuning. Additionally, the inclusion of an analysis on the correlation between large language model alignment methods, such as RLHF, and brain alignment would contribute to the comprehensiveness of the study. 3. It is recommended that the use of reading time in behavioral alignment be reconsidered to include the potential impact of word length. The analysis could potentially be more thorough if it differentiates between long and short words based on a defined word length threshold, and also takes into account the reading time adjusted for word length, such as by using a reading time to word length ratio. 4. Representational Similarity Analysis (RSA) and Brain-Score can both indicate brain alignment, and the authors might explore whether RSA could lead to the same conclusions.

Questions to authors

refer to the comments

Summary

This work investigates whether instruction-tuning can make LLMs process language similar to humans. In particular, this work studies: Brain Alignment: How closely LLMs’ internal representations match neural activities in the human language system. Behavioral Alignment: How similar LLM behaviors are to human behaviors during language tasks. Comprehensive and carefully-designed experiments are conducted to draw the conclsion: Instruction-tuning generally aligns LLM representations to human brain activity and instruction-tuning LLMs generally does not enhance behavioral alignment with human reading times.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

1. Very interesting research problem. since the instruction tuning has been a de-facto choice for training LLMs, this study can provide some insights into whether instruction tuning can lead to human-level or human-like intelligence. Whatever the answer is, the research question per se is important and worth investigating. 2. The experiment setup is valid and comprehensive: four types of langauge abilities, 25 LLMs (instruction tuned or not) with different sizes. 3. Intersting and meaningful findings.

Reasons to reject

1. It would be great to provide more insights/future directions to how to build more brain-/human-intelligence-aligned LLMs. The rest weaknesses here are more like a suggestion not weakness: 2. It would interesting to see the connections between the alignment and the training data, which will provide more insights to determine what's lacking in the current instruction tuning to build more brain-aligned LLMs. 3. I notice the behaviors of LLMs on reasoning tasks is not well aligned with human. I wonder whether it's a training data issue or model arch issue. There are reasoning-specific instruction tuning models such as Mammoth for math reasoning. It would be great if the models are specificly trained for a specific task can increase the alignment.

Response to the AC

We thank the reviewers for their feedback and the AC in advance for their meta-review. We also thank the reviewers for identifying our paper’s strengths: - **edNc**: comprehensive and carefully designed experiments, very interesting research problem, and meaningful findings. - **uLs2**: comprehensive analysis of instruction-tuning, using many LLMs from multiple model families. - **hcZZ**: interesting finding that brain alignment correlates better with world knowledge than reasoning. - **xjE7**: interesting to read, contributes knowledge to HCI and LLM-human alignment, and the task/setting is relatively novel. We summarize each reviewer’s suggestions/concerns and our responses: **Reviewer edNc**: - Suggested we discuss more insights to build brain-/human-intelligence-aligned LLMs. In our paper, we will add a subsection discussing papers with insights on LLM and human cognition and how to build brain-aligned LLMs. (References [1-7] in our rebuttal). - Suggested we study connections between brain alignment and training data. We explain that we instruction-tuned LLaMA-7B and an ablated version, and how that provides preliminary insights. - Suggested we test reasoning-specific models (e.g., Math). We explain that reasoning-specific models may not be appropriate as we limit our scope to the human language system and use datasets where participants read/listened to English text. **Reviewer uLs2**: - Noted that two figures in our paper could be clearer. We remove model names and adapt point sizes to model sizes for better readability. We provide larger figure versions with model names in the Appendix. - Suggested we test additional models, e.g., LLaMA2, Mistral. We explain we prioritized models with diverse model architectures and families (LLaMA, T5, GPT-2). Regardless, in our rebuttal, we show results for LLaMA2 and Gemma models, yielding similar results to our paper’s conclusion. - Suggested we study the effect of RLHF. We explain that we studied only one RLHF model, which is insufficient to draw rigorous conclusions. Our paper’s focus is instruction-tuning, for which we provide comprehensive results. - Asked about the effect of word length on our behavioral alignment method. We explain that we follow prior works in implicitly accounting for word length by computing the surprisal of a word with multiple tokens as the sum of surprisals of each token. Importantly, we use consistent tokenizers when comparing each vanilla LLM to its instruction-tuned version, thus controlling for the effect of word length. - Suggested we run other brain alignment methods (e.g., RSA) aside from Brain-Score linear predictivity. In our rebuttal, we add preliminary results for RSA and CKA, showing they are similar to linear predictivity. We focused on linear predictivity following prior works, but will add more comprehensive RSA and CKA results to our paper. **Reviewer hcZZ**: - Concerned that our paper has no new methodology. We cite the COLM review guide in explaining that developing new methods is only one of many forms of scientific contribution; our analysis of instruction-tuning and LLM world knowledge contributes new insights to the field. - Implied that since LLM performance is correlated to brain alignment (shown in prior work), it is expected that instruction-tuning improves brain alignment since it improves LLM performance. We explain why we are not confident in this generalization. (a) Prior works evaluate LLM performance using next-word prediction, whereas we study reasoning and world knowledge. (b) Contrary to the reviewer’s point, prior work showed that finetuning may result in no change or decreased brain alignment depending on the finetuning task and brain region. **Reviewer xjE7**: - Asked about the weakly positive correlations in our brain alignment results. We explain that some of our LLMs actually obtain brain alignment close to the fMRI “noise ceiling” (further explained in our rebuttal and paper), suggesting they align nearly all useful signals from fMRI data. We follow prior works in computing the noise ceiling. - Claimed that our study is not a causal analysis. We explain how our experiments showing that instruction-tuning improves brain alignment are, in fact, a causal analysis. - Noted a related work (reference [1] in their review) on instruction-tuning. We explain that our study is complementary to [1], with many key differences in datasets and methods. We will discuss this related work in our paper. - Suggested our behavioral alignment method should not be used, and yields different conclusions from [1]. We explain that our behavioral alignment method is commonly used in the literature. Also, it actually yields a similar conclusion to [1]: improving LLM performance does not enhance alignment with human reading behavior. We will add these clarifications to improve our paper. Thank you again! -Authors