Auxiliary task demands mask the capabilities of smaller language models

Summary

This paper looks at whether a "demand gap" exists in tasks designed for LMs, whether this gap is larger for "simpler" models, it draws on a cognitive analogy.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper raises an important point, that task effects may impact different models differently, and that there's a bias in how great this effect is given model size and training size. It reveals a strength of larger models (a robustness) or conversely, a weakness in the smaller models. Researchers should take this into account both for their evaluations and for deciding which models to deploy for their tasks. I appreciated the explicit paragraph on data contamination, and I which that were more standard that it is in NLP papers these days. The 1st, 2nd, and 5th paragraphs in the discussion were cogent and really summarized the paper and its important conclusions well.

Reasons to reject

I find it odd that the abstract assumes a competence-performance distinction in LMs. The distinction is widely but not universally accepted for humans in cognitive science, but it has been explicitly argued for decades that connectionist models (=> modern LMs) do not support such a distinction. Supporters of connectionist models of cognition saw/see this as a virtue. The consequence of this is that properties of the models relating to their architecture, eg number of parameters in this paper, and practical challenges emerging from those decisions, are fundamental to their "competence." They cannot be disentangled. While this doesn't evaluate the paper, it does bring the general narrative, as well as analogy with humans, into question. Take a look at Schwarz (1992, Connection Science), and Allen & Seidenberg (2013, Emergentist Approaches to Language) for examples. Buried under 3.3, there's a short paragraph disavowing direct parallels between child development and model scaling. This is of course true. Children doesn't grow more neurons as they develop, let alone by a factor of 70. However, since model size is one of the two measures of model power that this paper looks at, this severely undermines the analogy between childhood development and model power that the authors make throughout this paper. At the very least, this caveat should be brought up into the introduction, where I first noticed this problem, but a better solution would be to downplay the cognitive analogies being made here. That would also help solve the (lack of) competence-performance distinction issue raised above. Regarding demonstrating the "signature pattern," this relies both on the theoretical assumptions about competence-performance and also practical assumptions surrounding the demand gap. Is forced choice, which is often effectively classification, really the "same" cognitive construct as the production equivalent? They're certainly very different tasks from a traditional ML perspective. If they aren't, then what is being measured isn't really the "demand gap" as stated. They're two different competencies. Related to whether or not they're the same task further assumes that the task setups are in fact evaluating the problem that the researchers think they are. This is particularly crucial in the forced choice setting, which the authors sort of get at in footnote 2, but they don't take it far enough. One example of this that I found particularly compelling was the following, which looked at VQA with multiple choice answers. It turns out that their model achieved most of its performance without even being exposed to the image, which means that it was simply not actually doing VQA as intended. Instead it was picking up on biases in the Q and Q construction. It took aggressive debiasing of the data set to create an evaluation where the authors were confident that the model was actually performing VQA Wei-Lun Chao, Hexiang Hu, and Fei Sha. 2018. "Being negative but constructively: Lessons learnt from creating better visual question answering datasets." NAACL There was a more recent paper making the same point with regards to BLiMP. Because of the forced choice task setup and researcher-generated sentence pairs, models seem to be able to solve BLiMP without actually engaging in grammaticality. This is particularly relevant, since the authors rely on BLiMP for one of their evaluations. Héctor Vázquez Martínez, Annika Lea Heuser, Charles Yang, and Jordan Kodner. 2023. "Evaluating Neural Language Models as Cognitive Models of Language Acquisition." GenBench A better "demand gap" set up might, for example, compare solving math word problems with a lot of filler text and complex scenarios on one hand and more straightforward word problems on the other hand. These are more clearly the same ML task. Given all of the above, the 2nd past paragraph of the discussion comes off as probably nonsense. It should be removed. Typo at the bottom of page 5 "prompted asked"

Summary

This paper makes an interesting contribution to the study of LMs by looking at task demands as a potential moderator variable for task performance. The authors conduct a series of tests including varying the amount of training and find evidence that task demands do indeed pay a role.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The paper of well written, clear and does well in connecting a well known and rather basic concept from developmental psychology with LMs. The tests conducted are adequate and the conclusions floor logically from the findings.

Reasons to reject

I only have one concern (which I think can and should be addressed). The analysis is incomplete as it is essentially a descriptive analysis without any formal statistical testing. The authors could address this by looking at simple analyses such as an ANOVA or other models of the GLM family. This kind of Testing is not only desirable but essential: looking at individual comparisons without statistical justification to do so is incorrect.

Questions to authors

-

Summary

The authors study how task demands impact language model evaluation. Their approach is motivated from a developmental psychology perspective where it is well established that children might fail certain tasks due to higher task demands and not because they lack the capacity of interest. Applying this framework to language model evaluation, the authors ask whether task demands have a bigger effect on smaller models, thereby masking their “true capabilities”. The authors predict a demand gap, i.e., that less capable models (smaller models or those trained for less steps) suffer more from higher task demands, i.e., achieve lower performance, than more capable models. In the experimental section, the authors evaluate 5 different model families on 4 different datasets while controlling for task demand. The experimental findings are mostly consistent with the demand gap prediction. Finally, the authors discuss the implications of their work for LM evaluation, the study of emergence, and even developmental psychology.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The paper studies an important problem and is exceptionally well written. The research is well motivated and the authors present a clear hypothesis for what they expect to find. The experiments are well designed and sufficiently broad: testing 5 different model families, considering different sizes for every model (except Mistral), using 4 different datasets and 4 different ways of evaluation. The discussions are extensive and the authors do a great job of connecting their work to the previous literature as well as discussing the implications of their findings. Overall, this is a great paper and I really enjoyed reading it.

Reasons to reject

I only have two minor concerns and they are not strong enough to serve as reasons for rejection. I will still mention them here. - The Pythia models clearly behave as an outlier for 3 out of 4 evaluations. The authors could do a better job at discussing and investigating what is going on here (they briefly speculate about the number of training steps but I’m not fully convinced by that). - The meta linguistic evaluation considers only a single prompt (for each of the tasks where it is applied). It is however widely established that prompt engineering can have a dramatic influence on model performance. These results would be even more convincing if they would take different prompts into consideration.

Questions to authors

**Evaluation methods** - Are the answer options part of the input during forced choice eval? Or are you only extracting the logits from the answer tokens? This was not entirely clear to me. - It would be interesting to discuss the limitations of each evaluation method a bit more and also talk about whether they can be applied to any task. As an example, direct evaluation assumes that we already know the answer and therefore might not be applicable at test time. **Data contamination** - While I agree that performance per se is not so important here but rather the difference in performance, data contamination might impact the capabilities you are actually evaluating. If we assume that all models saw e.g., LAMBADA and bigger models are better on it, are we evaluating recall from memory or the actual task? I feel this could be discussed a bit more. **Error analysis for production and meta-linguistic (in the word prediction case)** - It would be interesting to look at the errors the models make in this case and see whether there are interesting patterns here. Maybe this could help to understand in which sense the model is misinterpreting or misunderstanding the task? **Figure 4** - It could be interesting to look at the rank of the correct answer (for the word prediction task) as a function of the training steps. **Pythia results** - The difference to other models might also be explained by the training data. I feel this aspect could in general be discussed a bit more. - It would be interesting to see the checkpoint evaluation (Figure 4) for pythia models as well.

Summary

This paper investigates how "task demands" in evaluating language models can mask their underlying cognitive capabilities. It argues that evaluations with greater auxiliary task demands beyond the targeted cognitive construct tend to yield lower performance, especially for weaker models. The authors adopt two evaluation contrasts to uncover the influence of task demands: 1) production vs forced choice, and 2) metalinguistic judgments vs direct probability measurements. These contrasts are applied to four test scenarios and five LLM families. The experimental results demonstrate that there is a "demand gap" where the higher-demand task results in lower performance compared to the lower-demand task, and this "demand gap" shrinks as model size/training increases.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

1. This paper explores and verifies an important issue, the "task demands" in assessing LLMs. Its core idea is that different abilities should be disentangled, so that other basic abilities like "interpreting questions" do not influence the measurement of the target ability. Studying "task demands" is important, as it can help us better understand the true performance when benchmarking LLMs. I have learned valuable knowledge from this paper, and I believe the NLP community should be broadly interested in it. 2. This paper is based on a concrete and traceable theory from developmental psychology, extending from assessing children to evaluating LLMs, which is reasonable and insightful. Considering that LLMs are currently widely used as human-like autonomous agents, this transferable research is worth encouraging. 3. The authors select four test scenarios (cognitive constructs) and considered two sets of high- and low-demand evaluation contrasts. They conduct extensive experiments on five LLM families (a total of 13 LLMs), demonstrating the existence of "demand gaps" and their shrinkage as the model size/training increases. The main claims are well supported by the experimental results. 4. The paper is very well-written and engaging.

Reasons to reject

Overall, this paper is good, but it appears to be under-explored or arbitrary on the following two points: 1. There is a lack of a clear definition and boundary of high-demand and low-demand tasks. What tasks are high-demand, and what are low-demand? Furthermore, what abilities are introduced by high-demand tasks that are not involved in low-demand tasks? Although the authors list some abilities like "interpreting questions", a complete and clear taxonomy or something similar is still missing. It is encouraged to supplement relevant psychological knowledge. 2. The selected evaluation methods (production, forced choice, metalinguistic judgment, probability measurement) and cognitive constructs (analogical reasoning, reflective reasoning, word prediction, grammaticality judgment) lack motivational explanations. It is unclear whether these selections are representative or can be generalized to other scenarios.

Response re. statistical model

Thanks for this clear response, this makes this paper much sharper. If you can explain this to the NLP audience (most will be [very] unfamiliar with these kinds of models) and maybe show the key test statistics and effect size of each effect (main effects and interactions, e.g., in a table) on the extra page, I'd be happy to increase my score.

Thank you for the response

Thank you for your rebuttal, though I don't find it convincing My comment about the competence/performance distinction is the to the applicability of this distinction to *models* just as you say. It is far from obvious that it even makes sense to talk about the distinction in reference to neural models in the first place, which is a fundamental assumption of the paper. This is why I brought up the connectionists. They argued that their neural *models* lacked the distinction and yet succeeded at their tasks, therefore humans lack the distinction. So there is relevant existing literature out there that disagrees with the fundamental assumption, yet it is not addressed. Yes, I saw the disclaimer in Section 3.3. It doesn't obviate the problems with how the paper is written.

Response to Authors

Thank you for your response. I appreciate the clarification on the second point. However, regarding the first point, I still recommend that the authors conduct a more extensive review of the psychology literature. Incorporating these two points into the revision will make the paper more convincing. Thank you, and I will maintain my score.