Summary
This paper aims to provide a theoretical explanation for key emergent abilities of LLMs (in-context learning, chain-of-thought prompting, instruction fine-tuning), in terms of Bayesian inference on latent intentions underlying the language.
Strengths
1. This paper provides a new look at the role of the link between language and underlying intentions in enabling emergent properties. It does so using both theoretical arguments and experiments.
2. The formal results are sound, as far as I could tell (though with some question, as described below).
Weaknesses
1. It remains unclear how the proposed account substantially improves over Xie et al 2022, which already explained ICL in terms of the recovery of an underlying state / intention. The overall claim, and the explanation of in-context learning in Section 5, are similar to Xie et al. 2022.
The paper additionally provides sections about Chain-of-Thought prompting (Section 6) and instruction finetuning (Section 7), but they are quite informal and unspecific.
The arguments in Section 6 are informal, using the idea that direct transitions between query and answer intentions may be rare, and that intermediate steps are more well-represented. It is left open how these ideas would be formalized and what precise assumptions are needed. In particular, as Sections 2&3 only assumed single intentions for the generative model underlying language (one intention is sampled and mapped to a text), it is unclear how the ideas in Section 6 (which refer to chains of intentions) can be formalized within the overall framework.
In Section 7, the idea seems to be that instructions provide information about the intention \theta, but this argument is again not formalized. Thus, it is left somewhat unclear how the formal results present a substantial advance over Xie et al 2022.
2. Section 5 considers prompts consisting of demonstrations of input-output pairs, and assumes that these correspond to some underlying intention \theta. The results in Section 5 show that, to the extent that underlying intentions are unambiguously recoverable, a predictor can correctly continue a prompt. However, this leaves open how an LLM would acquire the ability to infer the intention from such prompts, which typically have an unnatural repetitive structure, not well represented in the training set.
Questions
1. Issue with clarity: In Section 2, the definition of \epsilon-ambiguity: What is \theta_0? The condition seems to say that, whenever we sample \theta_0 and then sample x \sim q(x|\theta_0), then Pr(\theta_0|x) \geq 1-\epsilon(x) -- is this correct? Also, it seems Pr(.|.) is not defined. A plausible interpretation is that it is a Bayesian posterior for the prior q(\theta) and the likelihood q(x|\theta_0) -- is this correct?
2. Proof of Theorem 1: The last paragraph makes reference to neural networks and SGD, whereas the theorem statement makes no reference to these (it just refers to universal approximators, without reference to architectures or training procedures). Either the theorem needs to be rephrased, or the proof should be phrased for such general universal approximators.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.