Summary
Motivated by safety concerns, this paper studies if an LLM can infer a concept or fact without being trained on data that explicitly contains this fact and without using in-context learning. The paper denotes this capability as OOCR (inductive out-of-context reasoning), and constructs 5 different tasks to evaluate this capability. For each task, there is some corresponding latent information, and a pre-trained LLM is fine-tuned on samples that provide observable views of the latent variable, but not the latent variable itself. Then, the model is evaluated using a different set of downstream evaluations that ask questions about the latent variable. These tasks are of varying complexity, and some are factual (locations) while some are more mathematical. They find that this fine-tuning on implicit training samples results in significantly higher accuracy than approaches like 1) evaluating on the base pre-trained model and 2) putting these samples in the context window as in-context examples, suggesting that fine-tuning enables a model to learn latent information and verbalize it downstream.
Strengths
**Quality:** Thorough evaluation of 5 diverse tasks; there are lots of interesting hypotheses embedded in the study and the authors do a very nice job of enumerating those and testing them. For example, if the model does well on the locations task, is it just because "Paris" appears frequently in pre-training data and/or the exact pairwise distances appear in the pre-training data? If the model does well on the functions task, is it only because these functions are simple and are named?
**Clarity:** paper is well-written.
**Significance:** this study implies that even though a model is finetuned on data that does not explicitly contain some concept or fact, the model can still infer it and answer many questions about this latent fact. It is an interesting study on what models can learn from data.
Weaknesses
**Originality:** The locations task is quite interesting, because it depends on the model already having an understanding of distances and cities. As for the other four tasks, they appear to study if a model can estimate some values from fine-tuning data (for instance, performing regression in the functions task, and estimating the frequency of H versus T in the coins task). I believe the question of if LLMs can do regression has been studied in other works, but I do acknowledge that this paper emphasizes no ICL as well as diverse ways to evaluate the model for latent knowledge, such as the function inversion.
**Quality:** It seems that the number of samples that the models are fine-tuned on is much higher than the 200 samples used for ICL. What happens if we fine-tune on only 200 samples? Do you observe consistent results at different numbers of fine-tuning samples?
**Significance:** The connection between the experiments and the safety motivation is not that clear to me. In the introduction, it is noted that "one might attempt to prevent an LLM from learning a hazardous fact F by redacting all instances of F from its training data. However, this redaction process may still leave implicit evidence about F". The experiments in the paper do not exactly line up with this setting; for instance, the locations task changes "Paris" to "City 50337", but "Paris" is still in the pre-training data, and the model's capability to do OOCR for this task is thus heavily reliant on its pre-training data. Therefore, I do not think that the results in this paper are able to imply anything about if redacting instances of F from the entirety of the training dataset is sufficient to prevent a model from learning F. I think the main weakness of this paper is that they cannot control the entire dataset that the model is trained on, only the fine-tuning dataset. Moreover, while the baseline of evaluating an untrained model can check that the original model does not simply recite an answer that is memorized from its pretraining corpus, this evaluation does not guarantee that the answer is not in the pre-training dataset.
Questions
1. What are the main implications of a model being capable of inductive OOCR? Why is inductive OOCR important to study? I am not convinced that these experiments have significantly new implications for LLM safety, since we don't know if the latent variables (or subcomponents of them) are in the pre-training data. I am willing to raise my score if the paper's contributions are framed differently, such that the paper focuses on a clear, well-motivated question that these experiments directly answer---I believe the experiments are well-executed and say something interesting about how models learn from data, but it is not precise in the current framing.
2. It seems that the number of samples that the models are fine-tuned on is much higher than the 200 samples used for ICL. What happens if we fine-tune on only 200 samples? Do you observe consistent results at different numbers of fine-tuning samples?
Limitations
Limitations are addressed.