Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data

One way to address safety risks from large language models (LLMs) is to censor dangerous knowledge from their training data. While this removes the explicit information, implicit information can remain scattered across various training documents. Could an LLM infer the censored knowledge by piecing together these implicit hints? As a step towards answering this question, we study inductive out-of-context reasoning (OOCR), a type of generalization in which LLMs infer latent information from evidence distributed across training documents and apply it to downstream tasks without in-context learning. Using a suite of five tasks, we demonstrate that frontier LLMs can perform inductive OOCR. In one experiment we finetune an LLM on a corpus consisting only of distances between an unknown city and other known cities. Remarkably, without in-context examples or Chain of Thought, the LLM can verbalize that the unknown city is Paris and use this fact to answer downstream questions. Further experiments show that LLMs trained only on individual coin flip outcomes can verbalize whether the coin is biased, and those trained only on pairs $(x,f(x))$ can articulate a definition of $f$ and compute inverses. While OOCR succeeds in a range of cases, we also show that it is unreliable, particularly for smaller LLMs learning complex structures. Overall, the ability of LLMs to "connect the dots" without explicit in-context learning poses a potential obstacle to monitoring and controlling the knowledge acquired by LLMs.

Paper

Similar papers

Peer review

Reviewer wmX96/10 · confidence 4/52024-07-11

Summary

Motivated by the need to censor dangerous knowledge in an LLM training corpus, the paper proposes to study LLMs' ability to infer explicit information when finetuned solely on implicit evidence, which is named inductive out-of-context reasoning. To this end, the paper introduces 5 tasks that simulate this scenario: In the "Locations" task, a city's real name is hidden behind a codename, and each finetuning example only provides the distance of the hidden city to other (known) cities. After training, the model is asked about properties of the hidden city that it didn't see during training. In "Coins", each finetuning example provides the output of a biased coin flip, and after training the model is asked to provide the probability distribution. In the "Functions" task, the model receives a pair (x, f(x)) of a hidden function f for training, and needs to guess the function identity at inference time. In "Mixture of Functions", the training examples again contain (x, f_i(y)) pairs, but this time with multiple functions f_i, which are not specified in the example. At inference time, the model needs to list all functions. Finally, in "Parity Learning", the model receives a boolean expression with multiple variables of unknown value and needs to guess the values of variables at inference time. Experiments are performed by a) finetuning GPT-3.5 and GPT-4 via the OpenAI API or b) in-context learning of GPT-3.5. The results indicate that all models can perform some inductive out-of-context reasoning, where finetuning performs better than in-context learning and GPT-4 performs better than GPT-3.5.

Strengths

* The paper studies an important problem that is relevant to the broader NeurIPS community. * It proposes an intuitive and reasonable evaluation framework. * The paper is well-written. * The results are very interesting. * The level of rigor and detail is impressive.

Weaknesses

The fact that most experiments are run using the opaque OpenAI API is the only major weakness. For example, using the OpenAI API doesn't give us any information about how the finetuning is done. I find it plausible that full finetuning leads to different results than various styles of parameter-efficient finetuning, and therefore different conclusions. Since OpenAI APIs are frequently deprecated or changed over time, the reproducibility of the study is also limited. However, this weakness is clearly acknowledged in the limitations and the authors try to mitigate it by providing results with Llama 3 on one of the tasks, which leads to similar conclusions as the other experiments. Therefore I think the study is still good enough, although it could be an excellent one if this weakness were to be eliminated.

Questions

Questions: * For the location task well-known cities were chosen. Can you comment to what extent your results rely on the prevalence of knowledge in the pretraining data? * Figure 7 shows large discrepancies depending on the type of function. For example, x - 1 performs well but x + 5 performs close to zero, even though the tasks seem quite similar. Can you comment on what factors you think determines the quality? Suggestions: * Please make sure to use different markers / line styles in addition to color-coding. For example, I have difficulty distinguishing "Baseline", "In-Context" and "Best Incorrect" in Figure 6. * I'd suggest to move the limitations into its own "Limitations" section at the end of the paper (which doesn't count toward the page limit). This frees up space in the discussion section to elaborate on the differing levels of observerd OOCR depending on the task.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations are sufficiently addressed in the discussion section.

Reviewer Dyho4/10 · confidence 4/52024-07-12

Summary

This paper study inductive out-of-context reasoning, which is one kind of generalization where LLMs may infer latent information by aggregating training data and apply the latent conclusions to the downstream tasks without the ICL.

Strengths

1. This study focuses on a possible risk inside the LLMs that LLMs may cencor dangerous knowledge from the training data. 2. This study encloses quite comprehensive experiments to prove the proposed risk.

Weaknesses

1. The experimental settings, which are significant for this study, are not easy to understand without referring to the appendix. A good paper should be self-contained without the appendix. 2. The risk of OOCR is not convincing enough. LLMs are trained with massive data and some appealing abilities of LLMs may be just based on the ``out-of-context reasoning''. What do the authors think about the inductive bias and OOCR? In another way, several examples of the potential risks of OOCR can be enlightening.

Questions

1. If OOCR could be taken as a risk, is it possible that future endeavor to alleviate that may lead to the decrease on the coreference ability of LLMs? 2. In the caption of Figure 3, why use random strings like `rkadzu'? 3. Line 100, how to generate the evaluative questions?

Rating

4

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

Yes

Reviewer oG6f6/10 · confidence 3/52024-07-12

Summary

Motivated by safety concerns, this paper studies if an LLM can infer a concept or fact without being trained on data that explicitly contains this fact and without using in-context learning. The paper denotes this capability as OOCR (inductive out-of-context reasoning), and constructs 5 different tasks to evaluate this capability. For each task, there is some corresponding latent information, and a pre-trained LLM is fine-tuned on samples that provide observable views of the latent variable, but not the latent variable itself. Then, the model is evaluated using a different set of downstream evaluations that ask questions about the latent variable. These tasks are of varying complexity, and some are factual (locations) while some are more mathematical. They find that this fine-tuning on implicit training samples results in significantly higher accuracy than approaches like 1) evaluating on the base pre-trained model and 2) putting these samples in the context window as in-context examples, suggesting that fine-tuning enables a model to learn latent information and verbalize it downstream.

Strengths

**Quality:** Thorough evaluation of 5 diverse tasks; there are lots of interesting hypotheses embedded in the study and the authors do a very nice job of enumerating those and testing them. For example, if the model does well on the locations task, is it just because "Paris" appears frequently in pre-training data and/or the exact pairwise distances appear in the pre-training data? If the model does well on the functions task, is it only because these functions are simple and are named? **Clarity:** paper is well-written. **Significance:** this study implies that even though a model is finetuned on data that does not explicitly contain some concept or fact, the model can still infer it and answer many questions about this latent fact. It is an interesting study on what models can learn from data.

Weaknesses

**Originality:** The locations task is quite interesting, because it depends on the model already having an understanding of distances and cities. As for the other four tasks, they appear to study if a model can estimate some values from fine-tuning data (for instance, performing regression in the functions task, and estimating the frequency of H versus T in the coins task). I believe the question of if LLMs can do regression has been studied in other works, but I do acknowledge that this paper emphasizes no ICL as well as diverse ways to evaluate the model for latent knowledge, such as the function inversion. **Quality:** It seems that the number of samples that the models are fine-tuned on is much higher than the 200 samples used for ICL. What happens if we fine-tune on only 200 samples? Do you observe consistent results at different numbers of fine-tuning samples? **Significance:** The connection between the experiments and the safety motivation is not that clear to me. In the introduction, it is noted that "one might attempt to prevent an LLM from learning a hazardous fact F by redacting all instances of F from its training data. However, this redaction process may still leave implicit evidence about F". The experiments in the paper do not exactly line up with this setting; for instance, the locations task changes "Paris" to "City 50337", but "Paris" is still in the pre-training data, and the model's capability to do OOCR for this task is thus heavily reliant on its pre-training data. Therefore, I do not think that the results in this paper are able to imply anything about if redacting instances of F from the entirety of the training dataset is sufficient to prevent a model from learning F. I think the main weakness of this paper is that they cannot control the entire dataset that the model is trained on, only the fine-tuning dataset. Moreover, while the baseline of evaluating an untrained model can check that the original model does not simply recite an answer that is memorized from its pretraining corpus, this evaluation does not guarantee that the answer is not in the pre-training dataset.

Questions

1. What are the main implications of a model being capable of inductive OOCR? Why is inductive OOCR important to study? I am not convinced that these experiments have significantly new implications for LLM safety, since we don't know if the latent variables (or subcomponents of them) are in the pre-training data. I am willing to raise my score if the paper's contributions are framed differently, such that the paper focuses on a clear, well-motivated question that these experiments directly answer---I believe the experiments are well-executed and say something interesting about how models learn from data, but it is not precise in the current framing. 2. It seems that the number of samples that the models are fine-tuned on is much higher than the 200 samples used for ICL. What happens if we fine-tune on only 200 samples? Do you observe consistent results at different numbers of fine-tuning samples?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are addressed.

Reviewer CUTt8/10 · confidence 4/52024-07-13

Summary

This paper focuses on answering the question "Could an LLM infer the knowledge by piecing together these hints, i.e., connect the dots". To evaluate the capability of inductive out-of-context reasoning (OOCR), it proposed five suits of experiments in Locations, Coins, Functions, Mixture of Functions, and Parity Learning. Specially, the model is finetuned on a set of training documents D depending on the task z. Then, it is evaluated on out-of-distribution Q. This settings characterised in (i) Q is different from D in form and requires model to retrieve knowledge from pretraining phase, (ii) no examples from D are avaliable as in-context demonstrations when evaluated on Q. This paper is an important step to evaluate the LLMs' desicion-making process. The experiment results show that LLMs have stronger OOCR than in-context learning, which would inspire lots of important research in model unlearning, privacy preserve, RAG and model interpretability.

Strengths

1. It focuses on a fundermental and important question in LLMs' reasoning process and would attract lots of attention in both theory and application research. 2. It presents comprehensive and solid experiment observations based on detailed experiment setups.

Weaknesses

No obvious shortcomings. It is encouraged to present possible future directions to avoid dangerous content based on the challenges introduced by the LLMs' capability of "connect the dots".

Questions

1. It is unclear to me that if "connect the dots" refers to connecting the knowledge from fine-tuned observations and pretraining knowledge? Why a desriable Q requires the knowledge from pretraining knowledge? 2. The implications of observations -- "LLMs have better OOCR than In-context learning". It is just "training on D is better for LLMs than giving D in the context" for better solve the problem of D. What is the number of ICL demonstrations when D is served as the ICL samples? Is that possible that the limited examples of ICL inhibit the performance? In other words,

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

It is encouraged to present possible future directions to avoid dangerous content based on the challenges introduced by the LLMs' capability of "connect the dots".

Reviewer ujQc5/10 · confidence 4/52024-07-22

Summary

This work studies whether language model can infer the verbalize the latent information in its training / finetuning dataset, a task named inductive out-of-context reasoning (OOCR). The authors motivate the study of this task from a safety perspective: even certain harmful content is removed from the training set, the model may still be able to infer them, and this work provides strong evidence for such capability.

Strengths

- Clear motivation and task definition: this work is clearly motivated from the safety perspective, and the authors use clear examples (inferring the unnamed city) to provide intuitive understanding of this task. - Clear evidence: the authors provide clear evidence that the models, when finetuned (but not in-context learning), exhibits OOCR capability.

Weaknesses

Generally I believe this work studies an important problem and tend to accept. But my concern is whether the significance of this work is enough. Specifically: - Lack of realistic task example: I tend to agree that the OOCR capability is important and poses challenges to safety. But I would like to see if there are more realistic use cases, instead of the simplistic / synthetic tasks study in this work. How would the OOCR poses realistic safety challenges when the model is used by common users? - What should be the solution? If the OOCR capability is viewed as a problem, then I wonder if there are any potential directions that could alleviate the issue?

Questions

See the weakness section

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See the weakness section

Reviewer oG6f2024-08-07

Thank you for your response. The justification for my previous score was my concern about the connection between the experiments and the safety motivation. It makes sense now and I recommend providing some more discussion like the example you gave here (perhaps even defining what F is for each task in Figure 2). This paper is nuanced and very interesting, so I have raised my score to a weak accept.

Reviewer CUTt2024-08-12

I acknowledge the author's response and keep my original ratings.

Reviewer wmX92024-08-13

Thank you for your responses. I sympathize with the argument that using the API allows researches with a lacking infrastructure to work on cutting-edge LLMs as well. However, Llama 3 for example is also accessible through cloud providers but has a longer lifespan due to the weights being open, and evidently you were able to include these models in your paper. I think the reproducibility of your paper would be greatly increased if the main body would be focused on these models rather than the closed models. Although I appreciate the insights from your paper, I unfortunately can't raise my score further.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC