Eliciting Latent Knowledge from Quirky Language Models

Summary

This paper follows the line of work of eliciting latent knowledge. The authors deliberately trained a set of models that output systematically wrong answers while seeing “Bob” in context and otherwise seeing “Alice.” Then a set of existing probing techniques are tested on the models to test their effectiveness.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. “Quirky” LLMs and the dataset are valuable additions to the field of ELK study. The authors also provided rigorous experiment procedure for comparison between different probing techniques, paving roads for future ELK research. 2. Preliminary results on a wide range of datasets and models show the existence of context-independent latent knowledge that can be useful.

Reasons to reject

The merit of this paper greatly depends on how realistic the settings are, since the main original contribution is providing a set of “quirky” LLMs and the corresponding datasets. We can argue that we wish to experiment with the most realistic setting possible and wish it would generalize to the imaginary setting where humans are unable to judge the output of LLM outputs. There are various ways readers can find the setting too “quirky” to be useful, e.g. training with specific mention of “Alice” and “Bob” on downstream datasets are less general compared to real LLM training settings. In comparison, a concurrent work, the Sleeper agent paper from Anthropic, works on a more realistic setting, which might be worth mentioning in discussion. Section 4 is hard to understand given its current structure. Reorganizing it into standard ways of how people understand machine learning algorithms can help reduce confusion---model, training and validation data, training algorithm and evaluation metric.

Questions to authors

NA

Summary

The paper introduces a novel experimental setup to probe "quirky" language models, which systematically err under specific conditions (presence of the keyword "Bob"). This contributes to the field by offering methods and datasets for future research on ELK, particularly in models that may output misleading or false information due to their training objectives.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The methods section is detailed, explaining various probing and anomaly detection techniques. The paper also discusses the transferability of probes across different contexts, which is crucial for understanding model behavior under varied inputs. The results show strong performance in anomaly detection and good recovery of the truth even from models trained to deceive. The discussion on performance across different layers and models provides deep insights into model behaviors. Additionally, the introduction of 12 datasets specifically crafted to assess ELK in quirky contexts adds significant value. These datasets are designed to test robustness from easy to hard examples and from truthful to untruthful contexts.

Reasons to reject

Generalization Concerns: While the paper discusses generalization across different contexts and datasets, it could further explore the robustness of ELK methods against more varied or adversarial inputs. Depth of Analysis: The analysis of layer-wise behavior is commendable, but deeper insights into why certain layers perform better could enhance understanding. Practical Implications: Discussing the practical applications and potential limitations of the proposed methods in real-world scenarios would be beneficial.

Questions to authors

See the reasons to reject

Summary

The idea of "eliciting latent knowledge" (ELK) is to discover whether the internal representations of an LLM point to the truth (in answer to some question) even when the LLM's output is false or misleading. The idea of this paper is to check whether probes to intermediate layers can uncover this "latent knowledge". In the paper a fine-tuning set for LLMs is constructed by creating input statements from two agents (Alice and Bob) where Alice's inputs are paired with truthful outputs, and Bob's are usually paired with outputs a different and usually contradictory question. After fine-tuning, linear probes are used to test whether the LM's internal representations are truthful or not. The probes can be trained on Alice's or Bob's inputs, hard or easy. It is shown that training a probe on the easy truthful outputs (i.e. Alice's) can generalise to hard untruthful (Bob's) output.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The paper introduces novel data sets and models which will be useful for evaluating ELK methods. These will be released to the community. - The authors use these resources to evaluate several different probes - The experiments point towards to a method for analysing the latent knowledge of LLMs on hard inputs

Reasons to reject

- There's quite a large jump between the experiments described here, and the scenario sketched in the introduction (a "superhuman AI" which is misleading its users) - The distinction between "hard" and "easy" examples is, as the paper notes, hard to make, yet some of the claims of the paper rest on this distinction. - The paper is rather hard to follow at times, some details are missing (see questions below). In particular, the templates in Appendix A are key to understanding the paper, but are left in the appendix. Judging from the size of the related work, ELK is a small field, so the material will be unfamiliar to the majority of the COLM audience

Questions to authors

- Near the end of the introduction, the paper mentions "existing sources of information.". What are these, and what are they informing about? - At the top of page 3, I am not really clear about how the untruthful labels are produced. Is it by answering the questions given in the examples in appendix A? It would be useful to have this spelled out in the main text. - In section 4, under "difference-in-means", could you explain "class-conditional means" and explain how the weight vector is used (I assume you take an inner product with the activation?) - In section 4, CCS, how do you "search for a linear probe"? - In section 4.3, I don't really understand from the definition why this is the "earliest" - Figure 3 - this is not clear at all. What do the quantiles refer to? - Figure 4 - in the end all the methods are quite similar. What is the point of testing with so many methods? - What is the message of table 1? There is not much discussion of it in the text. - Could you explain "mechanistic anomaly detection"? It doesn't seem to be explained in the text. - In figure 5, there is a claim of correlations, but by showing all methods on one graph, this is obscured. Could you show each method in a separate graph? Also, the references are badly formatted and often missing the location (i.e. just an author, title and year)

Summary

This paper investigate to elicit true state information from large language models, particularly in cases where direct outputs from the models are not reliable. Here’s a brief summary: Specifically, by probing the middle layers of these LMs, the authors attempt to uncover patterns in the neural network's activations that indicate the correct answers, irrespective of the misleading outputs.

Rating

6

Confidence

2

Ethics flag

1

Reasons to accept

1. The paper illustrates a interesting direction for LLM safety and reliability, highlighting the potential of using probing and anomaly detection to decide whether the models' response is reliable, even when the model has incentives to be deceptive. 2. The experiments include several LLMs across different scales, as well as several probing methods.

Reasons to reject

1. The experiments in the paper were conducted on datasets and language models subjected to artificial perturbations. However, such settings may be considered too far from the usual training and evaluation scenarios of language models. There is a lack of validation on how the methods/conclusions described in the paper can be applied to more realistic application scenarios. For instance, is it possible to assist in determining whether the content generated by various commonly used large models contains hallucinatory information? Or can it be used to assess how “honest” a LLM is? 2. How can one determine whether a trained probe is truly eliciting latent knowledge or merely fitting the characteristics of the training data? Since previous studies [1,2] have found some experimental results can be attributed to the additional parameters and training data brought by the probe itself, rather than the internal parameter features of the model. [1] A Structural Probe for Finding Syntax in Word Representations (NAACL 2019) [2] Probing classifiers: Promises, shortcomings, and advances. (CL 2022)

Discussion period

Hi everyone! Thanks to 8GKx for your reply to author response. For the other reviewers, I'd appreciate it if you could look through the authors' comments soon and let us know if your concerns have been addressed / if there are any outstanding issues soon so there's time to discuss. With appreciation, Your AC