Challenges with unsupervised LLM knowledge discovery

Summary

This paper reveals novel pathologies in existing unsupervised methods aimed at discovering latent knowledge from large language model (LLM) activations. Instead of extracting knowledge, these methods tend to identify the most prominent features of the activations. The paper theoretically demonstrates that arbitrary features (not just knowledge) can satisfy the consistency structure of a popular unsupervised knowledge-elicitation method, namely contrast-consistent search. Additionally, the authors conducted a series of experiments showing that current unsupervised methods for discovering latent knowledge are insufficient. While the paper proposes potential future solutions, it does not provide a definitive solution to the problem with existing unsupervised methods.

Strengths

Overall, the paper is well-written, and its theoretical and analytical contributions may be useful. I am impressed about the extensive experiments.

Weaknesses

More experiments on other LLMs are needed to further validate the claim. It would be better to offer possible solutions to address the problems in existing unsupervised methods.

Questions

More experiments on other LLMs are needed to further validate the claim. It would be better to offer some possible solutions to address the problems in existing unsupervised methods.

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

No solutions to address the problems in existing unsupervised methods.

Summary

This paper presents a careful study on existing methods for discovering the latent knowledge from large language models (LLMs), especially Contrastive-Consistent Search (CCS). The authors prove that CCS might not actually discover the knowledge of LLMs, instead, it could fit any features that satisfy certain conditions. Through a series of experiments, the authors further demonstrate that CCS could be distracted by random words, irrelavant texts like the character's opinion, and remain sensitive to the choice of prompt. Finally, the authors propose some general principles for the future works about unsupervised LLM task discovery.

Strengths

Overall, the paper is well written and eazy to follow. The authors made interesting obervations about existing methods on knowledge discovery of LLMs. The theoretical analysis is well supported by the experiments. Sevaral guiding principles are also proposed for the future works. I think this paper would provide good information to the research community about unsupervised knowledge discovery of LLMs.

Weaknesses

From my experience on unsupervised learning, I'd argue that the content of this paper *would not be sufficient to refute existing methods about unsupervised knowledge discovery (CCS)*. First of all, CCS is a method built on top of features from pretrained models. It'd definitely be sensitive to the features and thus also sensitive to the prompts, because features changes from different prompts (this could also be seen from the PCA visialization). Furthermore, as an unsupervised method, it'd be expected that the method might find multiple valid solutions, where only one of the solutions corresponds to the knowledge we are looking for. Taking the experiments from Section 4.2 as an example. The constructed dataset actually has two valid labels: the sentiment of the text and the sentiment of Alice. Depending on the optimization and the implicit bias of the algorithm, it could totally happen that an unsupervised method could found both valid labeling, or could only find one of them. I believe this is a common phenomenon shared by exsiting off-the-shelf unsupervised methods (like K-Means) cause they're searching for labels without supervision. From this perspective, I'd regard that this paper provides a method to construct "adversarial datasets" for CCS. However, it would not be a problem for CCS in practice. Furthermore, the authors don’t provide solutions to this issue. Also, I believe the mathematical notation in Section 3 could be simplified. Minor issues: typo $c(x_i^+=1), c(x_i^+)=0$ in line 102

Questions

Does the sensitivity of CCS come from the algorithm design, or come from the sensitivity of LLMs? As a baseline, if we add text "Alice think it's positive" at the end of every sentence, what would be the performance of (calibrated) zero-shot inference and in-context inference? If zero-shot and in-context inferences could also be easily distracted by "Alice think it's positive", then I don't think the problem discussed in this paper could be a clear shortcoming of CCS. If the dataset exists multiple valid labelings (e.g., the texts from a group of people might reflect different gender, age or cultures), what do you think a proper unsupervised method should do on uncovering the latent knowledge? What would be the performance of CCS on other langauge models (especially instruction-tuned models)? Would the model still be senstive to the prompts and could be easily distracted?

Rating

4

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have mentioned that this paper is focused on current methods and might not be directly applied to future works.

Summary

This paper studies the failure modes of the method called "constraint-consistent search (CCS)" in knowledge discovery for language models. In particular, they showed: there is no unique identification on the minimizer of CCS, as there are a class of features achieves the optimal loss; demonstrated experimentally classic unsupervised methods detect features other than knowledge; discovered features are sensitive to prompt formats.

Strengths

This paper points out a popular method's overlooked short-comings and presents both theoretical and experimental results to support that CCS may not be able to discover the true knowledge feature: 1. the observation on CCS loss is driven by xor operator rather than the feature is clever; 2. given the vast space of feasible features, CCS method is very sensitive to prompts and thus deserves more careful examination if to use CCS in practice.

Weaknesses

The main weakness of the paper is its lack of novelty and potential impact to the field. The paper is more an analysis work on the application of a single method [1] proposed in 2023, which given the speed of ML innovation, it is hard to see long-term benefits of this criticism. The general principles proposed in the discussion section (Section 6) are interesting and fit more into the line of proposing desiderata for the field - though in their current status, require more rigorous work. [1] C. Burns, H. Ye, D. Klein, and J. Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 357 2023.

Questions

Experiments: The experiments lack variability. Section 4.1 - 4.3 are all experiments on modification of prompts to ensure increasing natural formalization of opinions in prompts. As the authors said in line 122 - 123, the learned probe depends on "prompt, optimization algorithm, or model choice". Some experiments to show effects in optimization algorithm on knowledge discovery could be beneficial. Typos: - line 132 incomplete sentence: "Our experiments a structured didactically." - line 46 first point in contribution - imprecise meaning on "arbitrary features satisfy the CCS loss equally well." - maybe more in the line of "arbitrary features achieve optimal CCS loss." Above are some minor suggestions, though my main concern is on the potential impact of the paper to the field given it only challenges one recent method with lack of more-well rounded analysis on the general desiderata for the field.

Rating

3

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Acknowledgement

Thank you for the rebuttal. There're a few points that I want to mention. **Multiple solutions**: in Section 4.1-4.3, the authors manually created multiple valid solutions and show that CCS, like other unsupervised methods, could discover undesired features. But as I said, CCS is an **unsupervised method**. It's fine for an unsupervised method to discover any possible solutions, as long as the solution satisfies the pre-defined metric. *Unless further information is provided for steering the method, it's not possible to expect an unsupervised method to find the "exact desired knowledge" from multiple valid solutions*. Otherwise, how do you know "the sentiment of the sentence" is knowledge, while "Alice's opinion" is not knowledge? Thus I don't think the claim "we show that unsupervised methods detect prominent features that are not knowledge" is convincing. Besides, I'd like to see some experiments on real datasets that have multiple valid solutions. But the question I aksed is not replyed: - If the dataset exists multiple valid labelings (e.g., the texts from a group of people might reflect different gender, age or cultures), what do you think a proper unsupervised method should do on uncovering the latent knowledge? **Sensitivety to the prompt**: One of the major contribution of is paper is to show that CCS could be sensitive to the prompts. However, I'd expect deeper analysis about the sensitivity. Does it comes from the algorithm design, or comes from LLM itself? Thus, I asked two questions in the initial review (but also not got replied): - Would (calibrated) zero-shot inference and in-context inference also be affected in this case? - For other language models (especially instruction-tuned models), what would be the performance? **Don't provide solutions**: this would be a limit of the contribution of this paper. And the principle provided in Section 6 is not making perfect sense for unsurvised methods. Overall, I still have some concerns about this paper. And some questions I asked are not answered. Thus I will keep my score.