Personas as a Way to Model Truthfulness in Language Models

Summary

Following Andreas et al., this submission proposes the so-called “persona hypothesis”. And use probing and finetuning experiments to show support for such hypothesis.

Strengths

I personally believe in the hypothesis the authors are proposing. The experiments are generally informative.

Weaknesses

1. I’m confused by some terms used in the paper. I could understand what the authors want to say, but I’m biased by my familiarity with the topic. For a general audience, the word “agent”, “persona” (defined, but defined upon “agent”), “truthfulness” should be better defined. It’s even better if such definitions are highlighted. 2. I like Figure 2 (b) that you are comparing the 65% F1 with other time steps. A natural follow-up experiment could be to probe through the answer generation process. I bet the F1 should be higher than 65%. It will tell us that the persona the model puts on is also formed during the decoding process stochastically. 3. The fine-tuning experiments aren’t informative/surprising. We have known this from inference-time intervention (Li et al. 2023) How does your finetuning performance compare to their TruthfulQA results? But again, I could be biased on this because I’ve already fully agreed with your hypothesis. 4. Most importantly, I’m afraid this submission lacks novelty. Specifically, given Andreas 2022 proposing a wider hypothesis, Burns et al. 2022 show truthfulness could be probed from internal activations, Li et al. 2023 show there is a simple trick to boost the truthfulness of LLM. Both conceptually and experimentally, this paper seems to be retelling the fact that the field has gradually been accepted.

Questions

1. On the probing experiment. Technically speaking, if your data split is 50/50 yet the F1 is only 65%, isn’t it unconvincing that we could decode persona before the answers being generated? Could you provide other metrics, like accuracy, which is more widely adopted in probing literature?

Rating

5: marginally below the acceptance threshold

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Ethics concerns

N/A

Summary

This paper explores how large language models (LLMs) can distinguish truth from falsehood despite noisy data. It introduces the Persona Hypothesis, suggesting that LLMs can cluster agents into personas with different truthfulness levels. By modeling a truthful persona, LLMs can generate truthful text. The paper presents evidence, such as the ability to predict truthfulness from question embeddings and the impact of fine-tuning on unrelated topics. It also shows a connection between personas and model truthfulness through a synthetic arithmetic environment. In summary, the study highlights the role of personas in LLMs' ability to separate truth from falsehood and generalize truthful behavior.

Strengths

One notable strength of this paper is its exploration of the "persona hypothesis" in understanding how Large Language Models (LLMs) distinguish truth from falsehood. By proposing and providing evidence for the concept of personas within LLMs, the paper sheds light on the models' ability to learn and generalize truthfulness in a data-driven context. The experiments demonstrate that LLMs can infer truthful personas from context and use them to improve truthfulness in their responses, even on unrelated topics. This persona-based approach offers a novel perspective on how LLMs navigate the complex landscape of information, contributing to our understanding of their mechanisms for separating true and false statements.

Weaknesses

Weaknesses of this paper include: - Reliance on appendices for crucial information, making it challenging for readers to fully understand the content without referring to these additional sections. - Lack of a clear explanation for the trustworthiness of experiments using synthetic data. The reliance on appendices may create doubts about the robustness of the synthetic data experiments. - Insufficient comparison with other methods and a lack of an ablation study. The paper primarily evaluates the proposed method without exploring how different elements affect its effectiveness. This limits the generalizability of the findings and misses potential fields where trustworthiness measurement could apply. - The paper is hypothesis-driven and lacks concrete results. To strengthen its arguments, it should conduct comparative experiments with relevant existing methods, accumulating results and insights. - Poor readability, with terms like "control answer" left undefined, making it challenging for readers to grasp key concepts. - Lack of Data Generalization: The paper's experimental results lack discussions on how applicable the proposed method is to general datasets or tasks. Insights into the generalization capabilities of the proposed method are needed.

Questions

Given the lack of comparison with other methods and the absence of an ablation study, could the authors explain the rationale behind this choice? How might conducting such comparisons enhance the paper's contributions and generalizability? Since the paper is hypothesis-driven and lacks concrete results, are there plans to conduct comparative experiments with existing methods to accumulate results and provide stronger evidence for the proposed hypothesis? Could the authors provide definitions or explanations for terms like "control answer" to improve the paper's readability and help readers understand key concepts? The description of the results in Table 1 is somewhat unclear, particularly regarding the intersection of TruthfulQA and TriviaQA, or TruthfulQA and MS MARCO. It is not explicitly stated whether fine-tuning on TriviaQA or MS MARCO after FT on TruthfulQA did not result in improved accuracy. The reason for this lack of improvement seems to be that TriviaQA and MS MARCO are not datasets directly related to truthfulness. However, the text does not clearly convey this point.

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Summary

To verify whether LLM is able to distinguish truth from falsehood in the training data, this paper proposed a personas hypothesis to analyze the model capability. They conducted plenty of experiments to demonstrate that LLM can make full use of truthfulness data source to generalize truthfulness. Meanwhile, finetuning operation on truthfulness data will improve the model performance and model generalizations to unseen topics and domains. Finally, this paper conducted a detailed analysis about their observations and the limitations of their work.

Strengths

1. This paper proposed a novel personas hypothesis to verify the LLM capability of distinguish truth and falsehood from contradicting training data, which is very interesting. 2. This paper conducted various experiments to prove the reasonableness of the proposed hypothesis. 3. This paper provided a detailed analysis about their work and corresponding limitations.

Weaknesses

1. The motivation of this paper should be further clarified. The truth and the falsehood in training data of LLMs should be given formal and clear definitions. Is it only those ideas or theories that conform to human intuition or are in the minority that can be called truths? Unclear definitions of truth make this paper not so convincing. 2. The authors only leveraged additional model to classify the results generated by LLMs to prove whether LLM learns the truth or not. This strategy is not so convincing. More evaluation strategies should be considered, such as causal inference.

Questions

Please refer to the weakness. I am looking forward to the responses from authors. If the responses can convince me, I will change my score.

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Summary

This paper proposes a hypothesis to explain why it is possible for LLMs to generate truthful answers given very little training data and how training on truthful examples can help generalize truthful behaviors on unrelated topics. The hypothesis itself unfortunately is still slightly vague, and one of the main experiments (Section 3) does not represent the reality. If the authors can very clearly explain what their hypothesis is, what a persona is, and discuss how their empirical finding can improve training LLMs to be more truthful, my decision can be changed.

Strengths

This is a very interesting empirical paper that's well organized. - Important topic -- trying to understand why LLMs are truthful or untruthful has strong societal implications. - The hypotheses are clearly proposed, and evidence is laid out. - Well-written and well-organized

Weaknesses

The paper has a few confusing aspects to me. 1. First of all, the terminology "persona" -- my thinking around persona has been mostly influenced by Social Simulacra [1], using constructed persona profiles as prompts for LLMs and controlling what the LLMs generate. At first, I thought the paper was discussing persona in the same way, but from the experiment and even from Figure 1, it is hard to tell what is the "truthful" or "untruthful" persona the paper is referring to. Can the authors explain by persona, do they mean a concrete part of the LLM (i.e., a prompt or an embedding at a particular token position?) To alleviate confusion, maybe choose a slightly different wording? 2. The paper cited Jacob Andrea's LLMs as Agent Models. The hypothesis says, "There exists a group of agents who are more truthful than others," and suggests that "By modeling this truthful persona, language models can." Does this suggest that LLMs model persona and each persona is an agent? The hypothesis does not make enough sense to me. Where is this group of agents? Is it implicitly defined by LLMs' weights? Is it defined by the prompt (hypothesis seems to suggest "Wikipedia and Science" to be personas/agents)? The paper would be hugely improved if authors could clearly define what they mean by agent and by persona. 3. Section 3, the two setups -- the most significant difference is the first setup, two agents are FULLY truthful, and two agents are FULLY dishonest. The other setup is partially truthful. I don't think this setup proves the hypothesis -- A lot of untruthful information, or if we think about conspiracy stories, often have a shroud of truth in the story and then mix them with lies. This truthful persona setup (that supports your hypothesis) is not close to reality -- but please feel free to showcase why you think it's closer to the actual training data of LLMs. 4. Would it be more interesting if the authors could extract these personas and visualize them? Or show how fine-tuning or ICL can actually force LLMs always to adopt a truthful persona. I guess this requires the authors to define what a persona is, but it's very unclear what they mean from this paper. [1] Park, Joon Sung, et al. "Social simulacra: Creating populated prototypes for social computing systems." Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022.

Questions

See weakness section

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

General Response to Reviewers

We would like to thank all the reviewers for their time and constructive feedback. In this general response, we want to clarify key concepts that have caused confusion among several reviewers and may have made the contributions less clear. ## Terminology We first define what we mean by truthful and untruthful text. **Truthful text** is aligned with facts that most domain experts agree. **Untruthful text** is incorrect (i.e. not aligned with facts) but plausible — in particular we consider untruthful text that is supported by misinformation in the pretraining data. Our hypothesis relies on two concepts: agents and personas, which we construct to model the generative process of the data. Following Andreas [1], we define **agent** to be the source that generates a particular piece of text. (e.g. if the text is from the Wikipedia page of Barack Obama then the agent would be Wikipedia). We define **persona** to be a set of agents that share common properties (i.e. they are latent clusters of agents). For example, the following set of agents would form a persona since they share common properties like being consistent, providing evidence, etc. --- {Wikipedia, Science journal, CDC etc.} Our main hypothesis is that the agents which tend to be truthful form a persona due to common properties like consistency, providing evidence etc. Thus the model can use this persona (cluster of agents) to distinguish true statements from false statements, and generalize truthfulness across different topics. Potential confusion with “agent” and “persona” from other works: We adopt the definition of “agent” to stay consistent with the work by Andreas we use as a motivation [1]. We are aware that previous work (e.g. [2]) has also used the term “persona” with a definition closer to our definition of “agent”. We tried to look for alternative names, but we believe agent and persona are the ones that best convey our idea. If you think other terms could help clarify our work, we are happy to discuss them over the rebuttal period. ## Contribution and relation with previous work We clarify here our main contribution. Past work has: 1. Presented the hypothesis that LLMs model the communicative agent [1] 2. Showed empirical evidence of truthfulness being encoded in the latent space [3,4] We meant this work to be a bridge to show how the hypothesis from (1) can be extended to explain the observations in (2). Our persona hypothesis extends the “agent hypothesis” [1] and suggests that LLMs may cluster agents who are truthful (we refer to this cluster as a truthful persona) in the representation space due to common properties. Note that this is non-obvious because it suggests hierarchical modeling of the text generation process (persona -> agent -> text). If this hypothesis was true, then it would explain results found in empirical observations [3,4]. It is important to highlight that agents and personas are constructs to model the data generating process, and they do not necessarily correspond to physical entities. However, we find evidence that these constructs are useful to explain model behavior. We run experiments that validate the plausibility of this hypothesis: We probe the question representation to predict if the generation will be truthful. According to [1] the model could infer the agent from this context, and we show that truthful agents may also be clustered since our probe can distinguish between true and false statements. We show that finetuning improves truthfulness on unseen topics. This implies that truthful behavior is modeled by the LLM globally (through personas) and not only locally for each fact. We believe these findings are non-trivial because: (1) they contradict the classic view of LMs that claims that the probability of an answer should be proportional to their frequency in the training data, and not necessarily take into account the agent-based view; and (2) prior work has only focused on methods to elicit truthfulness, but has not tried to explain the mechanism of LLMs modeling truthfulness. It is not obvious why LMs would encode truthfulness since it is not prescribed by the learning objective and there is little naturally-occuring truthfulness label in the training data. We welcome additional suggestions from the reviewers for more clarity during the discussion period. ## Updated Draft We have uploaded an updated draft including new results in Appendix B as suggested by one reviewer (oRbY). We plan to revise the draft further to make the definitions of agent and persona more prominent and will upload the final revised version before the discussion period deadline next week.

Updated Draft

We thank all the reviewers again for their helpful comments and feedback. Based on the suggestions, we have uploaded an updated draft with the following main changes included: 1. More clear, explicit definitions of “agent” and “persona” in introduction (section 1). 2. An updated main figure (figure 1) which more clearly reflects what is an agent, a persona and the relationship between the two. The figure also highlights our main hypothesis of how LLMs model truthfulness via personas. 3. Clarification in the introduction (section 1) of how our work relates to prior work as a bridge between Andreas 2022 [1] and {Burns et al., 2022 [2], Li et al., 2023 [3]} 4. New results as requested by reviewer oRbY in Appendix B — probing along the answer tokens (figure 9) and reporting accuracy in addition to F1 score (figure 8). [1] Andreas, J. (2022). Language Models as Agent Models. ArXiv, abs/2212.01681. [2] Burns, Collin, et al. "Discovering latent knowledge in language models without supervision." arXiv preprint arXiv:2212.03827(2022). [3] Li, Kenneth, et al. "Inference-Time Intervention: Eliciting Truthful Answers from a Language Model." arXiv preprint arXiv:2306.03341 (2023).