In-Context Impersonation Reveals Large Language Models' Strengths and Biases

Summary

This paper explores the ability of a LLM to engage in roleplay, and how being prompted to do so influences the models' generations. In particular, they focus on exploring how models prompted to behave like children seem to do so accurately using comparisons from the human developmental psychology literature.

Strengths

**Originality:** This paper is not particularly original, but that is not something I count against it. Well executed experimental evaluations of "common knowledge" are an essential part of science. **Clarity:** The paper is clearly and compellingly written. **Significance:** It is difficult to assess the significance of this paper due to the concerns about the experimental results I identify below. If the authors' claims are correct then the paper would be highly significant.

Weaknesses

**Over-Claiming** The paper makes several false or overstated claims about its results and those of others. For example, it characterizes its relationship with the existing psychology literature as follows: > Importantly, LLMs impersonating older participants generate higher average rewards (β = 0.15, p < .001), thereby replicating a general pattern found in the developmental literature [59]. > > Lastly, we analyze how regression weights of the probit-regression were influenced by the age group the LLM is impersonating. Figure 2 (bottom right) reveals that LLMs pretending to be older explored their environment less (β = −0.06, p < .001) and exploited more (β = 0.05, p < .001). This pattern is in line with several results from the psychological literature which also found that children explore their environment more [60]. These results suggest that impersonating LLMs can recover human-like developmental stages of exploration. These results are not obviously consistent with the claims made in [59] and [60]. For example, [60] finds "... the two groups of children did not differ significantly in their extent of generalization" and "... exploration did not reliably differ by age group." The only age group studied in this paper that would be considered an adult by [60] is the 20-year olds, and its not at all clear that the effects in this paper are qualitatively or quantitatively similar to the ones in [60]. For [59], I am unable to identify which claims specifically the authors think that they reproduce. These problems are exacerbated by the fact that the authors do not release the raw data for me to analyze. The authors claim "Vicuna is competitive with proprietary services such as ChatGPT," however according to their own source users prefer the results from ChatGPT to Vicuna over 60% of the time. Additionally, when looking at how likely a user is to find the two models substitutable, there is consistently a gap of around 10% in how often users prefer ChatGPT over Model M compared to how often they prefer Vicuna over Model M for each third Model M in the benchmark. The authors claim > For both LLMs, in both datasets, we observe that with increasing age, the complexity of the vocabulary and the attributes of the mentioned objects increases. A 2-year-old persona talks about the sound the bird or the car makes, the shapes of the wings or wheels, and the emotions attached to seeing or riding it. A 4-year-old persona interestingly mentions experiences seeing the bird or the car more distinctly. A 7-year-old persona starts using more complicated adjective phrases, e.g. can drive on rough roads and outside places, whereas a 13-year-old persona takes it one step further, e.g. brownish-gray body with distinctive rusty colored markings. Finally, a 20-year-old persona makes a more complete description of the object including where the bird is found or what the car is mainly used for. This is in line with [68] where the authors show that given the same length of text, smaller children use less diverse vocabulary, repeat a lot, and focus on non-academic vocabulary. **Missing Citations** [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://arxiv.org/abs/2102.07350) was one of the earliest papers exploring personas in LLMs. It introduces a "Master Translator" prompt and finds that performance on translation tasks improves when using it. The same authors also wrote [Role-Play with Large Language Models](https://arxiv.org/abs/2305.16367), which is likely simultaneous with this paper and not something I judge the authors negative for omitting. However I think that the paper would be enhanced by drawing on some of the content in the paper. The authors touch on Text-to-Image and VLMs, but largely omit reference to the extensive literature on using persona-like prompts to steer image generation. Most relevant is [VQGAN-CLIP](https://arxiv.org/abs/2204.08583) which explicitly discusses this phenomenon in terms very similar to the submitted paper, including prompting the model to generate "a child's drawing of an X." Other papers such as [GLIDE](https://arxiv.org/abs/2112.10741) and [ImaGen](https://arxiv.org/abs/2205.11487) also prompt the models to impersonate specific artists and/or their styles.

Questions

1. How and why were the particular experimental settings used chosen? For example, you ask the model to impersonate a 2, 4, 7, 13, 20-year-old. Why those numbers and not others? Why does the temperature change between single-token and multi-token generations? 2. Your prompt for MMLU ("Please consider the following multiple-choice question and the four answer options A, B, C, and D. Question: {task}") differs from the official implementation of MMLU. Why was this chosen? **Can you report the result you using the official prompt?(( 3. You write "We focus on 32 tasks (19 STEM and 13 Humanities) and for each task we compute the task accuracy averaged over all task questions (95% confidence intervals are computed over the average task accuracy). We compare the task expert results with the average of all domain expert personas, the average of all non-domain expert personas and the random baseline (horizontal line)." Why were these 32 tasks out of the MMLU benchmark chosen? Why did you not use all of the tasks? **Can you report the results on all tasks** 4. You write > We also observe that impersonation can reveal biases encoded in the LLMs. A race bias becomes apparent when we ask the LLMs to impersonate a “black” or “white” person. ChatGPT tends to describe both birds and cars better when posing as a white person. Vicuna-13B, on the other hand, provides better descriptions of cars as a black person. Gender biases are a bit less noticeable, but we still find Vicuna-13B giving better bird descriptions as a woman persona and ChatGPT identifying cars better as a man persona. While instruction-based fine-tuning [54] tries to remedy social biases encoded in LLMs to some extent, we can still expose them through in-context impersonation. This exposes a *difference* but it's unclear if it manifests a social bias. **Can you explain how these results connected to social biases?** 5. What do you believe the primary contribution(s) of the paper to be?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors did adequately.

Summary

This paper tries to understand whether LLMs can take on different personas when they do in-context generation. They have tested different settings: In a two-armed bandit setting, they ask LLMs to act as different-aged people; in a reasoning task, LLMs are asked to behave as different-level experts; and in a vision-and-language task, multiple roles have been set to the LLMs. The authors find while impersonation can improve model performance, such methods also reveal the biases in LLMs.

Strengths

- Test their hypothesis across different domains and with various persona aspects. - They have set the persona specific to each task, e.g, expertise-based persona for reasoning ability analysis. - Well-presented results.

Weaknesses

- LLMs seem to be sensitive to prompts [1]. And I'm wondering how stable the conclusion will be. - I'm curious to what extent the LLMs align with human behavior. For example, in the two-armed bandit task, the authors observe "LLMs could reproduce human-like developmental stages of exploration behavior". And I'd be curious how humans (from different ages) perform the same task. Similar to other tasks. [1] Arora, Simran, et al. "Ask me anything: A simple strategy for prompting language models." arXiv preprint arXiv:2210.02441 (2022).

Questions

- What if in each setting, you add a random persona? For example, in the domain expert case, you impersonate LLMs as "normal student" (or as "specialist"). I'm just curious how certain tokens ("expert") will impact the result.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

Language models can play different roles using prompting techniques (e.g., adding "you are a 3-year-old boy" at the end of the prompt). This paper investigates whether roles in the prompt affect language model performances on three tasks: multi-armed bandit, language-based reasoning, and visual category description. The paper found that: 1. Language models exploit more in multi-armed bandit tasks when prompted to be older. 2. Language models perform better when prompted to be task experts. 3. Language models describe objects better when prompted to be older. For certain object categories, language models show biases (e.g., describe cars better when prompted to be a man instead of a woman).

Strengths

- The paper focuses on an interesting and novel topic: how role-playing (impersonation) affects language model performances on downstream tasks. This is a less-known research question in the NLP community. - The finding about performance differences when describing certain object categories, such as cars, when prompting language models to be different genders provides a novel perspective to understand biases in language models and our society. - The paper is generally well-written and can be easily understood. However, the writing should be more focused. For example, in the first paragraph of the introduction, I did not get any connection between Min et al. found prompting LLM with random labels will not degrade model performance and your research topic impersonation. Similarly, there is no need to introduce CLIP training when introducing the visual category description task.

Weaknesses

- While the topic is to study how different roles affect language model performances, from experimental results, ages have the most significant impact on model performances, and other factors have much more minor effects on model performances. This seems obvious- it is easy to understand older people can exploit more in multi-armed bandit tasks and describe object categories more clearly. However, the most exciting finding, where gender/race can affect model performance on tasks, is not well studied. They only study two object categories from the visual object description task, and the performance difference between genders/races is much smaller and insignificant based on the plots. More systematic studies on language model biases during impersonation should significantly improve the work. - Experimental results can become more solid using more prompt variations and model families. Language models are well-known to be sensitive to prompts; the authors should mention how robust their results are when changing some words in the prompt. Moreover, more model families besides Vicuna are appreciated, especially on bias experiments. Will the finding about gender bias hold for different model families? - Besides the tasks with deterministic labels, it would also be interesting to show how impersonation affects task performances on non-deterministic tasks, such as social opinions or morality tasks. Since the major goal of impersonation is to simulate different people, it would be more interesting and convincing to show that language models indeed reflect these group opinions on non-deterministic tasks when prompted to be them.

Questions

- What are the dashed lines in Figure 3 (results on STEM and Humanities)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

See weaknesses. I'm happy to increase my rating if these are well addressed.

Summary

This paper presents a comprehensive evaluation on the downstream impact of impersonation with the use of LLM and multimodal model (specifically CLIP). There is a couple of interesting finding that LLM, when impersonated, aligns to some behavior carried by the people in the respective norm. This happens in ages, races, genders, and degree of expertises. The finding seems to suggest that one can exploit such phenomenon to make a large model perform better on some downstream tasks. This paper also suggest that such exploitation should be concerned. The arguments are of conflict, but also interesting.

Strengths

This paper introduced three types of evaluation using recent LLM and multimodal CLIP to probe LLM's downstream performances: exploration-exploitation, reasoning (MMLU), and shallow multimodal understanding. There is an very interesting finding that impersonation aligns to different behavior and downstream performances on these benchmarks. An immediate takeaway is that it gives an "interface" to easily boost downstream performance with a discovered stereotype to impersonate. The respective concern raised in this paper is also of value to the community. In general, I am happy to see such argument and findings to appear in the conference, as long as the argument goes well.

Weaknesses

A few downsides: - On one hand, this paper seems to suggest one can easy explore stereotypes to impersonate and use them to improve some downstream performances. On the other hand, this strategy is of special concern, and I believe, should be avoided. ***It's unclear what this paper's take is.*** A very simple and immediate counter argument to this paper is that, such exploitation aggregates the existing stereotypical biases in the real world. And if this concern is valid, then this paper shouldn't advertise it as a strength. - Not clear if the observation is robust against prompt variance. The only place seems to have robust eval is the two-armed bandit experiment. - The age-related evaluation is narrow. A more comprehensive range should go beyond age 20. This can be useful to probe for potential age biases and have a more comprehensive picture about the age "mimicking" phenomenon.

Questions

Please see the general comments.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Implicitly or explicitly, this paper suggests the exploitation of stereotypes to improve model performances on downstream tasks.

Excellent rebuttal, I now vote for acceptence

> [60] studied both exploration and generalization across different age groups. They found that both directed and random exploration were reliably higher for children compared to adults. Similarly, we find increased directed exploration, as measured by the effect of relative uncertainty, for LLMs impersonating younger ages. [59] has a section on “directed exploration” which corresponds to the effect of relative uncertainty in our probit regression. This whole section is about how children tend to use directed exploration more than adults. > > To further link our results to the literature on developmental psychology, we have modified this section: This pattern is in line with several results from the psychological literature which also found that children show higher levels of directed exploration than adults [60]. These results suggest that impersonating LLMs can recover human-like developmental stages of exploration in a two-armed bandit task. If life is seen as an exploration-exploitation problem, then younger agents should show higher amounts of directed exploration [F,G]. > > Lastly, we ensure reproducibility (including data generation) by releasing our code which is already included in the supplementary. Thank you for the extended experiments, especially involving older individuals and multiple prompts. I find this substantially more compelling, and it's the primary reason for my change of opinion. > Vicuna vs ChatGPT I have zero objection to using Vicuna (though it's not "open source," it's publicly available under the LLaMA 2 license. My sole concern is that I think you're overselling its performance. > Rather than arguing that LLMs faithfully represent the language of children, we qualitatively observe interesting patterns similar to humans. In our supp Sec3.1 with several text complexity metrics we quantitatively showed that the LLM descriptions of older personas are more complex than younger personas. Prior to your new experiments, I did not believe that the results you reported were qualitatively similar to those obtained by the cited literature. Your new, extended experiments have largely ameliorated this concern. > The conceptual “Role-Play with Large Language Models” paper (no experiments presented) was not publicly available at the time of our submission. My apologies, I had misread the date on that paper. I do think it's worth mentioning but will not dock you for choosing to not. > We have not investigated text-to-image models (there are many) as it is out of our scope but will mention them in our related work section, as they are also prompted with text. > Age groups & temperature: We chose key developmental stages of early childhood, childhood, adolescence, and adulthood. Nonetheless, we reran our experiments (with all prompt variations) for more ages. We discuss these results in the general comment, where we find a significant effect of age on the reward for ages until 20 (β=0.15,p<.001), but no significant effect from ages 20-60 (β=-0.01,p<.1). Different temperatures were chosen based on the recommended default values of each LLM as this will reflect the most common use case but the same value may not generalize across LLMs, e.g. text properties depend on a variety of factors including training data, model scale, model architecture. These experiments were extremely compelling. > Official MMLU prompt The official MMLU prompt is `The following are multiple choice questions (with answers) about {TOPIC}.\n\n`, as seen [here](https://github.com/hendrycks/test/blob/master/evaluate.py). > MMLU tasks: Due to space restrictions, we focus on the STEM and Humanities tasks but the expertise trends hold for the other domains. We include results for all domains in Figure 3, e.g for STEM the non-domain experts are all the non-STEM experts. The standard thing to do (see, e.g., GPT-NeoX-20B, LLaMA, LLaMA 2, and PaLM) is to report accuracies averaged over each of the four domains. That's what is recommended by the MMLU paper, and would give you plenty of space. > Social Biases Perhaps "differences" would be a more appropriate word? I do think that it's important to not call them social biases in the absence of evidence of power. **New score:** 7

Author's take and argument on the exploitation of biases should be clearly and non-briefly addressed in the draft

Thank you for the rebuttal. I agree that the exploitation of biases can be a measurement for LLM providers and this actually can be a good use case. However, this point should be clearly addressed in the draft. Emphasis on downstream performance improvement is essentially an aggregation of biases instead of an effort of mitigation.