Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas

Summary

Simulating human responses to a study from psychology using GPT-3.5 prompted with respondents demographic information, this study shows high fidelity, which is improved by also indicating the respondent's country, but deteriorates when using the respondent's language for prompting. Best fidelity is with English prompts, with lexical distance from English being negatively correlated with fidelity for other languages.

Rating

9

Confidence

5

Ethics flag

1

Reasons to accept

Addresses a topic of interest to a wide audience and societal impact, of cultural sensitivity of GPT. Grounded in psychological research and using data from robust experiments with human participants as a basis for comparison. Results are surprising and impactful as they reveal the capability of simulating human tendencies but confirm cultural biases in LLMs, and as simulating human responses with high fidelity across cultures is a useful task in itself.

Reasons to reject

Lacking details about the experimental setup of this study as well as crucial details from the Bos et al. study. (see questions to authors). The authors follow the analysis "closely, but not exactly", but it's not clear how they're different. Missing details about geographical, population-related and linguistic factors in the analysis of experiment 2, and about the correlation obtained with them. Testing the validity of translation only with GPT-3.5 is insufficient (we don't know how reliable this is without correlating it with human judgments) and it's also unclear how exactly this is done. Missing context and citations for studies that also investigated the effect of prompting country/language on LLM fidelity in simulating human responses: https://arxiv.org/abs/2306.16388 https://aclanthology.org/2023.c3nlp-1.7/ https://aclanthology.org/2023.c3nlp-1.12/ Averaging the relative deprivation ratings and replacing the original ratings seems arbitrary and unjustified. It is fair that it improves fidelity but what happens without doing this? The results from experiment 3 de presented in figure 4 with seemingly arbitrary x value, which is confusing. If there's no meaning to the x value it should be in a separate subfigure. Framing the results as revealing human-like characteristics, and claiming "LLMs exhibit persuasion and mobilization effects akin to humans", is unnecessary anthropomorphism. They are only simulators. Also, "LLMs" is to general since only GPT-3.5 was studied.

Questions to authors

Which language did participants in Bos et al.'s study were exposed to? What story did they read? The same as in the present study? What country was the story about? Their own? Given these details, can you speculate on why the prompting language yielded the opposite effect than that expected in experiment 3? How was the country masked in the present study? What is the justification to compare the sign of the country coefficients? Since they're set to have zero mean, the sign just means where each country lies relative to the mean of the countries in this specific set of countries, and may be very different if a different set of countries is selected. Figure 3 should be a table. P values should be given in decimals and not percents. Greek and Hebrew are not language families.

Summary

The goal of this paper is addressing the research question “How to effectively incorporate language in LLMs?” as well as “How precise is the nationality trait depicted by multilingual LLMs?” towards personalized and inclusive LLMs. Here, a semi-complete questionnaire (adapted from Bos study) is provided where numbers should be selected for each question along with news stimuli to the participants. The study contains >7K participants across 15 countries. 3 GPT-3.5-Turbo-1106 personas are created : 1) directly expressing the nationality 2) use of a side language (majority language of each country) 3) use of native language – Following each experiments, each participant’s persuasion and mobilization scores were collected while simulating the participant’s reaction to news stimuli. While modern LLMs have multilingual capabilities, there is bias due to shortage of data in other languages than English ( or other similar high-resource languages), this is a challenging problem specially when it comes to low-resource languages. Further, there are gender biases in certain languages that can be also present in English LLMs. Also, in tasks such as emotion detection, English LLMs perform better than other languages. LLM persona was introduced in 2023 where LLMs can showcase certain personality traits when interacting with an end-user. It is also shown that RLHF produces more liberal content than the original pretrained data that is more conservative. Experiment result was that languages that resemble English language the most have the higher agreement score and that translating to native language yields poor results. Further, providing the nationality of the participant as part of the prompt provides better results. Based on the experiments, better results are yielded when English is chosen as the language of prompting.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

- Authors have performed tests to ensure linguistic meaning of the survey was not lost during translation in different languages. - Comprehensive experiments across a diverse set of countries and a huge number of participants followed by detailed discussions. - This research paper provides insight in design multilingual LLMs that needs to be more inclusive. - The multidisciplinary nature of this work is beneficial to the community.

Reasons to reject

I am pro accepting this paper specially due to its multidisciplinary effect.

Questions to authors

I am not sure if using Google Translate would be best resource in this scenario since even in case of low-resource languages, Google Translate doesn't have the best performance. Could you please explain how you handled this situation? Also could you please elaborate a bit on if you handled gender biases present in certain nationalities? or what would you propose in this regard?

Summary

This paper replicates a psychological experiment from Bos et al. (2020), where predefined human profiles are simulated to study their reactions to news stimuli. The authors have access to the original human responses of Bos et al. (2020)'s experiment, and prompt GPT3.5 to generate simulated profiles. The authors test various combinations of information given to the LLM during prompting, but the main goal of the study is to explore how the various "metadata" information can affect the output of the LLM. I refer to it as "metadata" even if it is not, it is more a set of additional information about the human profile: nationality and language. Through a set of analysis and combinations of adding information about nationality, language, both, or neither, the authors explore how this affects the final prediction of the LLM. As far as I understand it, the final prediction of the LLM is a simple numerical value as response to one question precedes by a longer prompt presenting the human profile, the relative deprivation values of the human, as well as a news article. The paper is well written and interesting, but I have some doubts about some aspects of the motivation and claims. Please see weaknesses and questions for details.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- A nice contribution in the direction on grounding interdisciplinary research by replicating a study from psychology on LLMs. - The authors present a number of experiments and analysis of their results.

Reasons to reject

- Some aspects of the paper are not clear to me, and the authors fail to give clear arguments to support some of their decisions (I am not sure if I understand some of the decisions made by the authors, please see questions). - While I think that the idea of the work is interesting. Some I am not sure if asking GPT3.5 to give one numerical value means that we have a system that effectively simulates human behavior. - During their experimentation, the authors relied on machine translation to translate their prompts. To assess the correctness of their translation they used GPT3.5. While this might work and yield good results, it does not equate human evaluation. The authors fail to discuss this aspect in their limitations section, nor discuss the effect this can have on languages with higher lexical distant to English (which is one of their findings, that for languages with a high lexical distant from English, using English prompts is better). Maybe this is mainly a side effect of bad translation? It is anyways not discussed by the authors, and I did miss this discussion.

Questions to authors

- In experiment 2, I am not sure I understand the sentence "the simulation was run 12 times, each time simulating in the same language regardless of the nationality of the simulated participant". What is the single language used? Is this English? - In experiment 3: I am not sure I understand why you have approached this experiment in this way. If we want to test the effect of the language, shouldn't the prompt be change to the relative language of each country? Or is this to test if English yields the best results overall? I am not sure about the motivation here. - Did I get it correct that your experiments are based on one numerical value returned by GPT3.5?

Summary

The paper examines how information about language and the country of residence influences the replication of a multi-country experiment on political persuasion and mobilization. To this end, the paper relies on data from a political experiment involving ~7k participants from 15 European countries who read prompts about financial issues in their countries. The prompts attributed blame for the issues to either elites, immigrants, or both, and participants' persuasion and mobilization levels were assessed. The current research replicates this study, adding participants' demographic information and using an LLM to predict responses. Three experimental settings are tested where the prompts vary by including individuals' country of residence and modifying the language to compare the LLM's performance across conditions.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

Incorporating and examining cultural information embedded in LLMs is a crucial task, our field increasingly needs accurate approaches for capturing cultural representations and the role of language in shaping them. The paper has this valuable motivation. This paper makes a valuable point by investigating the correlation between various linguistic factors (e.g., distance from English) and cultural representation, exploring whether and how language influences cultural representation in LLMs. The research design is noteworthy for its reliance on human data from diverse countries, adding validity to the study of this important question.

Reasons to reject

The paper's approach to assessing the fidelity of LLMs in simulating human responses raises concerns. The chosen method, focusing on replicating regression results, seems inadequately justified. A simpler correlation analysis might have provided a more direct measure of how well the LLMs simulate individual responses. Additionally, the paper omits mentioning that the data and experiments are limited to European countries and languages. While this doesn't necessarily invalidate the findings, it's a surprising omission that should have been acknowledged. The generalization of findings to psychological traits is also problematic, as the dataset specifically examines the impact of two contexts on agreement with political arguments about a financial issue. A more appropriate dataset would have included psychological traits collected across diverse cultures, such as those found in the World Values Survey. Furthermore, while the paper discusses culture, it neglects to address the limited cultural variance within European countries.

Questions to authors

The evaluation method assesses 44 coefficients, 30 of which are country-level (C_i^P, C_i^M). However, the LLM is evaluated on replicating these coefficients even though the country of residence is not provided in the prompt. Could you clarify whether the expectation is for LLMs to infer this information from the language and text of the prompt, and if so, what linguistic cues or patterns the LLM might be utilizing for this inference? The relative deprivation appears to be a crucial independent variable in Bos et al.'s experiment. However, your paper mentions that "the three relative deprivation ratings in our prompt were set to the same average of the actual three relative deprivation ratings by the human participant in the original study" to enhance fidelity. Does this adjustment imply that LLMs struggle to capture the association between relative deprivation and persuasion (and mobilization), and if so, what are the potential reasons for this limitation and how does that impact the paper's main goal for replicating the human responses?

Increasing my rating

After reading the authors' detailed responses to reviewers, I am changing my rating to 8. I suggest containing some of these concerns/questions and responses in paper or its appendix or an accompanying website since if this is raised by us, it could be someone's else question too.