How Well Do LLMs Identify Cultural Unity in Diversity?

Summary

There has been much research regarding whether LLM can detect or even understand cultural diversity. This paper, instead, tries to understand whether LLM is sensitive to cultural unity, i.e., the common ground across cultures. To do this, the authors introduce a new benchmark dataset CUNIT that contains evaluation examples of a variety of concepts across 10 countries.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- This paper is generally well-written. I appreciate the authors include many examples in the main text as well as figures, which can help the reader to better grasp the idea. - The contribution is impactful. By benchmarking the LLMs' sensitivity to cultural unity, we can better understand the cross-lingual capability of these models. - The description of CUNIT is quite clear in Section 3. - The authors evaluate both LLM's results as well as human performance.

Reasons to reject

There are two limitations: - Before evaluating LLM's sensitivity to cultural unity, I think it is necessary to first qualitatively or quantitively evaluate whether an LLM knows or understands these concepts. - The authors did not mention the background of the two human annotators. I assume the authors want to set human performance as the upper-bound performance. However, since the concepts are across 10 countries, I am sure how well these human annotators are familiar with these concepts. So maybe a better human upper-bound performance would be to ask different people who are familiar with the concepts involved in each triplet. Some minor suggestions are listed in the section below.

Questions to authors

Figure 1 looks nice. However, the caption does not introduce the relations between each concept of interest itself and the features below it (And the features themselves). Therefore readers can get confused (I know they are described in later sections, but since the authors choose to place one figure on the first page to sell the paper, the authors should reduce possible confusion as possible as they can). Table 2: unfortunately, the authors seem not to mention \mu and \delta of the difference of similarity of the final triplet samples anywhere in the paper. I believe these are very important indicators of the difficulty in different granularities. The author mentioned that "A small level of similarity difference indicates a harder contrastive matching task". If I understand correctly from Table 2, "Small" indicates that the Sim. Difference is actually **large**, i.e., a value within ($\mu$+0.5$\delta$, 1], which should actually make the contrast task easier. It is quite confusing here. Are the authors mistaken the two granularity, i.e., "Large" and "Small"? Figure 3: the information is redundant since the similarity of A and B is the same as the similarity of B and A. The authors can remove the upper triangular part.

Summary

New benchmark for identifying cross cultural similarity of traditional concepts. Through semi automatic annotation of fine grained features for food and clothing concepts, concepts similarity is defined by shared features. LLMs are evaluated on distinguishing cross cultural concepts with high similarity versus low, a novel task as a step towards cross cultural communication facilitated by LLMs.

Rating

7

Confidence

4

Ethics flag

2

Reasons to accept

Well motivated, interesting and even groundbreaking goal statement, to enable cross cultural understanding and common ground facilitated by LLMs. The proposed task effectively operationalizes one aspect of this seemingly intractable goal. Expansion of traditional cross lingual semantic matching (lexicon induction etc) to the pragmatic level by focusing on concepts used in similar contexts. Creative and effective annotation process, where fine grained features are used to quantify concepts similarity for relatively objective quantification. There is very high agreement for the features annotation. Interesting results showing that LLMs lag behind humans in the purposes task.

Reasons to reject

Part of the annotation process is automated with ChatGPT, but the version is not specified, and neither is any indication of the performance (quality) of that automated step. Namely, it is used to summarize Wikipedia articles to extract excerpts relevant to the defined features; but we don't know what is the recall compared to a human doing this task or if there's a performance bias towards concepts from certain countries. While the idea of contrasive classification by number of shared features is interesting, I am not convinced it is justified to treat it as ground truth. Some concepts may be perceived by a well versed human as very similar despite having only a few shared features, maybe because some features are more important than others. And vice versa. The study here does not corroborate the calculated similarity with human judgments. Unclear experimental setup: a setting is mentioned where concepts names are masked and only features are available, but no results for this setting are presented in the main paper. Further, it seems odd to use this setup, since it seems like it relies purely on the arithmetic ability to calculate the Jaccard index from the lists of features, and does not require any cultural awareness. This also applies to the setting where both features and concepts are provided, since high performance can simply be explained by mathematical ability. No details about the annotators except that they are "from Eastern countries". The feature lists seem somewhat arbitrary and are not well motivated. For example many Jewish holidays are features, though they're not necessarily very cross cultural.

Questions to authors

"LLMs are still limited to capturing" should be "LLMs are still limited in their ability to capture"? "construct this feature list at the same while annotating" - redundant "at the same time"? In the framework of Hershcovich et al. [1], the contribution here is limited to the Common Ground dimension. What about the other dimensions? Would facilitating cross cultural communication require addressing Aboutness and Values too? Is each matrix in figure 3 symmetric? It may be easier to read if you drop everything under the diagonal to avoid repetition. Did you try Llama 3? It may be worth adding it to the next version. Also GPT-4. In figure 4, "significantly" refers to what p value and what statistical test? [1] Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and Strategies in Cross-Cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.

Ethics concerns details

No information on annotators or compensation

Summary

this paper proposes a framework to analyze to what extent English-based LLMs are able to identify cross cultural concepts, as a way to extend the current research focusing on the ability of LLMs to be culture aware, which so far have been centered mainly in terms of linguistic variations (usage, style). for this, the authors propose a task and a. dataset that encapsulates manually defined cross-cultural concepts and their features across ten countries, from which a probing task is designed to assess how well decoder-only LLMs detect concept with the highest cross cultural similarity. Results are somewhat expected : i) GPT is overall winner and ii) results are highly linked to frequency across languages

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

the topic is really relevant in terms of the amount of personalization LLMs are expected to provide. the dataset is represent a very valuable effort in terms of quality of annotations and expected diversity of the cross culture topics and seems to be well constructed and validated .

Reasons to reject

.

Questions to authors

1.Is just focusing on English centric LLMs a bit limiting for the scope of the problem ? Moreover, it seems counterproductive as people from non-English speaking countries are not, in the first place, willing to consume or interact with information provided by an English-based LLM. In that sense, while it consider the current results relevant, are we missing a very, very big picture by not considering a multilingual setting ? 2. I think this is actually a missed opportunity to increase the contribution of the paper: "We exclude South America due to the limited availability of its cross-cultural concepts on Wikipedia." As we know, most countries in South America uses Spanish as main language, but it is highly fragmented given historical and even geographical reasons. Therefore, having a single common language (Spanish), but that has clear cultural differences across more than 10 countries seems like a perfect scenario for these type of experiments (considering Spanish from Spain could even also make things more interesting as there could be an intercontinental setting) 3. i think one way of complementing the analysis and maybe to obtain more diverse results could bring known cultural structure for the analysis. For example, we know there are clusters that could be exploited , such as Korean - Japanese . Based on their historical ties and relationships (beyond their Chinese linguistic root), there are several terms in Korean that derive from Japanese . Is it possible to adapt the analysis to such setting?

Ethics concerns details

.

Summary

The paper addresses the important problem of cultural awareness. It introduces a dataset CUNIT with cultural concepts and their cultural associations and similarities across 10 countries to better facilitate cultural awareness of LLMs. The authors evaluate existing LLMs on this dataset and find that they are still limited in capturing cross-cultural associations.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The dataset is an important contribution to the community, because it can help LLMs to create a better common ground on cross-cultural concepts to improve cultural awareness. 2. The analysis is comprehensive and the experiment is well-designed. 3. The authors also show implication on data-centric factors and suggests the focus on long-tailed cultural knowledge.

Reasons to reject

The following are limitations general to many related problems not just to this paper in particular. 1. The evaluation method is a bit unnatural, it's a relative ranking task "sort the following cultural-specific concepts in descending order", but this is not how these LLMs will be used in practice: they will be more used in contextualized settings like a conversation. I wonder if the paper could discuss more about how to translate this task in a more grounded setting. 2. One thing I am not sure is whether it's the best practice to quantify the similarity with an actual number and how to use this number. LLMs are not super sensitive to numeric values, and it's also hard for humans to annotate if "bridal veil" is 30% similar to "honggaitou" or 35% similar.

Questions to authors

Can you say more about the annotation process? Do these annotators come from the same cultural background of the data or they will refer to some official resources?

Follow-up Response

Thank you so much for taking the time to read our response. We are glad that our answers have addressed some of your concerns. Regarding your point about the assessment of our framework emphasizing cultural unity, we would like to share some thoughts below. Given our focus on cross-cultural alignment, we can evaluate our cross-cultural similarity framework's practical benefits through downstream applications like matching translation. Following prior work on culturally-aware machine translation [1] and human translation theories [2], we learn that many culture-specific concepts in a source language lack direct equivalents in a target language. In these cases, professional human translators often use two translation strategies: (1) providing an additional explanation of the term in the target language and (2) substituting the concept with a culturally equivalent one that target language speakers can easily understand. Compared to traditional NMTS, mainly trained on parallel corpora, LLMs have a unique advantage in performing these explanation-based strategies. Therefore, our cross-cultural similarity framework could provide an initial step in assessing whether LLMs can identify culturally equivalent concepts. [1] Yao, B., Jiang, M., Yang, D., & Hu, J. (2023). Empowering LLM-based machine translation with cultural awareness. arXiv preprint arXiv:2305.14328. [2] Newmark, P. (1988). A textbook of translation (Vol. 66, pp. 1-312). New York: Prentice Hall.