Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore

Summary

This paper proposes to extend FACT-Score to be multilingual and shows various analyses across languages and geographies. They find that shorter Wikipedia pages in other languages provide less information to use for FACT-Score resulting in lower scores. They show this can be partially resolved through translation and ensembling of Wikipedia pages. They also show a western bias in FACT-Score. My score is marginally below as of right now, but I think with fixing the grammar and other issues it would go to marginally above. EDIT: increased my score, see response to their rebuttal.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The paper provides many interesting analyses into how different Wikipedia languages play a role in impacting FACT-Score values. - They propose several ways of countering this length bias that are intuitive and seem to help. Figures are well done and easy to understand - The topic is important when discussing the evaluation of LLMs across languages

Reasons to reject

1. Some of the experimental settings are unclear. For Table 3 and Figures 6-7, which references are being used? How many instances of data are used per language? Will these datasets be open-sourced? - Other questions include, how does each of the changes you made in terms of using open models impact these results? E.g. Does using Mistral cause a change? 2. There are a decent amount of mistakes in the paper in terms of spelling and latex references missing, which are minor and can be resolved but indicate a hurried submission 3. The paper title seems to indicate they extended Fact-Score itself, but in reality they applied it to new data - which is a fine contribution but somewhat misleading since there are no methodological contributions to the base FACT-Score.

Questions to authors

Minor comments: 1. The related work should be a cite paper with parentheses for many of the citations there (some are correctly using the name, others should be in parens). This should also be followed throughout the paper - only use the non-parens version when citing the name directly. 2. 5.2 should be low resource language**s** 3. Footnote 5 on page 6 has an undefined reference 4. Table 3 could be made clearer by multiplying the values by 100 and underlining less - it is a little busy

Summary

This paper the problem of factuality evaluation of multilingual LLMs. It introduces a pipeline, Multi-Fact, for multilingual factuality evaluation, adapting FactScore for diverse languages. This pipeline first translates texts in other language into English, and then using FactScore for evaluation. In experiments, this paper explores the potential and limitations of using non-English resources for factuality evaluation of long-form generation. The Multi-Fact tool can also be used to investigate cultural and geographic biases in LLMs.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

- Factuality evaluation in multi-lingual LLMs is an interesting and important research problem. - The proposed method is simple yet effective. - There are some interesting findings in the experiments, which are inspiring for future research on building multi-lingual LLMs. - The paper is well-written, and the idea is easy to follow.

Reasons to reject

- The novelty and technical complexity of the proposed method are quite basic. It mainly translates texts from other languages into English and then uses the FactScore tool for evaluation. There is no innovative approach or model created specifically for multilingual scenarios. - The experiments do not compare the proposed method with any baseline methods. It is unclear whether the proposed method is effective, even though it seems reasonable. - The experiments only use Wikipedia as a data source, which limits how widely the proposed method can be applied.

Questions to authors

N/A

Summary

It is nice that this paper tackles the multilingual challenge – this is very, very underexplored in our field. Overall, the paper makes a few design decisions that I would have probably done differently, but the authors do justify their decisions relatively well and I think I would be able to replicate their methods. In particular, I think the paper could be improved with two experiments: • Including at least one small experiment using a non-LLM translation compared to a translation LLM. • Trying to see how FActScore would work on a language without translation. Aka an experiment similar to section 4.1, but just done monolingually in the languages (such as Bengali generation and Bengali evaluation). One of my first questions when I read the paper was “Why translate with GPT-3.5?”. The authors do address this. It is ok for some languages, but if you are going to do automatic evaluation, another commercial solution like Google Translate may make more sense – especially for low-resource languages (Kocmi et al, 2023; Robinson et al., 2023). The authors make a nice point about gender in Google Translate and why they did not use it, but I would have still liked to have at lease seen one experiment showing what happens if you use that as well. A comparison even for one language with Google Translate and GPT-3.5. I would have also liked to have seen how the method works monolingually (aka not translating to English). Again, the authors make claims as to why they made these decisions such as “which makes it difficult to compare cross-lingual factuality if the knowledge base is different.” However, it would be nice to still see how it works in different resource settings. You are generating in that language, it could be another line or two in some tables. This feels like it could also answer some of my questions about translationese impacts. Overall, this paper is a nice attempt at multilingual fact checking, which is woefully underexplored. This alone is a worthwhile contribution to the field.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

There are very few papers looking at multilingual fact checking in LLMs. The field needs more of these.

Reasons to reject

I'd like to see the following two experiments: • Including at least one small experiment using a non-LLM translation compared to a translation LLM. • Trying to see how FActScore would work on a language without translation. Aka an experiment similar to section 4.1, but just done monolingually in the languages (such as Bengali generation and Bengali evaluation).

Questions to authors

Figure 5, the x-axis should be the same between a and b. Either (0.0-1.0) for both or (0.0-0.8) for both - not for 0.8 for one and 1.0 for the other.

Summary

This paper explores multilingual LLM evaluation by adapting existing factual evaluation method FActScore. The paper also studies the potential and limitations of using non-English resources such as non-English wikipedia and geographic biases in LLM.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

The paper adapts FActScore to a multi-lingual factuality evaluation framework, and experiment the method using 8 languages Providing insight about reasonably large non-English wikipedia articles can also serve as a viable source.

Reasons to reject

Overall, the originality of the work is limited, it uses GPT3.5 for language translation then adapts an existing factuality method. The paper can be improved by providing more details about FActScore, it is hard for readers to understand the approach with no knowledge of the method. Experiments only use GPT3.5 and GPT4 for evaluation, it is better to have more baseline methods such as llama, mistral, etc.

Thank you for the response - increasing my score

I like this work and think it is an incorrect topic. The authors have promised some revisions which I think are minor and easy to do. I think the paper will make for good conversation at the conference and agree it is an understudied area. Re: other reviewer concerns: I agree that this paper doesn't propose any new methods and the evaluation process is simple. However, I don't think that is a major blocker - simple analyses done well are important also. I agree with reviewer a4f9 that more models would be nice - perhaps the authors could use Together.ai (if they can't run them locally) to use some of the larger models like Llama3-70B (but of course you couldn't have put that in the paper since it wasn't out yet). I think the authors could and probably should do this so that there are reproducible results but again this wouldn't take very long to add.

About Atomic breakdown and factuality evaluation in non-English language:

**About Atomic breakdown and factuality evaluation in non-English language:** Accurate evaluation of the fact verification system (using non-English atomic breakdown) in non-English language is difficult for several reasons: * The original NPM implementation (which is crucial for reaching good accuracy) was trained on English data only and the tokenizer heavily disfavors non-English languages, which is why running the most effective method (ensembling with NPM) in non-English language is not possible [refer to the table at the very end of this comment]. Implementing non English NPM is a worthwhile objective, but out of scope from our project. We, thus, decide to only include results for Retrieval + ChatGPT method for doing atomic fact breakdown in non-English language. [We do not provide numbers for Mistral since Mistral is English only model] * Besides, to make a fair comparison between two languages’ factuality, while doing atomic breakdown in the corresponding language, we need to have the same ground truth corpus in both languages. To meet the later requirement, we translate wikipedia in the non-English language to English (as X→ EN usually leaves out less translationese than EN→X) * This setting is similar to Figure 4a. We sampled 10000 wikipedia articles from French and Korean wikipedia and then took the top 120 articles by length and asked GPT4 to generate biographies corresponding to those entities. We translate those 120 wikipedia articles from KOR/French into English for a comparison. * Since we do not have ground truth labeled data for these 120 entities in French/Korean, we only show the difference with the best performing method (Retrieval + Mistral + NPM using EN wikipedia) here (like Figure 4a) We only conducted this experiment for Korean and French (one latin and one non-latin script language) | Language | Wiki-X-breakdown-in-X | X-wiki-translated-to-EN-breakdown-in-EN | original-EN-wiki_breakdown-in-EN | |----------|----------------------------------|--------------------------------|-----------------------------------------| | French |-20.28 | -25.46 | -26.49 | | Korean | -17.36 | -34.27 |-35.86 | The column represents the difference with the best performing method (Retrieval + (Mistral + NPM)) with the column. The method is same for all the cases (Retrieval + ChatGPT): * **Wiki-X-breakdown-in-X:** We used atomic breakdown in KOR/FR and also verified against the corresponding wikipedia * **X-wiki-translated-to-EN-breakdown-in-EN:** We did atomic breakdown in English, and verified against the translated KOR/FR wikipedia, that has been translated into English * **original-EN-wiki_breakdown-in-EN:** Atomic breakdown in English and then verified against English wikipedia **In conclusion:** * It is difficult to accurately quantify whether doing atomic breakdown in non-English languages will yield good results or not given the large difference with the best performing method. * Still, it is possible that if there is a multilingual NPM model, the accuracy would go higher. * A qualitative look at the atomic breakdown of biographies generated in Korean shows that the generated facts are accurate. This suggests that once we have a good multilingual NP model, our method would work well in the Korean setting. The methodology comparison table (atomic fact breakdown in English): | Method | Arabic | Swahili | Bengali | Chinese | Korean | French | German | Spanish | Mean | |----------------------------|--------|---------|---------|---------|--------|--------|--------|---------|--------| | No Retrieval ChatGPT | -26.83%| -28.15% | -25.97% | -26.48% | -27.70%| -27.75%| -28.82%| -28.95% | -27.58% | | Retrieval + ChatGPT | -16.32% | -19.00% | -16.46% | -16.81% | -17.67% | -17.73% | -17.53% | -18.72% | -17.53% | | Retrieval + Mistral | -15.04%| -18.03% | -16.88% | -16.16% | -17.19%| -17.07%| -17.51%| -17.37% | -16.91% | | NPM | -9.74% | -13.27% | -10.52% | -9.60% | -9.81% | -14.00%| -14.87%| -15.25% | -12.13% | | Retrieval + Mistral + NPM | 2.80% | -1.57% | 2.19% | 2.89% | 0.63% | -1.26% | -1.27% | -1.15% | 0.41% | **Notes:** * ChatGPT refers to GPT3.5-turbo. * No Retrieval ChatGPT means simply asking ChatGPT whether a fact is correct or not * Retrieval + ChatGPT implies retrieving relevant passages from Wikipedia and then answering. * Retrieval + Mistral refers to retrieving passages and then asking Mistral 7B for judgement * NPM refers to Non Parametric Masked Language Modeling (Min et al ‘22), which requires having access to the Wikipedia article * The final method is the ensemble version of NPM and Retrieval + Mistral and it performs the best. This is the one we used throughout our paper for other analysis.