How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?

Summary

This paper investigates the robustness of LLMs to irrelevant information, particularly when such information is semantically related but irrelevant to the query. The study involves constructing various types of irrelevant information, ranging from semantically unrelated to related, and assessing their impact on the performance of LLMs.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. The robustness of LLMs to irrelevant information is a very critical issue in the field. 2. The authors provide a fine-grained analysis of the impact of irrelevant information based on its semantic relatedness to the query. 3. The structure of the article is well organized.

Reasons to reject

- The most important question is: the definition of "irrelevant information" in this paper seems to be unclear. - It is not well-explained why "semantically related information" can be considered "irrelevant". The distinction between semantically related information and concepts like incorrect contexts, misinformation, or conflicting knowledge, as mentioned in recent works [1,2,3,4], is not adequately addressed. - Why is the "Parametric Memory" in Figure 2 considered "relevant"? After all, Parametric Memory might not necessarily contain the correct answer. - In Table 4, the impact of question formats on the Misrepresentation ratio is quite apparent, especially for Llama2-7B. The free-form format clearly aligns more closely with real-world application scenarios. Why then does this paper primarily use the M.C. format as described in Section 3.6? - In Sections 4.1 and 4.4, the authors do not seem to mention the settings for question formats, and my understanding is that they default to using the M.C. format as discussed in Section 3.6. Perhaps the authors should also clearly specify the question formats used in each section. This has caused me a bit of confusion. - What should an ideal LLM behave? Should it adhere strictly to its own knowledge, completely ignoring external input? Or should it accurately identify the correct piece of knowledge from multiple external sources? - A more specific question about this, in section 4.2, "We adopt corresponding parametric memory as relevant information"? This is hypothesis unreasonable as parametric memory might be incorrect. It may be more reasonable to use the golden passage as relevant information. - The authors' use of parametric memory as relevant information seems to suggest that the model should always trust its own knowledge. If that is the case, why do we still need RAG? - The details or a formal definition of the Misrepresentation Ratio need to be provided more thoroughly. This is crucial for the reliability of the conclusions in the paper. For example, in section 4.4, if the LLM disregards its own parameter knowledge and relies on the golden passage, is this included in the Misrepresentation Ratio? - Some Questions Regarding Section 4.4 - Section 4.4 considers scenarios where multiple types of irrelevant content coexist. The authors only published the results of the Misrepresentation Ratio. Perhaps what is of greater interest is: which specific passage LLMs choose to trust, or which specific irrelevant content misleads LLMs (“Unrelated Information”, “Partially Related Information”, or “Related Information”). - Does Section 4.4 ensure that all five types of contexts provide different knowledge? If any two types of knowledge are the same, it could likely affect the model's behavior. - The conclusion that "CoT might cause over-reasoning due to misleading irrelevant information" is quite interesting. Has this conclusion been mentioned in other articles that you cited, such as [5]? Is this finding completely novel? - Additionally, sec 4.4, "To simulate such a scenario, we present LLMs with a combination of information that includes five pieces of irrelevant content (one 'Unrelated Information', one 'Partially Related Information', and **three 'Related Information'**)." Why including more 'Related Information' than 'Unrelated Information' and 'Partially Related Information'? - I suggest that the authors explain the concept of "over-reasoning" and why it is an undesirable situation. This concept is not very familiar to me at least. [1] Pan Y, Pan L, Chen W, et al. On the risk of misinformation pollution with large language models. EMNLP 2023 [2] Tan H, Sun F, Yang W, et al. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts for Open-Domain QA?. arXiv 2024. [3] Xu R, Qi Z, Wang C, et al. Knowledge Conflicts for LLMs: A Survey. arXiv 2024. [4] Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. ICLR 2024 [5] Over-reasoning and redundant calculation of large language models. EACL 2024

Questions to authors

see above

Summary

Language models are trained to use relevant information in query responses. This paper focuses particularly on the the ability of RAG LLMs to discriminate relevant information from other semantically related information, and on particular strategies used to improve performance on RAG tasks, by deliberately constructing semantically related distractors and evaluating on those. One model in particular, GPT-3.5 Turbo, was then tested to to see whether performance is improved by chain-of-thought (CoT) strategies, additional instruction, or additional "in context" labeled examples (ICL). Results indicate, as expected, that LLMs relatively rarely change their memory answers to irrelevant options when facing “Unrelated Information”, but that they are often distracted by partially related information. Methods commonly used to address this problem, surprisingly, can have a negative impact. With one of the best performers, GPT-3.5, results in Figure 5 indicate that CoT strategies have a negative impact, and even ICL can sometimes have a negative impact. The negative impact of CoT is perhaps the most surprising result, and the authors attribute it to "over-reasoning". An example supporting this conjecture is analyzed in appendix B3. For future work, it would be valuable to have a benchmark dataset to assess progress on these issues, and to allow a more comprehensive item analysis than can be provided by a examples in the appendix. This research is well-situated with respect to the literature and clearly presented.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

Deliberately constructed distractors is an interesting approach to assessing model robustness, and the negative impact of CoT and ICL is surprising -- further study should elucidate what is happening in cases like these.

Reasons to reject

While the work is fairly clearly explained, it would be a very considerable task to replicate.

Questions to authors

To facilitate replication of these results, it would be valuable to make components of the data construction and analysis available.

Summary

The paper presents an in-depth analysis into the performance of Large Language Models (LLMs), when asked to answer questions given irrelevant text as input (Irrelevant Context). This problem is crucial to ensuring the accuracy and faithfulness of LLMs, particularly what augmented with an Information Retrieval component, known as RALMs (Retrieval Augmented Language Models). The paper builds upon previous works in cleverly defining 3 distinct types of irrelevant contexts (irrelevant, partially relevant and related). It also provides automatic generation methods (using prompted LLMs, Wikipedia and Knowledge Base) to generate such contexts automatically. The experiments are rigorous, conducted on two QA benchmarks and using 4 LLMs of 3 different families (GPT, Llama, Gemini). The paper is extremely well written and clearly presents and supports its scientific findings.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

A1: The paper concerns an extremely important problem, crucial to the accuracy of LLMs in an information retrieval setting. A2: The paper provides thorough experiments that clearly demonstrate its findings and contributions. A3: The paper is extremely well written and clear.

Reasons to reject

R1: A few missing technical details and discussion points. Please see my “Questions to Authors” with respect to the issues that need to be corrected.

Questions to authors

Q1: Please add a brief discussion highlighting your differences and contributions from past work that also concern the robustness of LLMs to irrelevant contexts (Xie et al., 2023; Yoran et al., 2024). In particular, why isn’t the problem of “semantically related but irrelevant contexts” already captured in the top ranked Google passages, without ground truth answers, used by Yoran et al. [1]? Q2: In Section 3.4 in the “Related Information” paragraph: the three variants of “Related Info” contexts are stated to be automatically generated by GPT-4 Turbo, based on triple of (subject, relation, object). There are two issues with this: (a) the prompts for generating the contexts are not provided in the paper or appendix and need to be included; (b) there is a concern the GPT-4 might create false contexts which can directly contradict the original questions grand truth answer. It is (very) briefly touched upon in A.3 but I strongly feel should be expanded and included in the main paper. The aspect of contradicting information should also be discussed in 3.4, with respect to the GPT-4 context generation. Q3: In Section 3.5 it is stated that Facebook’s Contriver model [2] was used as the passage retriever. To the best of my knowledge, retrievers such as ColBERTv2 [3] are considered to be a better and more up-to-date retriever. Was there a particular reason for relying on Contriver instead of ColBERTv2 or Google Search API, as was done in prior works [1, 4]? I would appreciate including the reasoning behind using Contriver in the paper. Q4: Table 2 shows that for GPT-3.5 and Gemini Pro, Unrelated Contexts drastically increase model uncertainty (UR), significantly more than the Partially Related Contexts (between 15-29 percentage points). This result is counter-intuitive, as Unrelated Contexts have less to do with the final answer than Partially Related ones. However, this UR result is not discussed anywhere in text. I would encourage the authors to include a brief discussion in the updated version. Q5: In Section 4.4 “Limitations of Current Solutions” there is no mention of fine-tuning approaches to enhance robustness to irrelevant contexts [1]. This approach should also be addressed in this section, as it seems to have been successful at mitigating part of these issues, see [1]. To be clear, I do not feel that additional fine-tuning experiments are a must. I would appreciate adding a discussion on the role of fine-tuning in “LLM robustness to irrelevant contexts”. [1] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. ICLR, 2024. [2] Unsupervised Dense Information Retrieval with Contrastive Learning (TMLR 2021) [3] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction [4] Internet augmented language models through few-shot prompting for open-domain question answering

Response to authors rebuttal

I thank the authors for their response. Based on their response, my score remains unchanged. I feel that the paper should be accepted, considering these changes will be included in the revised version.

Kind Reminder from Authors

Hi Reviewer rBEp, Sincerely thanks again for your time and efforts in reviewing our submission. As the discussion period is ending soon, we wonder if there's any possibility that you can consider having a look at our reply. We would like to hear your feedback. If you have any other questions, please feel free to follow up anytime. We are committed to making any necessary revisions to further improve our work. Authors of Paper#148