Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents

Summary

This paper proposes a framework to investigate the knowledge conflict problem in the RAG scenarios. Compared with previous work, the authors claim that they use real conflicting documents rather than synthetic information, which is closer to the real-world setting. According to this paper, LLMs are easy to update their parametric memory when external conflicting evidence is presented in the context. Furthermore, the authors find a 'parametric bias' phenomenon: It's hard for LLMs to update their parametric memory if there is parametric memory-related information in the context, although it is incorrect and unrelated to the question.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

1. The knowledge conflict is an important problem in the RAG scenario, and the authors' findings are really helpful for the community. 2. This paper is easy to follow. 3. The experiments are sufficient and the analysis is promising.

Reasons to reject

1. In stage 1, the authors mention using in-context demonstration to elicit answers. However, could this introduce demonstration bias? The authors should clarify this aspect in the paper. 2. In stage 2, the authors mention using BEM to match the answers. What is the accuracy of BEM? Can it ensure matching quality? The authors should provide clarification on this in the paper. 3. Given the widespread use of closed-source LLMs in real-world RAG scenarios, it's essential to include at least one closed-source LLM to ensure the findings can be generalized to these models. 4. Why do the authors mention "tune the prompts" in Stage 3? I've checked the appendix, but I'm still confused about this point. Does it mean the authors have fine-tuned the models, or have they simply selected one of the best prompts? If it's the former, could this introduce another training bias since the parameters have been changed? If it's the latter, the authors should reconsider the terminology. 5. The first finding doesn't seem entirely new, as it has been reported in "Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts," which is also referenced in this paper. Specifically, Xie et al. found that LLMs are highly likely to update their parametric memory as long as the external evidence is coherent. Could you please justify the difference? (I will reconsider the score if the authors can address the points mentioned above to my satisfaction.)

Questions to authors

1. Could the phenomenon of 'masking reducing the likelihood of retaining parametric answers' be attributed to low-quality masking? It's possible that LLMs are strong enough to identify such obvious misinformation?

Summary

This paper studies knowledge conflicts under a new setup: incorrect parametric knowledge conflicts with ground-truth knowledge in context. The paper discovers parametric bias, which means knowledge updates can be impeded if incorrect parametric answers appear in the retrieved documents. Extensive experiments are conducted to demonstrate this.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

1. Probing the incorrect parametric knowledge to create knowledge conflicts is novel and interesting. 2. The experiments are comprehensive and well-designed. They fully demonstrate the main conclusion. 3. The paper is well-written and easy to follow.

Reasons to reject

1. While the setup differs from previous works, the main finding seems not very novel. 2. The paper does not provide a solution. It mainly focuses on analyzing and demonstrating a phenomenon. 3. The paper emphasizes knowledge conflicts under its setup are realistic. However, in reality, the retrieved knowledge might be noisy and not necessarily be the ground truth.

Questions to authors

1. Why the happening of knowledge updates is determined by exact match? Will changing it into semantic equality change the results much?

Summary

This work proposes to study another scenario of LLM knowledge conflicts: when the LLM parametric knowledge is wrong and the retrieved document contains evidence that supports the correct answer. Experiments demonstrate that models are much more likely to shift in this setting, while model behavior might be correlated with the occurrence of parametric/incorrect answers in the retrieved document.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

+ knowledge conflict is an important research question + the alternate scenario is interesting

Reasons to reject

- I'm not sure settings in previous works where the parametric answer is right and retrieved docs are wrong are "unrealistic". Retrieval could bring noisy, outdated, or irrelevant documents into LLM prompts, so it is definitely a valid setting. What could support the setting in this work, instead of claiming it to be more "realistic", is an empirical evaluation of retrieval-augmented LMs and see how often both scenarios happen. If the setting in this work is much more prevelant, it would strengthen the motivation of this work. - According to Table 3, it seems that in the setting of focus in this work (parametric knowledge is wrong, retrieved docs are right), LLMs overwhelmingly rely on the retrieved document >95-98% of the time. This probably indicates that LLMs consistently favor the retrieved documents under these settings and we just need to improve retrieval quality, and there probably is very little to investigate about knowledge conflicts in these scenarios. - Much of the analysis (most of Sec 4.3 and 4.4) focus on the $\mathcal{R}$ cases, i.e. the parametric answer is retained. However, it only constitute a very small fraction of cases (<5% usually). I wonder if more effort should be spent on investigating the $\mathcal{U}_i$ cases where the answer changed but still wrong. - An important point is that we often don't know right and wrong at inference time for either parametric answers or retrieved documents. The findings and proposed "interventions" rely heavily on the assumption that "parametric is wrong but retrieved is right", which might not be applicable when the correctness is not known beforehand. - The intervention experiments might be a bit trivial technically, simply adding or removing the parametric answer in the retrieved document. This might not be a great issue though. - It would be nice to present some qualitative analysis and/or examples in the main paper.

Summary

Past work on LLM knowledge conflicts has shown that models struggle to correctly use in-context passages to answer the question and fall back on the parametric answer that they have memorized. Follow-up work showed that this is less true if the context is modified sufficiently to serve as convincing evidence. However, all these approaches relied on artificial edits to the context passage. This paper proposes that LLM behavior in knowledge conflicts should be tested when (1) the parametric answer is incorrect and (2) the correct evidence passage is provided. Authors find that even in this realistic scenario where the context is not artificially edited, LLMs again predict their parametric answer a certain percentage of the time. Additionally, a large proportion of contexts result in the LLM in predicting a different incorrect answer. This behavior is established across several QA datasets and for 3 open-source models: Llama2-7B/70B, Mistral-7B, Mixtral-8x7B. They further study the cause of this concerning behavior. They identify a cause they name `parametric bias': the model predicts the parametric answer again when it reappears in the context even though there is a correct answer elsewhere. That is, there is a confirmation bias where models predict their parametric answer even if it occurs in unrelated contexts.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The identified `parametric bias' is concerning, and they provide additional evidence in the Appendix. In particular, they find that other datasets, such as the FreshLLMs QA dataset, also contain context passages with parametric answers (in addition to the correct answer) - This characterization of confirmation bias is dangerous for down-stream RAG applications where users expect models to reason about the context correctly - The `parametric bias' is established through 3 experiments 1. The models make a mistake even with the gold context if the parametric answer appears somewhere in the passage frequently 2. An experiment is conducted to show that masking the parametric answer from the context reduces the number of times the models retain the parametric answer (caveat discussed later) 3. An additional experiment shows that adding the parametric answer to the context with the phrase "Unrelated context:" promotes the LLM to generate the incorrect parametric answer

Reasons to reject

- I have one major question for the authors: why did you focus on the smaller fraction of answer retention errors over the larger fraction of errors where the model predicts a different but incorrect answer? - This is concerning because this category of errors contributes to a larger fraction of errors overall - In all ablation experiments, the fraction of errors of this kind seems to be increasing but are largely ignored

Questions to authors

1. Table 3: As stated before, what is in the error category $U_i$? 2. Table 4: Can yous separate out $P(R | a_p \subseteq c)$? Why did you choose to combine $P(R \cup U_i | a_p \subseteq c)$? 3. Sec 4.4.1: Since both $P(R)$ and $P(U_c)$ are decreasing, that means that $P(U_i)$ is increasing i.e. the model is predicting neither the parametric nor the correct contextual answer. Is the model predicting the masked (edited) parametric answer instead (thus putting it in the $U_i$ set? What do yo see is the error case? 4. Sec 4.4.1: How do you draw this conclusion? The last line is the definition of $R$ and not $U_i$. “This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category $U_i$. *This suggests that the model might be resorting to its original answer in cases when it cannot find the correct answer in the context.*” 5. Sec 4.4.2: What explains the increase in $P(U_i)$ (in 7 of 18 settings) when we add back the parametric answer with the phrase "unrelated context"?

Reply to the Authors

Sincerely thank you for the detailed response. I have adjusted my score accordingly. I encourage the authors to incorporate these discussions into the next version. One last comment: If I remember correctly, Xie et al. studied both correct and false contexts. Perhaps the authors should verify this and include the differences in the next version.