IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neurons

Summary

This work presented a framework IRCAN to locate key neurons for processing contextual cues, thereby mitigating conflicts between knowledge obtained from pre-training and knowledge within the context. Experiments on completion and multi-choice tasks showed that the IRCAN benefits the base model in knowledge conflict tasks.

Strengths

1. This paper is well-written and easy to read, and the authors present their methods and experiments very clearly. 2. The paper innovatively addresses the knowledge conflict issues by manipulating neurons. Experimental results validate the effectiveness of the method. Particularly, the improvement in results on knowledge completion tasks is quite significant. 3. Ablation analysis is sufficient for understanding the proposed method in depth.

Weaknesses

1. While the paper has demonstrated the effectiveness of the proposed IRCAN on 3 datasets, its performance on additional datasets remains unexplored. It would be beneficial for the authors to include more knowledge-related datasets in future versions of the paper to further validate the generalizability of the model. 2. In the completion task, the experiments compared the accuracy of the proposed IRCAN and baselines, and the IRCAN improved the performance by a large margin. However, the experiments are conducted on merely one dataset, weakening the generalization of the IRCAN, and there is no computing comparison, for example, GPU time-consuming.

Questions

1. This paper assumed that contextual knowledge was more reliable than parametric knowledge, and what if the context itself was fake or misleading? 2. How did the authors construct knowledge datasets along with the context? 3. The paper mentioned: “To calculate the attribution score Attr(n^{l}_{i}), we gradually change the activation value of a neuron n^{l}_{i} from v^{l}_{q_{i}} to v^{l}_{(c,q)_{i}}…”, what are the details of the activation value’s change process? 4. In section 3.3, what is the object W(n^{l}_{i}) for the reweighting? Activation value or the attention matrix? 5. For the evaluation of general abilities, results in Table 3 show that IRCAN can cause other knowledge-related tasks like MMLU and Winograd to fluctuate. What could be the cause of this fluctuation? Does that demonstrate that emphasizing the context may bring knowledge degradation?

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Various LLMs were employed in this paper, however, most of the models were 7B/8B. The authors could validate the proposed IRCAN in larger LLMs to complete the work in the future.

Summary

The paper addresses the valuable problem of mitigating parametric and contextual knowledge conflicts in LLM generation with a novel and reasonable method. It is well-written, with a comprehensive experimental design showing significant improvements in completion and multi-choice tasks. However, the evaluation is limited to short contexts, raising concerns about scalability to longer contexts. The assumption that contexts are contradictory to parametric knowledge may not always hold, and the method's performance on RAG tasks and its impact on inference speed needs further exploration. Additionally, unexpected performance results between llama3-8b and llama2-7b require explanation.

Strengths

- S1: The studied problem of mitigating parametric and contextual knowledge conflicts is of great value - S2: The paper is well-written and easy to follow - S3: Though simple, the proposed method is novel and reasonable for mitigating knowledge conflicts - S4: The experimental design is comprehensive and rigorous, and the results show significant improvement in completion and multi-choice tasks - S5: The discussion and results on preserving performances on other tasks are highly appreciated

Weaknesses

- W1: The evaluation is limited to datasets with relatively short contexts, it is questionable whether the proposed method can scale to long contexts. - W2: The paper assumes that the contexts are contradictory to the parametric knowledge. However, in many cases, only a tiny fraction of the context is inconsistent with the parametric knowledge, while the others are consistent or unknown. Is the proposed method still valid under these situations? - W3: The (partly) contradiction may frequently occur in RAG, it would be great to know how this method performs on RAG tasks. - W4: It seems the proposed method has to select neurons first before answering and this involves forward passes once with and once without contexts, will this process make the inference significantly slower? Some results and discussions on time complexity would be highly appreciated. - W5: As shown in Table 1, llama3-8b performs worse than llama2-7b, which is not as expected. Is there any explanation for this? - W6: Are the salient neurons selected specifically for each question (with or without context), or are shared across data points like this: https://arxiv.org/abs/2311.15983, why or why not?

Questions

Please refer to W1-W6.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper proposed a new framework, IRCAN, to enable LLMs to pay more attention to new knowledge in context and generate context-sensitive outputs. The framework first identifies neurons that significantly contribute to context processing by utilizing a context-aware attribution score derived from integrated gradients and then reweighting these neurons. Experiments show the framework can effectively mitigate knowledge conflicts while not harming the general abilities of LLMs.

Strengths

1. The proposed framework is novel and effective. 2. The analysis of the framework is comprehensive. 3. The paper is well written and easy to follow.

Weaknesses

1. The authors state they have discussed the limitations in the checklist, but no limitation section is found.

Questions

1. Why the paper does not compare the proposed framework with fine-tuning? How will instruction-tuning the model to be more sensitive to new knowledge in context help? While the proposed framework and fine-tuning both involve updating parameters and the proposed framework costs many hours that might not be much more efficient than fine-tuning, it seems natural to consider fine-tuning as a baseline. 2. What is the performance of instructing the LLM to pay more attention to the knowledge in context in the prompt? For the multi-choice task, it seems that simply instructing the model to adopt the knowledge in the context will be effective enough, which will cost much less compared to the framework.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors state they have discussed the limitations in the checklist, but no limitation section is found.

Summary

The paper introduces a novel framework, IRCAN, aimed at addressing knowledge conflicts in Large Language Models (LLMs). By identifying and enhancing neurons that are crucial for processing contextual cues using an attribution score derived from integrated gradients, the framework significantly improves the generation of context-sensitive outputs. Tested across various models and tasks, IRCAN not only enhances model performance notably but also integrates seamlessly as a plug-and-play solution with existing models, establishing new performance benchmarks in handling knowledge conflicts.

Strengths

1. The proposed method is plug-and-play and does not need additional training. 2. The authors conduct comprehensive experiments. 3. Steering the context-aware neurons seems effective on many large language models and many down-stream tasks.

Weaknesses

1. Lack comparisons between other steering methods, like steering probed directions or steering SAE's features. 2. The propose methods will cost more inference time. 3. I think knowledge conflict also has its benefits. It may help LLMs defend some jailbreaking or harmful uses. So increase the model's faithfulness to its context may introduce some safety problems.

Questions

Who you choose neurons? How do neurons compare with probed directions (https://arxiv.org/abs/2306.03341) or features from SAEs (https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html#assessing-interp/) ?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

As mentioned in weaknesses.

Thanks for the rebuttal!

Thanks for the responses, which addressed my questions. I will retain my positive scores.