Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration

Summary

The paper introduces UniKE, a novel multimodal editing method that addresses challenges in knowledge editing for Multimodal Large Language Models. UniKE unifies intrinsic knowledge editing and external knowledge resorting by vectorized key-value memories. By disentangling knowledge representations into semantic and truthfulness spaces, UniKE promotes collaboration between intrinsic and external knowledge editing, enhancing the post-edit MLLM's reliability, generality, and locality. Extensive experiments validate UniKE's effectiveness.

Strengths

1. UniKE establishes a unified framework for intrinsic knowledge editing and external knowledge resorting. 2. Extensive experiments demonstrate that UniKE consistently maintains excellent reliability, generality, and locality

Weaknesses

1. The presentation is confusing. For example, in intrinsic knowledge editing, how do you actual edit the intrinsic knowledge in FFN. Do you follow the same pipeline as T-Patcher or adding an additional neural network? Although intrinsic knowledge is considered as key-value pairs but they are different in the end. 2. The unified framework and disentanglement of knowledge representations into semantic and truthfulness spaces introduces additional complexity to the editing process. If I understand correctly from the implementation details, it requires more than 15k additional triplets to train the encoders. 3. In the experiments, I think /beta is another hyperparameter, but I did not find an experiment discussing it.

Questions

Why would you add extra 10 key-value pairs in the FFN of the last four transformer layers in intrinsic knowledge editing? It seems that \zeta is quite sensitive. Could you explain that?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Summary

This paper proposes UniKE, a novel multimodal editing method that establishes a unified perspective for intrinsic knowledge editing and external knowledge resorting. On this basis, the authors combine both types of knowledge editing methods, executing them in the latent space with a unified paradigm. Furthermore, this paper proposes to disentangle the knowledge representations into the semantic and truthfulness spaces, effectively enhancing the collaboration between intrinsic knowledge and external knowledge resorting. Extensive experimental results show that UniKE achieves promising results under various settings, ensuring that the post-edit MLLM maintains excellent reliability, generality, and locality.

Strengths

(1) I think knowledge editing for MLLMs is a relatively new topic. Previously, a few studies merely adapted existing knowledge editing methods from the NLP field into multimodal domain. To the best of my knowledge, this paper is the first to conduct a detailed and systematic analysis of the strengths and weaknesses of existing methods when applied to editing multimodal LLMs. (2) The proposed method is very novel and effective. Previous efforts in knowledge editing show significant differences between intrinsic knowledge editing methods and external knowledge resorting methods. In this work, the authors ingeniously convert in-context editing into the format of feature shifting, achieving a unification of the editing paradigms that can operate simultaneously within the same transformer layer with synergistic correlation. I find this to be a very inspiring design. Moreover, the design of knowledge collaboration is closely integrated with this unified paradigm. (3) The experiments are very solid and thorough, clearly demonstrating that UniKE effectively addresses multimodal knowledge editing tasks under various setups. Meanwhile, the authors have also provided the implementation code for the experiments. (4) I commend the authors for conducting an extensive set of ablations and analyses, which are very helpful in understanding the impact of each component within UniKE. (5) Additionally, I believe that the method proposed by the authors is not only applicable to knowledge editing tasks. By converting in-context learning into the representation space and avoiding the need to increase the context window space, it better synergizes with parameter update learning. I consider this to have significant implications for further studies on how to construct more powerful MLLMs.

Weaknesses

(1) In the NLP community, some studies will discuss the resilience to overediting [1] of knowledge editing methods by adopting the contrastive knowledge assessment [2] (CKA). Unlike the locality property that measures whether LLMs forget previous knowledge, overediting can be understood as excessive generalized to seemingly similar but unrelated samples. Although there may be no current work on multimodal editing that discusses the phenomenon of overediting, I encourage authors to add relevant experiments for a straightforward comparison of the resilience to overediting among each method (UniKE, MEND, T-Patcher, and IKE). (2) A more challenging task of knowledge editing is counterfactual editing, where the edited answer $y$ to the question $x$ can sometimes be counterfactual to the real world. A typical counterfactual editing dataset in the NLP community is called COUNTERFACT [3], which more accurately reflects the true effectiveness of knowledge editing methods by avoiding the effects of LLMs knowing this knowledge before editing. I encourage authors to construct multimodal counterfactual editing datasets and conduct more experiments to verify whether UniKE performs better in counterfactual editing scenarios compared to MEND, T-Patcher and IKE. [1] Zheng, Ce, et al. "Can we edit factual knowledge by in-context learning?." arXiv preprint arXiv:2305.12740 (2023). [2] Dong, Qingxiu, et al. "Calibrating factual knowledge in pretrained language models." arXiv preprint arXiv:2210.03329 (2022). [3] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

Questions

See the weaknesses.

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately discussed the limitations and potential negative societal impact.

Summary

UniKE is a unified framework for multi-modal knowledge editing that includes three main aspects: 1.Knowledge Separation: UniKE divides knowledge into factuality and semantic spaces to manage and coordinate different types of knowledge more effectively. 2.Knowledge Collaboration: In the factuality space, UniKE standardizes new knowledge based on a learned factuality distribution, enhancing reliability and generality. In the semantic space, it adjusts the integration of external knowledge based on relevance to the input samples, maintaining locality. 3.Multi-step Editing: UniKE supports single-step, multi-step sequence, and cross-task editing while maintaining high reliability, generality, and locality. Through these innovations, UniKE significantly improves performance in multi-modal knowledge editing tasks.

Strengths

1.The paper introduces UniKE, a novel framework that seamlessly integrates intrinsic and external knowledge editing, enhancing the model’s ability to handle complex multimodal information effectively. The motivation behind the work is clear, and the article is well structured, guiding the reader through the innovative approach and its benefits. 2.By disentangling knowledge into semantic and truthfulness spaces, the proposed method ensures robust collaboration between different types of knowledge, significantly improving the model's reliability, generality, and locality. The method's effectiveness is demonstrated through comprehensive experiments across various settings, consistently outperforming existing state-of-the-art methods. 3.UniKE's design allows for application across different multimodal models and editing scenarios, making it a versatile and robust solution for enhancing multimodal language models.

Weaknesses

1.The paper does compare UniKE with other intrinsic knowledge editing and external knowledge resorting methods and highlights its efficiency in several aspects. However, it lacks a detailed discussion on computational speed and resource utilization. 2.The reliance on fine-tuning for knowledge updates could lead to overfitting, especially if the model is frequently updated. This could impact the model’s generalization abilities, making it less effective in unforeseen or less frequent scenarios. 3.The paper lacks detailed explanations for the evaluation metrics in Line 240. In Table 2, across multiple experiments, it is unclear why there is little difference compared to the SERAC method in the T-Loc metric, but a significant difference in the M-Loc metric. 4.The paper lacks detailed information about the parameter "n" used in the contrastive learning formula, making it difficult to understand its impact on model performance, and does not discuss how different values of "n" might influence the results and effectiveness of the method.

Questions

Please address the comments in the weaknesses section.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations of the article have been discussed by the authors

Summary

This paper proposes UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Within such a unified framework, the authors further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of UniKE, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality.

Strengths

1. The paper is well-written and easy to follow and the motivation is clear and reasonable. 2. The paper gives a unified perspective on the intrinsic refinement of knowledge and the strategic reorganization of external knowledge, which can enhance subsequent research endeavors. 3. The experimental settings (one-step editing, sequential editing, cross-task editing) are fair.

Weaknesses

1. The authors only edit Qformer style MLLMs (MiniGPT4, BLIP2), more foundation models should be compared. e.g., LLava. Besides, the improvement on BLIP2 (Tab.1) seems incremental. 2. More recent model editing baselines should be compared to prove the effectiveness of the proposed methods. 3. The paper focuses on MLLM editing, what are the main differences between MLLM editing and LLM editing? What are the specific designs for multimodal models? I notice that some compared methods are proposed for LLM editing. Can the proposed methods be used on LLM editing? The authors should give some explanations and experimental results if possible.

Questions

Please refer to the weakness below.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper briefly mentioned limitations.

Official Comment

Thanks for your response. The Rebuttal addresses most of my concerns. I will raise my score to 7.