Summary
This paper proposes an algorithm for the unlearning of the visual recognition of concepts from single images in Multimodal Large Language Models (MLLMs) and introduces a benchmark dataset for evaluation. To achieve single image unlearning in MLLMs, the following process is conducted. First, based on the given image, Multifaceted Fine-tuning data is generated for four purposes: Aligning with Unseen Concepts, Assigning New Visual Descriptions, Decoupling Factual Knowledge, and Preserving Non-targeted Knowledge. Second, to ensure effective unlearning using the generated fine-tuning data, a Dual Masked KL-Divergence Loss, which considers token-level and vocabulary-level masking, is proposed. Additionally, the proposed MMUBench is designed as a dataset to evaluate unlearning the visual recognition of concepts from a given single image. Several metrics such as Efficacy and Generality are suggested to assess the success of unlearning. Finally, experimental results using both the proposed algorithm and existing baselines on MMUBench demonstrate that the proposed algorithm achieves superior results.
Strengths
1. As far as I know, this paper is the first to define the problem of unlearning in MLLMs and propose an algorithm and benchmark dataset for evaluation.
2. The algorithm proposed to achieve the unlearning goal set in Line 120 is well-designed with appropriate motivation to ensure successful unlearning.
3. Additionally, the proposed MMUBench offers a well-constructed dataset and various evaluation metrics to assess the achievement of the unlearning goal.
4. Finally, experiments using MMUBench have experimentally validated that the proposed algorithm achieves superior performance in MLLMs compared to existing algorithms.
Weaknesses
The weaknesses of this paper are as follows:
1. The structure of the paper makes it difficult to understand the entire content smoothly. Many crucial details are located in the Appendix, which hampers the reading flow. For example:
1.1) In Section 4.1, the authors refer to Figure 7 in the Appendix for explanation, but this information is also presented in the middle image of Figure 1. It would be better if the authors used Figure 1 to explain as much as possible.
1.2) Many parts that should be in the main text are in the Appendix. For instance, the motivation for DMK in Lines 164-165 should be mentioned in the main text. Additionally, a brief introduction to the MMUBench dataset in Lines 183-184 should be included in the main text. Even if it means reducing content in other parts, including these details in the main text would significantly improve the readability of the paper.
2. Are all the experimental results in the main text reported as the average results of multiple random seeds? It is necessary to conduct experiments with at least five seeds and report the standard deviation.
3. Lastly, what is the reason or motivation for targeting the unlearning of the visual recognition of concepts rather than factual knowledge in MLLMs? What are examples where the proposed unlearning goal is needed in the real world? The authors should include this information in the main text. Personally, although I am not an expert in MLLMs, I believe that unlearning factual knowledge might be a more necessary form of unlearning for MLLMs in the real world.
Questions
Please refer to the Weakness section. I positively evaluate the paper for defining the unlearning goal in MLLMs and proposing the corresponding algorithm and benchmark dataset. However, the paper's structure needs improvement, there is a concern about whether the experimental results consider multiple seeds, and there is a lack of explanation regarding the necessity of the proposed unlearning goal in the real world. These issues make it difficult to give a higher evaluation. If the authors address these concerns appropriately, I would be happy to raise the score.
Limitations
There is no potential negative societal impact of this paper.