Summary
The authors introduce a novel method, named MINER, to identify Modality-Specific Neurons in Multimodal Large Language Models (MLLMs). This approach comprises four main stages: 1. **Modality Separation**: The method begins by assuming that information within each modality's token set predominantly stays within that set, implying that cross-attention between modalities plays a minor role in MLLM tasks. 2. **Importance Score Calculation**: Instead of assessing the importance of individual activations for each sample, the method aggregates activations across tokens sets within a modality. The authors propose five distinct aggregation techniques for this step. 3. **Sample-Level Aggregation**: These aggregated importance scores are combined to generate a sample-level importance score. 4. **Neuron Selection**: Finally, neurons are ranked based on their importance scores, allowing for the selection of modality-specific neurons. This method offers a systematic approach to pinpoint neurons that are critical to individual modalities within MLLMs, advancing our understanding of how these models handle multimodal information.
The authors should the efficacy for their method on multiple tasks and multiple datasets.
Strengths
The proposed methodology has several notable strengths:
1. It explores the relatively unexplored area of identifying modality-specific neurons in Multimodal Large Language Models (MLLMs), offering insights into the internal workings of these complex models.
2. The authors provide comprehensive results across multiple MLLM architectures and datasets, accompanied by extensive ablation studies and in-depth discussions, demonstrating the robustness and applicability of their approach.
Weaknesses
In my view, there are several weaknesses in the paper's approach:
1. **Modality Separation Assumption**: The assumption of modality separation, as defined in the paper, is overly generalized and lacks robustness.
- **Dataset and Task Specificity**: Modality separation can only be reasonably argued for certain datasets and tasks. For example, it might be valid for specific questions in Visual Question Answering (VQA) but is unlikely to hold across all question types. Moreover, it seems especially problematic to assume modality separation for captioning tasks and datasets, where contextual understanding of the entire image is essential.
- **Dependence on Output Context**: Another critical consideration is whether modality separation should be treated as a function of the output. In captioning tasks, the entire image often correlates with the output, as generating captions requires broader contextual information. In contrast, the input (or prompt) for caption generation may hold less significance, further challenging the assumption of modality separation. Could you, if possible, conduct an ablation study to assess the significance of the output modality in the importance calculation?
- **Question on Importance Calculation**: This brings up a key question regarding the calculation of neuron importance: is the importance score determined solely based on the input, or does it also consider the model's output? If not, would including the model’s output in this calculation lead to different importance scores?
2. *Missing Definition*: H_L is not defined in section 3.2. No reference is present in any of the equation 2 or 3 on the same page.
3. **Dataset Size Impact**: Table 2 illustrates that different datasets yield different neuron importance scores. However, it's not clear if varying the quantity of data within a single dataset also impacts these scores. For example, if using 100% of the dataset identifies important neurons \(N = \{n_1, n_2, n_3, n_4\}\), would using only 50% or 33% of the dataset result in a different set of important neurons, or would it merely be a subset of \(N\)? If the importance neurons change, are they still importance neurons, or is there any underlying cause which we are not able to see.
4. **Computation Requirements**: The paper lacks an analysis of the computational resources needed to determine neuron importance. Since high-performance compute resources can be limited, an estimate of the required computational effort is essential to evaluate the feasibility of this explainability method.
Questions
**Questions and Suggestions for the Authors:**
1. Could you add a definition for \( H_L \) and clarify what is meant by the “hidden activation function”? This addition would enhance the readability of the paper. Additionally, Table 3 currently appears a page before Table 2, which disrupts the logical flow.
2. Please clarify the implementation details for Stage I (modality separation) in your proposed method. Refer to the outlined weaknesses for specific questions and concerns. Additionally, is there supporting empirical evidence for this assumption in the existing literature?
3. Please consider adding a section detailing the computational requirements of your approach. Address Weakness 4 by including specifics such as GPU hours, memory demands, and a comparison with other commonly available explainability methods. This will provide the readers with a clearer understanding of the computational resources needed for your method.
4. Please conduct an ablation study using datasets of varying sizes to analyze whether the neurons are a subset of the original set. Refer to Weakness 3.