Summary
This paper introduces an explainability approach for interpreting the internal representations of large multimodal models (LMMs). The authors train an image captioning model consisting of a pretrained image encoder and language model and a connector model. To extract interpretable representations, the authors use dictionary learning, decomposing representations into lower dimensional $U$ and $V$ matrices using semi non-negative matrix factorization as the optimisation objective. To interpret concepts in $U$ in the textual domain, the authors use the language model unembedding layer to extract the highest probability tokens associated with a given concept vector $u$. Likewise, to visually interpret a concept vector $u$, the authors find the set of images that maximally activate $u$. The authors provide quantitative evidence demonstrating that their method generates concepts which are well aligned with both the input image and the ground-truth captions. Additionally, they demonstrate their approach qualitatively produces well defined concepts with limited overlap between the tokens represented by different concepts.
Strengths
1. This paper introduces a promising novel approach to performing mechanistic interpretability on multimodal models. Given the increasing ubiquity of multimodal models there exists a clear need for interpretability approaches for this style of model. This work presents a viable dictionary-learning based approach I look forward to seeing other researchers building upon in the future.
2. The evaluation of the framework is comprehensive including both qualitative and quantitative results, both of which are essential for any interpretability method. Additionally, I was impressed to see the authors consider multiple approaches (e.g. PCA/KMeans) and sensible baseline models when evaluating their results, providing more confidence that the final approach taken was an appropriate method.
3. The paper is very clearly articulated, with the rationale well-defined, the approach clearly outlined and the results succinctly and clearly summarised.
Weaknesses
1. The authors provide good evidence that the concepts extracted by their method show minimal overlap with other concepts, however they do not address the alternative possibility, that their extracted concepts might represent more than one distinct concept. This phenomenon of feature “superposition” has been well documented in other model interpretability work (see [1][2][3]) and so it seems plausible that it may arise in the approach taken here. This seems especially likely given that the dimensionality of their concept dictionary is lower than the dimensionality of the internal representations. Though I do not think this should detract from the otherwise excellent contributions presented in this paper, I do think this at least warrants a brief discussion, and perhaps more qualitative analysis of the extracted concepts, to assess whether any evidence of feature superposition is observed.
2. Occasionally the authors make claims that go beyond the scope of this work. For example, in the abstract they state, “we present a novel framework for the interpretation of LMMs”. However, this claim seems too strong given that this approach is only really valid for image captioning models rather than large multimodal models more generally. Additionally, there are a few comments such as “we find the generalization of LLMs to multimodal inputs is an interesting phenomenon to understand” in 3.1 (under “Training”) and “the multimodal structure of internal token representations starts to appear at [later layers]” in 4.2 (under “Layer Ablation”). However these claims do not seem valid as the language model layers are frozen during training. It seems more appropriate to say that these extracted concepts represent language model concepts, and the connector model learns to transform the image representations such that they align with these language model concepts.
[1] Elhage, N, et al. "Toy Models of Superposition." arXiv:2209.10652 (2022)
[2] Arora, S et al. "Linear Algebraic Structure of Word Senses, with Applications to Polysemy." arXiv:1601.03764 (2016)
[3] Elhage, N, et al. "Softmax Linear Units" Transformer Circuits Thread (2022)
Questions
1. Why did the authors choose to use Opt-6.7B as the language model rather than more powerful similar sized models such as Llama-7B?
2. Why do the authors take the absolute activations of $V$ in (5)? Are these activations not guaranteed to be non-negative by the optimisation objective?
3. Did the authors consider trialling gradient-based feature visualisation approaches [1] in addition to taking images with the highest activations? This could be an alternative approach to build additional confidence that the extracted concepts do represent what qualitative analysis of the highest activating samples appears to suggest. I don’t expect the authors to perform analysis of this kind in this manuscript however it could be worth commenting on this as a future avenue of research, or alternatively raising any valid criticisms of this approach?
4. There is a typo in figure 1 (“Captionin”)
[1] Olah, C, et al. "Feature Visualization", Distill, 2017.
Limitations
The work presented here is only for a single, relatively small model, trained with a specific objective (image captioning). As such it is not clear that this approach will necessarily generalise to other multimodal models. The authors should touch on this limitation in the discussion or limitations section in the appendix.