Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

Summary

Whilst useful for many downstream tasks, CLIP’s vision-language representations are notoriously hard to interpret. The paper proposes to represent CLIP representations in a sparse, non-negative overcomplete basis of learnt interpretable directions using a standard dictionary learning technique. Not only is interpretability increased, but the core claim is that this does not harm the performance of the downstream applications significantly. Such a model would provide interpretability “for free”, in the sense that practitioners could opt for the interpretable version without any performance cost (which would constitute a significant achievement). --- Post-rebuttal: Following a productive discussion with the authors (who both clarified many experimental details and proposed revisions to the claims of the paper to better reflect its results), I think the paper will be of great value to the community and recommend its acceptance.

Strengths

- The work strives to address a vitally important task: to design models that offer interpretability at no cost to performance. Such an endeavor is particularly important practically in allowing practitioners to adopt interpretable models in practice (by not trading off performance). - The paper is very well-written, the applied methodology is appropriate, and the focus on CLIP representations means the paper has potentially great relevance and reach to practitioners. - Offering more than just transparency, I appreciate the method's multiple use cases--not only for discovering spurious correlations but also for model editing.

Weaknesses

# [W1] Sparsity/interpretability does trade-off accuracy I was very excited about the paper’s bold claim in the introduction to `provide interpretability, at no cost to downstream performance` ([L6], but also at [L349]) upon a first read. Unfortunately, from what I came to understand later from the results of the proposed dictionary in yellow plots in Figure 3, this claim appears false. There *is* a cost to zero-shot classification performance (which is arguably one of the most important downstream applications of CLIP). For example, small L0 norms lead to as much as ~10% accuracy drop on CIFAR100 (left-most plot), and what appears to be a ~20% performance drop on ImageNET1k using the raw dictionaries. A similar pattern of performance degradation for high sparsity is observed in Fig. 7 for retrieval tasks. Thus, it is simply not true that the method is at “no cost” to performance, and arguably it is not even at “minimal” cost (used later in [L52]). These results unfortunately do not support the paper’s core claims. Crucially, if I correctly understand the x-axis as quantifying the number of non-zero coefficients, this is even more problematic. High sparsity is desirable for interpretability [L164]—but we see from the experiments that too small an L0 term trades-off accuracy, and sometimes rather significantly so. This fundamental trade-off is a clear critical limitation of the work (in conflict with the paper's core claims) and puts interpretability and accuracy at odds—but it is not ever stated or discussed as a key limitation of the work. The authors should revise this core claim in light of the fact there is indeed a cost to performance increasing as a function of sparsity, and include a dedicated discussion of such limitation. Furthermore, I would expect to see many more experiments on the standard zero-shot datasets used in the original CLIP paper [1] to evaluate this core claim, not just on 3 (given this is zero-shot, these experiments are very fast to run with pre-trained dictionaries). This weakness in particular swayed my rating of this paper negatively, and I am happy to consider revising my score accordingly, should an adequate answer be provided in response. # [W2] Limited new technical contributions, but also lacking technical insights/comparisons to alternative existing solutions The paragraph in [L229] is rather brief and states that sklean’s Lasso solver is used. In the concrete application of sparse dictionary learning for CLIP representations in the paper, a representative reader would be very interested to see a much richer exploration of alternative approaches to solving the model objective. For example, in CLIP, how does this proposed solution compare to non-negative KSVD? How about NMF (with a sparsity penalty)? Or sparse autoencoders? Or learning this end-to-end with projected gradient descent (to project the coefficients onto the non-negative orthant), etc. Given that the paper does not offer any novel technical contributions (which is okay!), I think the paper would have been much stronger if at least some technical insights (about existing techniques) as they pertain to representation learning with CLIP were provided. At a minimum, a *discussion* of some of the other techniques listed above (and why they might not be appropriate) would be desirable, and experiments comparing the results from each to validate the proposed solution would make the paper’s contributions even stronger through technical insight. # [W3] Unnecessary mathematizing (minor) As a reader very excited about this paper’s application, I feel the second half of Section 3. “When sparse decompositions exist” is an unnecessary formalization ([L141] onwards). I’m unconvinced of the need to introduce “Proposition 1”—at minimum, it feels like a distraction from the other interesting results in the paper. As far as I can see, this proposition is not used at all in the paper, and no experiments are conducted to show the 5 assumptions are reasonable (aside from Assumption 3). The “assumptions” 1-5 listed do appear useful heuristics to reason about when Sparse decompositions are appropriate. As the authors also state [L133], concept linearity seems most crucial here. As a reader, what I care about is how these assumptions hold in CLIP in practice. As such, it would be much more useful to see instead the key experiment in Appendix B.5 discussed here in the main paper to support this critical assumption. To summarize this final weakness: my impression is that the paper’s clarity could be improved by (a) considering deferring the formalizations to the supplementary material, or reducing their significance/length in the main paper and (b) considering making the critical experiments validating the key assumption in Appendix B.5 more prominent. I believe the paper would be much more impactful and have greater reach if the key results were better highlighted, and math that is not strictly necessary dropped. --- * [1]: Radford, Alec et al. “Learning Transferable Visual Models From Natural Language Supervision.” International Conference on Machine Learning (2021).

Questions

# [Q1] Vocabulary choice and dataset dependence I’m not sure I would agree with the claim that the `efficacy of the decomposition is (in principle) independent of individual datasets`. Isn’t the vocabulary choice made in this work specific to LAION-400m? Perhaps using the WordNET nouns/adjectives as the vocabulary (and bigrams formed by common combinations) would be even more dataset/task agnostic? # [Q2] Mean centering As stated by the authors, the CLIP representations live on a hypersphere. Instead of taking the geometric mean in Euclidean space (as part of addressing the modality gap), did the authors explore taking the Frechet mean? This seems like it would better account for the geodesic distances between the points. # [Q3] Zero-shot performance in the limit What happens to the zero-shot performance in Fig. 5 in the limit of the l0 norm being equal to the dimensionality of the space? Shouldn’t we expect matching performance to the CLIP baseline? It would be insightful to extend the x-axis to observe how it performs in the limit.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

No, limitations are not adequated addressed, and the current limited discussion should be placed prominently in the main paper rather than the supplementary material.

Summary

This paper introduces a method for building sparse embeddings from CLIP in order to improve the interpretability of CLIP’s latent space. They formulate their objective as a sparse reconstruction problem, and under certain assumptions, demonstrate when finding a sparse representation is possible. Empirically, the authors show over three datasets that zero-shot accuracy and reconstruction errors are correlated, and performance is similar with sparse embeddings vs. without. They further illustrate the benefits of sparse embeddings for interpretability noting applications to reducing spurious correlation reliance on a Celeb-A benchmark, and identifying biases through representations.

Strengths

- This paper presents an important study toward understanding the latent space of vision-language models like CLIP which has become a very popular paradigm and backbone for many downstream tasks. This can help to improve the next generation of such models in training and downstream use. The use of text to interpret to solely understand the vision space and perform interventions is also important and makes the method relatively lightweight. It also makes the method extendable to researchers and practitioners outside of the space as only concept vocabulary is needed. - The paper is overall well-written stating clearly the assumptions of the work in Section 3, and the proposed method is also clear and intuitive to implement.

Weaknesses

- The primary weakness of this paper is that it is unclear to me what improvements come from generating sparse CLIP embeddings. Results indicate that there is no improvement in zero-shot accuracy due to some outstanding reconstruction error. In contrast, when removing spurious correlations it is typically seen as an improvement to model performance as the model would typically in this case then be evaluated in the OOD setting where the correlation is not present. From this perspective, I think the authors might benefit from changing to a setting more similar to this, where the results are more aligned with benefits from having an interpretable tunable mechanism, rather than the more ID zero-shot performance settings. This is started in the model editing, however it doesn’t lead to much improvement from correlation reliance, only mitigation of the capability to classify another attribute. In contrast performance goes down indicating some capability but perhaps not the right evaluation setting. - Similarly, I am uncertain about the case study evaluations for discovering spurious correlations. Results for the man-woman evaluation appear to only indicate the trend is present in 70/600. This seems like a relatively low number as the majority then do not have this bias. It is unclear what the impact would be in terms of downstream performance for classification on CIFAR-100, or a gender classifier trained on CLIP embeddings. The study seems inconclusive due to the results not being well contextualized 70 out of 600, and the lack of a downstream performance evaluation and improvement. - Finally, in multiple places, the authors mention improved human interpretability such as 244, 277, etc. however the authors never conduct any human evaluations. If the authors claim improved human interpretability, the authors should conduct such a human evaluation, otherwise it is only hypothesized improved human interpretability.

Questions

Have the authors demonstrated an evaluation setting where having the more interpretable concept features would improve performance? I believe some OOD setting such as spurious correlation could improve, but there may be many setting this is worse, as you are reliant on the concept features and may be worse for settings such as fine-grained classification where the concepts were not covered.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

- An additional limitation of the work not discussed by the authors is the reliance on writing a set list of word-level concepts. The approach is reliant on this set list having the desired concept when detecting spurious correlations, etc. As the authors have pointed out this also limits the type of concepts. - The authors have also not discussed how practical the assumptions proposed in Section 3 are, and whether they are correct in practice. This is important for demonstrating applicability of the proposed work.

Summary

This paper presents a method to explore semantic concepts in multimodal models of text and images. Specifically, the paper formulate semantic concept discovery problem as one of sparse recovery and build a novel method, Sparse Linear Concept Embeddings (SpLiCE), for transforming CLIP representations into sparse linear combinations of human-interpretable concepts.

Strengths

1. The paper is well written and reads well. 2. The technical method is generally make sense . 3. The contribution is enough for covering text-image multi-model method to an interpretable model, and the research topic is worth pursuing. 4. Sufficient experiments demonstrate the effectiveness of the proposed simple method. 5. The experiment designed in the paper is make sense.

Weaknesses

see questions

Questions

1 The connection and difference between your method with a multi model topic model? In my opinion, topic model is good at extracting interpretable concepts. 2 Although a lot of experiments were conducted in the paper, I think how to judge the concept discovery results is still not convincing enough. Whether to add manual evaluation experiments will be more convincing. 3. Why does negative concept weight perform better in Table 5? What does this mean?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness and question

Summary

This paper introduces a method to transform CLIP representations into sparse linear concept embeddings that are interpretable to humans. SpLiCE uses task-agnostic concept sets, demonstrating its versatility over prior works. SpLiCE provides interpretability without compromising zero-shot classification performance and shows further applications, including spurious correlation detection and model editing.

Strengths

SpLiCE suggests a novel method to interpret CLIP embeddings into semantic concepts, which could be a good way to understand the latent space of CLIP. As it is a task-agnostic approach, which does not constrain itself to a certain domain, it can be applied to various datasets to explore the CLIP embeddings. This task-agnostic concept shows its scalability across different datasets, including CIFAR100, MIT States, CelebA, MSCOCO, and ImageNetVal. Also, the authors demonstrate the efficacy of their method both quantitatively and qualitatively.

Weaknesses

[W1] I believe the choice of datasets is insufficient to show the efficacy of "task-agnostic" concpet sets. For example, how does SpLiCE work on bird identification dataset or CelebA? What are the top activated concepts in these datasets? [W2] Adding class labels to concept dictionary seems inappropriate from the perspecive of "concept decomposition," as in the case of ImageNet. If the top concept is class label itself, what is the need for concept decomposition? [W3] A user study on the interpretability of SpLiCE would be needed to quantify how interpretable this method is, as in [23]. [Miscellaneous] I think line 96's citation [22] should be changed to [23].

Questions

[Q1] Does the concept decomposition differ between correct and wrong samples (from the perspective of zero-shot classification) ? [Q2] As the effectiveness of SpLiCE heavily relies on the quality and comprehensiveness of the concept dictionary derived from the LAION dataset, is there a possibility that limitations or biases in this dictionary could directly impact the performance and interpretability of the method?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Please refer to the weaknesses and questions part.

Summary

This paper proposes to decompose the representations of the CLIP model using dictionary learning; where the components of the dictionary are composed of human understandable concept directions. The procedure here is as follows: a concept list is constructed from a filtered set of unigrams and bigrams from the LAION-400m captions, the concept dictionary is set to be centered CLIP representations for each pre-specified concept in the list, and an optimization problem that minimizes the reconstruction loss (subject to an L-1 penalty) is specified. Solving that optimization problem results in a weight vector that indicates how the CLIP embedding can be decomposed along the pre-specified concept direction. Given this formulation, the paper solves the problem for the CLIP model and shows that it does not lead to a performance loss given thousands of concepts. In addition, the paper presents several case studies showing how the approach can be used to edit downsteam classifiers built on top of clip representations, and discover spurious correlations.

Strengths

**Originality**\ This paper is the first that I know to decompose the representations of CLIP with known concepts using dictionary learning. The work is well motivated. **Quality and Clarity**\ The paper is well-written, and clear. The related work is also discussed, and each component of the workflow presented here is well-explained and clear. **Significance**\ This work, in my opinion, indicates that CLIP type models can be easily debugged using post hoc supervised concept learning. Overall, I think this opens up the door to more intriguing work along similar lines for larger and more complicated models.

Weaknesses

Overall, I discuss some of the challenges that I see with this work. I have not much substantive qualms with this work; I just think the presentation could be tightened. However this is up to the authors to either accept or reject. **Assumptions for the proposition not justified**: While the assumptions made to prove proposition 1 seem natural at first glance, it is not clear to me that we can claim that they should be true. One, there is a claim that CLIP captures 'semantic' concepts and not 'non-semantic' concepts (assumption 2). In Section 5.2, the authors even confirm experimentally that the assumption is not true. Of course, the model does capture some semantic concepts, but we cannot claim that CLIP cleanly partitions these concepts as described. This is because we know of failure cases where CLIP errs precisely because it does not just capture only semantic concepts. See this paper for example (https://arxiv.org/abs/2306.12105). The failures of the CLIP model that have been observed make it clear that this assumption is not true. Assumption 3 is somewhat problematic again because we have observed that CLIP representations indeed can be linear for *some* concepts, but it is not linear for all semantic concepts. In fact, how are we to know which concepts that CLIP considers to be semantic, and which it doesn't? Assumption 4 makes sense to me because CLIP is trained to 'align' both encoders. Assumption 5 also difficult to justify in my opinion, but I am willing to live with it. To summarize my challenge with this portion of the work, the proposition is correct, but we can't claim that CLIP behaves as described. I am bothered by these assumptions because the implication is essentially that CLIPs learns the data generating process defined here, which is not the case. This is actually a good thing because, as shown quite nicely in this work, the point of the concept dictionary 'layer' is to 'fix' the challenges with the original CLIP model. Overall, I don't think this section of the paper can be fully justified. **task-agnostic and without concept datasets, training, or qualitative analysis of visualizations.**: The bolded phrase is one of the motivations for the approach described here. This is not quite true. First, you need to collect the concept dataset, the processing done with the LAION dataset is *exactly* the process of collecting a concept dataset. You do need to 'train', in the sense that you can see the process of optimizing to obtain $w$ as training. Lastly, the layer results in concept scores that can be visualized as depicted in Figure 4, so the approach here also allows us to inspect . Overall, I think this claim is hyperbolic and should be relaxed. **Related Work and Discussion**: This work claims that it is not quite a concept bottleneck, but I disagree. It is exactly translating a concept bottleneck to CLIP. First, the methods does require annotations. To get the dictionary $C$, you need to collect a set of 10000 concepts, which are associated with particular images. The annotation process is exactly the process of manually constructing $C$. The concept list is supervised, meaning it is the authors that determine which concepts to include; of course, the process described here is not manual, but the authors should recognize that it is *still* a supervised process. The intervention procedure described in Section 6 is exactly how that is done in the CBM literature. In fact, the NMF formulation here is not new (see: CRAFT: Concept Recursive Activation FacTorization for Explainability for eg.). Overall, the approach described here is exactly what a CBM is, it is just not applied for classification. Having said all of this, I think it is fine to acknowledge the similarity.

Questions

**Unsupervised or residual concepts**: I am surprised that there wasn't any significant loss in performance between the Splice representations and the original CLIP representations. Is it that 15000 concepts is enough to capture the variation? It seemed like one would want to account for the 'residual' directions in the original embedding that the reconstruction doesn't account for. Is this needed? Or do the authors think we can always get away with this kind of supervision?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Appendix A discusses some of the limitations of this work.

Thanks to the authors; remaining question about probing in C.1

Thanks to the authors for their thorough reply! I appreciate the authors’ extra results, technical discussions, and re-organization of the paper based on the comments. I like this paper and appreciate its goals. To be clear, I think it’s reasonable to expect such a performance degradation for high sparsity levels. But the fact this happens should be made crystal clear, or else the paper runs the risk of being inadvertently misleading through the current claims in the abstract (and main paper). Some responses to the authors’ rebuttal: **Zero-shot performance** It is indeed helpful to interpret the results of Figure 3 again with the understanding that a sparsity level of below 32 is preferable according to human studies—similar to how it already appears in the discussion of related work, it would be insightful to include a brief discussion of this previous papers’ result to help interpret Figure 3 inline. I would respectfully disagree with the authors about the significance of the accuracy drop however: ~30 is near the upper bound of desirable levels of sparsity according to [49]. But this already brings a 4% drop in accuracy on a dataset as simple as cifar100--this seems to me a rather notable degradation. I don’t think one can use the word “small” to describe this. **Probing experiments** I do not quite understand the experimental setup in Appendix C.1. Wouldn’t “retaining performance” of CLIP mean that we match the performance of a linear probe trained on the CLIP embeddings’ training set and also evaluated on the CLIP embeddings’ test set? These are the capabilities we wish to retain. **Re-adding the residual error** This is indeed a smart idea for recovering performance. As the authors state though, the residual vector will remain uninterpretable. Therefore, I’m not convinced that this is a viable argument for the method offering interpretability whilst retaining performance.

Great thanks to the authors!

I appreciate the efforts the authors have made to address my concerns! Most of my concerns are well addressed, so I'll raise my support. I hope a more thorough user study will be conducted afterward. Thanks!

Discussion Summary

We would like to thank the AC for facilitating the review of our paper and the reviewers for providing constructive feedback and engagement during the discussion period. In this work, we propose SpLiCE, a method that leverages sparse coding to interpret CLIP embeddings by representing them as sparse combinations of semantic concepts. Our interpretable CLIP embeddings maintain high performance and unlock novel use cases, including bias detection, data summarization, distribution shift monitoring, and more. The reviewers' main concerns were with the lack of human evaluations, the limitations of our dictionary, and our theoretical analysis. We accordingly provided results from a small-scale user study following the suggested format of prior work, which highlighted the human interpretability of our method. We also further elaborated on the limitations of our proposed dictionary and discussed changes to our theoretical analysis in Section 3 of the paper. Following these rebuttals, reviewer abcD noted “most of [their] concerns were well addressed” and reviewer djnP “believe[s] it would be of great value to be presented at the conference”, and they increased their scores to a 6 and a 7 respectively. Reviewers nCHL, jbkR, and Vxod have not responded to our rebuttal as of yet. We will be sure to incorporate the changes suggested by the reviewers into our camera-ready version. We also intend to expand our user study in this time, as suggested by reviewer acbD.

Understood

**Theoretical Analysis**: "we can reasonably conclude that CLIP may approximately satisfy assumptions 1-5." I think even this point about sparse decompositions is probably not true, but I have no way to refute it. I think what you have showed here is that given a pre-specified basis, then you can find a decomposition of CLIP's representation in that basis. Is one with 10k directions sparse? I don't know. Overall, I think the appeal to the linear representation hypothesis stuff is not quite clear to me. I still think even the experiments in this paper show that you can't make the claim that you seem to be making here. However, I am willing to let this go since paper reviews are public, and the community can evaluate the claim itself. **CRAFT et al/NMF**: I don't agree with the point that you require feature visualization for CRAFT or NMF in general. I am not sure about CRAFT, but for traditional NMF there is no explicit need for feature visualization unless I am misunderstanding what you mean by feature visualization. One could argue that Fig. 4 is exactly the type of visualization most NMF methods use, which you also use here. Overall, I'll be keeping my score as is.

Reviewer Response to Author Rebuttal

Thank you for the comments and addressing many of my concerns. My primary concerns with the paper were the claims that (1) the representations and concepts were human interpretable, and (2) there was no decrease in performance. I see that concern (1) has been addressed, however I believe (2) has not been addressed sufficiently in the rebuttal and is still a clear limitation of the proposed work. Nonetheless, I am increasing my score following the addition of (1) in the rebuttal pdf as well as the desired inclusion of a more thorough study in the final version. Regarding (2), I am satisfied with the wording proposed by the authors in Review djnP. I believe authors should also address this wording in the Abstract of the paper as well: "In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, at **no cost** to downstream performance, by decomposing representations into semantic concepts." to reflect that there is at least a "small" cost.