B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable

Summary

This work proposes B-cosification, a method to transform a pre-trained network to a B-cos network [1,2]. Consequently, the transformed model can be finetuned for better explanations of the model behavior while retaining predictive performance. Experiments are done on various CNN and Transformer architectures, as well as a case study on CLIP. [1] Böhle, Moritz, Mario Fritz, and Bernt Schiele. "B-cos networks: Alignment is all we need for interpretability." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [2] Böhle, Moritz, et al. "B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

Strengths

* B-cos networks are an upcoming, interesting research direction. The need to train a network from scratch is commonly considered a limitation (e.g., see the Concept Bottleneck Model Literature [3,4]); therefore, alleviating this need is a valuable contribution to the research landscape. * The paper is clearly structured and well-written. Only towards the end of experiments, it becomes a bit cramped with information, which is minor. * Experiments are performed over a broad range of backbones. Also, I appreciate that the authors make the effort to actually measure the interpretability rather than just stating that it is interpretable. [3] Yuksekgonul, Mert, Maggie Wang, and James Zou. "Post-hoc Concept Bottleneck Models." The Eleventh International Conference on Learning Representations. [4] Marcinkevičs, Ričards, et al. "Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?." arXiv preprint arXiv:2401.13544 (2024).

Weaknesses

* My biggest uncertainty is about the stated (inherent) interpretability. Firstly, to me, I see B-cos(ified) networks more as a post-hoc gradient-based explanation method, which additionally regularizes during training for more faithful explanations. However, there is no certainty upon seeing an explanation that the model actually acted upon these highlighted pixels, which to me makes B-cos networks not inherently interpretable. In other words, a regularizer does not enforce that the model truly depends on the given explanations, and therefore, the explanation given is not necessarily faithful to the underlying model. Thus, I also disagree with the Abstract that states that the explanations are "model-faithful by design". * Followingly, such a method should be evaluated with respect to faithfulness to the underlying model. As such, I would be very interested in how B-cosified networks perform in some faithfulness sanity checks such as the Model Parameter Randomization Test of [5]. * Without any standard deviation, it is hard to assess the (statistical) significance of performance improvements. Also, multiple seeds (up to some degree) prevent the all-too-common hyperparameter optimization on the test set. * The CLIP explanations when B-cosifying CLIP are on par with GradCAM on the pre-trained weights. To be better requires the introduction of a new, arbitrary aggregation of value vectors. * If I understand it correctly, this new aggregation would also affect CLIP's performance. It is not explored how the model performance on CLIP Benchmark would be affected by this aggregation that appears to make CLIP more explainable. * While this work has high significance in today's world of foundation models, the methodological novelty is limited, as most B-cosification tricks have already been introduced in [2]. * In my opinion. speedups of 2x are not necessarily big enough to warrant that it's a huge improvement over training from scratch. * Page 1 appears to be missing some vspace after line 37. [5] Adebayo, Julius, et al. "Sanity checks for saliency maps." Advances in neural information processing systems 31 (2018).

Questions

* In line 295, is "interpolate" really the right word (it confused me slightly in my read). That is, if it was interpolating, then shouldn't Fig. 5a B-cos FeatureCLIP for p=1 be equal to B-cos CLIP? (Also, there are some minor reference naming issues with respect to the figure in line 292-297, and the legend of Fig. 5a needs to be updated) * While the Grid Pointing Game appears to be a good metric to measure how well the explanations are localized, it is not telling enough on a fine-grained level how informative the explanations are. While the qualitative figures 1 & 5b give some insights, into this, they're obviously cherry-picked (which I don't condone). What I like in papers with qualitative examples is that, in the Appendix, there are also some randomly chosen examples so that an interested reader can get a feeling for how the method actually performs. Thus, I would be interested in seeing some more qualitative examples that are (truly) randomly drawn. * How well would the proposed method be transferrable to different input modalities such as text?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

One could maybe write one more sentence with respect to which models are *not* B-cosifiable. Apart from that, limitations are adequately discussed.

Summary

This work builds on the recent line of works aiming to design inherently interpretable models. In particular, it considers the recently proposed B-cos networks and shows how pre-trained conventional CNN/VIT models can be converted into a B-cos network through fine-tuning. The authors first discuss which parts of conventional DNNs can be converted into an equivalent B-cos model and then propose two simple mechanisms that address the two missing components (increasing B and removing the biases) through fine-tuning. Results illustrate that the proposed B-cosified models are able to obtain equivalent accuracy to both the original model as well as the standard B-cos network, while obtaining similar localization results to the B-cos network in a shorter time due to the re-use of the original weights.

Strengths

The work addresses an important problem of converting pre-trained black-box models into inherently interpretable models. Instead of requiring complete retraining, which often restricts the applicability of these models, only requiring fine-tuning is a first step into the right direction. The paper is well structured and the different modifications required are presented in sufficient detail. The experimental evaluation on both CNN and Transformers as well the case study on CLIP shows the potential of the proposed approach.

Weaknesses

It would be beneficial to include a plot on how accuracy and localization score vary as fine-tuning progresses for one/some of the standard models (CNN/VIT). In Sec. 3.2.2, it is unclear how the bias is handled when performing the ablation study in Table 2. Also, the B-cosified model consistently outperforms the baseline in Table 2 and 3, do the authors have an hypothesis on why this fine-tuning process increases performance? Minor: It would be beneficial to also provide the number of fine-tuning epochs that were required in Table 4. Figure reference missing in Line 141.

Questions

For the ViT results, there are some more cases where the B-cosified networks performance degrades slightly compared to the black-box and the localization is slightly lower than for the B-cos model. While these differences are small, do the authors have a hypothesis on their reason? Are these induced by the interpretation of the GeLU?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations of the proposed method have been briefly but sufficiently discussed.

Summary

This work targets the interpretability of modern architectures via a process that the authors call B-cosification. Contrary to the original B-cos networks that are trained from scratch by architecturally enforcing alignment between inputs and weights, B-cosification constitutes a post-hoc method, aiming to convert/finetune exististing pre-trained models towards interpretability.

Strengths

This work aims to transform existing pre-trained models to "inherently interpretable" ones via a post-hoc b-cosification method. Contrary to other post-hoc approaches, this finetuning allows for essentially altering the properties of the network, bypassing some of the criticisms of post-hoc approaches. Overall, this work constitutes an interesting investigation of how to transform standard architectures to B-cos like ones.

Weaknesses

Even though this is an interesting approach, its novelty is limited. The authors explore how to finetune the model to match the b-cos alignment pressure, by exploring some architectural changes like normalisation and bias decrease. One of the main issues the proposed process is the complexity of the finetuning. Some of the ResNet architectures considered in the main text are trained for around 90 epochs, e.g., on ImageNet. That is a common number of epochs to train these models from scratch. Can the authors provide a summary/visualisation of the acc/localization with respect to the number of epochs? What is the behaviour of the base models when finetuned the same way as in the proposed b-cosification process (without b-cos)? Can the authors add another column to Table 4, denoting the accuracy of the standard trained from scratch b-cos networks? The text needs to be proofread. There are several sentences that lack clarity, typos and other issues, e.g., line 141 "see also Fig. XX, whereas conventional models use 3 channels". The authors are encouraged to remove the spacing altercations that do not follow the format of the conference.

Questions

Please see the Weaknesses section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have a limitation paragraph in the main text.

Summary

The authors discuss the B-cosification of a pre-trained model. The B-cosification involves changing the operations performed in the linear layers to one involving a cosine. Not all properties of B-cos models are in the end satisfied, yet performance in terms of accuracy and localization are on par with a fully trained B-cos model.

Strengths

- The method contributes to the current effort to turn pre-trained models into explainable ones without full retraining. - The performance dropped are limited both against the original model and the equivalent B-cos model

Weaknesses

- The method still requires a full fine-tuning of the weights. - The experimental setting is not very clear. - The presentation is not always clear and sometimes verbose.

Questions

* P1 L33: which are increasingly popular—can cost millions of dollars. What if I am not using dollars? Maybe time would be a better metric? Or energy? * P2 L38: the recently proposed B-cos Networks. Until here, B-cos was never defined. Since it is not a contribution of this paper, would you have a citation to guide the reader? * P2 L42: What does "functionally equivalent" mean? What do you mean with "alignment pressure"? Maybe instead of introducing these vague notions, why not use the opportunity given by this paragraph to explicitly say what B-cos consists of, namely (up to my understanding) replacing the linear operations of the hidden layers by a cosine-based function parametered with B? You can go further and say that for B=1, the operation is ("is" and not "is equivalent to") the classic linear matrix multiplication. "Linear" implies the absence of a bias term. Otherwise, that would be the "classic affine matrix multiplication". Nevertheless, one can stress this further. * P2 L45: "significantly more interpretable explanations (Fig 1)" What do you mean by "more interpretable explanations"? What I see in the third row of Fig1 is kind of a mix between the second row and an edge detector or an Integrated Gradient saliency map. Is capturing the shape or details of the "important part of the image" making an explanation more interpretable? Following this discussion, the result of GradCam+CLIP on the first example is preferable from my point of view because one can see that the model uses the help of the grass to actually predict a sheep. The same applies to the "[...] photo of a boat." * P2 L47: "On supervised settings, we find that B-cosified models often outperform" How is performance measured here? Classification accuracy or caption prediction? * P2 L53-57: I am a bit confused when you say "design[ing] inherently interpretable models". I do not understand your contribution as a new design but a conversion. This is a valuable and efficient contribution that aligns with current concerns, be it ecological, time efficient, or simply democratizing ML for the majority who do not have access to huge computational resources. This work fits into the recent line of research investigating how to alter the architecture of trained models to make them "interpretable." Why not stress this aspect more? * P2 L65-: I am unsure if the claimed results are that important to count as contributions. Again, what you mean by "significantly interpretable" is not clear. Could you please clarify? * Sec2: If you choose to attach your contribution to the line of "converting trained models" (you don't have to) to "interpretable ones", I suggest some works: * Stalder, S., Perraudin, N., Achanta, R., Perez-Cruz, F., & Volpi, M. (2022). What you see is what you classify: Black box attributions. Advances in Neural Information Processing Systems, 35, 84-94. * Aytekin, C. (2022). Neural networks are decision trees. arXiv preprint arXiv:2210.05189. * Gautam, S., Boubekki, A., Höhne, M. M., & Kampffmeyer, M. (2024) Prototypical Self-Explainable Models Without Re-training. Transactions on Machine Learning Research. * P3 L111: "Many common DNNs consist of a series of blocks of linear layers followed by non-linear ReLU activations [29], and are thus piece-wise linear functions1 : i.e., for every input x, they effectively compute a linear transformation of that input: y(x) = W(x)x + b(x)" What are W and b? I assume the model's input and output are x and y, respetively. If it is "effectively [...] a linear transformation", why are W and b functions of x? * P3 L113: "dynamic linear" Why do you need to introduce this notion? What is dynamic in an affine matrix multiplication? * P3 L116: "This linear mapping W(x)". The same goes here: W cannot be a linear mapping/function if we are talking about "a piece-wise linear model". * P3 L120: Why does a "complete explanation" requires "y(x)=W(x)x"? Two sentences later, you actually contradict this claim. I fail to grasp why having a bias is so problematic if most of the backward operations are based on the gradient. Could you clarify? * P4 L123: Can you clearly define "alignment pressure" and why it is necessary to be introduced? From this paragraph, I understand that the conversion from line (B=1) to (B>1)-cos model has already been investigated in [6] and [7]. Has it? If it has been, this jeopardizes the relevance of your main contribution. * P4 Eq1: How do you compute $cos(x,w)$? I understand that $x$ is a vector, $W$ is a matrix, and $\times$ is the element-wise multiplication. * P4 L127: Why do we need the notion of "dynamic linear"? If it has to do with W being a function of x: if B=1, W does not depend on x. * P4 L141: Missing reference with Fix XX. * P4 L146: "In particular, we show that a conventional model can be framed as a functionally equivalent B-cos model as in [7] with B =1, which additionally employs bias terms. Only upon modifying these two aspects, i.e., biases and B, does the model need to be fine-tuned to adapt the weights to those changes." These two sentences are not clear. * P4 L150: "As mentioned in Sec. 3.1, B-cos models use input representations with six color channels" This was not mentioned in Sec 3.1. This seems to be an atavism from an older version of the paper. Please fix. * P4 "Input Encoding and Normalisation": The operation is straightforward: B-cos models requires 6 channels: so inputs need to be transformed according to Eq2 and the first layer of the B-cosified models is discarded and replaced by one with 6 channels (implying a full training thereof). * P5 "Activation Functions." You start the section by saying that activations are unnecessary, yet you introduce them. Why? * P5 "...unit norm weights, ..., which the authors motivated by the fact that the only way any given neuron can achieve its maximal output is by increasing the weight-input alignment, which in turns[typo] leads to the improvements of the explanations." What is "weight-input alignment"? I fail to see why normalized weights improves the explanations. You could streamline this section by leaving the justification of the B-cos meaningfulness to an introductory paragraph and focusing on the changes required by your approach. There is no need to justify or make claims about the benefit of B-cos since it is not one of your contributions here. * P5 Section 3.2.2 None of the tables present statistical tests, which makes it difficult to draw any conclusion, especially given that the results are quite similar from one setting to another. * How many times the experiments were repeated? What are the standard deviation? Which result is significantly better? * What is the localization score? * P6 L204: "setting B = 2 and then fine-tuning, yields performance that is on par with learnable B parameters," If this holds, it is also on par with Linear B 90 epochs. * P6 Table 4: The increments are quite small. Have you considered running a t-test to identify significant differences? Is the "speedup" with respect to the fully trained B-cos model? What about the "speedup" with respect to the vanilla models with classic matrix multiplication? * P7 L245: "Specifically, we find that averaged across architectures, B-cosified models outperform B-cos models trained from scratch by 2.31 pp, with an average training speedup (to match performance) of 2.96x." This is an important result. Why not highlight it in a separate section/ablation study? * P8 Interpretability: In the text you refer to paper [6] while in the caption of Fig2 you refer to [7] (These papers share essentially the same figures). Fig2 would be better positioned closer to its text. Finally, interpretability is defined in paper [6] (or rather in paper [6] of paper [6]) as the performance at the "grid pointing game". If I understand it, it is related to the sum of the GCAM attribution to each class of the grid cell. My understanding of gradient-based explainable methods is that the scale of the output saliency maps is inconsistent. Is it valid? If yes, how does it affect the score returned by the Pointing Game? How robust is this score against possible spurious relationships between grid cells? How does performing well in this game relate to the interpretability of the model's explanations? * P8 Fig4: What are Natural/Specialized/Structured Data? * P9 L284: "We find that the B-cosified models significantly outperform the Text2Concept approach and achieve accuracies that are more similar to the original CLIP’s zeroshot and linear probing accuracies." Text2Concept is a quite different approach. In the original paper thereof, they report performance close to CLIP's. How do you explain this difference? What is different in your setting? Do you train supervised or self-supervised? * Evaluating Model Interpretability. In the image classification case, you compute the explanation based on a single label; the equivalent case here is thus to back-propagate using the Isn't the average similarity equivalent to the similarity with an out-of-distribution point in the text embedding? Why did you choose p=7 and p=19? Overall I did not quite understand this section. I am not very familiar with CLIP, so it is not easy for me to follow what is happening.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

None

Reply to Author Response

I would like to thank the author for providing the additional clarifications as well as the additional results on how the localization scores change during fine-tuning. After reading the rebuttal as well as the other reviewer's comments, I will retain my 'accept’ score and believe that this will make a valuable contribution to the conference.