DISCOVER: Making Vision Networks Interpretable via Competition and Dissection

Summary

This paper proposes a Dissection of Competitive Vision Networks (DISCOVER) to build interpretable neural networks. It aims to understand the neuron functionality for image inference. Moreover, this paper introduces the Jensen-Shannon divergence between the probability of concept presence and the probability of neuron activation for network dissection. Experimental results evaluate the effectiveness of the proposed CVNs.

Strengths

1. Relatively interesting idea of the proposed CVNs for network dissection. 2. high interpretability of the proposed framework evaluated in experiments.

Weaknesses

1. Limited novelty. DISCOVER method is similar to DMoE [ref-1] combined with the work [ref-2]. Please clarify their difference. 2. Lack of more visualization results to evaluate the interpretability over other methods. [Ref-1]. Deep mixture of experts via shallow embedding. Uncertainty in artificial intelligence. PMLR, 2020. [Ref-2] Interpretable Neural Network Decoupling. In ECCV, 2020.

Questions

Refer to the Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

This paper proposes to use stochastic local winner-takes-all (LWTA) layers in vision networks combined with CLIP-Dissect to improve the ability of a neuron to capture a concept. A stochastic LWTA layer groups pre-activation neurons into $B$ blocks of $U$ features each and within each block, chooses one feature with probability according to the softmax of the pre-activation neurons in the block. The chosen feature is preserved while the rest are zeroed out. The paper tests DeiT-T, DeiT-S, and ResNet-18 architectures on ImageNet and Places365 replacing original activation functions with stochastic LWTA. The classification performance slightly improves when each block is of size 2, or $U=2$, and degrades as $U$ increases. The stochastic nature of LWTA allows the authors to use CLIP-Dissect with the Jensen-Shannon Divergence (JSD), which they do. The learned concepts for the different investigated neurons, such as “green,” “red,” “trains,” and “snowmobiles” match the images those neurons are activated for. The text embedding similarity of found concepts to ground truth ImageNet label is higher with LWTA neurons and JSD than normal neurons and WPMI.

Strengths

The presentation is clear and the motivation in the opening sections is well-expressed. Also, extensive code is provided to support the approach. The idea of introducing competition between neurons and having one win out seems intuitive and suited for interpretability, the stated goal of this paper. The results are promising (the introduced module does not harm performance too much, and potentially increases performance if a very mild form of it with $U=2$ is used, and some quantitative evidence for superior interpretability is provided).

Weaknesses

It’s not fully clear whether the approach is being presented as an ante-hoc method or a post-hoc method. It seems one could argue that needing to train a network from scratch with the LWTA nonlinearity makes it an ante-hoc method. It also seems one could argue that since LWTA is a drop-in replacement for any non-linearity, it functions more as an enhancement to any ante-hoc or post-hoc methods applied on top of the architecture rather than an ante-hoc/post-hoc method itself. Could the authors clarify this? The evaluation could be more thorough and more convincing. Here are some suggestions, not all of which might be necessary: - Significant decline in performance is stated as a drawback of other ante-hoc methods, which this method is arguably an instance of; is it possible to compare classification performance with one or more baseline ante-hoc approaches? - Can you provide some sort of comparison between $U$ and the interpretability of the proposed approach? Such as, a plot of $U$ vs. cosine similarity between intermediate neuron description, or comparing concepts found with $U=2$ to $U=16$. Since $U=2$ achieves the best performance, it’s not clear why we would prefer $U=16$. The argument that $U=16$ achieves higher sparsity and thus higher interpretability intuitively makes sense, but is this supported empirically? - In my opinion, an additional interpretability metric to evaluate the potential of CVNs for interpretability purposes would improve the paper. Would it make sense to use the IOU-based interpretability metric from the Network Dissection paper? If that would require too much additional experimentation, is some sort of more lightweight added experiment/comparison possible? Miscellaneous typos/comments: - L177: Doesn’t $\mathrm{CE}(y_i, f(X_i, \hat\xi))$ denote the cross entropy, not $f(X_i, \hat\xi)$ - L340: “the resulting networks yielded on par or even improved performance, even when using up to 4% of active neurons per example” seems a bit of a stretch; going from original network to most sparse competition-based network for DeiT-T on ImageNet-1k, the accuracy dropped from 72.2 to 70.5, for ResNet18 on Places365, from 53.5 to 49.5. I would call these drops significant. I would call the drop of DeiT-S on ImageNet-1k from 77.0 to 76.7 “on par”, but not improved.

Questions

I put the most relevant questions in the “Weaknesses” section. - In L228, it is claimed “In the context of inner-product neurons, we directly obtain the presence probability via Eq. (2).” But aren’t vision transformer neurons also spatially arranged, like convolutional neurons? I.e. for a given feature there are different instances of it which corresponding to a spatial map, corresponding to the different embedded patches. So do you also average over spatial positions for transformer?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations were adequately addressed.

Summary

This paper proposes a novel method for interpreting vision networks by combining an architecture that incorporates stochastic local-winner-takes-all activations with the CLIP-DISSECT framework. The author makes use of the probabilistic interpretation of LWTA as a categorical distribution to derive a similarity measure for LWTA activations based on the Jenson Shannon Divergence. They show the benefit of their approach by comparing the discovered interpretations to ground truth labels for conventional and competitive (LWTA) networks.

Strengths

The studied problem of interpretability is very important, the idea is well motivated, the background and related work is informative, and the writing is clear. The accuracy of the LWTA adapted transformer is very encouraging and suggests the usefulness of competition independently of interpretability. The improvements in automated neuron description, are encouraging.

Weaknesses

Clarity could be improved: The figures are not very helpful. Especially fig 1 could be improved by simplifying it (e.g. without showing individual neurons and weights). In my opinion Fig 2 is not necessary and could instead make space for a much needed figure to illustrate the DISCOVER framework. The math feels a bit convoluted with too many new symbols introduced in each paragraph. Again, simplification could help understandability. The paper spends quite time explaining how LWTA and DISCOVER can be used for convolutional networks. However, the main results of the evaluation focus on transformer architectures for which it is less clear how LWTA is incorporated (only replacing the ReLU inside the MLP block or also in the attention and residual branch?). I suggest switching focus and making conv architectures much less prominent. The evaluation feels incomplete. The main evidence for the efficacy of DISCOVER are: 1. the qualitative results in Figure 3, which are only partially convincing. Eg. the "aquaculture" neuron seems to react to a highway, and the "dashboard" neuron mostly picks up on hifi equipment. 2. Table 2, which shows that, compared to non-LWTA-networks) the textual embedding of the discovered neuron descriptions is more similar to the corresponding ground truth label. However the CLIP cosine similarity score is hard to interpret and the reported gains seem mild. For a paper that focuses on interpretability, I find the results hard to interpret. As it stands, the paper has me convinced that stochastic LWTA is interesting and should be studied further, but fails to convince me of the added value of DISCOVER over e.g. CLIP-dissect.

Questions

How exactly is LWTA incorporated into the transformer?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The paper does discuss two relevant limitations.

Summary

The paper proposes DISCOVER, a novel framework for creating interpretable vision networks. It utilizes stochastic Winner-Takes-All layers and multimodal vision-text models to uncover the specialization of each neuron. The framework achieves high activation sparsity, allowing for direct examination of concept representations. It also introduces a new similarity metric for network dissection based on Jensen Shannon Divergence. Experimental results demonstrate improved classification performance and provide a principled framework for text-based descriptions of neuron representations. The paper pioneers the training of LWTA networks on large datasets, yielding interpretable functionality with a small subset of neurons per datapoint.

Strengths

The strengths of this paper include: 1. Novel Framework for Network Dissection: The paper introduces a novel framework, DISCOVER, for creating interpretable vision networks. It combines stochastic Winner-Takes-All layers and multimodal vision-text models, providing a unique approach to uncovering neuron specialization. 2. Improved Classification Performance: Despite not performing hyperparameter tuning, the DISCOVER framework achieves comparable classification performance. 3. Interpretable Similarity Metric: The paper introduces a new similarity metric based on Jensen Shannon Divergence to match neuron representations to concepts. 4. Pioneering LWTA Networks: The paper contributes to the literature by being one of the first to train large-scale LWTA networks using substantial datasets and architectures.

Weaknesses

1. The WTA approach in this neural network framework draws inspiration from previous methods employed before the advent of deep learning. For instance, techniques like bag of words, Locality-constrained Linear Coding (LLC) (Wang et al., "Locality-constrained linear coding for image classification," CVPR, 2010) and a long etc, also involved neuron competition for activation. 2. It would be valuable to have a clearer understanding of the level of sparsity in the network without WTA. While the paper mentions that in a network without WTA sparseness is "input specific," providing some statistical analysis or comparisons of the sparseness level with non-WTA networks could provide insights into the necessity and effectiveness of the WTA mechanism. 3. The paper assumes that sparse representations lead to more interpretable units, but the connection between sparsity across neurons and human interpretable concepts needs further clarification. It would be beneficial to elaborate on the theoretical arguments or underlying assumptions supporting this claim. Additionally, providing empirical evidence or previous research that supports the notion that sparse representations indeed enhance interpretability would strengthen the argument. 4. The qualitative results provided in the paper are limited, particularly regarding the comparison between applying the WTA mechanism and not applying it. It would be insightful to include examples of units that are not interpretable when the WTA mechanism is not utilized. This comparison would help illustrate the added interpretability that the WTA approach brings to the network and provide a more comprehensive understanding of its impact on the model's performance. 5. The dependence on the interpretability similarity metric raises questions about the generalizability of the results. While the paper introduces a new similarity metric based on Jensen Shannon Divergence, it would be valuable to investigate the robustness of the findings using alternative metrics and visualization techniques. Demonstrating the consistency of the results across different evaluation methods would strengthen the validity and generalizability of the proposed framework for increased interpretability. Exploring the impact of alternative metrics and visualization techniques could provide a more comprehensive understanding of the interpretability achieved by the network. By addressing these concerns, the paper can further enhance its contributions by providing more insights into the necessity and effectiveness of the WTA mechanism, establishing theoretical justifications, and expanding the qualitative analysis to better understand the interpretability aspect.

Questions

As I mentioned earlier, there is a concern regarding the lack of clear evidence establishing a direct relationship between sparsity and interpretability. While the paper assumes that sparse representations lead to increased interpretability, it would be beneficial to provide results that demonstrate this relationship across different levels of sparsity, utilizing various interpretability measures. By systematically examining the impact of different sparsity levels on interpretability using diverse evaluation metrics, the paper could provide stronger evidence for the claim that sparsity enhances the interpretability of individual units. This analysis would contribute to a deeper understanding of the relationship between sparsity and interpretability and provide more robust support for the proposed framework.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

I believe the limitation section could take into account the points I raised in weaknesses, as they were not mentioned.

Summary

The work proposes a interpreatable network that uses concepts from competition networks specifically there are for a specific output multple copies of the neuron and the neurons compete to provide the winning activation for that output. The method proposes MLP and convolutional versions of their network. The work also provides evidence of interpreatability by using CLIP-DISSECT and shows that their method is more interpreatable while using same number of parameters as DiET.

Strengths

1. The method is quite simple and elegant in its motivation 2. Existing modules can be swapped with the proposed block without any changes

Weaknesses

1. The method is essentially a stochastic version of maxpooling where the deterministic version is exactly maxpooling. Is there an ablation to show how the stochastic version is more appropriate than a deterministic one? 2. Having 2 competetors seems to provide the best performance whereas using more decreases the performance of the network. Is this because of using less "effective neurons" (as the model size kept same) or with more competition the network inherently starts degrading? If it's the second case than it does not put competition networks in a good light. 3. It seems wasteful that with more competition exponentially lower number of neurons in the model are activated. 5. Can it be used in networks for other tasks? Segmentation, object detection etc.?

Questions

See the weakness. 1. Gumbel softmax generally requires a bit of finetuning. How stable is the gumbel softmax training on different datasets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Thank you

Thank you for extensive comments on my review. The review adresseses many of my concerns. > 1. The method is essentially a stochastic version of maxpooling where the deterministic version is exactly maxpooling. Is there an ablation to show how the stochastic version is more appropriate than a deterministic one? Answer is mostly convincing. Thank you for additionally pointing to citing [1]. Argument against is that reduction can be easily undone for maxpooling, (e.g. done in backpropagation). > 2. Having 2 competitors seems to provide the best performance whereas using more decreases the performance of the network. Is this because of using less "effective neurons" (as the model size kept same) or with more competition the network inherently starts degrading? If it's the second case than it does not put competition networks in a good light. Answer is mostly convincing. Although if current setup is suboptimal it puts the work at a disadvantage. The work is skipping a step in the discovery process. There should exist a well optimized LWTA based network to start with, before performing any kind of analysis on their advantages. > 3. It seems wasteful that with more competition exponentially lower number of neurons in the model are activated. Answer is mostly convincing. I agree with reviewer kXaB that lower number of neurons may not necessarily mean better interpreatability. There are still 50 neurons interacting with each other in the next step. There is no guarantee that each neuron is going to be specialized. > 4. Can it be used in networks for other tasks? Segmentation, object detection etc. The answer is satisfactory. > Q1: Gumbel softmax generally requires a bit of finetuning. How stable is the gumbel softmax training on different datasets? The answer is satisfactory.

Response to rebuttal

I thank the authors for the clarifications. I understand now that advertising the usefulness of stochastic LWTA is part of the contributions of this paper. In particular, highlighting the fact that it can help with interpretability without sacrificing performance. Because of that, I raise my score to borderline.