This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations

Summary

This paper proposes a new interpretable prototype-based classifier, called ProtoConcepts. Unlike existing prototype-based classifiers that use one-to-one comparisons, ProtoConcepts learns prototypical concepts using multiple image patches. This approach aims to make it easier to identify the underlying concept being compared, allowing for richer and more interpretable visual explanations. Experiments show that this modified "this looks like those" reasoning process can be applied to various prototypical image classification networks without affecting accuracy on benchmark datasets.

Strengths

1. The authors make a valid point that utilizing a single training image patch as a prototype can be insufficient for users to comprehend the concept the prototype represents. For example, when presented with a "blue circle" prototype, it is unclear whether the concept is related to the color "blue" or the shape "circle." 2. The paper (Table 1) demonstrates that ProtoConcepts can be integrated with various prototype-based classifiers, such as ProtoPNet, TesNet, and ProtoPool, showcasing its adaptability across different methods.

Weaknesses

1. It is definitely not a new idea in the XAI community to use multiple images/patches to visualize a concept. This is a common practice in the works of concept-level explanations, like [TCAV](http://proceedings.mlr.press/v80/kim18d/kim18d.pdf), [ACE]( https://arxiv.org/pdf/1902.03129.pdf), and so on. However, none of them are discussed in the paper; 2. The main issue of the paper lies in the evaluation of interpretability. It is unclear whether these examples are cherry-picked. Also, given that the source code of the method is unavailable at this moment, it is unknown how good/bad the explanation results would be for many other cases. If the examples are not cherry-picked, it would be better for the authors to clearly state this in the paper or use examples in a fixed order (e.g., always use the first example of each class). It is hard to be convinced that the interpretability of ProtoConcepts is good by showing several separate examples. I would suggest that the authors conduct a carefully designed human user study. The user study results should better convince users that the method brings about a better understanding of the model's behavior. User studies are often conducted in the work of concept-level explanations.

Questions

See weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1. The novelty of the paper appears to primarily lie in the combination of two distinct lines of research within XAI - concept-level explanations and prototype-based classifiers. 2. The absence of user study results and the unavailability of source code make it difficult to be convinced about the interpretability of the explanations generated by the proposed method. See more details in the part of "Weaknesses".

Summary

The paper proposes a modification of the prototype layer of the existing prototype-based networks. Existing methods rely on a single training image patch as a prototype. The main drawback of this is that it is difficult for the user to understand the meaning of the prototype from a single visualization. The paper proposes to learn prototypical concepts which are visualized using multiple training image patches. In this way, the semantic of each prototype is less ambiguous.

Strengths

_Clarity_: the paper is generally well-written and structured clearly. The figures showing image examples are very helpful, especially for readers that are not experts on prototypical networks. _Significance_: the paper addresses a relevant problem which is well known to practitioners using prototypical networks. The problem is filling the gap between what the prototypes represent and the human understanding of prototype semantics. Previous works tried to tackle it. The proposed changes do not badly impact the overall performance of the prototypical networks; indeed, the accuracy is comparable to previous methods. _Quality and originality_: the two novel losses are technically sound. The idea of representing the prototypes as a ball in the latent space is interesting. The solution can be applied to several types of prototypical networks. _Reproducibility_: the code will be made available upon acceptance (it was unavailable to the reviewers). The supplementary material reports the training parameters.

Weaknesses

As reported in the introduction, the proposed solution should allow the user to determine the semantic meaning of each prototype with less ambiguity. However, the experiments do not report one or more metrics that show this reduction of ambiguity with respect to the plain prototypical networks. The paper and the supplementary material reports some example of concrete cases but not an overall evaluation. The optimal solution would be to run an evaluation with real users, but I understand that it is expensive. The paper [1] proposes activation precision as an interpretability metric (section 4.2) (the segmentation masks are provided with CUB-200). Given that Cub200 dataset provides part location annotations for the images, another metric could measure the degree to which the multiple visualizations of the prototype refer to the same bird part. [1] Barnett 2021, "IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography"

Questions

- what is the main advantage of your solution with respect to learning standard prototypes (e.g., as done in ProtoPNets) and visualizing it on multiple most activated training images? In other words, the original work on ProtoPNets visualizes the prototype on the most activated training image; what if you just take more images? - How is k set in the experiments? Why does the value of k change depending on the prototype-based model?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors address the limitations. I would like to stress a point that is also discussed in the limitations. The user expertise influences the impact of the method on identifying the concepts: given the same set of prototype visualization, an expert user has a different interpretation of them with respect to a non-expert.

Summary

This paper extends the patch comparison of prototypical part-based classifiers from one-to-one patch to one-to-multiple (e.g. from this looks like that to this looks like those). They compare their proposed network with Proto-PNet variants and show no classification accuracy improvements and little evidence of better interpretability. The proposed method was tested on CUB and Cars as previously done in ProtoPNet variants. I have read the author’s rebuttal and adjust the rating accordingly!

Strengths

1. The research is well positioned in the literature of prototypical part-based classifiers. 2. The extension from 1vs1 to 1vsmany is interesting and could help improve human understanding tasks that all prototypical part-based classifiers fall short on [1] [1] hive: evaluating the human interpretability of visual explanations

Weaknesses

There are multiple weaknesses of this paper, mostly about the novelty. 1. I see the extension is incremental though well motivated. There is no surprise in the classifier accuracy compared to other ProtoPNet variants. I always see (all of them) they perform roughly the same. I believe the "big fish" is to really make this prototypical part-based explanations usable for humans. 2. Although it was motivated by the lack of interpretability of prototypical part-based classifiers, the analysis of model interpretability is superficial and not spot-on. I believe maybe this work also fails to help humans [1] as its siblings [2,3]. The authors could improve the interpretability analysis by running more extensive study (both automatic and human studies) rather than dissecting sample-wise visualizations that could be easily cherry-picked and misled. 3. I think the paper lacks the analysis of computation cost (i.e. 1vs1 and 1vsmany comparison). [2] ProtoPNet [3] ProtoTree

Questions

The 1vsmany patch-wise comparison has been studies and they even conducted human studies on interpretability. How this work differs from them [4]. [4] visual correspondence-based explanations improve ai robustness and human-ai team accuracy

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper improves interpretability for prototypical learning. Most previous methods use a single prototype as an interpretation -- and this is not as informative as we don't know what portion of the prototype corresponds to our target image. The authors introduce a *prototypical ball* to interpret what prototypes are used for inference. Instead of selecting a single prototype, training samples from this prototypical ball are sampled.

Strengths

*Originality*: I am not completely informed on all prototypical learning methods, but it appears that this method is original. In addition to the novel use of a ball for multiple visualizations, they describe a training scheme specifically for their method. This contributes to the originality of their work. *Quality*/*Clarity*: The paper is generally presented well, with descriptions of previous work and their drawbacks, as well as what contributions improve the current work. I think some of the descriptions of previous work is not very clear, but I understand space constraints preclude a better description. I believe that the main contribution of this paper can be more clearly elucidated in the figures, which I describe in ``Weaknesses" section. *Significance*: I think this work is important to the community. While it matches performance of previous methods, it presents a new way of interpreting the results of algorithms. Integrating this novelty into previous works is adds to its significance.

Weaknesses

* A better explanation of interpretability would stregthen the paper. The results do improve interpretability, but the fundamental question (that is introduced in the paper) still remains: why are these selected as prototypes? There are more images and we humans can make connections and inferences, but it's still unclear as to why these are the prototypes within the ball. I do see this explicitly addressed in Section 5, but I still think it is a weakness of the paper. I think an analysis like your reference [16] would strengthen your paper signficantly. I don't know if it is critical to this paper, but it would be a good follow-up paper. * Figure 2 is fairly unclear. It's difficult to understand what your method is actually doing. It's not clear how the logits are obtained either. Overall, if I were to look at that diagram without the help of the text, it would be quite unclear as to what your novelty is and how it is implemented. The ball that you introduce, which is your main contribution, is missing from the figure. A couple small things: * Please add reference for ProtoPool in Figure 1 and line 209. * Reference [16] and [18] look very similar -- are these the same? * The text "ProtoConcepts Layer $g_p$" in Figure 2 looks compressed compared to the rest of the text in that figure.

Questions

* Do you have an ablation on the loss scaling parameters in the supplementary (Tables 2,4)? How were these $\lambda$ chosen? How does performance vary with this? * Why was $k=10$ chosen for $\mathcal{L}_{\text{clstk}}$? Do you have an ablation on this? * Continuing from the section above, I think a figure showing the ball, it's center, and radius with its prototypes would help readers understand this work more clearly. For example in Figure 1, the multiple visualizations of prototypical concepts doesn't clearly identify how those multiple visualizations are obtained (i.e. you could use the same figure if the results were the same using a k-NN algorithm) (same for figure 2) * In equation three, I think it's important that you show the full loss equation i.e. $\mathcal{L} = \mathcal{L}_{\text{other terms}} + \lambda_1 \mathcal{L}_{\text{clstk}} + \lambda_2 \mathcal{L}_{rad} $. It would be clear as to what the loss weights mean in Section 4.1.1

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the negative societal impacts.

Response to Rebuttal

I appreciate the authors' response. I take the point that the concept-level explanation methods like TCAV and ACE are posthoc methods while the proposed method in the paper is inherently interpretable. However, the work of concept-level explanation methods should definitely be discussed in the final version of the paper. Also, note that while TCAV requires known concepts, ACE does not. The human study results supplied are vital in persuading readers about the interpretability of ProtoConcepts. The results should also be included in the final version. I raised my rating of the paper to 6.

Concerns remained about the evaluation and novelty

I would like to thank the authors for the efforts in the rebuttal. After reading your rebuttal and reviews from other reviewers, my replies are: 1. Not only me, the other reviewers also question on the novelty of this work (wBiV in Q1, x5Kg in W1). 2. I genuinely value the authors' efforts in conducting the human study, but I remain unconvinced by the results. * The report lacks specifics regarding the human study—like the number of participants and their expertise (whether they were experts or laypeople), etc… * Your comparison with the random choice (50%) and ProtoPNet—a model with nearly random choice accuracy—doesn’t adequately capture human utility due to the weak baselines employed. Given that, my primary concerns are still about the novelty and evaluation methodology of this paper. I raise the score (to 4) and am open for more discussion.

Response to rebuttal

Thank you for your answer and for taking the time to run a user study. The user study improves the evaluation part. I agree with Reviewer x5Kg that the user study should be added to the final version of the main text. I agree with your explanation about visualizing the prototypes on multiple most activated training images. Would it make sense to have an additional experiment to demonstrate this also visually by showing the user the same number of visual explanations for both ProtoPNets and ProtoConcept? I expect that ProtoConcept wins the comparison because its multiple visualizations are faithful and of higher quality. It is not only a matter of having more multiple visualizations but also of their quality and interpretability. This can also be shown by computing the activation precision metric (see IAIA-BL paper), which should be higher for ProtoConcept. This may partially address the charry-picking issue raised by Reviewer p3JV and Reviewer x5Kg. I raised the score to 6.