Concept Bottleneck Models (CBMs) tackle the opacity of neural architectures by constructing and explaining their predictions using a set of high-level concepts. A special property of these models is that they permit concept interventions, wherein users can correct mispredicted concepts and thus improve the model's performance. Recent work, however, has shown that intervention efficacy can be highly dependent on the order in which concepts are intervened on and on the model's architecture and training hyperparameters. We argue that this is rooted in a CBM's lack of train-time incentives for the model to be appropriately receptive to concept interventions. To address this, we propose Intervention-aware Concept Embedding models (IntCEMs), a novel CBM-based architecture and training paradigm that improves a model's receptiveness to test-time interventions. Our model learns a concept intervention policy in an end-to-end fashion from where it can sample meaningful intervention trajectories at train-time. This conditions IntCEMs to effectively select and receive concept interventions when deployed at test-time. Our experiments show that IntCEMs significantly outperform state-of-the-art concept-interpretable models when provided with test-time concept interventions, demonstrating the effectiveness of our approach.
Paper
Similar papers
Peer review
Summary
The intervenability of concept bottleneck models has been taken for granted thus far. There have been numerous methods that tried intervening or propose policies to intervene. However, previous methods did not go so far as to optimize models for interventions. This work proposes intervention-aware training and extends Concept Embedding Models for this purpose. With experiments across MNIST, CUB, and CelebA, the authors demonstrate that IntCEM achieves on-par downstream performance compared to CEMs, and investigate the intervention performance with two sets of automated studies. **After the rebuttal:** I read and acknowledged the authors rebuttal. I am leaning towards an acceptance position after the rebuttal, given that the authors clarified my questions. I find the paper's message interesting and the presentation to be clear and well-written, the reason I am not giving a higher score is the lack of real-world settings to test the method in more complex settings and where the models do not need concept annotations at the training time.
Strengths
1. Optimizing a model for intervention awareness is an interesting and strong idea. Many works did indeed attempt interventions, whereas it is infrequent to optimize the model for this purpose and the intervenability has been taken for granted. I do have a question about whether the current form of this is useful (Q1), yet I still believe the idea is worth sharing with the community. 2. I believe having an intervention policy has implications beyond interventions per se. By looking at the Rollout Loss, I could imagine the policy revealing locally important / less important concepts and performing selective inference. While this is a hypothesis that needs verification, this could be an interesting direction to think about. 3. (Minor) I appreciate the clarity in the writing. Formalization is clear and easy to digest even though there are nuanced tricks to make the method work.
Weaknesses
1. In my understanding, there is a disconnect between the promise/message of the paper and the experiments. Concretely; - 1.1 In many parts of the paper, there is a theme around getting the models ready for expert feedback, which can make sense. However, I do not fully understand how we can infer that the current method achieves something related to a user or an expert. - 1.2 In this regard, a major weakness is the lack of a user study. There are many arguments in the paper that touch upon optimizing the model for user interventions, whereas there are no experiments that verify this argument. An experiment where the users used these models, intervened on them, and IntCEM performed better than baselines would support the argument here. - 1.3 Furthermore, the selection policy encodes a certain belief: That the users will choose concepts that would improve downstream performance. An alternative user metric would be showing that users prefer the concepts chosen by the selection policy. This may or not may not be the case. One could have also hypothesized that the policy can pick concepts that the users would also like to intervene on more than other policies, but again, there is no supporting evidence for this argument either. In general, I believe this is an important disconnect that needs addressing. 2. The usability of the method suffers from the requirement to have concept annotations at training time, as do vanilla CBMs and CEMs. However, there are still concept-based models that could be worth exploring in this context that are already cited in the paper as [26, 27] which are Post-hoc CBMs and Label-Free CBMs that also scale to ImageNet/COCO. The authors make the argument that the method would complement those architectures as well, and I can believe this hypothesis, but this needs verification. 3. The above weakness also leads to another one, which is the lack of experimental settings closer to the real-world such as ImageNet, COCO, or medical datasets that were tested before in the above papers. I do think experiments would benefit from expanding beyond CUB and CelebA. Personally, I believe this would also allow impactful user studies. 4. It is hard for me to interpret the numbers in Sections 4.2 and 4.3. This is also connected to the 3rd point above – but the improvements in MNIST or CUB seem 1-2%. Further, as a minor comment, while there are Figures, all of the y-axes are in different scales, and I cannot really tell the numbers.
Questions
1. In terms of the story, while the premise of intervention awareness sounds interesting at face value, I still think the case needs to be made for such a need. I believe it could help to give a concrete use case before laying out the method. E.g. L118-119 authors mention an interesting example, “may be hard for the model to predict on its own”. Can authors make it more concrete, give real world examples / formalize this? When would it hurt to not be intervention aware? When would we benefit from being intervention aware? 2. I would like to ask the authors’ reasoning about the first weakness I raised in the Weaknesses section. If intervention awareness is indeed to improve the interaction between the model and the human, how should we think about the current results? Is there a way to deduce that models are more ready for humans? Can we infer that humans would prefer this notion of intervention awareness over previous methods (or not having it)? 3. In my understanding, according to the loss function after L194, the selection policy $\psi$ is optimized to select the concept that would increase the probability of the ground-truth label being predicted the most, upon the intervention. Would this not bias the selection policy towards more typical interventions for the class (e.g. where $P(Y|C)$ is high)? For instance, say that a bird specie X is 95% of the time red but 5% of the time blue, whereas other birds are more often blue. Then, wouldn’t this loss function incentivize the selection policy to pick the color concept for blue X birds? 4. Similar to the above, while Section 4.3 studies the performance improvement from the policies, it is not clear what these policies pick. Do the authors have any analyses that explain what the chosen concepts are? For instance, are these the concepts that are specific to a class? Are these the most common concepts for the class, are there any other patterns there? 5. (Minor) I would appreciate having the Tables in addition to the figures, I’m sorry if I missed them but I could not find them in the Appendix either. Numbers could be hard to read from the Tables.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
I think the most significant limitation is the lack of an actual real-world study to verify the claims. In my understanding, it is not possible to infer this from the current results. I would appreciate hearing the authors' reasoning about this.
Summary
The authors proposed a modified version of CEM that is better at receiving human test-time intervention by explicitly incorporating intervention into the training stage. Specifically, an intervention prediction module is trained to behavior clone an optimal-greedy intervention policy (Skyline). The CEM is trained to optimize for the correctness of the initial guess as well as the prediction after the predicted intervention. The idea of the method is simple and straightforward. Extensive experiments explore design choices and show strong results in practical settings.
Strengths
* The paper is well-written. Extremely easy read and all concerns I have about the method are covered in experiments. * Simple and effective idea anchored on the premise that models not explicitly trained to receive intervention might not handle intervention well (empirical support in Fig 3, 4). * Questions proposed in P6L221 - 230 effectively guides reader through the work and the authors' through process. * The authors care about how this method could realistically be applied in real-world settings. * Heuristics for tuning hyperparameters for the losses are provided. * Considers realistic settings of applying IntCEM where human-intervention might not be greedy-optimal and show empirically that IntCEM still performs better than baselines under non-optimal interventions (even random intervention). * Observe desirable properties for application in the real world (P7L265)
Weaknesses
* The only weakness is lacking human study but this could be conducted rather straightforwardly. Given the extensive study on intervention policies, I believe this to be merely nitpicking. * This is obviously a work that would induce plenty of follow-up work. Perhaps adding a section for what types of design choices have not been explored yet (e.g. other RL methods besides BC) could help stimulate the research community.
Questions
* How severe is information leakage for CEMs? This is tangent to this work's contribution but since this work is based on the CEM framework, perhaps the authors could comment on whether IntCEM succeeds because of the leakage of downstream task information into the concept embeddings? * Typo: * P2L39 artefact -> artifact
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes.
Summary
The authors in this paper proposed a novel method of improving test-time interventions for Concept Bottleneck Models (CBMs). Although many CBM works showcase their ability to do intervention, none of them explicitly motivate the learned model to do well on intervention during training phase, hence hindering their ability and reliability to do intervention correctly. This work builds on top of concept embedding models (CEMs) and constructs an additional probability distribution (parameterized by neural networks) to learn what concepts to intervene on. To learn this distribution, the authors proposed a composite of three losses, where the first loss (Rollout loss) follows a greedy strategy of selecting concepts that yield the maximum probability of the given class, the second (task prediction loss) penalizes wrong predictions with many interventions, in which the penalty scales with the number of intervention performed. Finally, the third loss concept loss is just binary cross entropy to promote the correctness of the concept prediction. Empirical results on MNIST and CUB (and their variants) show that the proposed formulation performs competitively against state-of-the-art CEMs/CBMs that also utilizes interventions to improve performance. The authors also argued that the sampling from the learned distribution is much more scalable than current state-of-the-art methods.
Strengths
The paper is well-written and is organized in a chronological and clear manner. Introduction sufficiently motivates the problem formulation. The background and related work is written neatly so that the paper is sufficiently self-contained. The description of the method and notation is also clear and easy to follow. And experiments have clear objectives and empirical results clearly demonstrate the effectiveness of their method. Given CBMs are a very popular topic, and that there is a lack of work in the current field that focuses on improving test-time interventions in CBMs, I consider the formulation proposed in this work of significantly novel. It definitively addresses a limitation of CBMs/CEMs and proposed a good formulation that addresses that limitation.
Weaknesses
There are no major weaknesses in this paper. But a few things that I suggest the authors to add for the sake of completeness: As the main selling point of interpretable-by-design methods are interpretability, it feels a bit odd that authors did not add any examples of interventions on a particular sample. Even though the metrics clearly show the method is effective, a few concrete test samples and use of intervention can better illustrate what interventions do, how the posterior of the task prediction changes before/after interventions, etc. Potentially, illustrations could help readers who may perhaps be from a different field.
Questions
1. I understand that “the single MC sampling per training step” is valid, but what is the main motivation for doing so? Is it to improve training speed? And do you speculate, (even better, show with ablation studies) that improving this estimation can improve convergence when optimizing your objective? 2. In Figure 3, can you explain why intervention could cause a drop in performance for Join CBM-Logit? Since the proposed objective explicitly optimizes for CE after intervention, would you agree that your method trained to prevent a drop in performance after intervening from happening? 3. How long does it take to train typical model with the proposed objective? I would imagine the rollout loss can be slow when T is large. Does restricting T to be small heavily impacts the performance of the trained model?
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
The limitations of this work is addressed in the main manuscript and is well-stated. The answer to my questions above can potentially brought up as limitation, and if so, I would suggest the authors to add that to the limitation section.
Summary
The authors introduce IntCEMs, an extension of Concept Embedding Models designed specifically to react correctly to external interventions to the learned concepts. Compared to regular CEMs, IntCEM feature two additional elements: a policy that, essentially, guesses what interventions a human expert would do on the model for any given input, and a penalty that regularizes the model to achieve high accuracy under interventions. The idea is to ensure the model performs well even after interventions - under the assumption the model requires interventions to ``ask for help'', i.e., to obtain the ground-truth label of certain concepts at test time. Experiments show that in fact IntCEMs outperform CEMs and CBMs on four data sets at test time under interventions, even from a different policy. **Post-rebuttal update**: Increasing the score by one point under the assumption the authors will fairly highlightly leakage as a limitation of CBMs/CEMs/IntCEMs and its interaction with interventions in the paper. I am still a bit concerned about impact, but the contents of the paper are good quality overall.
Strengths
+ English is good, narrative is generally good but parts of it are confusing, see below + Motivation is sensible + Proposed approach is also sensible + Empirical evaluation is generally positive, with only some minor drops in concept quality depending on choice of \lambda_roll + Related work is sufficient
Weaknesses
- One issue is limited significance, in the sense that the authors tackle a problem that affects only a certain type of operation (namely, a *specific* type of interventions - not all possible interventions) performed on a certain type of model (CEMs, CBMs). It is true that the performance gain of IntCEM vs CEM under interventions can be substantial (CelebA in figure 3), but it is generally more modest (CUB, CUB-Incomp; the diff in perf for MNIST-Add and MNIST-Add-Incomp are somewhat biased by the difference at Groups Intervened = 0, which is not due to extra robustness to interventions). This is not a deal breaker, and indeed my score is positive, but it explains why I decided not to go above weak accept. - Writing is a bit confusing. It took me a while to understand where the ``learning to receive help'' fits into the abstract and introduction. The idea is that models can ask users to help them at test time by - essentially - requesting the ground-truth label of certain concepts, which means that this work is closer in spirit to concept-level active learning (but at test time) rather than to interventions proper. These have a causal connotation and can be used for changing the concepts to *any* value, not only to their ground-truth values. See for instance all the literature on algorithmic recourse. The target task should be made more explicit, at the bare minimum in the introduction. I also feel the text abuses greek letters (all of section 3) a little, hindering readability. Every time I see, say, \eta, \psi, \omega... in the text, I have to look back at their definition. The $c$ letter also appears in too many variants. More generally, the text feels quite compressed. - Evaluation is only carried out against CEMs and CBMs. Other self-interpretable models exist and have been published recently. Considering the focus on CBMs, this is not a major issue, but it is an issue nonetheless. Minor issues worth fixing ------------------------- - line 102: the range {0, 1}^k is wrong, \tilde c_i is set to 0.5. - Considering \mu_i also appears in the definition of \tilde c_i, Eq 1 is probably too complex for what it needs to achieve. Is it possible to simplify it?
Questions
Q1. It was shown by: Mahinpei et al. "Promises and pitfalls of black-box concept learning models", 2021. Margeloiu et al. "Do Concept Bottleneck Models Learn as Intended?", 2021. that CBMs suffer from "concept leakage", whereby learned concepts encode information from unintended sources (like unobserved concepts), with negative consequences of interpretability. I am wondering whether 1) How IntCEMs fare in this regard; specifically, I am only aware of two self-interpretable models that specifically address this, namely: Havasi et al. "Addressing Leakage in Concept Bottleneck Models", 2022. Marconato et al. "GlanceNets: Interpretable, Leak-proof Concept-based Models", 2022. A brief comment on IntCEMs vs leakage would be welcome - this is an important issue. 2) Robustness to interventions can prevent leakage - the link is that the model is specifically trained to be robust against replacing a (possibly leaky) concept prediction with a (presumably non-leaky) concept annotation. Q2. Why did you compare only against CEMs and CBMs? There are other self-interpretable models out there, like the two aforementioned works, part-prototype networks, concept whitening, self-explainable neural networks, and many others. They all support interventions.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
I think so, in section 5 there is a brief discussion on some limitations (e.g. time complexity). The issue of concept leakage is not mentioned though.
Response to the Rebuttal
I thank the authors for their detailed rebuttal. > Message of the paper and experiment design Thank you for the discussion. I do believe now I better understand the notion of intervention here. > Policy Thank you for the clarifications. Of course, I see that the policy is dynamic and not a static one. My typicality question was a very simple and empirical one - in practice what concepts end up being intervened on, and are they simply the typical concepts? This should be a simple statistic compute, if I am not mistaken. If the answer is yes, I think this has important implications. > Extension to other kinds of concept models and datasets Thank you for this discussion. I find the argument authors raised to be completely fair. Due to the overlap in the exclusion period and moving forward I will not consider the inclusion of P-CBM or Label-Free CBM results. I would still find it informative to add a (short) discussion of this in the limitations. Similarly, I still find the lack of larger datasets to be a limiting factor to demonstrate how much the method would transfer to real world problems. > IntCEM improvements and presentation of results Thank you for creating this table. > Overall Comment I thank the authors for an informative rebuttal. I find several of the authors' arguments convincing and will revise my score accordingly. I would still appreciate if it were possible for the authors to answer the simple question I asked above. To clarify, while I am leaning towards an accept position and find the ideas interesting, the reason I do not increase my score further is due to the lack of real-world settings (larger datasets, more complex problems that would clarify that results transfer beyond MNIST/CUB) and user studies which would strengthen the significance of the work.
Thank you so much for your reply and for leaning towards acceptance after our rebuttal. We really appreciate the time taken to reply to our rebuttal and your careful answer. Below we reply to your specific concerns. Please let us know if there are any further concerns and/or questions after our replies below. ### Typicality Apologies on our end for misunderstanding your concern. We now see what you mean, and we agree with you. Empirically, we observe that certain concepts are selected with a higher probability at earlier steps of rollouts of our policy $\psi$. For example, we observed that "breast pattern" and "wing shape" were two concepts that were often selected in the first few steps by $\psi$ for the IntCEM we used to generate the plots in Figure 1. Intuitively, these seem to be highly informative concepts for some underrepresented classes. This is indeed an interesting observation, and we will discuss it in Section 4.3 while also including a histogram showing these statistics as part of the appendix in our updated manuscript. Nevertheless, for the sake of fairness, we are not allowed to include this histogram as part of this reply, and therefore we apologize that we cannot provide this figure at this moment. ### Large datasets and user studies Our evaluation consisting of studies on task performance, concept performance, receptiveness to different intervention policies, and study of our own learnt intervention policy across five tasks (three of which are real-world datasets) unfortunately left us with very little room for adding even more baselines, datasets, or well-crafted user studies. Nevertheless, we hope that the publication of our method in an easy-to-use Pytorch-based installable library will enable future research to be easy to carry regardless of the task or setup. As you suggest, however, we will make sure to include a short discussion of the need for future larger-scale studies, as well as user studies, in our limitations subsection.
Thank you for the detailed reply. I have only one comment. > [...] leakage may lead to detrimental interventions because a CBM’s downstream label predictor may learn to rely on information leaked from concept representations [...] > This issue does not arise in CEMs and IntCEMs because [...] one essentially performs a swap between embeddings, allowing the model to maintain any leaked information from the input/other concepts/task labels as part of the embedding of each concept. This is a fair point. However, based on your reply, I take that IntCEMs are susceptible to concept leakage. This is an issue in and of itself. Of course, leakage is not the focus of this paper and I do not expect the authors to drastically change the method or the text to address it. But I think it is worth informing readers of this issue (as it /does/ impact interpretability of the learned concepts) and that there exist works that specifically address it. In other words, I would be disappointed if the authors chose to dismiss it.
Dear Reviewer Zgwp, Thank you for getting back to us regarding this point. We agree that leakage may have implications beyond allowing IntCEMs/CEMs to have higher accuracy, and those should be properly studied in what could probably be a paper on its own. As discussed in our rebuttal, we will discuss our findings regarding leakage in our updated discussion section and include the experiment mentioned above in a new appendix. Nevertheless, our paper's main claims, results, and observations still hold. Moreover, we believe this preliminary leakage study on concept-based architectures, together with the discussion we will include, will open up the door for immediate future research and increase the potential impact of this work. For that, we thank you and the other reviewers for bringing this to our attention.
Reply
> More explicit definition of “interventions” I do appreciate this change, thank you. > IntCEM’s performance gains Thank you for the clarification. > Leakage-free CBMs [...] effectively operates like a CBM in practice and the extra leakage bypass it offers is a constrained version of the bypass allowed by embeddings in CEMs. Based on your general reply, would I be correct in thinking that a "less leaky" model is likely to see smaller benefit from better chosen interventions? I'm just curious. > concept interventions in GlanceNets are not too different from those in Hybrid-CBMs [3], [...] My understanding is that Hybrid-CBMs are CBMs with additional unsupervised concepts. Presumably these are used for prediction. GlanceNets do not use unsupervised concepts for prediction. In what sense are their "concept interventions" similar? Also, what sense Hybrid-CBMs "underperform"? > Concept Whitening, SENNs, ProtoPNets I see. I think these points are valid, at least at a high level. Thank you for the explanation. Edit: fixed the formatting.
Thank you for replying to our rebuttal; we really appreciate your feedback and the thorough review. Below we reply to the questions raised in the comment above: > "Would I be correct in thinking that a 'less leaky' model is likely to see smaller benefit from better chosen interventions?" This is a really good question. We think a thorough study of this may be needed to answer your question. However, if we were to speculate, our intuition from this work is that models that allow more information through their bottlenecks, and that are able to maintain that information after an intervention is performed (i.e., the intervention is not destructive), may be able to perform better after interventions. This is because the label predictor has more information it is able to exploit after an intervention is performed. Nevertheless, we emphasize this is **not** a sufficient condition as seen in Hybrid-CBMs. In these models, as one increases the amount of extra unsupervised capacity in the bottleneck, interventions become less effective due to (1) the model learning possible shortcuts directly from the input features to the task prediction via the unsupervised neurons, and (2) the lack of clarity on how one should modify these unsupervised neurons when one intervenes on a specific known concept. Notice that CEMs and IntCEMs circumvent this shortcutting by introducing training-time regularisers that implicitly encourage the distribution of concept embeddings to be distinct when the concept is "on" than when it is "off". > Hybrid-CBMs and GlanceNets You raised a good point, and we acknowledge that GlanceNets do not use the extra unsupervised "concepts" for prediction, while Hybrid-CBMs use the unsupervised capacity in their predictive process. We meant that they were similar in that, during interventions, only the supervised concepts are modified (without touching the unsupervised neurons in the bottleneck). This leads to "ill-defined" states during interventions as the change intended from intervention is not propagated to all relevant neurons in the bottleneck. The original CEM paper empirically showed that this leads to Hybrid-CBMs being significantly less receptive to interventions than all other baselines they compared against (including CBMs and CEMs). If GlanceNets would use the unsupervised concepts during prediction, we would expect a similar situation to develop. More importantly, however, precisely because GlanceNets do not use the unsupervised concepts for prediction and they use them only for reconstruction for the open set recognition, we expect interventions to suffer significantly when the set of training concepts is incomplete w.r.t. the downstream task, a common setup in real-world tasks (e.g., $\texttt{MNIST-Add-Incomp}$, $\texttt{Cub-Incomp}$, $\texttt{CelebA}$). Because of this, and equally as important because GlanceNets were not designed or even evaluated to intake interventions, we decided to focus our evaluation on models that are very likely better at receiving interventions such as CEMs.
Reply
Thank you again for engaging in the discussion. I'll keep your points in mind when discussing with the other reviewers. To be clear, I still think the problem being tackled is relatively minor, however the paper's contents are definitely there. Given the authors promised to discuss leakage in more detail, and assuming they will discuss this as a limitation of IntCEMts, I will increase my score to 7.
Many thanks to the authors for the response. I do agree that information leakage may or may not be (un)desirable and explicitly stating this in the paper is a good way to inform readers about the potential catch with this method working.
Thank you
We would like to thank all reviewers for their valuable reviews and for engaging with us in the discussions that followed. Your feedback has allowed us to significantly improve our manuscript.
Decision
Accept (spotlight)