State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining concepts in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.
Paper
Similar papers
Peer review
Summary
The paper proposes State2Explanation, a framework for training RL agents in such a way that both the human and agent benefit, the Protégé Effect as the authors state. The basic idea is to learn a joint embedding space with "temporal" concepts that actively helps the agent train better by shaping their rewards. At test time, these explanations are provided to humans to also help train them.
Strengths
A strength of the paper is I feel the motivation. Explainability in RL is harder than supervised learning due to the temporal added component, so approaching this problem is well motivated. In addition, the interplay between agent and human is also very exciting, I love the idea of the two helping and playing off each other. I feel this is a great way to impose human understandable concepts onto an RL agent's policy. I also appreciate the rigour in the user study (although I need some clarifications in the rebuttal), it's clear to me the authors tried here.
Weaknesses
What I always look for in a paper is a single nice idea, and I am not certain this paper really contains one, let me explain by iterating your claimed contributions. Regarding the desiderata: (1) the idea that a 'concept should be grounded in human domain knowledge' is not a new idea I feel. The original T-CAV paper noted this by e.g. using zebra stripes to explain zebra classifications, rather than e.g. logit values. Simple saying the same thing is true in a sequential setting doesn't seem that original to me. I'm not aware of a great many people suggesting q-values are a good explanation. (2) The idea that a concept should relate to the task goal is an interesting one, and could be useful in certain context, but I feel it doesn't generalize well. For example, if I'm in a self-driving car going from a => b, and halfway through the journey the car brakes to avoid hitting a pedestrian, your framework seems to posit the explanation should be "I avoiding killing the human so I could still get to point b". I feel that's a questionable belief system here we are teaching the agent. (3) is a nice idea, but I don't feel it's particularly surprising, the idea that explanations should be robust and generalisable is well agreed upon. The framework you propose does seem quite interesting to me, but reward feedback loops are not my area of expertise, so I defer to my colleges to evaluate the novelty of such a framework. Section 5.2: I am slightly concerned that the explanations are not always matching in Figure 3, does that mean that this method will inevitably give "wrong" explanations? Also, I don't know what i, j, and k are in the figure? Now I personally believe it matters more that it just helps people in a user study, which you show it does, so that's great and I feel makes up for this. Section 5.3: If I understand this correctly, your method doesn't seem to show much difference here at all, and in fact Fig 4a seems to show the baseline outperforming it? User Evaluation: I appreciate the attention to detail here, but a lot of details are missing or (I feel) badly explained. I don't really understand how people's performance could get worse in group $E_{A}$, all they are being shown is the action as an explanation, so the idea their ATS would get worse seems extremely odd to me, and makes me feel I am misunderstanding something here. I tried to explore the appendix to understand what was happening, but didn't have much luck. I think there's some citations that you might also like to be aware of regarding human friendly concepts in sequential settings. Ji, Ying, Yu Wang, and Jien Kato. "Spatial-temporal Concept based Explanation of 3D ConvNets." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. Kenny, Eoin M., Mycal Tucker, and Julie Shah. "Towards interpretable deep reinforcement learning with human-friendly prototypes." The Eleventh International Conference on Learning Representations. 2023.
Questions
1. During the training there's no feedback from the human right? If I understand, the human defines the concepts, and then these help the agent train better? (but Fig 4 seems to show this isn't the case?) 2. Whilst you used a CNN in connect 4, does this scale to harder Deep RL problems? 3. Is my understanding that Fig 3a shows the explanations are often "wrong" accurate? 4. What is i, j, and k in Fig 1? 5. How did you pick a sample size in the user study? Although it seems fairly rigorous, I don't see any power analysis, and you aren't clear what your attention checks are? It also seems strange that you had 66->60 and 98->90 in the two domains, a perfect balance for 15 in each group. I might recommend using Prolific instead, it seems to have less issues compared to MTurk which is full of bots and LLMs. I am happy to revise my review after the rebuttal.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
This is ok
Summary
Inspired by the Protege effect, learning and developing explanations should provide a dual benefit, both to the readers of the explanations, and to the developers of the explanations. Based on this idea, the paper proposes State2Explanation, an algorithm to learn joint embeddings between state-action pairs and concept-based explanations. This allows for reward shaping, which benefits the explanation developer, while also providing explanations. These claims are validated for agents in two different reinforcement learning settings: Connect 4 and Lunar Lander.
Strengths
1. Provides concrete desiderata for a concept is, including the need for generalizibility and its relationship to the task goal 2. Reward shaping through explanations is potentially novel, and explores use cases of explanations beyond explainability 3. Thorough evaluation investigates the impact of various additions to the model, making it clear what the impact of Information Filtering (InF) and Temporal Group (TeG) are 4. User study demonstrates real-world viability model through a human explainability lense
Weaknesses
1. Concepts for domains are dependent on expert knowledge for state-action pairs, making it unclear how easy it would be to generalize beyond well-studied games. This is especially the case when state-actions pairs tend to get very large. 2. An additional baseline would make it clearer what the impact of each study condition was in Section 6. In particular, a baseline with no intervention/information would make it clear how much the rise in performance is due to the additional practice attained from the PreTest when completing the PostTest. 3. If concepts for Lunar Lander are derived from the existing domain reward function, then claiming that $M_{LL}$ "informs reward shaping comparable to expert-defined dense reward functions" seems to follow from the definition of the concepts rather than an indicator of the performance of the algorithm.
Questions
1. For reward shaping, how is the amount of reward determined/amount the reward function is changed by? 2. In general, who would annotate the explanations, and what is the impact if there are multiple explanations for a single move/conflicting annotations?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
4 excellent
Limitations
The paper discusses limitations
Summary
The paper proposes a framework to incorporate explanation concepts to sequential decision tasks. The framework can be applied both to the training of the agent by improving RL and to provide explanations to end-users during deployment. The framework is tested using two simple games, Connect 4 and Lunar Lander, where it shows it help improve training performance and user performance.
Strengths
The main strengths of the paper are: 1. It provides an unified framework for explanations which can be used both for training and deploying. 2. The explanation framework is based on user-understandable concepts and terms. 3. The paper provides a sound theoretical and empirical analysis of the proposed framework. 4. The paper performs user testing and subsequent data analysis in a methodologically sound manner, rare in NeurIPs papers.
Weaknesses
The main weaknesses of the paper are: 1. The paper condenses two much information in very little space, including impossible to read figures. It is very hard to read. 2. The paper fails to discuss how difficult will be to create frameworks for more complex scenarios where there are more and more complex concepts. For instance, consider a system making decisions about buying and selling stocks. 3. Because of 2, the ideas presented in the paper may never be applicable to real-world problems. 4. It provides a very brief description of how the concepts are actually created, stored, and verified. 5. The paper does not discuss situations where the explanation is wrong or inadequate for a situation (for instance, there is a better move), and how that impacts training and user performance.
Questions
1. How do you expect the framework to perform when it is applied to more complex tasks and domains, both in terms of describing the actual concepts and on the performance during training? 2. How hard is to identify, create, store, and debug concepts? 3. What kind of wrong explanations it produces? How often? How that impacts training and user performance?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
4 excellent
Limitations
The paper does not do a good job of discussing how it can scale to real-world applications. By not doing so, it failed in identifying an important limitation of the work.
Summary
The authors propose a unified framework called State2Explanation (S2E) that combines learning a joint embedding model between state-action pairs and concept-based explanations. The authors draw inspiration from the Protégé Effect, which suggests that explaining knowledge reinforces self-learning. They propose that concept-based explanations can benefit both the RL agent and the end-user by improving the agent's learning rate and the end-user's understanding of the agent's decision making. The S2E framework is designed to inform reward shaping during an agent's training and provide explanations to end-users at deployment for improved task performance.Results on Connect 4 and Lunar Lander demonstrate the success of S2E in providing a dual benefit.
Strengths
The paper introduces a novel framework S2E. The authors suggest that explaining knowledge to the agent can improve its learning rate, while providing explanations to end-users can enhance their understanding of the agent's decision making. The framework is supposed to be useful in providing explanations understandable in various applications.
Weaknesses
There are no significant weaknesses from my point of view, while there are some improvements can be made. Although the paper mentions experimental validations in the Connect 4 and Lunar Lander domains, the proposed method should be tested on more complex tasks to test its effectness. The authors should also consider how concepts can be better selected beyond expert-defined thresholding.
Questions
- To be specific, how concept candidates are selected when performing InF? - How thresholds are determined via qualitative methods? Is there any results showing that different concepts selected will influence the final performance?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See weaknesses and questions.
Response
Thank you for your clarifications. On the topic of the user evaluation my question was the following: Would the post-test agents not outperform the pre-test agents because the post-test agents see 3 more scored games, getting more practice? Therefore, would it not be more fair to have some set of agents practice -> explanation -> practice, and others do practice -> no explanation -> practice, then compare each?
Further Clarification on User Eval to Reviewer g9Rv
I appreciate your follow up. There seems to be an underlying misunderstanding that we did not catch earlier. In our user evaluation in Section 6, the RL agent’s policy in the user study is fixed and the agent is not learning in any way during our user study. We’d like to make this distinction very clear. The S2E framework provides a dual benefit of concept-based explanations to the RL agent and end-user, but at completely different stages. When we discuss the positive effects of concept-based explanations from S2E on agent learning, that occurs *during the agent’s training time*. The impact of S2E on agent learning is discussed in Section 5. When we evaluate the benefit of concept-based explanations on end-users in Section 6, the RL agent has already been deployed, meaning the RL agent is already trained and its policy is fixed. Therefore, our user evaluation in Section 6 is focused on seeing how different explanation types of a fixed RL agent help participants improve their task performance and understanding. At the beginning of Section 6 we will reiterate this distinction to avoid future misunderstandings. With the above clarifications, we’d like to now respond directly to your questions. **Q: “Would the post-test agents not outperform the pre-test agents because the post-test agents see 3 more scored games, getting more practice? ..Would it not be more fair to have some set of agents practice -> explanation -> practice, and others do practice -> no explanation -> practice, then compare each?** There are no “post-test agents” or “pre-test agents” and there is no RL agent learning or practicing in the study. Instead, human participants play games in the “post-test” and “pre-test” stages. Furthermore, all participants play the same number of games, and no user receives more practice. Potential improvement in participant performance due to learning effects is accounted for in the metric. Please see our original rebuttal response on how our metric, ATS, takes user learning effects into consideration since the ATS metric measures the *difference* between the Pre-Test and Post-Test scores. In this manner, we account for participants’ initial performance on the “PreTest” and measure whether the explanations provided in the “Explanation stage” help to provide any *improvements* to their “Post Test” scores. Also, we’d like to point out that your question in the original review about having an additional baseline is still valid for human participants. That is, we could have had an additional baseline where human participants in the “Explanation Stage” received no feedback. However, as we mention in our original rebuttal response, our current study conditions are fair comparisons given that we account for learning effects as well as compare against relevant baselines utilized in prior work (Action-Based & Value Based Explanations). Please let us know if the above clarifies your understanding of our user study or if you have any other clarifying questions. Thank you!
Response
Thank you for your response. I think there was some confusion over my use of the word "agent" as I was referring to human agents (AKA participants). If I understand correctly, do you guys measure the impact that having/not having the explanation has on the post-test score? Or do you measure the difference between post-test and pre-test score? If it's the latter, then how do you ensure that the participant seeing 3 more questions during post-test did not boost their post-test score over their pre-test score?
Response to g9Rv about additional "None" Study Condition + above question
Thank you for the follow up. Our ATS metric measures the difference between participant “Post-test” and “Pre-test” scores. We believe conducting the additional "None" study condition is the best way to analyze concerns about learning effects. Therefore, last week, we recruited participants from AMT and conducted the Lunar Lander and Connect 4 user studies with the “None” condition. Just as a recap, in the “None” condition, participants receive no explanations during the “Explanation Stage”. Each “None” condition includes 15 participants. Note, in total, we recruited 18 participants for Lunar Lander and 16 participants for Connect 4; however, we filtered 3 and 1 participants, respectively, based on our filtering criteria in Appendix E. Note, I am unable to attach the updated graphs from the paper since there's no option, it seems, to share a PDF and no links are allowed in our responses. However, the tables below summarize the post-test ATS scores. The values in the table are the values currently plotted in Figure 5 and Figure 6 in our paper. To both of these Figures, we will also add the “None” study condition results. Recall from the paper that the ATS metric adjusts the participants’ average “Post-test” score by their average “Pre-test” score (see Appendix E.3). Therefore, the table shares the average adjusted “post-test” ATS per study condition. **Table Analysis:** The main takeaway to observe from the tables is the marginal difference between “None” and “Action-Based” for both Lunar Lander and Connect 4. These results show us that even without any information during the “Explanation Stage”, participants are not able to significantly improve their post-test scores from their pre-test scores. **Statistical Significance:** Our statistical analyses show that our S2E concept-based explanations significantly improve participant ATS compared to the “None” study condition. We perform a one-way ANOVA with a Tukey HSD post-hoc test for both Lunar Lander and Connect 4 as stated in our paper. In Connect 4, we find significant improvement on participant ATS with our S2E Concept-Based explanations in comparison to “None” (t(70) = 3.076, p<0.05). In Lunar Lander, we find significant improvement on participant ATS with our S2E Concept-Based explanations that include TeG and InF abstractions in comparison to “None” (t(98) = 3.148, p <0.05). We will report these details in Section 5. **Connect 4 Table of average ATS score per study condition** | Study Condition | ATS: Post-Test| | -------- | ------- | | None | -0.038 | | Action-Based | -0.041 | | Value-Based | -0.104 | | S2E Concept-Based | 0.215 | | GT Concept-Based | 0.256 | **Lunar Lander Table of average ATS score per study condition** | Study Condition | ATS: Post-Test| | -------- | ------- | | None | -0.051 | | Action-Based | -0.049 | | Value-Based | -0.068 | | S2E Concept-Based | -0.043 | | S2E Concept-Based w/TeG | 0.100 | | S2E Concept-Based w/InF | 0.034 | | S2E Concept-Based w/TeG+InF | 0.156 |
Response
W: single idea missing.. No problem, as I say that’s not my area of expertise, so I defer to the other reviewers to rate that aspect. *** W: desiderata.. I understand, I agree with the sentiment of the desiderata, I just feel that it’s not really a significant contribution by itself, although a nice “aside” in the paper, something which is “backed up” by the subsequent experimental evidence. *** W: Sec. 5.2 + Q3: Ok I understand, thanks for the clarification here. Again, I don’t feel it’s a huge issue if explanations are occasionally wrong, it really depends on the downstream application we have in mind. E.g., on a Mars land rover I do feel they have to give certain guarantees, but if you’re just using the to improve task performance in another less sensitive way, I could imagine that’s ok. No pressure, but it could be worth mentioning. *** W: Sec. 5.3 Ok thanks, for what it’s worth I personally struggled a bit to understand this, so it probably would be worth cleaning up the writing a bit. *** W: User Eval.. If you could explain this in even more detail I would appreciate it, I still just really don't understand how this could happen. Even the experimental design? *** Q1: Thank you, and just to clarify, these explanations are for actions right? Not a series of actions? That is, at each action taken, the user must interpret a separate explanation? Rather than say, one explanation for the next 20-40 actions? *** Q2: Ok *** Q4:: Ok thanks. *** Q5: It is good practice to have an equal number of study participants per study conditions to conduct fair statistical analysis comparisons. 100% agree. After filtering participants, we continued to recruit participants to ensure we had 15 participants per condition. But why 15? Why not 10? Why not 50? My concern is that it is well known you can “p-hack” your way to significance by simply increasing your sample size, that’s what a power analysis is for. We filtered participants by analyzing visualizations of the participants’ games. We determined a participant to show no effort if they only played a single action during the entire course of any of their pretest or posttest games. Was there a large amount that played e.g. 2 actions per game? A single action seems a fairly arbitrary cutoff. We did not conduct a power analysis in this particular study given that power analyses require making estimations about each study conditions’ mean or variance which we did not deem appropriate to assume. We will add these details to Appendix E.3. Ok. Usually you would observe similar studies done prior and base your assumptions on that. I know it is difficult, but user studies are not easy to do well. I’d appreciate if you could reply to the remaining concerns, thank you!
Response to GSRV - Part 1
Thank you for your follow up. We respond to your comments and concerns below. I've split my responses into two parts. This is part 1. **“Desiderata + single idea”:** We also agree that our main contribution is the S2E framework. The desiderata is also a contribution given that we are expanding the existing definitions of concept-based explanations in sequential decision making problems. **“Sec 5.2 + Q2”:** We agree and will add such details in the limitations section. **“Sec 5.3”:** We will improve the clarity of the current text in Sec. 5.3. **Q: “these explanations are for actions right...** In both user studies, during the “Explanation Stage”, the user sees an explanation per action taken by the RL agent in all study conditions. In the case of the concept based explanation with temporal grouping (TeG) condition, some explanations are grouped across identical sequential actions (see Sec. 4.3 and 6.2). However, the user still sees a single explanation per action, they are just made aware that such explanation holds true for “N” steps. For example, if the lunar lander agent “fires the right engine” 5 times consecutively. A concept-based explanation w/TeG would be: “Fire right engine for the next 5 steps to decrease agent tilt”, and the user would see such an explanation across the 5 steps. In comparison, a concept-based explanation without TeG would be “Fire right engine in this step to decrease agent tilt”, and the user would see such an explanation for each of the 5 steps.
Response to GSRV - Part 2
Thank you for your follow up. We respond to your comments and concerns below. I've split my responses into two parts. This is part 2. **User Eval: If you could explain this in even more detail..** In our previous response, we provide one hypothesis on the phenomenon of decreased user performance when exposed to “action-based” or “value-based” explanations. That is, these explanations may cause end-users to project wrong understandings to agent actions, and therefore their performances decrease in the post-tests. To study exactly why such decrease occurs requires performing more, in-depth qualitative user analyses such as conducting semi-formal interviews after the user studies to probe participants more about their experience and understand where such decreases may be coming from. **Experimental design:** We detail our user study in Appendix E.1 and provide visuals of the study in Appendix E.4. The user study consists of four stages in order: “practice” stage, “pre-test stage”, “explanation stage” and “post-test” stage. Prior to the “practice stage”, participants are introduced to the task, and the game they will be playing. After the “post-test stage”, the user also answers a few survey questions in a questionnaire. The “practice stage” allows users to play 2 practice games–participants are told these are unscored games, and they are meant as practice to get users further familiarized with the game. The “pre-test stage” consists of playing 3 scored games. Users are told these games are an assessment and are scored. The “explanation stage” consists of the user interacting with a well-trained agent as the agent plays the game. The user is able to click through every action the agent plays in the game and depending on the study condition, the user sees an accompanied explanation type for each action played. In Connect 4, the user watches a Connect 4 RL agent play 3 games. In Lunar Lander, the user watches a Lunar Lander RL agent play 1 game. The “post-test stage” consists of playing 3 more scored games. As our analysis, we study how explanations provided in the “explanation stage” may help improve the user’s task performance. Therefore our metric is ATS, adjusted task score, is computed as a difference between the aggregated “post-test” task scores and the aggregated “pre-test” task scores. The single game’s task score is computed as an aggregation of user rewards received during the game (see Sec. 6). **Q5: But why 15?...sample size.,.power analysis"** The best way to address your concerns about our sample size per study condition having enough power is by demonstrating a priori power analysis as well as post-hoc power analysis. Please be aware that priori power analyses, as I mentioned earlier require setting assumptions. We will set the minimal assumption of effect size since we believe estimating group means are unrealistic when conducting such study for the first time. All power analyses are conducted in R using pwr.anova.test (prior power analysis) and power.anova.test (post-hoc power analyses). We will include these results in the appendix. The priori power analyses assumes an estimated power of 0.8 and estimated effect size of 0.4. This effect size is moderate, considering Cohen’s guidelines for effect size [1,2]. With these assumptions, we find N=15.89 for Connect 4 and N=13.09 for Lunar Lander. Our current study has N=15 for both studies, which falls within these estimated N values. We also perform a post-hoc power analysis given after performing the user studies we have the actual group means per study condition. The post-hoc power analyses show that our N=15 has a power of 0.99 and in Lunar Lander our N=15 has a power of 0.96. Thus, our statistical analyses reported have high statistical power. Overall, the above analyses show that N=15 is a reasonable number of participants to conduct for our user studies. 1. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates. 2. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155 **Q5: A single action seems a fairly arbitrary cutoff...** Playing a single action was an obvious tell-tale that a participant was not performing the study with effort and was trying to finish as fast as possible. We visualized every participant’s actions in all games. There were only two distinctions we could fairly make. Participants who pressed a single button (single action) until the end of most games, and participants who did not press a single button and instead played actions that were more varied, showing effort. Trying to determine a more nuanced cutoff for filtering did not seem appropriate and could induce additional biases.
Thanks to the authors for the information in the rebuttal. I confirm I have read the rebuttals provided by the authors.
Thanks
Thanks to the authors for the information in the rebuttal. I have read the author's response and fellow reviewer's feedback. I believe that the authors have addressed most of my concerns.
Decision
Accept (poster)