Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples

Summary

This paper poses the problem of learning a controller from a batched dataset containing time-series observations of the world, actions, and a value estimate (e.g., for determining how to treat a patient given the results of medical tests based upon patient outcomes). This problem is considered with a POMDP formalism with execution traces. The paper imposes assumptions about the linearity of the mapping from the belief space to the value space in order to create some analytical properties about how wrong the model might be or to mitigate the model error. The algorithm, ABC, is evaluated against baselines in a set of batched-data experiments, including in a healthcare applicaiton.

Strengths

+The paper addresses an important problem of bringing safety to machine learning +The paper clearly states assumptions and presents logical arguments and definitions to support its thesis. +The evaluation shows positive results and does so in important domains (e.g., healthcare)

Weaknesses

-The paper seems to be addressing the problem of offline reinforcement learning without actually addressing offline reinforcement learning. Though, this paper does cite a plethora of papers that address this topic. As such, it is difficult for the reviewer to properly contextualize this paper in this relevant prior work. -The paper appears to be missing a number of baselines for offline reinforcement learning, as shown below. Some of the baselines chosen do not have access to the same set of information available to the ABC algorithm (e.g., the BC model). Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A. and Mordatch, I., 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, pp.15084-15097. Kumar, A., Zhou, A., Tucker, G. and Levine, S., 2020. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, pp.1179-1191. -The paper develops theory to bring accountability to this problem. However, the results section provides some relatively simple computational examples and a qualitative description that is in the eye of the beholder. It would have been better to provide a clearer, more convincing test to demonstration that there are clear guarantees and fulfilled analytical properties. -One could have considered a human-subject experiment to evaluate whether this approach is really "accountable." Literature on accountability could have been considered as well. For example, see: Kim, B. and Doshi-Velez, F., 2021. Machine learning techniques for accountability. AI Magazine, 42(1), pp.47-52.

Questions

-Why is this problem different than offline reinforcement learning? -Why are offline RL baselines not included? -How do the results provide convincing evidence of accountability in a non-superficial manner? -For what class of problems is the linear assumption for the mapping from belief to value reasonable? -What is the computational complexity of the approach? -How scalable is the approach with the size of the state space? -By relying on belief space modeling, why is this approach relevant to large, real-world problems?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper does not mention "limit" once.

Summary

This paper investigates imitation learning in scenarios with limited data. The proposed approach (ABC) involves utilizing a linear combination of the belief space to generate accountable decisions. The authors evaluate the performance of ABC in simulated and real-world healthcare scenarios, highlighting its ability to effectively handle batched control tasks while maintaining a high level of performance and accountability.

Strengths

1. The paper is presented in a clear and easily understandable manner. 2. This article investigates a highly meaningful direction, which generates reliable strategies through offline data.

Weaknesses

1. Assuming that the value is a linear combination of belief states is a strong assumption, which may not hold true in certain tasks, particularly those that are harder and more complex. 2. The optimization problem (Eq. 9) seems to be ill-defined. Eq. 9 could have an infinite number of solutions because the effect of $l\circ b$ is equivalent to $cl \circ c^{-1}b$, where $c$ represents a scalar. This would have an impact on the distance calculation when solving Eq. 10. It would be beneficial to address the limitations of the optimization procedure and conduct additional experiments to examine the influence of different solutions to the optimization problem (Eq. 9). 3. The paper lacks a discussion and comparison with offline reinforcement learning, which is an important related work and a strong baseline. 4. The evaluation is confined to simple environments. For more challenging tasks, please refer to [1]. Reference: [1] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

Questions

1. Could you provide the results of offline reinforcement learning methods in the test environments? 2. Can you provide the environment results obtained when applying the proposed approach to more challenging environments?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

This work is primarily focused on simple environments and does not address more complex scenarios.

Summary

This work proposes accountable batched control with five desirable properties. The design is motivated by the fact that the reward or feedback of trajectories are hard to obtain in high-stake responsibility-sensitive applications. The minimal hull subset of the decision corpus is constructed for the decomposition of the value function for each candidate action. Then the optimal policy is selected in terms of the weighted value function. This work demonstrates the promise of the five properties on one real-world healthcare dataset and one simulated maze environment.

Strengths

After the rebuttal I tentatively raised my score from 5 to 6. --- The paper is quite obscure and not easy to understand, making it challenging to grasp the complete understanding of the methodology it presented. However, I would appreciate it that the authors included a video presentation in the appendix and saved my time. In particular, the animation of the algorithm makes it easier to understand. The introduction of the batched control is a seemingly novel contribution. The experimental setup and analysis are presented in a concrete and well-written manner, albeit deferred to the appendix due to space constraints. In addition, the authors attached an anonymous link to their code implementation.

Weaknesses

While I may not have a comprehensive understanding of the literature, it appears that the focus of this paper leans more towards reinforcement learning rather than the chosen primary area of interpretability and explainability. I am uncertain about the reasons behind the batched controller possessing the five advantageous properties in comparison to other well-known methods, including Q-learning, model-based RL, and behavior clone. Although I skimmed through the Appendix B, the interpretation is still unclear.

Questions

1. The term *decision corpus* is never explained. How does it differ from the trajectory in the offline data? 2. The batched controller appears to neglect the historical decisions and fails to account for the temporal correlation among actions. How can you guarantee that the policy will receive the optimal accumulated rewards?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have include a separate broader impact section in the Appendix.

Summary

The paper presents the Accountable Batched Controller (ABC) based on the example-based explanation framework as a solution for offline control in responsibility-sensitive applications. Through experiments on simulated and real-world tasks, the method shows accountability, conservation, and adaptability.

Strengths

The paper proposed a novel method for accountable batched control with decision corpus, theoretically proved the existence and uniqueness of the decomposition under mild conditions, and conducted solid experiments to verify the effectiveness and desired properties of the method. The paper is clear-presented and well-organized.

Weaknesses

To further improve, more experiments on real-world control tasks are needed.

Questions

More in-depth analysis of Table 2 may help gain deeper insight regarding the suitability for low-data regimes. Why is section 5.1 highlighting P1-P3?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

As the authors also mentioned, the current accountable batched control method is limited to low-dimensional control tasks and may not perform as well in high-dimensional control systems, limiting its potential contribution.

Summary

I found the paper somewhat hard to read and understand, so here I’ll present a summary that’s quite different from the author’s presentation. In offline RL, or other settings where there is a performance metric to optimize, we can consider two simple baselines: 1. Nearest neighbors: For each action, find the most similar transition(s) in the dataset, and use those to estimate the value of the action, then take the action estimated to be best. 2. Supervised learning: Train a model to predict the value of each action, and use that to estimate the value of actions. The advantage of (1) is that we get a notion of explainability (visualize the nearest neighbors that were used to estimate action value), but the disadvantage is that it does not work well (because similarity in the input space may not mean that decision-making will be similar). The advantage of (2) is that it works better, but is less explainable. So the first idea is that we can get the best of both worlds by still using (2) to train a model, but then use the embeddings (i.e. the activations before the final linear layer) as inputs for a nearest neighbors approach. However, this can still have problems: in particular, for a new test point, the nearest neighbors may all be very tightly clustered but far away from the test point. Ideally, in such a situation, we would find nearby points in a variety of _different_ directions, and average them, so that our estimates are interpolations rather than extrapolations in the embedding space. So, instead of finding the nearest neighbors in our dataset, we find a minimal set of points from the dataset such that the current embedding falls within the convex hull of those points (or, if no such set exists, the embedding is as close as possible to the convex hull). We automatically discard any actions that are far away from the best convex hull, since they are likely OOD. This gives the author’s method: ABC. (In the paper’s presentation, the embeddings are called “beliefs”.) The authors test their method in a variety of settings: 1. Heterogeneous Pendulum: Similar to classic Pendulum, except that there is a 50% chance for the action effects to be swapped. 2. Maze: A 2D setting with a wall separating the start and goal states, with two openings in the wall. 3. Ward: Healthcare task, in which the task is to predict whether or not to use an oxygen therapy device. In heterogeneous Pendulum, the authors show that ABC performs slightly better than model-free RL and model predictive control, and much better than other baselines. They also show the effect of ablating $\epsilon$, the hyperparameter that controls which actions are considered OOD. In Maze, the authors collect data from a variety of different behavioral policies, which have to be composed together to solve the task, and show that ABC is capable of this. There are two different ways to solve the task, corresponding to the two openings in the wall. The authors show that ABC can show both methods of solving the task, and that when visualizing the points forming the minimal convex hull for the resulting actions, they can be attributed to the behavior policies that used the same hole in the wall to solve the task. They also show that by increasing the proportion of different behavioral policies, you can control which of the two solutions ABC is more likely to use. On Ward, the authors show that ABC performs on par with behavior cloning (BC) using a multilayer perceptron, and performs better than k-nearest-neighbors and BC using a linear model.

Strengths

1. Once I understood the idea, I found it simple and intuitive, with a clear story about why it should be helpful. 2. The application of machine learning to healthcare is important, and accountability and conservatism are important properties to ensure in such a setting. 3. There are a variety of experiments demonstrating the claimed properties of the method.

Weaknesses

**Properties of ABC** The authors list five properties that ABC satisfies. I agree with the author’s points that ABC works better with low data (at least relative to kNN) and that ABC can be used in the reward-free setting (at least for continuous actions spaces). However, I’m not convinced of the other three properties: 1. Conservatism: The authors claim that ABC is conservative, I believe because they filter out actions that have a belief corpus residual that is too high. While I think the authors are probably correct, I don’t think their experiments show it: in all of the experiments that compare against baselines, black-box methods perform about the same as ABC, even though black-box methods are not normally “conservative”. 2. Accountable: The authors claim that ABC is accountable because for any action taken by ABC, we can identify data points in the training dataset that make up the convex hull that determined that particular action, and show those to the user. However, there isn’t even a qualitative evaluation of how useful such explanations are. The closest is Figure 5, which visualizes the belief corpus as points on a 2D grid whose axes are uninterpretable (belief dimensions 1 and 2) relative to the test data point, but looking at that figure I do not feel like I have understood very much about ABC’s decision in that setting. 3. Adaptive: To show that the ABC is adaptive, the authors perform an experiment in which they change the composition of the dataset on which ABC is trained, and show that this affects ABC’s behavior. But by this standard, essentially all algorithms are adaptive, including the baselines they compare against (e.g. behavior cloning, which they say is not adaptive in Table 1). It’s not clear why this is a unique advantage of ABC. (Incidentally, on accountability, the author’s technique is extremely similar to the technique of presenting maximally activating dataset examples to explain neuron activations, a common technique for explainability in supervised learning.) **Additional comparisons** I would like to see the authors compare ABC to the first method in my summary, i.e. training a model to predict value / actions, and then using k-nearest-neighbors on the embeddings (activations before the final linear layer). This can be thought of either as a baseline, or as an ablation (as an ablation, it corresponds to ABC without the convex hull aspects). This would be helpful in understanding the effects of the various design decisions the authors make. If performing an experiment on accountability, then I’d like to see a comparison to the dataset examples technique applied to the kNN-on-embeddings model discussed above. **Disagreement with Section 5.2 claim** Section 5.2 notes that there are two hyperparameters: “the number of uniformly sampled actions and the threshold”. It claims that these can be unified into a single hyperparameter, the _effective action size_. However, the experiment doesn’t support this: it simply sets the number of sampled actions (which we’ll call $n_A$) to 100, and then shows the effect of varying the percentile threshold $\epsilon$. The experiment that should be run would be to use a variety of settings of _both_ hyperparameters, and then check whether runs with similar effective action sizes $\epsilon \times n_A$ have similar performance: if so, then it would be justified to only think about the effective action size. However, my guess is that this will not be the case. **Minor issue with the theory** (Note: set notation doesn't seem to be working below) Proposition 3.10 is false because of the requirement that the convex hull contain $d_b + 1$ examples. For example, suppose $d_b = 2$, $b_t = [1, 0]$, and $\mathcal{D} = \{ [7, 0], [3, 0], [-1, 0] \}$. Note that $b_t = 0.5 * [-1, 0] + 0.5 * [3, 0]$, and so if we have $\mathcal{C} = \mathcal{D}$, then $b_t \in \mathcal{CB(C)}$, and so $r_{\mathcal{C}}(b_t) = 0$ as required by Proposition 3.10. Definition 3.9 requires the minimal corpus subset $\tilde{\mathcal{C}}(b_t)$ to have 3 elements, and which means that $\tilde{\mathcal{C}}(b_t) = \mathcal{D}$. However, the decomposition on the minimal hull is not unique, since we have both $b_t = 0.5 * [-1, 0] + 0.5 * [3, 0]$ as well as $b_t = 0.75 * [-1, 0] + 0.25 * [7, 0]$, contradicting Proposition 3.10. The issue is that you require the convex hull to contain $d_b + 1$ examples. If you remove that restriction, then in the example above $\tilde{\mathcal{C}}(b_t) = \{ [-1, 0], [3, 0] \}$ and then the decomposition is unique, as desired. (I believe your current proof would also work if you remove the restriction. Currently, it doesn’t work because you remove an element from $\tilde{\mathcal{C}}(b_t)$ and call that a contradiction, but it is actually not a contradiction because the newly created set no longer has $d_b + 1$ elements.)

Questions

**Overall view and suggestions for the authors** I quite like the idea in this paper, but currently I think the evaluation and presentation of the idea are not good enough, and so I am recommending rejection. However, I think there is the seed of a good paper here, and would likely be quite excited about a version of the paper that looked more like: 1. Discussing embeddings as a useful way to get a nice structured representation, perhaps considering kNN-on-embeddings as the baseline. 2. Identifying linear interpolation in the minimal convex hull as an alternative to the kNN decision criterion. 3. Conducting a series of experiments that demonstrate the value of the convex hull idea, focusing particularly on questions like: (a) Are belief corpus residuals better at OOD detection than nearest-neighbor distances? (b) Does linear interpolation in a convex hull lead to better performance than kNN on embeddings? (c) Do the points in the convex hull provide a better explanation of the selected action than the k nearest neighbors in embedding space? I think it is quite plausible that convex hulls do better on all of these metrics, but the current experiments don’t show it. **Note on confidence** I’ve selected a confidence of (4) below, but I want to note that I am not very familiar with related literature, and in particular I know very little about accountability in the healthcare setting. As a result, I cannot evaluate (1) the originality of the work (maybe convex hulls have been explored before), and (2) whether the authors compared to state of the art techniques. **Questions** I’m interested in responses to any of the weaknesses I listed above, but in addition, I have some questions on specific details: 1. Why do you require that the minimal corpus subset $\tilde{C}(b_t)$ have $d_b + 1$ elements? 2. In Section 3.5, $\epsilon$ appears to be an absolute threshold for the belief corpus residual, but in Section 5.2, it appears to be a percentile for the belief corpus residual. Which is it? 3. In the Maze environment, what is the performance measure for the behavioral policies that make up your dataset? Is it cumulative reward on the final task (i.e. going from (0,0) to (16,0)) or cumulative reward on each individual task (i.e. going from (0,0) to (8, 16), going from (0, 0) to (8, 8), etc), or something else entirely?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The weaknesses listed above are not present in the paper. Suggestions for improvements are in the previous sections. The paper applies ABC to discrete settings, and also says that it can work in reward-free settings, but the idea for reward-free settings only works for continuous action spaces, not discrete ones. This is mostly not a big deal since the reward-free setting is not currently a major focus of the paper.

Raising my score

Thanks for the response! It has addressed most of my concerns, and I am raising my score from 3 to 5 (and contribution from 2 to 3). My main remaining concerns are: 1. While it is true that with the authors’ method it is possible to identify the data points in training that affect the action taken, it is not clear to me how much accountability this provides. 2. I find it hard to square the results of Appendix F2 and the failure of the kNN + embeddings method with the success of black box methods, which makes me think I’m misunderstanding something about the paper. 3. As mentioned in the review, I find the presentation of the paper quite confusing.

Missing attachment

Unless I am mistaken, I believe the authors were going to attach a document: "We conducted additional experiments to better address the reviewer's concern and provide results in the attached PDF file due to space limitations." However, I don't believe I see this document. Can the authors confirm?

Raise my score

I thank the reviewers for their detailed response to all of my questions. I thoroughly read the other reviewers' feedback and the authors' response. In general, I like the neat idea of finding the minimal hull. However, the authors also admitted that their method primarily focuses on high-stake decision making process, instead of the conventional offline reinforcement learning setting. That is why I hesitated to raise the score to 6 rather than 7 or above.

Thank you for the response

I want to thank the authors for the detailed response, I have considered them, and I maintain my original score.

Follow-Up Author Response on Presentation to Reviewer fhj3 and fQBY (Part 1/3)

### General Response on Presentation In this work, we address the important yet previously underexplored problem of accountability in decision-making. To bridge the gap between example-based explanation in XAI and our problem setting, it is necessary to introduce multiple **novel yet non-trivial formal definitions**: - Definition 3.1 Corpus Subset, - Property 3.2 Belief Space Linearity, - Definition 3.5 Belief Corpus Subset, - Definition 3.6 Belief Corpus Convex Hull, - Definition 3.7 Belief Corpus Residual, - Definition 3.9 Minimal Hull and Minimal Corpus Subset Those formal definitions might seem obscure at first glance, yet they are **crucial and necessary**. This is because only with those formal definitions, we are able to provide the two key propositions - Proposition 3.8: Estimation error bound of value estimation - Proposition 3.10: Existence and uniqueness of the decomposition the former proposition provides a theoretical guarantee of the value estimation, while the latter proposition aids the stability and robustness of the proposed method. **We recognized during the rebuttal period that providing a high-level roadmap, as of the proof sketch in technical literature, to anchor the flow of our method, can be helpful.** Therefore, we integrated the advice from our reviewers and made the following revision: 1. Re-write the introduction, with more concrete examples and clearer motivations 2. Add a roadmap section before formally introducing the method 3. Re-organize the related work section, highlighting the connection and difference from Offline-RL --- ### **1. Re-organized Introduction** (new contents highlighted with shaded boxes) In recent years, batched control that uses pre-collected data to generate control policies has gained attention due to its potential to reduce the costs and risks associated with applying control algorithms in real-world systems, which is especially advantageous in situations where real-time feedback is challenging or expensive to obtain. > However, in many critical real-world applications such as healthcare, the challenge is beyond enhancing policy performance. It requires the decisions made by learned policies to be transparent, traceable, and justifiable. Yet those essential properties, summarized as _Accountability_, are left largely unaddressed by existing literature. > In our context, we use _Accountability_ to indicate the existence of a _supportive basis for decision-making_. For instance, in tumor treatment using high-risk options like radiotherapy and chemotherapy, the treatment decisions should be based on the successful outcomes experienced by previous patients who share similar conditions that were given the same medication. > Another concrete illustrative example is the allocation decisions of ventilator machines. The decision to allocate a ventilator should be accountable, in the way that it juxtaposes the potential consequences of both utilization and non-utilization and provide a reasonable decision on those bases. > In those examples, the ability to refer to existing cases that support current decisions can enhance reliability and facilitate reasoning or debugging of the policy. > To advance offline control towards real-world responsibility-sensitive applications, five properties are desirable: > - (P1) Controllable **Conservation**: this ensures the policy learning performance by avoiding aggressive extrapolation. > - (P2) **Accountability**: as underscored by our prior examples, there is a need for a clear basis upon which decisions are made. > - (P3) Suitability for **Low-Data** Regimes: given the frequent scarcity of high-stake decision data, it’s essential to have methods that perform well with limited data. > - (P4) **Adaptivity** to User Specification: this ensures the policy can adjust to changes, like evolving clinical guidelines, allowing for tailored solutions > - (P5) **Flexibility** in Strictly Batched Imitation: this property ensures a broader applicability across various scenarios and data availability. > To embody all these properties, we need to venture beyond the current scope of literature focused on conservative offline learning. In our work: > - Methodologically, we introduce the formal definitions and necessary concepts in accountable decision-making. We propose the Accountable Batched Controller (ABC), which makes decisions according to a decomposition on the basis of the representative existing decision examples. > - Theoretically, we prove the existence and uniqueness of the decomposition under mild conditions, guiding the design of our algorithms. > - Practically, we introduce an efficient algorithm that takes all the aforementioned desired properties into consideration and circumvented the computational difficulty. > - Empirically, we verify and highlight the desired properties of ABC on a variety of batched control tasks, including five simulated continuous control tasks and one real-world healthcare dataset.