Regardless of the particular task we want them to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task settings to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.
Paper
Similar papers
Peer review
Summary
The authors propose a novel method for constraint learning from expert demonstrations by an optimal CRL policy. The idea behind the algorithm is to frame the problem as a zero-sum game between a policy player and a constraint player, using inspiration from a two-player zero-sum game expression of IRL and CRL. With the reward function available, expert demonstrations, and a class of potential constraints containing the ground truth constraint, the authors propose to minimize the cost to go subject to being at least as safe as the expert, i.e. to obtain a larger cost than the expert policy on all constraints in the set. The authors derive no regret convergence bounds for the learned constraints and deal with degeneracy issues by extending the problem to the multi-task framework with asymptotic guarantees. They present simulation results on high dimensional robotics tasks.
Strengths
Very sound paper, and very clean and ingenuous framing of the problem as a game. I really enjoyed reading the paper as the work elegantly composes concepts from IRL, CRL, GT and no regret learning. I also see it as being of big significance for a very important problem, that of learning constraints from demonstrations. Overall very solid work.
Weaknesses
- Although the algorithmic contribution is very strong, it seems that the practical algorithm section of the paper is much weaker and feels rushed. The single task results seem very simplistic while the multi-task maze example is way too briefly discussed. There is one sentence describing the threshold after which correct recovery of the constraints occurs, an aspect that would deserve a more extensive presentation and analysis. - Along that line, figure 5 deserve to be reworked for clarity, it mixes too many things in together with little readability. - There is no discussion on the computation costs of running the algorithm and on its data requirements. The authors only mention succinctly that CRL is slower than RL (by how much?), but it would be useful to know how many demonstrations one needs for typical robotics scenarios and how much compute the recovery of constraints needs.
Questions
- How do the authors expect their solution to scale for systems where the state is much higher in dimension than the simple examples presented. For example do they think that their method could be a direction to learn constraints from sequences of images in applications such as autonomous driving or flight? - (Optional) out of curiosity, I am wondering if the authors have any insight on how much more challenging it would be to consider constraints in multi-agent environments?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
- The authors mention the lack of real world experiments as a limitation, I could not agree more but would go even further in suggesting that the proposed simulations are insufficient to help readers fully grasp the potential of the method. The paper would warrant a higher rating should that aspect be improved. - I am awaiting the authors response to better understand whether the method would actually be practical to use for real life applications and robotics problems of interest.
Summary
The paper proposes a novel inverse constraint learning (ICL) approach that leverages constrained reinforcement learning (CRL) and a game-theoretic formulation. Specifically, the paper shows that the game-theoretic view of inverse reinforcement learning (IRL) naturally extends to ICL, by forming the Lagrangian of CRL. The resulting single-task ICL algorithm is shown to Pareto-dominate the expert policy in terms of its expected cumulative reward and the constraint violation. However, the recovered constraint may be too conservative and generalize poorly under a new task. To address this issue, the framework is extended to incorporate expert demonstrations under multiple tasks that share the same constraint.The paper shows that the multi-task ICL framework is expected to approximately Pareto-dominate the expert policies. Simulation results suggest the efficacy of the approach in continuous and discrete MDPs.
Strengths
* Unlike prior ICL methods, the proposed framework is built on the game-theoretic approach. This brings in multiple benefits: 1) the form of the reward function can remain general; 2) conditions on approximate Pareto dominance can be derived, which gives theoretical justification of the proposed approach; 3) the resulting algorithm is relatively straightforward to implement. * Figures 1 and 2 give a nice visual representation of the proposed approach. Algorithm 1, 2, and 3 are presented in a consistent and organized way that clarifies the differences among CRL, single-task ICL, and multi-task ICL.
Weaknesses
* The most concerning point is that the paper provides no comparison to prior work in the experiment. Some important insights of the proposed approach is shared by the seminal work of [1]. Specifically, both approaches 1) observe that access to reward functions enables constraint learning by looking at the differences between safe demonstrations and unsafe optimal demonstrations; 2) can leverage safe demonstrations from multiple tasks. To strengthen the contribution of the paper, it should include qualitative and quantitative comparison to [1] in Section 4 (where applicable), to clarify the practical advantages of the proposed framework. This might involve evaluating the ICL algorithm in more realistic problems than the toy task of maze navigation, since the authors claim better scalability of their approach compared to [1] in realistic problems (in Section 2). In addition, the paper should give a more detailed description of [1] to better highlight the similarities and differences. * There are multiple places in the paper where a variable or an acronym is used without definition: 1) $c^*$ in Theorem 3.1; 2) “FTRL” in line 153. Also, some figures in Section 4 are confusing and hard to interpret. Specifically, I believe that the blue cells in Figure 5(b) and (c) represent start and goal cells. However, the same color is used in other sub-figures to represent the constraints. It is recommended to use different colors to represent the start and the goals so that readers can make sense of Figure 5 as a whole. For further points concerning the presentation of the paper, please see the questions below. [1] Chou, Glen, Dmitry Berenson, and Necmiye Ozay. "Learning constraints from demonstrations with grid and parametric representations." The International Journal of Robotics Research 40, no. 10-11 (2021): 1255-1283.
Questions
* What is $c_i^k$ in Algorithm 3? Shouldn’t it be just $c_i$ since the constraint is shared across all the $K$ tasks? * The order of minimization and maximization is swapped between (6) and (7). Does the equality hold because of the standard minimax theorem in game theory? If so, are there any conditions imposed on the form of $J$ in order for the equality to hold?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
* The paper assumes access to demonstrations of the optimal safe policy for a given task. While the availability of safe demonstrations is a reasonable assumption,optimal demonstrations may be too demanding to ask for a human demonstrator in real-world settings [2]. It would be quite interesting to see how the proposed ICL framework handles suboptimal demonstrations; in theory, the objective function of equation (4) does not seem to require the optimality of the expert under the given reward function $r$. [2] Xu, Haoran, Xianyuan Zhan, Honglei Yin, and Huiling Qin. "Discriminator-weighted offline imitation learning from suboptimal demonstrations." In International Conference on Machine Learning, pp. 24725-24742. PMLR, 2022.
Summary
This paper proposes to use inverse constraint learning (ICL) in multi-task scenarios. It uses game solving to describe the ICL problem, and then illustrates the learning algorithm under the single-task and multi-task problems. The authors give a theoretical analysis to explain under what conditions the learned reward can be generalized. The author shows the correctness of the proposed algorithm in both single-task and multi-task control experiments.
Strengths
- The motivation of this paper is clear and reasonable to extract common safety-constraint from multi-task. - The paper considers whether the learned reward and constraint can be generalized theoretically, and gives two conditions and related proofs.
Weaknesses
- The paper did not give a comparison with the ICL algorithms mentioned in the related work, and did not show the advantages of the algorithm in both learning or safety concerns. - The experimental settings of the paper is too simple, and only consider two constraints that do not require complex approachs to learn. It makes the machine learning method shows no advantages on human priors. - The authors did not give the results of robotics control tasks (both in simulation and real-world) that really need safety-constraints, such as robotic manipulation with human.
Questions
- Can the author supplement the experimental results on more complex robotics control tasks? For example, consider such a manipulation task as follow. We hope that the force or speed of the robot arm will not be too large while picking up an object. Can the algorithm learn such complex safety-constraints? - The authors' experiment in multi-task scenarios essentially improves the diversity of samples through randomization. We generally don't think this is a real multi-task RL scenario because all tasks are navigation. Can the author explore diverse but related manipulation tasks (like push and pull an object) in the open-source taskset (such as [meta-world](https://meta-world.github.io/)) to show the effectiveness of the proposed method?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
It is good that the paper focuses on the safety problems of reinforcement learning for robotics. However I suggest that the authors use more realistic robotics tasks (at least in simulation) that really need to consider safety-constraint to verify the method. Regardless of the comparison with related works or the complexity of the task, the experiments in the paper are not solid enough to support good motivation and theoretical results.
Summary
Broadly, the paper address the challenge of safety constraints for robots, in particular the difficulty of handcrafting these. They propose learning safety constraints from expert demonstrations, particularly in a multi-task setting where each task reward is known and the safety constraints are task-agnostic. They review related work on inverse RL (IRL), constrained RL (CRL), inverse constraint learning (ICL), and multi-task ICL. Compared to existing ICL approaches, they claim their approach is more general, provides more guarantees, and is simpler to implement. They extensively formalize ICL, including explaining relevant approaches to IRL and CRL. As a brief recap, they compare ICL to IRL: IRL has an outer loop that searches over potential rewards, with an inner loop that uses RL to find an optimal policy. ICL has an outer loop that searches over potential constraints, with an inner loop that uses CRL to find a safe-optimal policy. In either case, the optimized policy is compared to the expert demonstrations to revise the learned reward or constraint. They also explain a challenge with single-task ICL, namely that ICL may learn an overly-conservative safety constraint that simply forbids all states not visited by the expert demonstrations. This motivates their exploration of multi-task ICL, where the safety constraint is assumed to be shared among the tasks. Their primary novel contribution of the paper is an extension of a single-task ICL approach to multi-task: "we alternate between solving K CRL problems and updating the constraint based on the data from all policies". Using it, they prove that they can exactly identify a shared constraint given enough expert policies, under certain very strong assumptions. They also prove a weaker but more practical theorem: if they have enough tasks, they can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance). For the single-task case, they evaluate their approach in the simulated ant environment in PyBullet and MuJoCo benchmarks. They choose arbitrary velocity and position constraints. They represent their learned constraints as neural networks, mapping from the state space of their agent to a bounded scalar in the range [0, 1]. They show that their approach can recover these ground-truth constraints, under certain simplifying assumptions (linear constraints; usage of a subset of the agent state as input). For multi-task, they use the PointMaze environment from D4RL, with each task variant having one of two start positions and one of 10 goal positions. Using their approach, they are able to approximately learn the ground-truth constraints (namely the maze's blocked cells). They also show that naive approaches for combining 10 separate learned constraints from single-task ICL don't work well.
Strengths
Their premise, that safety constraints are difficult to engineer and should be task-agnostic, is accurate and relevant to this domain. Their approach is a straightforward extension of an existing single-task ICL approach to multi-task. It works well and matches the intuition that safety constraints should be task-agnostic. Rigorous math in section 3 to formalize their work.
Weaknesses
In section 3, it's not clear what is prior work and what is their novel contribution. In 3.1, the authors copied several sentences verbatim from "Inverse Reinforcement Learning without Reinforcement Learning" (https://arxiv.org/pdf/2303.14623.pdf, section 3.1). The use of copied phrases like "we solve" suggest the authors, themselves, did this work. In their single-task and multi-task results, they don't compare to any baselines, for example "Learning constraints from demonstrations" from Chou et al. For readability, it would be helpful if they numbered references.
Questions
Section 3.4 and 3.5 mention needing expert policies. However, the premise of your work is that you have access to expert demonstrations and reward functions, but not expert policies. Can you clarify? A suggestion for positioning this line of work: instead of "safety constraints", I might argue that a more broad (and thus more valuable) set of constraints to be learned from human demonstrations is "task-agnostic human-behavior constraints", which would encompass safety plus other conventions and preferences. E.g. coming back to the example in your introduction, if I ask my friend to make toast, I'd be surprised if they set a plate down upside-down or rinsed the bread under the faucet. >we consider a setting in which we have access to demonstrations of the optimal safe policy for a task, along with knowledge about the task’s reward Can you describe this setting in practical terms? It seems that your approach has stronger requirements, namely (1) access to not just the rewards received by the demonstrations, but access to the reward function itself, and (2) access to a simulator or other mechanism to train a policy using CRL. As a clarifying example: if you had access to expert trajectories of a real-world robot including full scene state and reward annotations, across multiple tasks, this alone would not be sufficient to learn safety constraints using your method, right? >In short, if we observe enough tasks, we are able to learn a constraint that, when optimized under, leads to policies that approximately Pareto-dominate that of the expert. Can you elaborate? What is the specific Pareto-dominance here? My understanding is: you can learn a constraint that, if used to optimize a policy, leads to policies that don't exceed the constraint-violation of the expert demonstrations and do meet or exceed the reward of the expert demonstrations (within a tolerance). However, I don't intuitively follow why this should be your goal for the learned safety constraint. For Figure 2, can you offer an intuitive explanation? For example: = can you label the axes? = why is an expert policy a single point in this space? = what does the boundary of the polytope represent? = what does this mean? "diversity of expert policies, none of which are optimal along the reward vector"
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
No concerns here
Summary
This paper addresses the challenge of learning safety constraints for agents in various tasks from expert demonstrations, instead of manually specifying them. The authors extend inverse reinforcement learning (IRL) techniques to the space of constraints, aiming to learn constraints that prevent highly rewarding behavior that the expert could have performed but avoided. However, the constraint learning problem is ill-posed and tends to result in overly conservative constraints. To mitigate this issue, the authors utilize diverse demonstrations from multi-task settings to learn a tighter set of constraints. The method is validated through simulation experiments on high-dimensional continuous control tasks.
Strengths
- The authors develop a multi-task variant of inverse constraint learning, which consists of multiple policy players aiming to maximize task-specific rewards and a constraint player determining a single constraint that all policy players must follow. - The proposed approach reduces the chances of choosing degenerate solutions. - The inverse constraint learning problem is formulated as a zero-sum game involving a policy player, who seeks to maximize rewards while adhering to potential constraints, and a constraint player, who selects constraints that impose maximum penalties on the learner compared to the expert. - The proposed method demonstrates its effectiveness in various continuous control tasks. When applying restricted function classes, the technique can retrieve ground-truth constraints for certain tasks.
Weaknesses
The author argues that previous ICL problems have not been well-formulated, leading to solutions that only allow expert actions. However, it is not clear how the proposed method addresses this issue, aside from utilizing a greater number of demonstrations from a multi-task setting. It would be interesting to determine whether the proposed method can work for tasks beyond maze locomotion, which lack complex interactions with the environment. For example, in manipulation tasks, safety constraints are more critical as the robot must interact with and alter the environment.
Questions
The author could better address the concerns mentioned in the weaknesses section. It is worth noting that the reviewer is not well-versed in this field and does not specialize in IRL or ICL problems. Thus, the overall rating may be subject to change based on feedback from other reviewers and the author's rebuttal.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Consider removing Section 4.2. A more intuitive explanation of Figure 2 would be helpful.
Response to authors
I thank the authors for their clarifications. I have no other questions and maintain my review score.
Please clarify "human prior"
Dear Reviewer 3RKE, In order to give the authors an opportunity to address your W2, can you briefly respond to their question regarding a "human prior"? Thank you very much!
Thanks for your rebuttal and apologies for the lateness of my reply. >variety of baselines (e.g. the max, mean, or individual single-task constraints) These seem like weak baselines. It's not clear that they are "prior state of the art" or otherwise a well-regarded existing approach for recovering shared safety constraints from multiple tasks. Do you have any references to prior published work that utilize/advocate any of these approaches? I will continue reviewing your rebuttal and post a longer response later today.
Re:
Thanks, we look forward to your full response! Re: the multi-task baselines: to the best of our knowledge, there is only one other published work that considers the multi-task ICL problem, which is that of Chou et al. As we discussed in our global response, re-implementing their method requires access to several custom solvers that differ based on the particular family of constraints one wishes to search over, making it difficult for us to directly compare. We therefore turned out attention to single-task baselines, of which we compare to several reasonable ones.
>[Chou et al] did not release their code, it would be quite an undertaking for us to re-implement their methods on the problems we consider. As I'm sure you would agree, this (very) inconvenient fact doesn't exempt your work from comparison against strong baselines. >much of the prior work in inverse constraint learning focuses on tabular problems, while most of the work in constrained reinforcement learning focuses on relatively low dimensional tasks. So, we would argue that compared to the prior art, we are considering strictly harder settings. Unfortunately I don't have enough confidence in this domain to agree/disagree here. >It would be interesting to determine whether the proposed method can work for tasks beyond maze locomotion, which lack complex interactions with the environment. I agree with this reviewer's comment. In particular, the complexity I'd like to see is not just high dimensionality, but also long horizon length (e.g. avoiding a hot stove until it has cooled) and low predictability (e.g. keeping a larger distance from a pet or child that might move erratically). I'm not able to increase my rating. I suspect that a proper review of your work requires a deeper understanding of IRL, CRL, and ICL than I can offer. There's a lot of information in your rebuttal and reviewer discussions that I wasn't able to absorb, so I'm going to lower my confidence score.
We thank the reviewer for their candor. We'd like to respond on a few points. 1. Beyond the fact that it is "very inconvenient" to implement the method of Chou et al., it is also extremely easy for us to construct problems for which our method readily applies but one cannot apply the method of Chou et al. For example, if we make one of the blocked sections in the maze a *circle* instead of a square, one can no longer use the "axis-aligned rectangles" technique of Chou et al. to recover a reasonable constraint. It would be rather shocking to us if a deep neural network could not fit this circle. Furthermore, Chou et al.'s method requires knowing the constraint family (e.g. "axis-aligned rectangles") a priori, while ours does not. This makes our techniques far more easily applicable to real-world problems where one does not usually know the exact parametric family of the constraint they are trying to fit, a limitation acknowledged by Chou et al. 2. The *minimum* horizon task we consider is of length 1000, while our maze task goes up to several thousand. So, we would argue that we are already learning constraints on long-horizon behavior. If there are any pieces of our above responses or our responses to other reviewers that could benefit from a more in-depth explanation, please let us know. We thank the reviewer for their engagement with our work!
>This makes our techniques far more easily applicable to real-world problems This is fine and a strength of your approach. But it sounds like there is a (restricted) problem you could construct where both approaches could apply, and then they could be compared. My earlier comment: >In particular, the complexity I'd like to see is not just high dimensionality, but also long horizon length Your response above: >The minimum horizon task we consider is of length 1000 I was referring to the horizon length of the constraint, not the task. Consider learning a safety constraint around avoiding a hot stove. I'm assuming your state space includes the current on/off state of the stove (which we can imagine a smart home sensor or perception system providing), while your state space doesn't include the current stove-element temperature, as this is much harder to provide. Thus your approach needs to learn a safety constraint that is based on a memory of past environment state (how long ago did it get turned off? how hot did it get before it was turned off?).
Re:
Ah, thank-you for the clarification. While we are learning over-the-horizon constraints (i.e. a maximum cumulative cost over an episode), our method does not directly handle the sort of partially observed problem you describe above (i.e. a problem where the learner needs to perform some sort of filtering over history to extract the state required to represent the constraint). That being said, even in the safe RL setting (where we know the constraint a priori), it is not clear whether we can solve partially observed problems well without strong assumptions. This is because the information gathering actions the learner might need to take to reduce their uncertainty over the underlying state of the environment might end up violating the constraint (e.g. if the constraint was to never get close to a hot stove and the learner is only able to measure a temperature when they are close to an object). So, while we agree that this is an interesting question, it seems orthogonal to our paper, which focuses on constraint inference. However, we would be happy to clarify that we assume full observability in our paper.
Re:
Hi, In response to your request and that of another reviewer, we spent some time thinking about how we could reasonably faithfully implement the method of Chou et al. on the problems we consider. At a high level, their method requires 2 steps: 1) For each expert trajectory, perform a random search in trajectory space starting from the demo to try and compute the set of trajectories that are higher reward than the demo. 2) Solve a constrained optimization problem over a known parametric family that labels each expert trajectory with +1 and each learner trajectory with 0. Note that in contrast to our method, Chou et al. perform a *single* constraint estimation, rather than an iterative procedure. For 1), they use hit-and-run sampling that ensures good coverage over this set of trajectories in the limit of infinite sampling. However, for tasks of the horizon (1000+) and state-space dimension (~30) we consider, it seems like one would need an astronomical number of iterations to ensure good coverage (i.e. an $\epsilon$-net would require $\frac{30^{1000}}{\epsilon}$ samples). This is part of the reason hit-and-run sampling is not used in practice for solving even regular Mujoco tasks (and, to the best of our knowledge, is mostly applied on bandit problems). While we weren't able to find descriptions of the horizons of the task Chou et al. consider, we would be rather surprised if they were in the thousands. If search in the space of trajectories is infeasible, we could instead try search in the space of *policies*, as is much more standard on long-horizon problems. So, we propose initializing a policy with behavioral cloning (similar to starting from the expert trajectories), performing maximum entropy RL (a search procedure with good coverage properties), and returning the set of trajectories from the replay buffer that are of a higher reward than that of the expert. For 2), they use a variety of mixed-integer LP solvers. In essence however, they are trying to solve a classification problem. Of course, with their assumption of an optimal expert (or a "boundedly suboptimal" expert with a known sub-optimality gap), they can solve the strict feasibility problem. Because we assume that the expert is safe in expectation (rather than uniformly), it is natural to consider a relaxation of this problem (i.e. a convex relaxation of the 0/1 loss). We experimented with several such relaxations when developing our method and found the $\ell_2$ relaxation (i.e. treating the problem as a regression problem with different targets for learner and expert demos) to work the best in practice. It therefore seemed the most fair for us to use the same relaxation for their method. As a reviewer pointed out, for our single-task experiments, it is clear how to specify the "parametric family" the constraint fits in (e.g. a linear threshold), so we believe the above is a faithful implementation. So, in short, we perform the same constraint learning procedure over a different source of learner data. We implemented the above method on our position task and display the results below: | Algo. | $\|c^* - c\|$ (↓) | $J(\pi_E, r) - J(\pi, r)$ (↓) | $J(\pi_E, c^*) - J(\pi, c^*)$ (↑) | |---------------|-----------------|-----------------------------|---------------------------------| | ICL (iter 1) | 0.052 | 569.277 | **270.111** | | ICL (iter 10) | **0.000** | **-72.149** | 133.59 | | Chou et al. | 0.311 | 384.772 | -1958.372 | We see that our method performs better in terms of all 3 performance criteria we evaluate under. As implemented, Chou et al.'s method appears to produce an inaccurate constraint that leads to a significant safety violation (the bottom right cell in the above table), even when compared to the first iteration of our iterative procedure. We hope these experiments help resolve any remaining concerns you might have. Please let us know if you have any other questions.
Re
We also provide the result for velocity constraints: | Algo. | $\|c^* - c\|$ (↓) | $J(\pi_E, r) - J(\pi, r)$ (↓) | $J(\pi_E, c^*) - J(\pi, c^*)$ (↑) | |---------------|-----------------|-----------------------------|---------------------------------| | ICL (iter 1) | 0.249 | 12.267 | -244.25 | | ICL (iter 20) | **0.006** | **-46.771** | **-21.602** | | Chou et al. | 0.246 | -28.315 | -134.58 | One again, our iterative method performs better than the baseline. Both this and the results in the previous table were averaged over three seeds. We would be more than happy to add these results to the paper. We also spent some time thinking about implementing Chou et al's method on our maze problem. Even if we ignore the difficulties with picking a parametric family that captures the set of all possible mazes, ignore that we were learning constraints on raw x / y positions rather than in the 10 x 10 grid space, and focus purely implementing step 1) on the grid (i.e. sampling shorter paths than the expert took), we would need to generate an number of trajectories that we are incapable of storing even a reasonable fraction of in memory. Specifically, we wrote a program to perform the dynamic programming calculations and it told us we would need to generate 7,126,295,908 trajectories to fully implement the multi-task version of their method. If we had to guess, this is why Chou et al. evaluate their method on smaller grids than we consider (as one pays exponentially in the height / width of the maze). We hope that the computational intractability of implementing their multi-task method combined with their poor performance on the simpler single-task problem is sufficient evidence of the superiority of our method.
I believe that my concern is because I am lack of experience on this domain (safe RL). So till now I understand that the single-task experiment is showing that, if there is an unknown position or velocity constraint (defined by a linear function), the policy can learn not to break the constraint? The human prior that I have mentioned is that, the position and velocity constraints can be fully considered and easily learned with simple reward shaping. But if the safe RL area aims to set a task with the totally unknown constraints and only shows the training process can discover the constraints, I agree that it is too strict to evaluate this paper in a much more realistic view. I can raise my score due to that I did not understand the safe RL area, and the rebuttal has addressed my main concern. But I am also willing to see the future work on the real robotic tasks that really face to the safety problems.
Re:
Thanks! Yeah, the single-task experiments show that we are able to learn safe policies without knowledge of the ground-truth constraint.
We thank the reviewer for their thoughtful comments. As we get closer to the end of the discussion period, please let us know if we were able to answer all of your questions or if there is anything else we could clarify.
Reply to Authors
Thanks for your reply. I have no further questions. As mentioned in the original review, I have never worked on this field so that I am not confident for the comments for your paper. I believe that the final decision will be more dependent on other reviewers.
We thank the reviewer for their thoughtful comments. As we get closer to the end of the discussion period, please let us know if we were able to answer all of your questions or if there is anything else we could clarify.
Acknowledgement of Author Response
Dear authors, I apologize for my delayed response. Thank you for providing detailed explanations, especially around the differences between your work and the work of Chou et al. I would highly recommend that you add the discussion in the body of the paper, at least in the Appendix (and summarize the key points in the main paper as much as possible). Thank you also for conducting the experiments with suboptimal experts. It is indeed strong empirical evidence that can differentiate your work from the majority of prior work that requires a soft-optimal expert. As a result, I acknowledge the author response and am going to raise my score. However, I still believe that a baseline comparison is highly desirable for a more objective evaluation, at least in the simpler single-task setup. (For instance, it seems that a linear position constraint in Figure 4 could be learned via parametric or grid-based constraint representation that some prior work can handle.)
Re:
Hi, In response to your request and that of another reviewer, we spent some time thinking about how we could reasonably faithfully implement the method of Chou et al. on the problems we consider. At a high level, their method requires 2 steps: 1) For each expert trajectory, perform a random search in trajectory space starting from the demo to try and compute the set of trajectories that are higher reward than the demo. 2) Solve a constrained optimization problem over a known parametric family that labels each expert trajectory with +1 and each learner trajectory with 0. Note that in contrast to our method, Chou et al. perform a *single* constraint estimation, rather than an iterative procedure. For 1), they use hit-and-run sampling that ensures good coverage over this set of trajectories in the limit of infinite sampling. However, for tasks of the horizon (1000+) and state-space dimension (~30) we consider, it seems like one would need an astronomical number of iterations to ensure good coverage (i.e. an $\epsilon$-net would require $\frac{30^{1000}}{\epsilon}$ samples). This is part of the reason hit-and-run sampling is not used in practice for solving even regular Mujoco tasks (and, to the best of our knowledge, is mostly applied on bandit problems). While we weren't able to find descriptions of the horizons of the task Chou et al. consider, we would be rather surprised if they were in the thousands. If search in the space of trajectories is infeasible, we could instead try search in the space of *policies*, as is much more standard on long-horizon problems. So, we propose initializing a policy with behavioral cloning (similar to starting from the expert trajectories), performing maximum entropy RL (a search procedure with good coverage properties), and returning the set of trajectories from the replay buffer that are of a higher reward than that of the expert. For 2), they use a variety of mixed-integer LP solvers. In essence however, they are trying to solve a classification problem. Of course, with their assumption of an optimal expert (or a "boundedly suboptimal" expert with a known sub-optimality gap), they can solve the strict feasibility problem. Because we assume that the expert is safe in expectation (rather than uniformly), it is natural to consider a relaxation of this problem (i.e. a convex relaxation of the 0/1 loss). We experimented with several such relaxations when developing our method and found the $\ell_2$ relaxation (i.e. treating the problem as a regression problem with different targets for learner and expert demos) to work the best in practice. It therefore seemed the most fair for us to use the same relaxation for their method. As a reviewer pointed out, for our single-task experiments, it is clear how to specify the "parametric family" the constraint fits in (e.g. a linear threshold), so we believe the above is a faithful implementation. So, in short, we perform the same constraint learning procedure over a different source of learner data. We implemented the above method on our position task and display the results below: | Algo. | $\|c^* - c\|$ (↓) | $J(\pi_E, r) - J(\pi, r)$ (↓) | $J(\pi_E, c^*) - J(\pi, c^*)$ (↑) | |---------------|-----------------|-----------------------------|---------------------------------| | ICL (iter 1) | 0.052 | 569.277 | **270.111** | | ICL (iter 10) | **0.000** | **-72.149** | 133.59 | | Chou et al. | 0.311 | 384.772 | -1958.372 | We see that our method performs better in terms of all 3 performance criteria we evaluate under. As implemented, Chou et al.'s method appears to produce an inaccurate constraint that leads to a significant safety violation (the bottom right cell in the above table), even when compared to the first iteration of our iterative procedure. We hope these experiments help resolve any remaining concerns you might have. Please let us know if you have any other questions.
Thank you for the additional experiment
Dear Authors, Thank you for conducting the comparative study and providing the details of the experimental setup. The design of the experiment seems reasonable to me, and the empirical results clearly exhibit superiority of ICL over the prior work. I am going to raise the score once again as all my concerns so far have been addressed. Thank you for your hard work despite the short period of time.
Re:
Thank you -- we appreciate your continued engagement! For completeness, we also include below the result for velocity constraints: | Algo. | $\|c^* - c\|$ (↓) | $J(\pi_E, r) - J(\pi, r)$ (↓) | $J(\pi_E, c^*) - J(\pi, c^*)$ (↑) | |---------------|-----------------|-----------------------------|---------------------------------| | ICL (iter 1) | 0.249 | 12.267 | -244.25 | | ICL (iter 20) | **0.006** | **-46.771** | **-21.602** | | Chou et al. | 0.246 | -28.315 | -134.58 | One again, our iterative method performs better than the baseline. Both this and the results in the previous table were averaged over three seeds. We would be more than happy to add these results to the paper. We also spent some time thinking about implementing Chou et al's method on our maze problem. Even if we ignore the difficulties with picking a parametric family that captures the set of all possible mazes, ignore that we were learning constraints on raw x / y positions rather than in the 10 x 10 grid space, and focus purely implementing step 1) on the grid (i.e. sampling shorter paths than the expert took), we would need to generate an number of trajectories that we are incapable of storing even a reasonable fraction of in memory. Specifically, we wrote a program to perform the dynamic programming calculations and it told us we would need to generate 7,126,295,908 trajectories to fully implement the multi-task version of their method. If we had to guess, this is why Chou et al. evaluate their method on smaller grids than we consider (as one pays exponentially in the height / width of the maze). We hope that the computational intractability of implementing their multi-task method combined with their poor performance on the simpler single-task problem is sufficient evidence of the superiority of our method.
Decision
Accept (poster)