Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraints

Summary

The paper proposed a trust-region method for handling multiple constraints in safe RL for CMDPs. If the TRPO-like gradient calculation with multiple constraints is not feasible, the authors proposed a gradient integration method to calculate feasible gradients. The authors also proposed a TD-$\lambda$ method to calculate the target distribution of distributional critic. Experimental results have shown that the proposed methods seem to outperform baselines.

Strengths

## Originality The gradient integration is novel and looks useful for the infeasible initial stage when solving CMDP. Computing the TD $\lambda$ for the distributional critic is also novel to me. ## Quality The paper is well-written and includes many technical details, which is good for understanding and reproducing the work. ## Clarity The paper is very clear. Especially, I like the two figures which explain the ideas very well and I get the idea immediately. The notation is a little bit messy but clear enough to read. ## Significance The TD-$\lambda$ method for distributional critic might have a broad effect on other distributional RL algorithms, but I am not sure if similar approaches have been proposed to solve the problem.

Weaknesses

1. The paper has novelty but the contributions are not clearly demonstrated. From the results in Figures 3 and 4, I did not see too much performance improvement, solid constraint satisfaction, or faster convergence to feasible policies compared to baselines. Given this paper focuses on practical contributions, the performance results are not convincing. 2. Two contributions of this paper are not related. Gradient integration and TD-$\lambda$ seem to stand alone on their own. The authors should explain why you put these into one paper and how they contribute together to improve the safe RL. 3. The gradient integration problem is fragile and might have some problems in the finite-convergence theorem. I will put more discussion in the questions section.

Questions

1. I got confused about the figures of the first column in Figure 4. The curves and shades are not continuous. Please explain why. *Updates after rebuttal: Clarified in the reply. They are random samples.* 2. For Figure 1, what if the $g_1$ and $g_2$ are parallel to each other (no feasible region), or have an arbitrarily small relative angle? Then you integrate them to near zero and get stuck. Then I doubt the conclusion about finite-time convergence to feasible region. You might need more assumptions like the feasible region does exist. Do you want to comment on these situations? 3. Following problems in 2, it seems like no guarantees like the proposed gradient integration could be faster than naive methods could be made. Then how exactly can safe RL methods benefit from the gradient integration? 3. Practically, Solving TRPO is already computationally heavy. Locally linearization of multiple constraints and objectives will cause more approximation and further harm the performance. If you only would like to handle multiple constraints, many straightforward approaches could be used like first optimizing a weighted sum of multiple constraint critics. How does the proposed algorithm outperform these intuitions then? Please carefully explain these questions. I will consider increasing the score if the authors give me a reasonable response.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors addressed the limitations well.

Summary

This study focuses on multiple constraints setting in safe reinforcement learning, and an interesting method is proposed by leveraging gradient integration methods. Moreover, the feasibility of multi-constraint problems is addressed, TD distribution method is introduced to decrease the estimation bias.

Strengths

1. Multiple constraints are considered. 2. The code is provided. 3. The writing quality is good.

Weaknesses

1. Some papers are not investigated, e.g., [1] and [2] [1] Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., ... & Knoll, A. (2022). A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330. [2] Garcıa, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1), 1437-1480. 2. The experimental results do not convince me, as shown in Figure 3, other baselines also present better performance than this study, e.g., the WCSAC method. 3. Could you compare the method with PPO Lagrangian?

Questions

1. Can we handle multiple constraints by averaging the multiple constraints as one constraint? 2. The essential contribution is to design the distributional critics with low biases. How about the distributional actors for estimating different types of constraints? 3. What is the difference between PCPO and this method? In PCPO, they also make projections to optimize reward and cost. 4. Why does the $J(\pi)$ like the formation in page 3, line 93? 5. Why do we need a Shannon entropy? 6. How does the study have Equations (4) and (5)? Could the study provide analysis to prove them? 7. Since the method claims the algorithm is efficient and computation efficiency is better than other baselines, could the study provide the sample complexity?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

1. The cost is assumed as convex. However, in most cases, the constraints may be nonconvex. 2. The computation complexity and sample complexity should be provided to prove the effectiveness of this study.

Summary

The paper tries to address the problem of safe RL with multiple constraints with a safe distributional actor-critic (SDAC) approach. The approach includes a gradient integration method to manage the infeasibility issues in multi-constrained problems and a TD$(\lambda)$ target distribution to estimate risk-averse constraints with low biases. Experimental results show that the proposed approach outperforms the baselines in both single- and multi-constrained tasks.

Strengths

Safe Rl with multiple constraints is an important problem. The presented approach uses a gradient integration method to address the infeasibility issues of the trust-region-based algorithms with theoretical guarantees and proposed a TD$(\lambda)$ loss to reduce the estimation bias of the critics. Sufficient experimental results and ablation studies are provided to support the claims of the paper.

Weaknesses

1. The writing of the paper can be improved. There are little background information and intuitions introduced, which makes the paper hard to read. For example, In Section 2 when introducing the trust-region method with a mean-std constraint, it would be better if the authors can introduce the intuitions behind the equations instead of just putting the equations there, because [Kim and Oh, 2022a] is not a very well-known paper. Also in Section 3.1, little information is provided about the intuition of equation (9). 2. See “Questions”.

Questions

1. Seems the performance of the algorithm depends a lot on $\alpha$ and $\lambda$, then how can we choose them in the experiments? 2. Since the algorithm is designed to deal with multiple constraints, could the proposed algorithm deal with constraints with different priorities, for example, collisions with humans should be avoided prior to collisions with obstacles? 3. In Figure 3(a), WCSAC with both $\alpha=0.25$ and $\alpha=1.0$ satisfies the cost threshold in all environments while in some environments it also has higher rewards compared with SDAC. Could the authors provide some intuition about these observations?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors discuss the limitations well in Section 6.

Summary

The paper presents a safe reinforcement learning (RL) algorithm called SDAC for handling multiple constraints in safety-critical robotic tasks. SDAC incorporates risk-averse constraints and makes two key contributions: a gradient integration method for handling infeasibility issues and a TD($\lambda$) target distribution for estimating risk-averse constraints. Experimental results show that SDAC outperforms safe RL baselines, achieving fewer constraint violations and faster constraint satisfaction.

Strengths

In general, the work is solid and contributes quite some novel ideas for safe RL. - The formulation of safe RL problem with multiple constraints is very important for real applications. - The infeasibility issues in constrained optimization are explicitly considered. - The risk-averse safety measures are practical to safety-critical problems.

Weaknesses

- While the measures have low bias, it is still important to consider the bias when making decisions. - The presentation of the technical details, particularly the quantile regression part, is challenging to comprehend. It might be beneficial to refer to Dabney's paper, which provides a clearer explanation of the related content. - From the empirical analysis, the proposed method cannot ensure safety during the early stage of training, despite the theoretical guarantees.

Questions

- What is the purpose of adding a policy entropy term in line 91? Is it necessary, given the use of trust-region methods, and how is it maximized? - Is there an implicit assumption that the distribution is Gaussian considering the risk measure in Equation 2? - What is the reason for balancing bias and variance? Even though the bias is low, are there still some negative effects? How can we mitigate them, especially in the context of risk? - How will the algorithm work if the constraints completely conflict with each other? - When the cost signal is binary, multiple constraints can be taken as one constraint. Then, a method like WCSAC or CPO can also be used. In this case, what are the main advantages of SDAC?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the limitations are clearly stated at the end of the paper.