Response to Reviewer ZGMh
Thank you for the response!
**1.If the framework is extended to (state, action)-dependent cost as defined by authors, then it seems no hope to handle the large action case even if function approximation is used since the definition of unsafe states always requires to maximize over all actions.**
If we extend to (state, action)-dependent cost, it indeed needs to explore all actions to receive the safety feedback of each action. One possible avenue to avoid this exploration is to posit that the cost function exhibits a certain favorable structure, (e.g. $c(s,a)= \theta^T \varphi(s,a)$ for some $\theta \in \mathbb{R}^d$ and known features $\varphi(s,a) \in \mathbb{R}^d$, as in existing works [1,8].) and then estimate the cost for all $(s,a)$ exploiting the structure without exploring all actions. This is an interesting future direction.
**2. I am still not fully convinced by the reasons that the cost feedback can be given in the reward-free setting. In some cases, the reward function can also depend on the environment, which does not change frequently.**
The RFE setting is motivated by the fact that in practice, the reward function can be difficult to specify and change frequently by manual crafting [2,3], or there can be multiple rewards of interest in the same environment [4].
In contrast, the cost function can heavily depend on the environment such as barrier placements [5] and the gesture of the robot [6]. Hence, it can be immutable and knowable. Previous safe RFE work [7] also receives the cost feedback during the exploration phase. Moreover, from a theoretical perspective, access to information about the cost function becomes imperative. Devoid of this information, ensuring safety constraints during the exploration phase becomes an impractical pursuit.
We thank the reviewer again for the detailed and valuable comments. Please let us know if you have any further questions. We will be happy to answer them. If you find our response satisfying, we wonder if you could kindly consider raising the score rating of our work?
Thank you very much!
**References:**
[1]. Amani S, Thrampoulidis C, Yang L. Safe reinforcement learning with linear function approximation, ICML, 2021.
[2]. Leike J, Krueger D, Everitt T, et al. Scalable agent alignment via reward modeling: a research direction, preprint 2018.
[3]. Fu J, Singh A, Ghosh D, et al. Variational inverse control with events: A general framework for data-driven reward definition, NIPS 2018.
[4]. Wu J, Braverman V, Yang L. Accommodating picky customers: Regret bound and exploration complexity for multi-objective reinforcement learning, NIPS 2021.
[5]. Thananjeyan B, Balakrishna A, Nair S, et al. Recovery rl: Safe reinforcement learning with learned recovery zones, IEEE Robotics and Automation Letters, 2021.
[6]. Thomas G, Luo Y, Ma T. Safe reinforcement learning by imagining the near future, NIPS 2021.
[7]. Huang R, Yang J, Liang Y. Safe exploration incurs nearly no additional sample complexity for reward-free rl, ICLR 2023.
[8]. Wachi et al. Safe policy optimization with local generalized linear function approximations. 2021.