RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

Summary

This paper studies and proves the applicability of two risk-aware objectives to Preference-Based Reinforcement Learning (PbRL), i.e., iterated and accumulated quantile risk objectives. The authors design an algorithm called Risk-Aware-PbRL (RA-PbRL), which can optimize both iterated and accumulated objectives. Furthermore, the authors provide a theoretical analysis of the regret bounds. The results demonstrate that the regret bounds of algorithm RA-PbRL under both the iterated and accumulated objectives are sublinear with respect to the number of episodes. Finally, the authors present empirical results to support their theoretical findings.

Strengths

1. The studied problem, i.e., applying the iterated and accumulated risk-aware objectives to PbRL, is relevant and useful for some applications such as healthcare and AI systems. The considered reward model depends on the feature of the trajectory, instead of the feature of the state-action pair, which is more general than the prior works. 2. The authors consider both the iterated and accumulated risk-aware objectives, which encompass the popular CVaR objective. In addition, the authors also design an algorithm and provide regret analysis for both objectives. The algorithm design and theoretical analysis are well executed.

Weaknesses

1. The proposed algorithm is very straightforward, which seems to simply combine the confidence set construction and the risk-aware objective. This algorithm is computationally inefficient and hard to implement in practice. Can the authors explain more on how to implement this algorithm? 2. What is the intuition behind the factor $\min_{\pi,d} \omega_{\pi}(d)$ in Theorem 4.1? In particular, why the probability that the feature is not zero will influence the regret? More discussions on the regret due to reward estimation are needed. 3. It seems that $L_G$ appears in the upper bound (Theorem 4.2) but not in the lower bound (Theorem 4.4). Why the authors said that “it demonstrates the significant impact of LG” below Theorem 4.4? In addition, it seems that $dim_{T}$ appears in the lower bound, but not in the upper bound. Can the authors explain more on it? 4. This paper needs careful proof-reading. There are many typos. For example, the factor $\min_{\pi,d} \omega_{\pi}(d)$ in Theorems 4.1 and 4.3, and the $O(…)$ notation in Theorems 4.1 and 4.2. In Line 130, “use” should be “used”. In Line 197, the “and” should be moved to the front of “$V^{\pi}_i$ in Eq.5 …”?

Questions

Please see the weaknesses above.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Please see the weaknesses above.

Summary

This paper incorporates risk-awareness into Preference-based Reinforcement Learning (PbRL). Specifically, it tackles the issue that under PbRL, the reward is episodic, meaning that it can only be computed on full trajectories. The authors adapt both iterative and accumulated quantile risk objectives to deal with episodic rewards. Additionally, the paper presents an algorithm (RA-PbRL) to incorporate these objectives into PbRL. Lastly, the authors provide regret boundaries for both iterative and accumulated quantile risk objectives with RA-PbRL.

Strengths

[Quality, Clarity] The paper is well written (particularly the introduction). As far as I could follow the mathematical development is robust, with regret upper- and lower-bounds being established for both types of risk considered.

Weaknesses

* **W1** [Quality]: As the manuscript already mentions, the experimental setting is very simple. It would have been interesting to train more complex settings, particular from actual human preferences. * **W2** [Significance]: I think more could be done to stress the important of risk-awareness in the PbRL setting. After reading the paper, it was still not clear to me which applications would benefit from RA-PbRL.

Questions

In PbRL, it is often assumed that the given preferences $o_i$ may be noisy. Do authors assume perfect preferences? If not, it would be interesting to analyse how the amount of noise in the preference affects the regret bounds.

Rating

7

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

The limitations have been correctly addressed.

Summary

This paper focuses on the theoretical analysis of risk-aware preference-based reinforcement learning and introduces Risk-Aware-PbRL (RA-PbRL) to optimize both iterated and accumulated risk-aware objectives.

Strengths

- This paper proves that both iterated and accumulated quantile risk-aware objectives can be applied to one-episode reward settings. This may provide a theoretical foundation for future episodic RL or PbRL methods focusing on risk-related objectives. - This paper provide analysis of regret guarantee on the proposed risk-aware algorithm.

Weaknesses

- I think it is necessary to clearly clarify all the strong assumptions, not only the linear reward function assumption, as these strong assumptions typically cannot be met in real-world control scenarios. This would make the theoretical results in this paper more applicable and useful for researchers using deep PbRL methods to address real-world control problems. - Why is the regret of RA-PbRL higher than PbOP when $\alpha$ is small (Fig. 1(b) and 2(b))? Can the authors provide an intuitive explanation for this phenomenon?

Questions

See above.

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

None.

Summary

This paper studies preference-based RL (PbRL) where instead of the expected return, the agent optimizes a risk measure based on preference feedback. The authors study two settings called "iterated" and "accumulated" quantile risks, otherwise known as nested and static risks. They provide sublinear regret bounds for both approaches and instantiate a hard-to-solve MDP to establish a lower bound.

Strengths

The proposed research is well-motivated, as it combines the accessibility of preference-based feedback with risk-sensitive RL, which is crucial for safety-critical applications. If correct, the contributions are meaningful, as they englobe both nested and static risk measures. The text has a nice flow. I have to say that I reviewed a previous version of this submission and the structure is much clearer now.

Weaknesses

- Some used terminologies are non-standard for the risk-sensitive RL (RS-RL) community. For example, the authors discuss "iterated" versus "accumulated" risk measures. As far as I know, most of the RS-RL works name these "nested" versus "static" risk measures [1,2,3]. This confused me, as I could not understand the abstract until the middle of the introduction. - line 62: "the optimal policy becomes history-dependent, which is more general than assuming the trajectory reward is a linear function of the sum of per-state features". Why? This deserves more explanation, as it justifies the novelty of this work compared to Chen et al (2023). - Previous works have studied RL under trajectory feedback, although not in the PbRL setting, see e.g., [4]. Therefore, contribution 3. seems overstated. [1] Hau, J. L., Petrik, M., & Ghavamzadeh, M. (2023, April). Entropic risk optimization in discounted MDPs. In International Conference on Artificial Intelligence and Statistics (pp. 47-76). PMLR. [2] Hau, J. L., Delage, E., Ghavamzadeh, M., & Petrik, M. (2024). On dynamic programming decompositions of static risk measures in Markov decision processes. Advances in Neural Information Processing Systems, 36. [3] Tamar, A., Chow, Y., Ghavamzadeh, M., & Mannor, S. (2015). Policy gradient for coherent risk measures. Advances in neural information processing systems, 28. [4] Efroni, Y., Merlis, N., & Mannor, S. (2021, May). Reinforcement learning with trajectory feedback. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 8, pp. 7288-7295).

Questions

- The authors tackle nested and static CVaR with the same methodology. Classically, the static formulation encounters time consistency issues that do not appear in nested CVaR. Why isn't it the case here? Could the authors discuss time-consistent risks in the preference-based setup? - What does Assp. 3.1 formally say? How should it be interpreted and how restrictive is it? - lines 142-148: Why can we restrict policy-search to deterministic policies even in the preference-based setting? Is it an underlying assumption or is there a formal result stating that we do not lose optimality? - Are the results from Sec. 3.2 novel - especially the recursions (5) and (6)? - It is perhaps a philosophical concern but I would be glad to have the authors' feedback. If the feedback is preference-based, what is the difference between average and risk-sensitive return maximization? In other words, doesn't the agent's objective express itself directly through the preference outputs? This question is crucial to me because I currently do not see the motivation for RS-PbRL. *Minor comments and suggestions* - Please follow the formatting guidelines for section titles - Secs. 2, 5 - line 58: "may not be applicable"; line 60: "cannot" - line 125: "and the reward"; line 126: "the trajectory embedding dimension" - Assp. 3.1: "We" - capital letter - line 133: " At each iteration"; line 134: "unlike standard RL"; line 137: "a Bernoulli distribution" - lines 139-140: "It is aware" --> "It is known" or "We are aware that"; line 157: "risk-measures" - line 160: "a random variable" (remove distribution); line 161: "then" --> "so" - line 163: What is $Z$? Should it be $X$? - line 176: "denotes the" - lines 178-179: Reformulate "the policy should be history-dependent policies" - line 185: "The proof is" - Lemma 3.5 and 3.6: "For a tabular MDP and a reward of the entire trajectory being decomposable as ..." - line 191: "given the current history" - line 199: "The proof is" - line 207: "We define" - line 211: "regardless of iterated or accumulated" - line 214: "of the entire episode training" - line 218: "that minimizes regret" - lines 221-22: "We present the algorithm called RA-PbRL and establish and upper-bound regret for it" - line 225: Remove "The algorithm"; "defined" --> "described" or "presented"; "which main part" - line 229: "choose a policy"; line 232: "find an estimated optimal policy" - line 236: "we initialize"; line 237: "we observe history samples"; line 239: "use" (present) - Algo 1, line 9: "Receive trajectory ... and preference"; line 10: italic $k$ - lines 253, 254: "with policy $\pi$" - remove "the" - line 274: "hard-to-learn constructions" (hyphen and "s") - lines 275-6: "are unavoidable in some cases" - line 282: "For a comparative analysis"; line 286: "aligned ... with RA-PbRL"

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Although some limitations are not addressed in this work (see previous remarks), most are properly described in Sec 6.

Thank the authors for their response

Thank the authors for their response. I tend to maintain my score.