Summary
This paper tackles robust reinforcement learning in large state spaces, where the transition kernel is accessible only in a nominal setting. The authors demonstrate that the $f$-divergence, R-contamination, and the $l_{p}$ norm are computationally infeasible in the context of robust RL for large state spaces. To overcome this limitation, the authors propose two new tractable uncertainty set formulations suitable for large dimensions: double sampling (DS) and the integral probability metric (IPM).
DS involves independently and identically distributed state sampling following a transition kernel $p^o_{s,a}$. The IPM corresponds to the robust Bellman operator, but with a regularization term on the norm of weights. Both approaches are enabling Robust RL in scenarios previously hindered by computational complexity.
The paper introduces a new algorithm, the Robust Natural Actor Critic (RNAC), for training both a critic and an actor for the proposed IPM and double sampling uncertainty sets. The authors provide convergence guarantees for the RNAC algorithm.
Finally, the authors demonstrate the efficency of their approach via two applications: one involving the suite of MuJoCo environments and the other, a real-world robotics application. These practical applications lend credence to the theoretical contributions and the robustness of the proposed approach.
Strengths
- The authors provide two straightforward and computationally feasible uncertainty set formulations for large state spaces.
- The paper offers substantial theoretical contributions.
- The real-world robotics application lends credibility to the paper's robustness claims.
Weaknesses
- The convergence guarantees are valid for $(s,a)$-rectangular uncertainty sets, which is rather limiting. However, it is commendable that the authors have acknowledged this limitation in their work and suggested it as an avenue for future research.
- The paper seems incomplete without the appendix. The need to constantly refer to the appendix disrupts the flow of reading.
Questions
- In lines 175-177, why did you not apply regularization to the bias parameter in the last layer? Moreover, this seems inconsistent with the provided implementation (lines 201-206 in the `RNAC-ppo/ppo_continious.py` file), where regularization is applied to all the critic's layers.
- Doesn't the IPM's proposed regularization ultimately reduce the Lipschitz constant of the learned function, making the agent less sensitive to state variations?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors mention their limitations as future research directions. We appreciate the authors for being upfront.