Summary
This paper focuses on the meta-reinforcement learning setting, where the agent learns a meta-policy that adapts to new tasks with few observations. However, traditional meta-RL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty. This may make the agent fail to generalize to risk tasks. In this paper, the authors introduce CVaR into the meta RL framework to solve this problem. They theoretically prove that an unbiased gradient estimator is available in the meta RL setting, which allows a general critic module. They then propose the Robust Meta RL algorithm (RoML) algorithm to generate a robust version of any given MRL algorithm, by identifying and oversampling harder tasks throughout training. Experiments on the gym locomotion tasks demonstrate the advantages of the proposed method.
Strengths
1. This paper is well-written and easy to follow. The figures are clear and illustrative for understanding the idea.
2. The theoretical analysis shows evidence to support the advantages of the proposed method.
3. The experiments also support the claim that the proposed method achieves better adaptation to unseen tasks.
Weaknesses
1. I am a little confused about the setting of robust meta-reinforcement learning. What does robustness mean in meta RL? Is there any disturbance to the dynamic system or observation? It seems that the setting mentioned in this paper is that some tasks are harder than others.
2. The main contribution of this paper is extending the CVaR-PG framework to meta RL setting and eliminating the limitation of the biased gradient estimator. However, the intuition behind the difference between using CVaR in RL and meta-RL settings is not clear. If the task space is not discretized (as mentioned in the caption of Figure 1), the boundary of RL and meta RL will be unclear. So we can still have both easy and hard situations in the RL setting and assume there is an unknown distribution of the difficulty. I think the authors should give some hints in the introduction section about why using CVaR in meta RL is better than in RL. Does the advantage come from the discretized task indicator?
3. The experimental evaluation is not thorough enough. First, the experiments do not use standard meta-RL benchmarks, such as Meta World. Second, the experimental setting is not consistent with the motivation “Standard MRL methods optimize the average return over tasks but often suffer from poor results in tasks of high risk or difficulty.” There are no particular high-risk or difficult tasks in the environments （a high-risk task is defined as the high mass in the mujoco environment）. The example in Figure 1 looks good but is not mentioned in the experiment part. Third, potential simple baselines are missing. For example, oversampling the hard task according to the return. This could be a general trick for all meta-learning methods, so I think it should be a fair baseline.
Questions
Points 1 and 2 of the weaknesses are my questions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The limitation of the proposed method is discussed in the conclusion section. The algorithm requires control over the selection of training tasks. This assumption is acceptable since it is possible to define and control different tasks in both simulation and real-world cases.