$f$-Policy Gradients: A General Framework for Goal Conditioned RL using $f$-Divergences

Summary

This paper studies goal-conditioned reinforcement learning and proposes a framework based on f-divergence. The authors show that minimizing the f-divergence between the agent’s visitation and the proposed goal distribution leads to the optimal policy and discuss the practical implementations, connections with previous GCRL methods.

Strengths

1. The paper is well written and easy to follow. 2. The motivation to study the divergence between the state visitation and goal distribution is straightforward, which is good. 3. The analysis is thorough. The discussion between the proposed framework and previous methods is also interesting.

Weaknesses

1. The authors state that they are focusing on sparse-reward (1 if the agent reaches the goal and 0 otherwise). However, $p_g(s)$ will become a Dirac distribution and the f-policy gradient objective makes no sense in this case. 2. As the authors admit, the algorithm and analysis are meaningless in a full Dirac distribution setting. The authors choose to "add an ϵ probability to all other states". Will this uniform ϵ addition + f-PG performs better than uniform reward shaping + standard PG with GC-reward? Intuitively, such uniform addition will still lead to sparse training signal. 3. It is not very clear about the advantage of the proposed framework. In the strict sparse-reward setting, the framework degrades. Even in dense-reward settings, the analysis only states that optimizing the f-PG objective moves to the optimal policy, which is also true for standard PG methods in GCRL that are based on goal-conditioned (dense) rewards only. Are there any efficiency advantage of f-PG?

Questions

See the weakness section above.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors only discussed some future avenues of their work. I suggest the authors to discuss more about the benefit over the shaping rewards GCRL algorithms, and discuss the previous theoretical results established for shaping rewards methods.

Summary

The paper focuses on goal-conditioned reinforcement learning (RL), where the goal-conditioned reward function is often sparse. Consequently, standard goal-conditioned RL algorithms suffer from the exploration problem, thus requiring the need to shape rewards. The paper proposes a a new formulation that minimizes the $f$-divergence between the policy's state-visitation distribution and the goal distribution. The paper theoretically shows that minimizing such objective yields the optimal policy and provided empirical evaluations on two maze tasks.

Strengths

- As far as I know, the $f$-divergence between state-visitation distribution and goal distribution is a novel contribution. - The paper theoretically shows that optimizing their objective yields the optimal policy. - Good visualizations on demonstrating the policies' behaviours.

Weaknesses

- I think $\gamma$ should not be introduced as it does not really contribute to the idea. Also, since we wish to maximize $\mathbb{E}_\theta\left[\eta\right]$, the aim to achieve the task as quickly as possible is there. - How is $\int \eta_\tau(s) ds = T$ in theorem 4.3 when we are not assuming fixed horizon $T$---this is based on page 3, on line 131, where the paper assumes that the trajectory ends when the agent reaches the goal or if maximum horizon $T$ is reached. - It seems like if the paper is using the fixed-horizon setting, then theorem 4.3 will make sense, in terms of being similar to maximizing the expected return. - Furthermore, I believe we can actually relate the policy error even if the trained policy is sub-optimal. It will be nice if there is some results regarding suboptimality. - On page 5, second last paragraph, why do we not just assign equiprobability to the correct goal states, but $\varepsilon$ to all other states? It seems like we already have such access anyway. Otherwise, is this not a "coverage" assumption? - If I am not mistaken, under Gaussian policies, $f$-PG with FKL essentially recovers some sort of L2 distance. I think for reaching tasks this is naturally a good and dense reward, which may explain the better performance than other approaches. - It will be interesting if the paper provides plot describing the correlation of the "constructed" reward and the "natural" dense reward (i.e. L2 distance between state and goal). - It will also be nice to show a task aside from maze and analyze whether FKL still performs better in general. - The paper should further describe how the state-visitation distribution $p_\pi$ and the goal distribution $p_g$ are trained, maybe in the appendix.

Questions

- Based on Figure 4, the paper suggests that the maximizing entropy of state-visitation matters. - When computing the success rate, are we sampling from the policy, or taking the mean of the policy? - How would PPO with entropy-regularized objective (or SAC) perform? - The paper proposes to learn the distributions separately using kernel density estimators (KDE). - It seems like this is not really the consequence of the theory, but just a choice, is that right? - If it is indeed a choice, I am not completely sure if the instability of the discriminator plays a role. - In the related works section, there were few sentences indicating instability due to non-stationary rewards (in particular, under the subsection shaping rewards). Isn't this approach also non-stationary? Shouldn't there be similar instability as other approaches then? **Possible typos** - On page 2, line 52: "insufficiency" instead of "insufficient" - On page 2, line 90: "learning" instead of "Learning" - On page 3, line 94: "expert's" instead of "expert" - On page 3, line 102: "learning" instead of "Learning" - On page 3, line 133: The expectation should also be over the goal $g$ - On page 3, line 137: The distribution should include $\mu_0(s_0)$ - On page 3, equation (1): $p_\theta(\tau ; g)$ instead of $p_\theta(\tau)$ - On page 3, equation (2): $(s) d\tau$ should be $d\tau$ on the numerator - On page 4, line 160: $\pi$ instead of $p_\pi$. - Figure 2, legend: $fkl-rew$ should be $fkl-new$ - Appendix, on page 6, line 86: Should be "Theorem 4.3" instead of "Theorem 4.2"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

- The paper requires an assumption of knowing the goal function. That is, reward $r(s, a, s';g) = \mathbb{I}(s' = g)$. Designing such a goal function can be challenging due to hardware constraints in real-life applications, notably in robotics tasks.

Summary

The paper proposes to minimise the f-divergence $J(\theta) = D_f(p_\theta(s) \| p_g(s))$ between the desired state-distribution $p_g(s)$ and the current one $p_\theta(s)$. Clipped likelihood-ratio policy gradient is used to estimate the gradient. Kernel density estimation used to estimate the state visitations. Experiments on point reaching in grid world and in continuous-state maze are provided.

Strengths

The paper is straightforward to follow. The idea and motivation are clear. Originality: this niche has been explored in a number of works. The paper proposes a slightly novel approach, but its merit is not clear as it is only tested on simple point grid world and maze environments. In contrast, the baselines have been shown to work on robotics environments and scale quite well. Quality: the paper is OK as a first step in validating the idea, but to really make claims about this algorithm, comparisons on harder environments are necessary. Important details are missing, e.g., it is not really clear from the paper how exactly the gradient is estimated and how KDE was used to estimate visitation frequencies, etc. Clarity: the paper is mostly clear but it can be improved. E.g., the lemmas and theorems in Sec. 4 are obvious. It would be better to show Algorithm box there and provide more experiments. Significance: currently quite low. Basically, experiments use f-KL, which is the well-known MaxEnt RL. So, the benefit of introducing the whole f-divergence story is unclear.

Weaknesses

It is unclear how the authors deal with having $p_g(s)$ in the denominator of the ratio, since by definition it is a delta function. Some epsilon-modification is proposed for grid world, but it doesn't seem like it would scale to any reasonably high-dimensional task.

Questions

See comments above

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

The paper does not provide a limitations section. Only future work is mentioned in the conclusion, but this is different from limitations. The authors are encouraged to add the limitations section.

Summary

This paper proposes a novel framework for goal-conditioned RL called f-Policy Gradients (f-PG), which minimizes the f-divergence between the agent's state visitation and the goal distribution to provide dense learning signals for exploration in sparse reward settings. The paper derives gradients for various f-divergences to optimize this objective and shows that entropy maximizing policy optimization for commonly used metric-based shaping rewards can be reduced to special cases of f-divergences. The paper compares f-PG with standard policy gradient methods on grid world environment as well as the Point Maze environments. It shows that f-PG performs well in all three environments and better than the baseline shaping rewards. The paper also illustrates how the learning signal evolves with the policy and performs an ablation to compare different f-divergences on their performances on the three Point Maze environments.

Strengths

Firstly, a new framework for goal-conditioned RL which provides dense learning signals for exploration in sparse reward setting is proposed. Secondly, it is proved that minimizing the f -divergence (for some divergences) recovers the optimal policy. Furthermore, the analytical gradient for the objective looks very similar to a policy gradient which allow authors to use established methods from the policy gradient literature to come up with an efficient algorithm for goal-conditioned RL. Finally, empirical evidence in experiment part demonstrate that f-Policy Gradients outperform standard policy gradient methods and baseline shaping rewards in challenging environments. Overall, the paper presents a promising approach for addressing the exploration challenge in goal-conditioned RL.

Weaknesses

In this paper, utilizing f-divergence to minimize the mismatch of an agent’s goal-conditioned state visitation to this target distribution is the main contribution. Besides, theoretical as well as empirical analysis has indicated the effectiveness of the methodology. However, some weaknesses still exist and remain to be polish up. (1) Firstly, f-divergence is not a complete metric in math. Some shortcomings or limitations of it are not been discussed. (2) Secondly, some notations in the paper are not explained clearly ( e.g. “trajectory dependent state visitation” in section 3, ). (3) Furthermore, Experiment environments which are adopted in the paper seem to be too simple. Effectiveness of the methodology may not illuminated. Furthermore, some well-known methods in sparse reward setting are not compared. (4) Novelty of the paper seems to be relatively weak. While the main body of the paper is well-written, there is space for improvement. I defer some of my issues in the appendix to "Questions".

Questions

Q1: What will be like if replacing f-divergence with some other mathematical complete metrics (i.e. L2-norm or wasserstein distance)? It is suggested to explain the motivation of utilizing f-divergence more. Q2: How does f-PG compare to some famous baselines such as curiosity-driven[1] method and DIAYN[2]? It is suggested to add some Comparative experiment。 Q3: Ablation studies about different components of the methodology are suggested to added. Q4: When the state of environment is in the form of image, is there any difference about the computation of f-divergence? Q5: Some notations in the paper are not explained clearly ( e.g. “trajectory dependent state visitation” in section 3, ). [1] Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018). Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355. [2] Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018). Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

It would be better if the paper conducts more experiments from the aspects of ablation and comparison with other baselines. Whether effective or not in MARL setting may also be a good extension for the paper.

Rebuttal Acknowledged

Thank you for your response and clarification, especially with regards to the relation to MaxEnt RL. It would be important in the future submission to explicitly contrast this method to MaxEntRL and to f-max / FAIRL in the paper since these are the two that immediately come to mind. And although I understand your point about the algorithm being on-policy, I believe it may be hard to publish unless you show that some version of your algorithm is competitive with the baselines on Walker or Hopper or more Fetch environments.