Summary
The authors address credit assignment in deep reinforcement learning to improve sample efficiency. They propose Hindsight Distribution Correction Estimation (H-DICE), a variant of Hindsight Credit Assignment (HCA) that attempts to mitigate the instability of importance-sampling ratios in the distribution correction. The method is evaluated using PPO in a new GridWorld, LunarLander, and several MuJoCo environments where reward information is forcibly delayed until the end of the episode. The tested baselines are PPO and other variations of it that use HCA.
Strengths
- Overall, the paper is well organized and easy to follow.
- The references in the paper are relatively thorough and shows research effort from the authors. However, I feel that the discussion misses a number of important references related to credit assignment and overstates the novelty of the work (see weaknesses below). The related work section is currently relegated to the appendix, but this should be featured in the main paper in my opinion.
- Other than the violation of the Markov reward property (see weaknesses below), the authors consider a diverse range of environments and follow reasonable experimental practices. I think there is potential for a publication in the future if the authors are able to more fairly evaluate their method in these tasks. I like the new GridWorld environment as a focused credit-assignment study, and perhaps it could be modified to demonstrate the issues tackled by the authors in a more principled way. Unfortunately, the current empirical results do not support the hypotheses or claims made by the authors, as I discuss below.
Weaknesses
- The main contribution is incremental, essentially applying an existing technique from DualDICE to the existing framework of HCA. There are no new theoretical results or analysis of how the proposed method mitigates the purported instability of HCA.
- The experiment results are undermined by the “delayed-reward” setting introduced by the authors. During an episode, rewards are accumulated, and the final sum is presented to the agent only upon episode termination. This makes the reward function extremely non-Markov, which the authors acknowledge (Section 4.1). However, this is a major issue. All of the baselines (which are just PPO with or without various forms of HCA) rely on a value function assuming Markov states; it is impossible to learn $V(s)$ in this partially observable setting, since the agent does not have access to the history of observations. Notably, this is not a matter of credit assignment; the given baselines are formulated based on the Markov assumption that does not hold in the experiments. Without this assumption, the value function is not well defined. There is simply no way for these methods to learn $v_\pi$ or $v_*$, even with an infinite amount of data—the agent does not have the information available to make the problem learnable. On the other hand, H-DICE does not learn a value function, and I suspect this is the main reason it is able to outperform the baselines. Despite this, the scores achieved in the MuJoCo environments are low compared to the standard benchmark, making me question how well the H-DICE agent is actually learning these tasks and whether the authors’ goal of a “demonstrably stable” method has really been achieved. I would like to see how H-DICE performs in comparison to PPO on the standard MuJoCo benchmark or other Markov-reward environments. If the authors’ method is as successful at credit assignment as they claim, then it should be able to outperform PPO in these environments, too.
- Some claims about credit assignment in reinforcement learning are incorrect. The authors say that PPO with GAE (i.e., $\lambda$-returns) “does not perform explicit credit assignment.” This is simply not true. The $\lambda$-return very explicitly assigns credit on the basis of temporal recency, with less credit given to states far into the future. In fact, all deep RL methods perform credit assignment to some extent (usually through the value function itself), because if they did not, they would simply be unable to learn anything. Perhaps what is meant instead is that these methods are assigning credit only based on temporal information and not more sophisticated techniques. Still, there are many papers about more advanced credit-assignment techniques for deep RL (e.g., [1,2,3] and the cited HCA paper) despite the authors’ assertion that that there has been “relatively little consideration given to issues of credit assignment.” See the references of these papers for other related works.
- PPO is incorrectly described as an off-policy method, but in reality it learns from on-policy experience. The importance-sampling ratio in the objective comes from the sample approximation in TRPO [4]. The clipping is applied by PPO to generate a surrogate objective that aims to simulate the trust-region effect.
**References**
[1] Safe and efficient off-policy reinforcement learning. Remi Munos, Tom Stepleton, Anna Harutyunyan, Marc G. Bellemare. 2016.
[2] Hindsight experience replay. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba. 2017.
[3] Expected Eligibility Traces. Hado van Hasselt, Sephora Madjiheurem, Matteo Hessel, David Silver, Andre Barreto, Diana Borsa. 2021.
[4] Trust region policy optimization. John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, Pieter Abbeel. 2015.