Summary
This paper studies off-policy biases in multi-step goal-conditioned reinforcement learning (GCRL). It focuses on successful trajectories and divides the biases into two types: shooting bias and shifting bias, which correspond to TD errors of multiple steps and the errors of the action-value function Q at goal states. For the former, the paper suggests that sometimes this bias is helpful and proposes to use quantile regression to utilize this type of bias. For the latter, the paper proposes to truncate the multi-step target to the step in which the goal is first reached. An empirical study shows the proposed algorithm, BR-MHER, consistently performs well on the six tested domains.
Strengths
This paper has a very clear structure and is well organized. It first gives an introduction to GCRL and different types of biases. Then, it provides a literature review on multi-step off-policy (goal-conditioned) RL and quantile-based regression, which is used to utilize potentially useful shifting bias. It then analyzes the multi-step target in GCRL and introduces two types of biases, followed by corresponding treatment. Finally, metric and empirical results are given to illustrate the effectiveness of the improved algorithm. Overall, the logic is clear.
Weaknesses
The major weaknesses are its presentation and soundness.
For the presentation, the paper needs to improve its preciseness and rigorousness. First of all, there are a lot of typos and undefined symbols, which make it difficult to understand the accurate message intended to be delivered (see Questions for an unexhausted list). Apart from these, the paper is also not precise and rigorous in its exposition. For example, 1) in Eq. (1), it defines the policy as the greedy policy of the $Q_\theta$ function, while it is actually a parameterized policy and may be different from the greedy policy; 2) the reuse of $\delta_\theta$ in Eq. (3) is really confusing as it was defined as the TD error already; 3) In Eq. (4) and Eq. (5), a) $r_i^{\tau^g_\pi}$ is not defined; b) the exponent of the $\gamma$ in the final term is off; c) does Eq. (4) hold? If yes, how?
For its soundness, the biases that the paper proposed are either not precise or not well justified. Despite the effort to isolate the defined off-policy biases from other biases (like over-optimistic bias and hindsight bias), the shooting bias seems to couple with the bias from the target network. As for the shifting bias, it appears that it is just an estimation error instead of a bias due to the off-policy issue of HER. At least from its current presentation, it seems farfetched to call the defined shooting and shifting biases off-policy biases.
Questions
Questions that may affect the assessment:
1. What doesn’t an episode or an n-step trajectory terminate upon reaching a goal? And why can’t we just set the values of goal-reaching states to zero?
2. How can we get Eq. (4)? Isn’t $Q_\theta$ the estimation of the action value function?
Clarity and typos (that have a low impact on the assessment independently):
1. The naming of the algorithms is a bit confusing. In the appendix, all the three variants of the proposed algorithm, TMHER($\lambda$), QR-MHER, and BR-MHER, are using the $\lambda$ target for learning, which is only indicated in TMHER($\lambda$). This may cause confusion that QR-MHER and BR-MHER don’t use the $\lambda$ target.
2. The paper focuses exclusively on deterministic environments, which is mentioned in the introduction but should also be explicitly stated in the preliminary section. Defining $\mathcal{T}$ to be a deterministic transition function will help make this clear.
3. In Eq (1) on page 3, what is $\mathcal{B}$? Is it a dataset?
4. Before section 4 on page 3, “can truncate” should be “can be truncated.”
5. In the first equation (unlabeled) on page 3, 1) $\pi_\phi$ should be $\pi_\psi$; 2) looks like $\theta$ and $\bar{\theta}$ are swapped, which is really confusing.
6. On page 3, the details of the derivation of Eq. (2) are not in Appendix A, which I assume is mistakenly written as the derivation of Eq. (3) in A.8.
7. Several typos in the derivation in A.8: 1) Should the $t+1$s on the left-hand side be $t$? 2) What are $s$ and $a$ in the first equation? 3) Are the subscripts for all $s_{t+1}^\tau$s off? 4) What’s the definition of $r_t$?
8. Be consistent with the naming of the bias. For example, in section 5 on page 4, “shift” should be “shifting,” and “shoot” should be “shooting.”
9. On page 5, “Quatile” should be “Quantile.”
10. On page 6, there is no Eq. (8).
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.