Summary
This paper addresses the problem of sample-efficient exploration in sparse-rewards deep reinforcement learning. It builds on previous literature on the maximization of the state entropy as an exploration objective, which was mostly used in reward-free settings, proposing a value-conditioned state entropy objective along with the task reward. The latter pushes the agent to maximize the state entropy while penalizing the entropy over the estimated values of the visited states. This value-conditioned state entropy is implemented through an off-the-shelf non-parametric conditional entropy estimator. Finally, the paper compares the performance of value-conditioned entropy maximization against standard entropy maximization in MiniGrid, DeepMind Control, and Meta-World.
Strengths
- The paper proposes an intrinsic reward can be easily incorporated in any previous method;
- The experiments show that the value-conditional objective brings improved/matching performance in a variety of domains;
- The paper clearly presents ideas and contribution, and also include some compelling visualization of the conditional entropy estimation.
Weaknesses
- The intrinsic reward computation requires robust estimates of the value function, which is a challenge per se, especially in sparse-rewards domains;
- As mentioned by the authors in their conclusion, the theoretical ground for value-conditional state entropy is unclear.
Questions
This paper advocates for using state entropy bonuses to accelerate RL in sparse-rewards tasks, instead of training an entropy maximizing policy in reward-free settings, which is the most common mode of previous works.
To adapt state entropy bonuses to the demand of the sparse-reward task, which requires to deal with the exploration-exploitation trade-off, the paper proposes to condition the entropy of the visited states with the value of those states, effectively driving exploration towards rewarding regions of the state space.
This interesting notion leads to a methodology that can be easily incorporated in previous algorithms while benefitting the resulting sample efficiency in a variety of domains.
However, the paper does not provide a formal theoretical understanding of the proposed approach, which leaves some doubts on the generality of their results.
Anyway, I am currently providing a slightly positive score while encouraging the authors to improve the theoretical ground for value-conditioned entropy bonuses, and especially how the estimation error of the value function impacts the learning process.
**(Clarification on the objective)**
From my understanding of Algorithm 1, the proposed method maximizes the value-conditional state entropy over the samples taken from the replay buffer, and not just the samples drawn with the latter policy. While this is common in previous works as well (e.g., Liu and Abbeel, 2021), I am wondering whether looking at the value-conditional entropy of the latest samples is more appropriate in a setting where external reward is also present. Can the authors discuss their implementation choice and comment pros and cons of the two alternatives?
**(Sensitivity to $\beta$ also for VCSE)**
The experiments report an interesting analysis on how the value of $\beta$ affects the performance of A2C+SE. It would be interesting to inspect also the sensitivity of VCSE to the corresponding $\beta$ value.
**(Comparison with CEM)**
A recent work (Yang and Spaan, CEM: Constrained entropy maximization for task-agnostic safe exploration, 2023) considers state entropy maximization under cost constraints. I am wondering whether their approach could be also used to address the problem of reducing task-irrelevant exploration, e.g., by introducing appropriate value constraints. Can the authors discuss the comparison between their solution and CEM, possibly highlighting pros and cons of the two?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Some limitations of the paper and the proposed methodology are reported in the first paragraph of Section 6. I believe that a formal discussion on how the value estimation error affects their method would be extremely valuable.