Summary
This paper addresses the problem of maximum state entropy exploration in environments without rewards. Especially, it proposes a method to learn an history-based policy maximizing the entropy over the states sampled in a single trajectory. The method, called $\eta\psi$-Learning, combines predecessor representations, to keep memory of the previously visited states, and successor representations, to predict the states that will be visited in expectation under the current policy. The introduced algorithm is tested against MaxEnt in a set of toy domains and some continuous control tasks, it is also briefly tested against VariBAD in gridwolrd domain.
Strengths
- The paper proposes a simple yet effective method to learn non-Markovian policies that maximize the state entropy induced by a single trajectory;
- The introduced method remarkably improves the performance of MaxEnt, here taken as a representative algorithm for maximum state entropy exploration with Markovian policies;
- The ideas are presented with clarity and with some nice visualizations (e.g., Fig. 1 and 3) to support the intuition.
Weaknesses
- It is unclear how the procedure can be scaled up to more challenging domains, e.g., image-based inputs;
- The method is only tested against MaxEnt, which is not the state-of-the-art method for state entropy maximization, especially in continuous control tasks.
Questions
This is an overall interesting paper, tackling a relevant problem raised by (Mutti et al., 2022a) on how to learn non-Markovian policies for maximum state entropy exploration efficiently. Especially, the authors propose to circumvent the inherent computational intractability of the problem with a well-motivated function approximation approach, in which the current decision is conditioned on a compact representation of the history and a forecast of the future state visitations under the policy.
Although the paper does not introduce particularly novel or surprising ideas, it presents the method with clarity and showcases a brilliant performance for $\eta\psi$-Learning, at least in comparison with MaxEnt. While those might be sufficient reasons for acceptance, I believe that including a discussion on how this procedure can be scaled up to more challenging domains, such as image-based tasks, would make for a significant leap in the value of the paper.
*(Major. How to scale up the approach?)*
The experiments show a remarkable improvement over MaxEnt, but previous works (e.g., Mutti et al., 2021) have demonstrated how Markovian policies maximizing the state entropy can be learned in challenging domains. Instead, it is unclear how the predecessor and successor representations of $\eta\psi$-Learning can be scaled to truly high-dimensional settings. While the method may be trivially adapted to continuous settings via discretization, it is unclear whether it can match the performance of previous works based on $k$-NN entropy estimators. Can the authors discuss avenues to scale their approach to high-dimensional domains? Estimating $\psi_\theta$ in practice would be significantly easier than state density estimation?
*(Estimating the state visitations instead of the entropy)*
In $\eta\psi$-Learning, the quality of an action is evaluated in terms of the future visitations it will induce under the policy, from which the entropy of the trajectory can be then computed. However, the task of estimating the future state visitations seems strictly harder w.r.t. what is really needed to make an optimal decision, i.e., an estimate of the entropy that future visitations will induce. I am wondering whether there is a way to directly target entropy estimation instead of future state visitations in $\eta\psi$-Learning.
*(Experiments. MaxEnt performance)*
The learning curve of MaxEnt is quite underwhelming: While the inferior performance of Markovian policies against $\eta\psi$-Learning is reasonable, I would have expected to see some learning progress for MaxEnt as well, especially in domains in which the random policy is far from the optimal strategy, such as in ChainMDP and RiverSwim. Can the authors clarify what is happening with the learning curve of MaxEnt?
*(How the approach relates to Forward-Backward representations?)*
A previous paper (Touati and Ollivier, Learning one representation to optimize all rewards, 2021) also addressed unsupervised reinforcement learning through representations of the past and future visitations, which they call forward-backward representations, bearing some similarity with the predecessor and successor representations of $\eta\psi$-Learning. Differently from this paper, they make learning these representations the actual unsupervised objective, and they advocate for directly employing forward-backward representations for zero-shot adaptation to the supervised task, instead of running state entropy maximizing policies first. Can the author compare $\eta\psi$-Learning with this previous approach, and explain whether the successor representation might be directly exploited in a supervised adaptation phase?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper does not include an explicit assessment of the limitations of the presented approach, which would add significant value to the paper in my opinion.