Summary
Entropy regularization is commonly used in on-policy policy gradient methods like PPO to improve performance and stability. However, because entropy regularization only maximizes policy entropy in visited states, it has led to the development of Maximum Entropy Reinforcement Learning (MaxEnt RL), which aims to maximize the cumulative reward and the entropy of trajectories. Algorithms such as Soft Actor-Critic (SAC) have been extensively researched in the off-policy RL, but there is a lack of work for on-policy methods. In this paper, the authors propose an on-policy actor-critic algorithm called EAPO within the MaxEnt RL framework. This algorithm separates the entropy objective from the MaxEnt RL objective and introduces an entropy critic. This separation enables finer control over the entropy term and makes it easier to adjust the balance between the entropy term and the value term. Through experiments conducted in Mujoco and Procgen environments, EAPO demonstrates improved performance compared to PPO.
Strengths
1. There have been many approaches to improving the performance of on-policy algorithms by using entropy regularization to increase exploration and enhance stability. There has also been a significant body of research on using the MaxEnt RL framework, which increases the expected trajectory entropy in off-policy algorithms like SAC to improve performance. However, the use of MaxEnt RL in on-policy algorithms has been underexplored. Through research that combines PPO with MaxEnt RL, a new algorithm was developed, and experimental results demonstrated better performance compared to PPO.
2. By combining the on-policy RL algorithm PPO with the MaxEnt RL framework, both stability and performance were enhanced. Moreover, separating the entropy objective from the estimate has several advantages. Firstly, the entropy term can have a distinct discount factor, allowing for effective reduction of the horizon, addressing the long trajectory issue present in traditional MaxEnt RL. Additionally, by using Generalized Advantage Estimation (GAE) for the entropy term, the trade-off between variance and bias can be controlled. Separating the objective function and introducing a new entropy critic enables independent optimization of the entropy term while fine-tuning control.
3. Off-policy algorithms may suffer from data overfitting due to the reuse of learning data and a lack of generalization in unseen environments. By merging MaxEnt RL, which is robust to distributional shifts in environments, with on-policy RL, generalization was improved. Experiments showed enhanced performance in unseen environments. Furthermore, the inherent simplicity of on-policy RL and EAPO, along with the generalization property of MaxEnt RL, open up new avenues for applications in competitive and robust RL fields.
Weaknesses
1. Approaches like the soft advantage function and GAE have been explored extensively in previous off-policy MaxEnt RL research, and there have been studies combining on-policy RL with the MaxEnt RL framework. Since the performance difference between simply adding entropy reward to PPO and the proposed algorithm appears to be similar in the experiments, additional contributions would be valuable.
2. The direction of integrating MaxEnt RL into PPO, which traditionally used entropy regularization alone, is promising. It would be even more motivating to highlight the advantages that arise from combining MaxEnt RL with on-policy algorithms in this paper. The paper could benefit from demonstrating the inherent simplicity and lack of overfitting in unseen environments with experiments, both strengths of on-policy algorithms in this paper. For example, a comparison with off-policy RL algorithms in the Procgen unseen environment would be beneficial. Additionally, performance experiments in the MuJoCo environment would be good not only to compare PPO, but also to off-policy algorithms and actor-critic algorithms.
3. The paper's contribution of separating the entropy objective from the MaxEnt RL objective and proposing a new entropy critic is valuable. It would be advantageous to demonstrate experimentally whether the associated benefits materialize. While it is possible to fine-tune control with additional discount factors and hyperparameters in the experiments, it would be beneficial to include an ablation study. Additionally, demonstrating the reduction in horizon as an advantage through experiments would strengthen the paper's findings. Furthermore, showing the decrease in variance in the entropy term due to the use of GAE in experiments would increase the validity.
Questions
1. Although an entropy critic was added to optimise the entropy objective function separately, the experiments utilized parameter sharing with the existing value critic. This approach offers the advantage of reducing computational costs. However, it appears that using this method is similar to not adding a new entropy critic and instead incorporating the final layer into the existing value critic, then separating and training only the objective function. I'm also curious about the performance difference when parameter sharing is not used compared to when it is used.
2. In this paper, it was mentioned that EAPO's theoretical guarantees are unclear in a regularised MDP setting. I'm curious if adding the MaxEnt RL framework independently to the on-policy policy gradient theoretically does not hinder convergence. Additionally, if we were to use TRPO (Trust Region Policy Optimisation) instead of PPO as the baseline in the setting proposed in this paper, could we potentially establish theoretical guarantees?
3. In the experiments with MuJoCo environments, it appears that entropy reward augmented PPO is an interpretation of adding entropy rewards to the existing PPO. I'm curious whether entropy reward augmented PPO kept the entropy regularisation intact or removed it and simply added entropy rewards. Additionally, I would like to know your thoughts on using both entropy regularization and the maximum entropy RL framework simultaneously.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.