Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

Summary

This paper proposes a new framework for Maximum Entropy Reinforcement Learning using Energy-Based Normalizing Flows (EBFlow). The authors argue that this framework offers three main advantages: Firstly, the soft value function can be obtained without estimation. Second, the framework combines policy evaluation and policy improvement in one iteration. Finally, it does not use Monte Carlo methods and interacts with the environment effectively. To achieve this goal, Learnable Reward Shifting (LRS) technique which involves reward shifting for the value function using a reward shifting network and Shifting-Based Clipped Double Q-Learning (SCDQ) technique that employs two reward shifting networks for the value function and utilizes the minimum value function similar to double Q-learning have been proposed.

Strengths

1. The paper is well-structured, and the overall content is easy to follow and intuitive. The detailed background explanations allow readers to progress through the main text with a solid understanding of the prior work. 2. The experimental domains are diverse, clearly demonstrating the strengths of the proposed algorithm in an easy-to-understand manner. The experiments are well-structured, and the detailed explanations make it easier for readers to follow the setup. 3. The proposed algorithm does not use Monte Carlo-based methods, allowing it to select the proper action for training during each step without waiting for the end of an episode. This enables the policy to interact with the environment more effectively 4. Existing SAC-based algorithms often rely on value function estimation, which can introduce errors. The proposed algorithm can obtain the soft value function without estimation, thus avoiding such errors.

Weaknesses

1. Despite the fact that the proposed method is the first to apply EBFlow to reinforcement learning (RL), the novelty of the algorithm may appear weak, as there are several existing RL methods based on flow-based models. Besides applying EBFlow to the RL structure, were there are any additional considerations or advantages that were taken into account to enhance its performance or applicability in the context of RL? If such points exist, highlighting them would help to emphasize the contributions of the authors' proposed technique. 2. Introducing new frameworks generally necessitates providing sufficient mathematical proof to validate their efficacy, even for basic techniques or simple conjectures. It would be beneficial if the paper included mathematical proofs demonstrating that the proposed framework converges to the optimal policy. For example, the techniques presented in the paper, namely Learnable Reward Shifting (LRS) and Shifting-Based Clipped Double Q-Learning (SCDQ), have a significant impact on performance. However, the paper does not provide adequate mathematical justification for these techniques. Including such proofs would greatly enhance the confidence in the proposed method. 3. It would be beneficial for the paper to compare its methods with more recent state-of-the-art algorithms. The comparison with SAC, published in 2018, feels a bit outdated. Given that reinforcement learning is a rapidly evolving field, including comparisons with more recent methods would provide a clearer understanding of where this framework stands in the current landscape. Such comparisons would greatly enhance the paper by placing the proposed method in the context of the latest advancements.

Questions

1. Please answer the questions in the weakness section 2. According to the appendix, the proposed method is said to be 2.3 times slower than SAC. Given this difference in speed, a time-wise comparison of the algorithm's performance would be very helpful to understand its efficiency relative to SAC. 3. In Figure 3, SAC and MEow show similar performance in the Mujoco domain, but there is a noticeable difference in performance in the Isaac Gym domain in Figure 4. Could you provide a further analysis on the reasons for the occurrence of this difference? It would provide valuable insight into the strengths and limitations of the proposed method. Minor Issues\ Typo in line 149: "Jocobian" should be "Jacobian."\ Misreference in section 4.1: It should refer to Fig. 2, not Fig. 1.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

They have handled it well.

Summary

This paper proposes to use the energy-based normalizing flow (EBFlow) to represent the policy and value functions in maximum entropy (MaxEnt) RL. Specifically, the paper builds the connection between EBFlow and MaxEnt RL by linking the conditional unnormalized density that depends on the input to the action value function Q and the constant that is independent of the input to the value function V. As a consequence, the Boltzmann policy defined by Q and V can be expressed exactly using a mapping in EBFlow. By updating the current Q estimate to approximate the action value of the Boltzmann policy, the algorithm would spontaneously update the Boltzmann policy. Hypothetically, the algorithm should converge to the optimal policy under entropy regularization. Using an existing implementation of EBFlow in the relevant literature, the paper proposed MEow based on this formulation. Experiments on four MuJoCo environments and six Issac Gym environments demonstrate the learning performance of MEow.

Strengths

The strengths of this paper are as follows: 1. The connection between EBFlow and MaxEnt RL revealed in this paper is very interesting and, to my knowledge, novel. Given that MaxEnt is a popular approach in the RL literature, the issue addressed in this paper, learning the action value function and the Boltzmann policy, as well as the proposed approach, will be interesting to a broad set of audiences in the RL community. 2. The paper includes abundant comparative and ablation experiments. Notably, the ablation study covers the effect of the two components in the practical implementation, the difference between the deterministic and stochastic evaluation, etc. 3. The paper is well-written and mostly clear.

Weaknesses

1. There is a potential concern about the empirical investigation: The proposed method is tuned over the target smoothing parameter, while the baselines are not tuned. Why is the target smoothing parameter tuned per environment? What would happen if the proposed method also used the same, fixed smoothing parameter? 2. This is a minor point. The organization of the paper can be improved. Specifically, the background section is too long, which takes up the space for more interesting discussions presented in the appendix. If space allows, the main text should include the network architecture in Figure A5, as it may be unfamiliar to a lot of readers. In addition, the limitation, especially the computational aspect of the proposed approach, should be discussed in the main paper. Other suggestions that do not affect the evaluation: * It would be great if there is at least some qualitative comparison between the proposed method and baselines using MaxEnt RL in the multi-goal environment. * Typos between Lines 254 and 276: The reference to Figure 1 should be to Figure 2.

Questions

Some questions that may affect the evaluation: 1. Are deterministic policies used for other algorithms in the evaluation in Figures 3 and 4?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations of the work are discussed in the appendix, which should be moved to the main text. Another unmentioned potential limitation is the sensitivity to the target network smoothing parameter.

Summary

The paper presents a new framework for Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) using Energy-Based Normalizing Flows (EBFlow). Traditional MaxEnt RL methods, particularly for continuous action spaces, typically utilize actor-critic frameworks and alternate between policy evaluation and policy improvement steps. The proposed EBFlow framework integrates these steps into a single objective, eliminating the need for Monte Carlo approximation while enabling efficient modeling of multi-modal action distributions. The framework was experimentally validated on the MuJoCo benchmark and high-dimensional robotic tasks in Omniverse Isaac Gym, demonstrating superior performance compared to existing baseline methods.

Strengths

- By combining policy evaluation and policy improvement into a single training process, the proposed method simplifies the optimization process and avoids potential optimization errors inherent in alternating updates. - The use of normalizing flows facilitates efficient sampling and exact calculation of probability density functions, which addresses the inefficiencies associated with Monte Carlo methods in traditional MaxEnt RL approaches. - Experimental results indicate that the EBFlow framework outperforms widely-adopted baselines on both the MuJoCo benchmark and complex robotic tasks, showcasing its effectiveness and robustness.

Weaknesses

- There are some unreasonable aspects in the organization of the content. For example, Section 3.2 appears somewhat abrupt; it is suggested to move the content of A.4 into the main text. The discussion about inference in Section A.3 is also very important and should be included in the main text. - The experimental section does not sufficiently highlight the advantages of the MEow method. The experiments in Section 4.1 on toy examples are not enough. Can comparisons be made from the perspectives of inference time, training time, representation capability, etc.? - While the integration of policy evaluation and improvement steps simplifies the training process conceptually, the implementation details of the EBFlow framework may introduce additional complexity that needs to be managed.

Questions

- GFlowNet points out that the MaxEnt RL paradigm has deficiencies in terms of diversity. Specifically, MaxEnt RL tends to achieve target states that are "further" from the initial state. Does MEow also inherit this defect of the MaxEnt RL paradigm? - Energy-based normalizing flow is a generative model, and its original loss is to maximize log-likelihood. In Algorithm 1, this loss does not seem to be present. In my view, MEow uses the network structure of energy-based normalizing flow, but the loss still follows the MaxEnt RL paradigm. In this case, could the title of this paper be potentially misleading? - Figure A5 shows that the MEow network uses a hypernetwork module. Why was this module introduced? Is this a common architecture for energy-based normalizing flow? - What are the learning dynamics observed during the training process? Are there any notable differences in the convergence behavior compared to traditional actor-critic methods? - How does the proposed framework scale with increasing complexity of the environment or the state-action space? Are there any specific limitations or challenges observed during experimentation? - How robust is the EBFlow framework to different hyperparameters?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The author has already discussed the limitations ofthe method and its potential impacts in the paper, and provided possible solutions.

Summary

The paper introduces a new MaxEnt RL framework called Meow based on Energy-based Normalizing Flow, which integrates the policy evaluation steps and the policy improvement steps and results in a single objective training process. Besides Meow enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation and supports the modeling of multi-modal action distributions.

Strengths

1. The proposed method is innovative and interesting, which utilizes the special property of Energy-based Normalizing Flow, and thus enables the consistency between the actor and critic and a single objective training process. 2. The paper is overall well-written. The background and related works are introduced appropriately. 3. The experimental results are comprehensive, involving the mujoco and Issac gym benchmarks.

Weaknesses

1. The calculation of the derterminant is usually time-consuming. So the reviewer wonders how the training time of Meow is compared with that of the classical off-policy RL methods like SAC. 2. In the experiments, Meow is just compared with several classical RL methods for continuous control and lacks the comparison with advanced RL methods [1, 2] with other generative models like diffusion model and consistency model, which also supports the modeling of multi-modal action distribution. [1] Yang L, Huang Z, Lei F, et al. Policy representation via diffusion probability model for reinforcement learning[J]. arXiv preprint arXiv:2305.13122, 2023. [2] Yue Y, Kang B, Ma X, et al. Boosting offline reinforcement learning via data rebalancing[J]. arXiv preprint arXiv:2210.09241, 2022.

Questions

1. How is the training time of Meow compared with other methods? 2. How is Meow compared with other advanced RL methods with the utilization of generative-model-based policy? 3. Could the authors explain why Meow can learn a good reward shifting rather than a trivial shifting, like for any $s_t$, $b(s_t)=0$?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have clearly presented the limitations in the paper.

Summary

The paper presents a new method for MaxEnt RL based on the recently proposed energy-based normalizing flows (EBFlow). The adoption of EBFlow allows us to overcome two major issues with training MaxEnt RL algorithms: (I) sampling from an energy-based model (of the policy) and (ii) approximating the soft value function (with MC methods).

Strengths

* **Contribution**: I think the contribution of this work is significant and novel. The issues discussed with MaxEnt RL are important. The idea proposed is novel, it naturally emerges from the EBFlow framework and it's seamlessly applied to the MaxEnt RL domain. The approach solves the problems mentioned and it demonstrates superior performance compared to other standard approaches. * **Presentation**: the presentation is clear and well-organised, and the method is described in detail.

Weaknesses

* **New and old challenges**: while the approach resolves some issues with the current MaxEnt RL methods, it still suffers from typical deep RL issues (e.g. overestimation, for which the authors proposed SCDQ) and it introduces additional challenges (e.g. exploding Jacobian determinants, for which the authors introduce LRS). Normalizing flows also introduce additional constraints on the network's architecture, a problem which I believe is strongly mitigated by the introduced LRS (?) * **Required clarification on LRS**: see Questions

Questions

* Typo at line 149, "Jocobian" * I am not sure if the role of the Learnable Reward Shifting (LRS) is clear to me. The authors say that they were inspired by the reward-shifting literature. However, I don't see the LRS term working the same way as classical reward shaping, as a state-dependent non-linear function ($b_\theta$), without any particular constraints (I don't see any stated) wouldn't preserve the properties of the original functions. I think the role of the learnable reward shifting is to have some form of "baseline" in the Q and V functions that simplifies the estimation process (since it's using a more flexible learning architecture), and so it avoids the explosion of the Jacobian. Could the authors provide clarification on this? * I would be interested in seeing a comparison between the learned reward shifting term and the value function (V) in a plot (it would suffice to see this in the Appendix)

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

Limitations are addressed in Appendix. I think the authors should mention this in the main text.

Follow-up questions on hyperparameter tuning

Thank you for the rebuttal and additional experiment results on SAC’s sensitivity on the target smoothing factor, $\tau$. I think the difference in the sensitivity to $\tau$ of SAC and MEow is very interesting and worth further investigation. In addition, I wonder what the hyperparameter tuning process is in the Issac gym environments. Are both SAC and MEow tuned (for both $\tau$ and $\alpha$)? If yes, how?