FlowPG: Action-constrained Policy Gradient with Normalizing Flows

Summary

Handling constraints in reinforcement learning is a fundamental problem that has application in areas such as robotics and resource allocation. A common solution is to incorporate a projection step to compute feasible actions, which involves a computationally expensive optimization solver in the loop. This can be prohibitively slow, especially when constraints are non-convex. When used as part of the policy with a differentiable solver, it can led to the zero gradient problem if the policy output is far outside the feasible space. To circumvent the need for an optimization solver, the authors propose to learn an action mapping which respects the constraints with high probability by design. Specifically, they train a normalizing flow offline on samples from the distribution of feasible actions. These samples are generated via Hamiltonian Monte-Carlo for continuous action spaces and probabilistic sentential decision diagrams for discrete action spaces. During RL, the policy then outputs a latent action, which is mapped to the space of feasible actions via the normalizing flow. The weights of the flow are frozen, but the gradients are still propagated through the network to train the policy. They evaluate their approach, FlowPG, on two continuous control robotics tasks and a discrete resource allocation problem. FlowPG achieves the highest average return across ten random seeds for all problems and has the lowest number of constraint violations (prior to a projection step). When it does violate a constraint, the magnitude of the violation is lower, meaning it is closer to the feasible action space. And it achieves these benefits with reduced wall clock time compared to the best baseline.

Strengths

- Addressing action constraints in reinforcement learning is an important problem and critical for many real-world robotics and decision making tasks. - FlowPG is a novel solution to this problem and effective in improving performance while reducing constraint violations and wall clock time, all while being fairly straightforward to implement. - The finding that a uniform prior reduces constraint violations when used in conjunction with RL is a useful insight. The bounded support of the uniform distribution also works nicely with the fact that policy gradient algorithms often work well when the output is passed through a squashing function to respect box constraints on action limits. - Constraint violations which do occur, however infrequently, can still be remedied by solving an optimization problem. Importantly, this does not need to be done when performing the policy updates, just during rollouts, which makes training more efficient. - The paper is well organized and clearly written. It does a good job explaining the novelty and results and provides enough information to support its claims. In particular, I liked the visualizations in Figure 3 of the learned action space for the Reacher task.

Weaknesses

- The tasks considered in the paper are fairly standard benchmarks and showcase the effectiveness of FlowPG. However, the constraints in each task are relatively simple, with only Half Cheetah having constraints which depend on a portion of state. This makes it difficult to gauge how this approach will scale to more challenging, state-dependent constraints. Tasks with more complex constraints would significantly strengthen the paper. - It seems difficult to scale the sample generation procedure to higher-dimensional action spaces. And for state-dependent constraints, it seems challenging to span the relevant portions of the state space which will be visited by the intermediate and final policies used during learning. Again, more challenging tasks would help prove the concept. It may be that an iterative approach which refines the flow on relevant portions of the state space will be necessary. - There does not appear to be any discussion of how the discrete actions in the BSS environment are handled. My guess is that the flow is trained on the integer values, but the output will still be continuous. We can then just round the output, but this could result in constraint violations. A better discussion of how this is handled in the main paper would help. - There is a lengthly discussion of gradient computation for the normalizing flow, but this is usually just handled by the deep learning framework. It seems unnecessary to get into these details unless the form of these gradients is leveraged to improve the speed of training. If auto-differentiation is still used as normal, then this feels a bit like filler and could be replaced with more relevant details about training, the tasks, or results. - The motivation for using a normalizing flow, rather than other generative models, seems missing in the paper. The authors do mention something, but it felt a bit hand-wavy. If we were optimizing a stochastic policy with an on-policy algorithm, such as PPO, having a tractable likelihood would be really important. This would be a great motivation for using a normalizing flow. But since we are using a deterministic policy and training with DDPG, it seems more arbitrary.

Questions

- How do you think this approach would scale to more complex, state-dependent constraints? Would it be too difficult to generate samples which cover the state space sufficiently? And if so, would an iterative approach which refines the flow on states encountered under the current policy be feasible? - How are the discrete actions in the BSS environment handled? If the output of the flow is rounded, does this lead to more constraint violations? - Is the form of the normalizing flow leveraged to more efficiently compute gradients while training? Or are we using standard auto-differentiation? - What is the rationale for using a normalizing flow over other generative models in the deterministic policy case? - How do you think this approach would work for an on-policy algorithm, such as PPO, which requires the log-likelihood? - How was the "hard wall" for handling constraints implemented with HMC, which needs a differentiable log-likelihood? Were they just barrier functions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss how their approach can still lead to constraint violations, requiring the use of an optimization solver in this case. However, they show that the probability of constraint violation is significantly reduced. A possible limitation not really discussed is scalability due to the need for sample generation to train the flow. It may be hard to scale this approach to higher-dimensional action spaces and constraints which heavily depend on the state. The Half Cheetah task definitely shows it can work, so at the very least, this is a good preliminary study to illustrate the potential of the approach.

Summary

This paper solves the problem of action-constraint reinforcement learning. The author utilizes the Flow model to learn a projection from the action to the latent variable, and then integrate DDPG to construct the FlowPG framework. Empirically FlowPG outperforms its competitors in both fewer constraint violations and faster in elapsed time.

Strengths

1. The manuscript is clearly structured and well presented, and it is easy to read. 2. It seems interesting to apply the generative model in constraints optimization problem. Introducing the flow model is novel in action-constraint scenarios, and it effectively avoids solving a QP problem after the policy network. 3. The authors utilizes HMC and PSDD in sampling from the valid actions set.

Weaknesses

The action-constraint problems is actually somehow similar with constraint-RL problem, and there are some important works such as [1,2] and other related works. It is suggested that the authors should provide a discussion about whether other constraint RL methods could be applied in the action-constraint scenarios, and provide some experimental result if possible. [1] Constrained Policy Optimization https://arxiv.org/pdf/1705.10528.pdf [2] Safety-Constrained Reinforcement Learning for MDPs. https://arxiv.org/abs/1510.05880 Another issue is that the action-constraint RL problem could be seen as offline RL problem (invalid actions space could be OOD actions). There are various offline RL methods appearing [3,4] in recent years ([5] also applies the flow model), and the authors are suggested to consider the offline RL methods in this manuscript. [3] Off-Policy Deep Reinforcement Learning without Exploration. https://arxiv.org/abs/1812.02900 [4] Stabilizing off- policy q-learning via bootstrapping error reduction. https://arxiv.org/abs/1906.00949 [5] APAC: Authorized Probability-controlled Actor-Critic For Offline Reinforcement Learning. https://arxiv.org/abs/2301.12130

Questions

1. There are some confusions in generating valid action space, even with HMC and PSDD, could it be assured that the $\tilde{a}$ could be transformed to a valid action after the flow model projection in Figure 2(a). In addition, as we want to maximize the cumulative reward in RL problem, should the input of the flow model be ($\tilde{a}, s, s'$) where $s'$ is the next state. It is also noted when $\tilde{a}$ becomes a valid action $a$, the next state $s'$ may also changes 2. The author claims that flow model is more efficient than VAE and GAN for data generation. Could the author provide some ablation study on this issue, especially for generating valid action samples. 3. The experiments seem not convincing enough. The authors only considers Half-Cheetah and Bike Sharing System. Actually Half-Cheetah belongs to D4RL tasks, and there are many other tasks inside (some could be constraint RL tasks), it is suggested to consider more dataset for comparisons. 4. In addition, the MCMC style methods are often time-consuming, how is the entire training time, compared with other competitors?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see weakness and questions

Summary

This paper provide a new method for ACRL, which incorporate normalized flow methods to alleviate the action violation problem. It achieves better results on MuJuCo compared with other methods.

Strengths

- Introduce normalized flow into action control -- which maps the original hard-to-control action space into another easy-to-control space. - Design a HMC-PSDD framework to efficiently train the flow model. - Find an appropriate prior distribution for the flow model.

Weaknesses

Main issue: - An ablation study for PSDD may be necessary -- as we don't know the quality of generated valid actions. For example, a possible baseline can be: interact with the environments many times to get the possible valid actions. Minor issue: - DDPG is relatively outdated. More recent SoTA methods should be added into the experiments.

Questions

I am not an expert in ACRL so I wonder if there are other ACRL benchmarks? Only 3 environments seem not enough.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No obvious negative societal impacts.

Summary

The paper introduces a novel action-constrained reinforcement learning (ACRL) algorithm called FlowPG, which utilizes a normalizing flow model to generate actions within the feasible action region. Experimental results demonstrate that FlowPG effectively handles action constraints and outperforms two existing ACRL algorithms by reducing the number of constraint violations.

Strengths

The paper is well-written and provides good motivation.

Weaknesses

The paper lacks the ablation study to validate importance of the proposed HMC and PSDD. The advantages of training speeds are not obvious.

Questions

Q1. The motivation behind the use of Hamiltonian Monte Carlo (HMC) and probabilistic sentential decision diagrams (PSDD) is not sufficiently clear. Additionally, an ablation study is necessary to validate the importance of these proposed techniques. Q2. The paper claims that the proposed method achieves faster training speeds compared to the baselines. However, Figure 4(a) shows no clear advantages in terms of convergence speed for FlowPG compared to the two baselines. This discrepancy needs to be addressed and clarified. Q3. The paper would benefit from including more comparisons between the proposed method and recent ACRL algorithms. By providing such comparisons, the authors can further highlight the strengths and weaknesses of their approach and provide a more comprehensive evaluation.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

see Questions

Thank you for your rebuttal

Thank you for your detailed reply. Your response has addressed some of my concerns, and I do appreciate your work. **Other generative models such as VAE and GAN** The section 1 in common response is our strong enough to convince me, and comparisons between flow model, wgan and vae are somehow weak. More empirical studies are suggested. **Entire training time when considering HMC data generation** Thank you for supplemental studies. I am OK with about the training time study.

Response

Thanks for the additional experiments and explanation, that address my concerns.