Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration

A promising technique for exploration is to maximize the entropy of visited state distribution, i.e., state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle in a supervised setup with a task reward, where an agent prefers to visit high-value states to exploit the task reward. Such a preference can cause an imbalance between the distributions of high-value states and low-value states, which biases exploration towards low-value state regions as a result of the state entropy increasing when the distribution becomes more uniform. This issue is exacerbated when high-value states are narrowly distributed within the state space, making it difficult for the agent to complete the tasks. In this paper, we present a novel exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, our method prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. We demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks. Source code is available at https://sites.google.com/view/rl-vcse.

Paper

References (63)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer FYPf7/10 · confidence 4/52023-06-27

Summary

This paper addresses the problem of sample-efficient exploration in sparse-rewards deep reinforcement learning. It builds on previous literature on the maximization of the state entropy as an exploration objective, which was mostly used in reward-free settings, proposing a value-conditioned state entropy objective along with the task reward. The latter pushes the agent to maximize the state entropy while penalizing the entropy over the estimated values of the visited states. This value-conditioned state entropy is implemented through an off-the-shelf non-parametric conditional entropy estimator. Finally, the paper compares the performance of value-conditioned entropy maximization against standard entropy maximization in MiniGrid, DeepMind Control, and Meta-World.

Strengths

- The paper proposes an intrinsic reward can be easily incorporated in any previous method; - The experiments show that the value-conditional objective brings improved/matching performance in a variety of domains; - The paper clearly presents ideas and contribution, and also include some compelling visualization of the conditional entropy estimation.

Weaknesses

- The intrinsic reward computation requires robust estimates of the value function, which is a challenge per se, especially in sparse-rewards domains; - As mentioned by the authors in their conclusion, the theoretical ground for value-conditional state entropy is unclear.

Questions

This paper advocates for using state entropy bonuses to accelerate RL in sparse-rewards tasks, instead of training an entropy maximizing policy in reward-free settings, which is the most common mode of previous works. To adapt state entropy bonuses to the demand of the sparse-reward task, which requires to deal with the exploration-exploitation trade-off, the paper proposes to condition the entropy of the visited states with the value of those states, effectively driving exploration towards rewarding regions of the state space. This interesting notion leads to a methodology that can be easily incorporated in previous algorithms while benefitting the resulting sample efficiency in a variety of domains. However, the paper does not provide a formal theoretical understanding of the proposed approach, which leaves some doubts on the generality of their results. Anyway, I am currently providing a slightly positive score while encouraging the authors to improve the theoretical ground for value-conditioned entropy bonuses, and especially how the estimation error of the value function impacts the learning process. **(Clarification on the objective)** From my understanding of Algorithm 1, the proposed method maximizes the value-conditional state entropy over the samples taken from the replay buffer, and not just the samples drawn with the latter policy. While this is common in previous works as well (e.g., Liu and Abbeel, 2021), I am wondering whether looking at the value-conditional entropy of the latest samples is more appropriate in a setting where external reward is also present. Can the authors discuss their implementation choice and comment pros and cons of the two alternatives? **(Sensitivity to $\beta$ also for VCSE)** The experiments report an interesting analysis on how the value of $\beta$ affects the performance of A2C+SE. It would be interesting to inspect also the sensitivity of VCSE to the corresponding $\beta$ value. **(Comparison with CEM)** A recent work (Yang and Spaan, CEM: Constrained entropy maximization for task-agnostic safe exploration, 2023) considers state entropy maximization under cost constraints. I am wondering whether their approach could be also used to address the problem of reducing task-irrelevant exploration, e.g., by introducing appropriate value constraints. Can the authors discuss the comparison between their solution and CEM, possibly highlighting pros and cons of the two?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Some limitations of the paper and the proposed methodology are reported in the first paragraph of Section 6. I believe that a formal discussion on how the value estimation error affects their method would be extremely valuable.

Reviewer pL2M6/10 · confidence 4/52023-07-05

Summary

This paper present a exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, it prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. The experiments demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks.

Strengths

(1) The exploration method proposed in this paper show that maximum value-conditional state entropy (VCSE) exploration successfully accelerates the training of RL algorithms. It can be used for reference in other RL algorithms. (2) For section 3, the development process of the entropy estimator is very clearly described. (3) For section 5, extensive experiments have been conducted to demonstrate the effectiveness of the algorithm proposed in this paper.

Weaknesses

(1) The writing of this article seems difficult to understand in some aspects,especially for certain formulas. (2) The motivation of this article is not fully explained. (3) To be honest,it lacks a certain degree of novelty. As is well known, the methods used in this paper like “k-nearest neighbor entropy estimator” and “KSG conditional entropy estimator” are all previous work. We just utilize some of them. (4) The main body of this paper is clear to understand, here is space for improvement. I defer some of my issues in the appendix to "Questions".

Questions

(1) In line 117, what is the meaning of z and z’? (2) In Equ.(1),what is the meaning of dx ? It seems unexplained. (3) In section 5, the selected experimental environment is relatively simple with a low dimension. Can we select some complex environments with a high-dimension state and action spaces for verification? (4) In Fig1, how to do “State Norm” and “Value Norm”? (5) In Fig1, why do you only choose the third one(kNN (k=3))instead of the others?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

In section 6, the authors talk about some limitation and future directions about this paper. At present, the potential negative social impact of this work has not been identified.

Reviewer MwTR7/10 · confidence 4/52023-07-07

Summary

The paper proposes an improvement over a popular intrinsic reward based on state entropy. Instead of encouraging large state entropy uniformly over all states which may deviate the policy toward failure states, the proposed intrinsic reward motivates the agent to maximize the value-conditioned state entropy. The value-conditioned state entropy is estimated with a classical kNN method.

Strengths

The intrinsic reward is an important direction of improving RL exploration and sample efficiency, and how to balance between instrinsic and extrinsic reward is a challenge for intrinsic reward design. This paper proposes a novel method to achieve this goal --- conditioning the state entropy on the state value to partition the state space and focus on state entropy over states with similar value. The paper is well written and the proposed method is strongly supported by a large range of experiments.

Weaknesses

There is no significant weakness that I notice except for an ablation study on the effect of k for the kNN.

Questions

Besides the effect of k, I wonder how if authors have tried other entropy/mutual information estimator (for example, the one based on contrastive learning) and why ending up selecting the kNN one.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

kNN method for entropy estimation usually suffers in high-dimensional space, but the proposed method works well in environments with visual observations. So I don't find any significantly limitation.

Reviewer CsYB4/10 · confidence 3/52023-07-17

Summary

The paper investigates an exploration technique that maximizes the entropy of visited states while taking into account the expected return of the states. The goal is to explore the part of the state space that is both less visited while avoiding too much exploration for the low-value states.

Strengths

The paper is overall well written and presents an interesting approach to taking into account an estimate of the expected return together with state entropy exploration.

Weaknesses

My main concern is that it is a bit unclear how the presented algorithm actually enforces more exploration on the high expected return states as compared to previous approaches. In Figure 8, I'm unsure how VCSE will have a different behaviour than SE given that the agent needs to at least visit once the reward to be able to see that there is a lower bonus by keeping exploring the low expected return states. In addition, previous approaches that use a combination of intrinsic and extrinsic rewards should also tend to visit more the high expected return parts of the state space. Additional minor comments: - I'm unsure why "supervised setup" is mentioned in line 4 (abstract) and line 29. - line 91: $\gamma$ can't be 1. - Typos: line 105

Questions

Can you clarify my main concern (see above). If so, I would happily increase my score.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors adequately address limitations.

Authorsrebuttal2023-08-14

Dear reviewer MwTR, Thank you for your reply! We're excited to hear that our response successfully answered your questions. We'll make sure that our rebuttal response is fully incorporated into the final draft. Thank you very much, Authors

Reviewer CsYB2023-08-16

The answer didn't clarify my doubts

In Fig 8, the sub figures a) and b) seem to have only one reward at the bottom right. It is still unclear to me why VCSE works better in this case. For the minor comment, $\gamma$ can't be one if the horizon is infinite (which seems to be the case here). The expected return would be unbounded.

Authorsrebuttal2023-08-17

Further Clarification by Authors

Dear Reviewer CsYB, we respond to each of your comments one-by-one in what follows. A1. Further explanation on how VCSE can work better than SE in Figure 8. First of all, we would like to clarify two things. In Figure 8, RL agents should still do additional exploration for exploiting the task reward after encountering the first reward, because the observation is partially observable and thus there could be states with higher value estimates. Moreover, both SE and VCSE obtain rewards at the similar timestep while it is not that visible in the heatmap from Figure 8. Now we will provide a more detailed explanation of how VCSE can work better than SE in this setup. Because the RL agent tries to exploit the task reward, the agent initially begins to visit more states around the high-value states. However, because SE aims to encourage uniform coverage by visiting both low-value and high-value states similarly, the intrinsic reward for low-value states begins to increase. Then the agent starts to do excessive exploration around low-value states, instead of further exploring states around the rewarding state at the bottom right state in the map. This makes it difficult for the A2C+SE agent to quickly learn to solve the target task. Moreover, we further showed that this issue is not easily addressed by adjusting the scale of intrinsic rewards in Figure 4. On the other hand, VCSE can avoid this issue because it does not consider states with different values when calculating intrinsic rewards. This means that VCSE can still encourage exploration around high-value states, which allows the agent to visit states with higher values and quickly learns to solve the target task. We hope that this explanation answers your question. Please let us know if there still is anything not clear, we will try to further clarify this in the final draft. — A2. Discount factor Thank you for pointing this out. We will incorporate your comment in the final draft.

Reviewer pL2M2023-08-18

In this round of feedback, the author provided detailed explanations and modifications to the questions I raised. Especially the description of novelty and innovation can enhance the overall understanding of the article, and it is recommended to supplement it in the relevant parts of the article. In terms of expanding the experiment, in order to dispel my doubts, the author added relevant experiments to support it, which makes the article more competitive. Finally, I suggest that the author further sort out the entire article to make its context clear. Therefore, I agree to increase the article by 1 points.

Authorsrebuttal2023-08-21

Dear reviewer pL2M, Thank you for your reply! We'll make sure that our additional clarification, experiments, and editorial comments are fully incorporated into the final draft. Thank you very much, Authors

Reviewer FYPf2023-08-18

After response

I want to thank the authors for their detailed response, which is properly addressing my concerns. I am updating my evaluation upwards, and I really encourage the authors to include some bits of the provided clarifications and additional sensitivity analysis to a final draft of the paper. As for how to evaluate the impact of approximate value functions. I think the best way to do so would be through a theoretical analysis that shows how the value function error propagates to the objective function. Of course, this might be non-trivial. In terms of experiments, it would be great to compare the performance of VCSE with a learned value function against "fixed" value approximators, which I guess would derail learning completely when the quality of the approximation is not good enough. Also, an ablation study on timescale separation between value learning and policy learning would be interesting.

Authorsrebuttal2023-08-21

Dear reviewer FYPf, Thank you for your reply! We will make sure that our clarifications and additional experiments are fully incorporated into the final draft. Moreover, we would really appreciate your suggestions on additional experiments with fixed value functions and separated actor-critic learning phases. We will try our best to design a more concrete experimental setup and include the results in the final draft. Thank you very much, Authors

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC