Effective Experiences Collection and State Aggregation in Reinforcement Learning

Abstract

In reinforcement learning systems, learning agents cluster a large number of experiences by identifying similarities in terms of domain knowledge and replace the groups with a representative prototype. Our method addresses two key challenges in reinforcement learning: the difficulty of transferring continuous state domain into the discrete state space, and the need for a good compromise between exploration and exploitation. To tackle the former challenge, the adaptive state aggregation algorithm with a decision tree helps the agent to learn the optimal policy in continuous state space. For the latter challenge, this paper proposes an adaptive state aggregation algorithm, which uses information entropy to evaluate the probability of state-action-dependent exploration and the value for $\varepsilon $ is determined using the information entropy instead of manual tuning. Meanwhile, a Tabu search is used to lift the efficiency of exploration. For the planning algorithm, the time required for global exploration depends only on the metric resolution, and not on the size of the state space. The simulation experiments demonstrate the effectiveness of the proposed method, which adaptively partitions the state space into exclusive subspaces and, meanwhile, obtains a good compromise between exploration and exploitation.