Learning to Sample with Local and Global Contexts in Experience Replay Buffer

Experience replay, which enables the agents to remember and reuse experience\nfrom the past, has played a significant role in the success of off-policy\nreinforcement learning (RL). To utilize the experience replay efficiently, the\nexisting sampling methods allow selecting out more meaningful experiences by\nimposing priorities on them based on certain metrics (e.g. TD-error). However,\nthey may result in sampling highly biased, redundant transitions since they\ncompute the sampling rate for each transition independently, without\nconsideration of its importance in relation to other transitions. In this\npaper, we aim to address the issue by proposing a new learning-based sampling\nmethod that can compute the relative importance of transition. To this end, we\ndesign a novel permutation-equivariant neural architecture that takes contexts\nfrom not only features of each transition (local) but also those of others\n(global) as inputs. We validate our framework, which we refer to as Neural\nExperience Replay Sampler (NERS), on multiple benchmark tasks for both\ncontinuous and discrete control tasks and show that it can significantly\nimprove the performance of various off-policy RL methods. Further analysis\nconfirms that the improvements of the sample efficiency indeed are due to\nsampling diverse and meaningful transitions by NERS that considers both local\nand global contexts.\n