A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation

Marginalized importance sampling (MIS), which measures the density ratio\nbetween the state-action occupancy of a target policy and that of a sampling\ndistribution, is a promising approach for off-policy evaluation. However,\ncurrent state-of-the-art MIS methods rely on complex optimization tricks and\nsucceed mostly on simple toy problems. We bridge the gap between MIS and deep\nreinforcement learning by observing that the density ratio can be computed from\nthe successor representation of the target policy. The successor representation\ncan be trained through deep reinforcement learning methodology and decouples\nthe reward optimization from the dynamics of the environment, making the\nresulting algorithm stable and applicable to high-dimensional domains. We\nevaluate the empirical performance of our approach on a variety of challenging\nAtari and MuJoCo environments.\n