Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions

Off-policy evaluation in reinforcement learning offers the chance of using\nobservational data to improve future outcomes in domains such as healthcare and\neducation, but safe deployment in high stakes settings requires ways of\nassessing its validity. Traditional measures such as confidence intervals may\nbe insufficient due to noise, limited data and confounding. In this paper we\ndevelop a method that could serve as a hybrid human-AI system, to enable human\nexperts to analyze the validity of policy evaluation estimates. This is\naccomplished by highlighting observations in the data whose removal will have a\nlarge effect on the OPE estimate, and formulating a set of rules for choosing\nwhich ones to present to domain experts for validation. We develop methods to\ncompute exactly the influence functions for fitted Q-evaluation with two\ndifferent function classes: kernel-based and linear least squares, as well as\nimportance sampling methods. Experiments on medical simulations and real-world\nintensive care unit data demonstrate that our method can be used to identify\nlimitations in the evaluation process and make evaluation more robust.\n