Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

Current work in explainable reinforcement learning generally produces\npolicies in the form of a decision tree over the state space. Such policies can\nbe used for formal safety verification, agent behavior prediction, and manual\ninspection of important features. However, existing approaches fit a decision\ntree after training or use a custom learning procedure which is not compatible\nwith new learning techniques, such as those which use neural networks. To\naddress this limitation, we propose a novel Markov Decision Process (MDP) type\nfor learning decision tree policies: Iterative Bounding MDPs (IBMDPs). An IBMDP\nis constructed around a base MDP so each IBMDP policy is guaranteed to\ncorrespond to a decision tree policy for the base MDP when using a\nmethod-agnostic masking procedure. Because of this decision tree equivalence,\nany function approximator can be used during training, including a neural\nnetwork, while yielding a decision tree policy for the base MDP. We present the\nrequired masking procedure as well as a modified value update step which allows\nIBMDPs to be solved using existing algorithms. We apply this procedure to\nproduce IBMDP variants of recent reinforcement learning methods. We empirically\nshow the benefits of our approach by solving IBMDPs to produce decision tree\npolicies for the base MDPs.\n