Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model

In high-stake scenarios like medical treatment and auto-piloting, it's risky\nor even infeasible to collect online experimental data to train the agent.\nSimulation-based training can alleviate this issue, but may suffer from its\ninherent mismatches from the simulator and real environment. It is therefore\nimperative to utilize the simulator to learn a robust policy for the real-world\ndeployment. In this work, we consider policy learning for Robust Markov\nDecision Processes (RMDP), where the agent tries to seek a robust policy with\nrespect to unexpected perturbations on the environments. Specifically, we focus\non the setting where the training environment can be characterized as a\ngenerative model and a constrained perturbation can be added to the model\nduring testing. Our goal is to identify a near-optimal robust policy for the\nperturbed testing environment, which introduces additional technical\ndifficulties as we need to simultaneously estimate the training environment\nuncertainty from samples and find the worst-case perturbation for testing. To\nsolve this issue, we propose a generic method which formalizes the perturbation\nas an opponent to obtain a two-player zero-sum game, and further show that the\nNash Equilibrium corresponds to the robust policy. We prove that, with a\npolynomial number of samples from the generative model, our algorithm can find\na near-optimal robust policy with a high probability. Our method is able to\ndeal with general perturbations under some mild assumptions and can also be\nextended to more complex problems like robust partial observable Markov\ndecision process, thanks to the game-theoretical formulation.\n