Re-understanding Finite-State Representations of Recurrent Policy Networks

We introduce an approach for understanding control policies represented as\nrecurrent neural networks. Recent work has approached this problem by\ntransforming such recurrent policy networks into finite-state machines (FSM)\nand then analyzing the equivalent minimized FSM. While this led to interesting\ninsights, the minimization process can obscure a deeper understanding of a\nmachine's operation by merging states that are semantically distinct. To\naddress this issue, we introduce an analysis approach that starts with an\nunminimized FSM and applies more-interpretable reductions that preserve the key\ndecision points of the policy. We also contribute an attention tool to attain a\ndeeper understanding of the role of observations in the decisions. Our case\nstudies on 7 Atari games and 3 control benchmarks demonstrate that the approach\ncan reveal insights that have not been previously noticed.\n