Augmenting reinforcement learning with imitation learning is often hailed as\na method by which to improve upon learning from scratch. However, most existing\nmethods for integrating these two techniques are subject to several strong\nassumptions---chief among them that information about demonstrator actions is\navailable. In this paper, we investigate the extent to which this assumption is\nnecessary by introducing and evaluating reinforced inverse dynamics modeling\n(RIDM), a novel paradigm for combining imitation from observation (IfO) and\nreinforcement learning with no dependence on demonstrator action information.\nMoreover, RIDM requires only a single demonstration trajectory and is able to\noperate directly on raw (unaugmented) state features. We find experimentally\nthat RIDM performs favorably compared to a baseline approach for several tasks\nin simulation as well as for tasks on a real UR5 robot arm. Experiment videos\ncan be found at https://sites.google.com/view/ridm-reinforced-inverse-dynami.\n