Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning

Accurately predicting the dynamics of robotic systems is crucial for\nmodel-based control and reinforcement learning. The most common way to estimate\ndynamics is by fitting a one-step ahead prediction model and using it to\nrecursively propagate the predicted state distribution over long horizons.\nUnfortunately, this approach is known to compound even small prediction errors,\nmaking long-term predictions inaccurate. In this paper, we propose a new\nparametrization to supervised learning on state-action data to stably predict\nat longer horizons -- that we call a trajectory-based model. This\ntrajectory-based model takes an initial state, a future time index, and control\nparameters as inputs, and directly predicts the state at the future time index.\nExperimental results in simulated and real-world robotic tasks show that\ntrajectory-based models yield significantly more accurate long term\npredictions, improved sample efficiency, and the ability to predict task\nreward. With these improved prediction properties, we conclude with a\ndemonstration of methods for using the trajectory-based model for control.\n