Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

A limitation of model-based reinforcement learning (MBRL) is the exploitation\nof errors in the learned models. Black-box models can fit complex dynamics with\nhigh fidelity, but their behavior is undefined outside of the data\ndistribution.Physics-based models are better at extrapolating, due to the\ngeneral validity of their informed structure, but underfit in the real world\ndue to the presence of unmodeled phenomena. In this work, we demonstrate\nexperimentally that for the offline model-based reinforcement learning setting,\nphysics-based models can be beneficial compared to high-capacity function\napproximators if the mechanical structure is known. Physics-based models can\nlearn to perform the ball in a cup (BiC) task on a physical manipulator using\nonly 4 minutes of sampled data using offline MBRL. We find that black-box\nmodels consistently produce unviable policies for BiC as all predicted\ntrajectories diverge to physically impossible state, despite having access to\nmore data than the physics-based model. In addition, we generalize the approach\nof physics parameter identification from modeling holonomic multi-body systems\nto systems with nonholonomic dynamics using end-to-end automatic\ndifferentiation.\n Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/\n