NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spaces

Learning low-dimensional latent state space dynamics models has been a\npowerful paradigm for enabling vision-based planning and learning for control.\nWe introduce a latent dynamics learning framework that is uniquely designed to\ninduce proportional controlability in the latent space, thus enabling the use\nof much simpler controllers than prior work. We show that our learned dynamics\nmodel enables proportional control from pixels, dramatically simplifies and\naccelerates behavioural cloning of vision-based controllers, and provides\ninterpretable goal discovery when applied to imitation learning of switching\ncontrollers from demonstration.\n