PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

We propose PIGLeT: a model that learns physical commonsense knowledge through\ninteraction, and then uses this knowledge to ground language. We factorize\nPIGLeT into a physical dynamics model, and a separate language model. Our\ndynamics model learns not just what objects are but also what they do: glass\ncups break when thrown, plastic ones don't. We then use it as the interface to\nour language model, giving us a unified model of linguistic form and grounded\nmeaning. PIGLeT can read a sentence, simulate neurally what might happen next,\nand then communicate that result through a literal symbolic representation, or\nnatural language.\n Experimental results show that our model effectively learns world dynamics,\nalong with how to communicate them. It is able to correctly forecast "what\nhappens next" given an English sentence over 80% of the time, outperforming a\n100x larger, text-to-text approach by over 10%. Likewise, its natural language\nsummaries of physical interactions are also judged by humans as more accurate\nthan LM alternatives. We present comprehensive analysis showing room for future\nwork.\n