We study the problem of dynamic visual reasoning on raw videos. This is a\nchallenging problem; currently, state-of-the-art models often require dense\nsupervision on physical object properties and events from simulation, which are\nimpractical to obtain in real life. In this paper, we present the Dynamic\nConcept Learner (DCL), a unified framework that grounds physical objects and\nevents from video and language. DCL first adopts a trajectory extractor to\ntrack each object over time and to represent it as a latent, object-centric\nfeature vector. Building upon this object-centric representation, DCL learns to\napproximate the dynamic interaction among objects using graph networks. DCL\nfurther incorporates a semantic parser to parse questions into semantic\nprograms and, finally, a program executor to run the program to answer the\nquestion, levering the learned dynamics model. After training, DCL can detect\nand associate objects across the frames, ground visual properties, and physical\nevents, understand the causal relationship between events, make future and\ncounterfactual predictions, and leverage these extracted presentations for\nanswering queries. DCL achieves state-of-the-art performance on CLEVRER, a\nchallenging causal video reasoning dataset, even without using ground-truth\nattributes and collision labels from simulations for training. We further test\nDCL on a newly proposed video-retrieval and event localization dataset derived\nfrom CLEVRER, showing its strong generalization capacity.\n