Since visual perception can give rich information beyond text descriptions\nfor world understanding, there has been increasing interest in leveraging\nvisual grounding for language learning. Recently, vokenization (Tan and Bansal,\n2020) has attracted attention by using the predictions of a text-to-image\nretrieval model as labels for language model supervision. Despite its success,\nthe method suffers from approximation error of using finite image labels and\nthe lack of vocabulary diversity of a small image-text dataset. To overcome\nthese limitations, we present VidLanKD, a video-language knowledge distillation\nmethod for improving language understanding. We train a multi-modal teacher\nmodel on a video-text dataset, and then transfer its knowledge to a student\nlanguage model with a text dataset. To avoid approximation error, we propose to\nuse different knowledge distillation objectives. In addition, the use of a\nlarge-scale video-text dataset helps learn diverse and richer vocabularies. In\nour experiments, VidLanKD achieves consistent improvements over text-only\nlanguage models and vokenization models, on several downstream language\nunderstanding tasks including GLUE, SQuAD, and SWAG. We also demonstrate the\nimproved world knowledge, physical reasoning, and temporal reasoning\ncapabilities of our model by evaluating on the GLUE-diagnostics, PIQA, and\nTRACIE datasets. Lastly, we present comprehensive ablation studies as well as\nvisualizations of the learned text-to-video grounding results of our teacher\nand student language models. Our code and models are available at:\nhttps://github.com/zinengtang/VidLanKD\n