When automatically generating a sentence description for an image or video,\nit often remains unclear how well the generated caption is grounded, that is\nwhether the model uses the correct image regions to output particular words, or\nif the model is hallucinating based on priors in the dataset and/or the\nlanguage model. The most common way of relating image regions with words in\ncaption models is through an attention mechanism over the regions that are used\nas input to predict the next word. The model must therefore learn to predict\nthe attentional weights without knowing the word it should localize. This is\ndifficult to train without grounding supervision since recurrent models can\npropagate past information and there is no explicit signal to force the\ncaptioning model to properly ground the individual decoded words. In this work,\nwe help the model to achieve this via a novel cyclical training regimen that\nforces the model to localize each word in the image after the sentence decoder\ngenerates it, and then reconstruct the sentence from the localized image\nregion(s) to match the ground-truth. Our proposed framework only requires\nlearning one extra fully-connected layer (the localizer), a layer that can be\nremoved at test time. We show that our model significantly improves grounding\naccuracy without relying on grounding supervision or introducing extra\ncomputation during inference, for both image and video captioning tasks. Code\nis available at https://github.com/chihyaoma/cyclical-visual-captioning .\n