Optimizing Latency for Online Video Captioning Using Audio-Visual Transformers

Video captioning is an essential technology to understand scenes and describe events in natural language.To apply it to real-time monitoring, a system needs not only to describe events accurately but also to produce the captions as soon as possible.Low-latency captioning is needed to realize such functionality, but this research area for online video captioning has not been pursued yet.This paper proposes a novel approach to optimize each caption's output timing based on a trade-off between latency and caption quality.An audio-visual Transformer is trained to generate ground-truth captions using only a small portion of all video frames, and to mimic outputs of a pre-trained Transformer to which all the frames are given.A CNN-based timing detector is also trained to detect a proper output timing, where the captions generated by the two Transformers become sufficiently close to each other.With the jointly trained Transformer and timing detector, a caption can be generated in the early stages of an event-triggered video clip, as soon as an event happens or when it can be forecasted.Experiments with the ActivityNet Captions dataset show that our approach achieves 94% of the caption quality of the upper bound given by the pre-trained Transformer using the entire video clips, using only 28% of frames from the beginning.