Straight to the Point: Fast-forwarding Videos via Reinforcement Learning Using Textual Data

The rapid increase in the amount of published visual data and the limited\ntime of users bring the demand for processing untrimmed videos to produce\nshorter versions that convey the same information. Despite the remarkable\nprogress that has been made by summarization methods, most of them can only\nselect a few frames or skims, which creates visual gaps and breaks the video\ncontext. In this paper, we present a novel methodology based on a reinforcement\nlearning formulation to accelerate instructional videos. Our approach can\nadaptively select frames that are not relevant to convey the information\nwithout creating gaps in the final video. Our agent is textually and visually\noriented to select which frames to remove to shrink the input video.\nAdditionally, we propose a novel network, called Visually-guided Document\nAttention Network (VDAN), able to generate a highly discriminative embedding\nspace to represent both textual and visual data. Our experiments show that our\nmethod achieves the best performance in terms of F1 Score and coverage at the\nvideo segment level.\n