DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization

The recent growth of web video sharing platforms has increased the demand for\nsystems that can efficiently browse, retrieve and summarize video content.\nQuery-aware multi-video summarization is a promising technique that caters to\nthis demand. In this work, we introduce a novel Query-Aware Hierarchical\nPointer Network for Multi-Video Summarization, termed DeepQAMVS, that jointly\noptimizes multiple criteria: (1) conciseness, (2) representativeness of\nimportant query-relevant events and (3) chronological soundness. We design a\nhierarchical attention model that factorizes over three distributions, each\ncollecting evidence from a different modality, followed by a pointer network\nthat selects frames to include in the summary. DeepQAMVS is trained with\nreinforcement learning, incorporating rewards that capture representativeness,\ndiversity, query-adaptability and temporal coherence. We achieve\nstate-of-the-art results on the MVS1K dataset, with inference time scaling\nlinearly with the number of input video frames.\n