See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization

In recent years, abstractive text summarization with multimodal inputs has\nstarted drawing attention due to its ability to accumulate information from\ndifferent source modalities and generate a fluent textual summary. However,\nexisting methods use short videos as the visual modality and short summary as\nthe ground-truth, therefore, perform poorly on lengthy videos and long\nground-truth summary. Additionally, there exists no benchmark dataset to\ngeneralize this task on videos of varying lengths. In this paper, we introduce\nAVIATE, the first large-scale dataset for abstractive text summarization with\nvideos of diverse duration, compiled from presentations in well-known academic\nconferences like NDSS, ICML, NeurIPS, etc. We use the abstract of corresponding\nresearch papers as the reference summaries, which ensure adequate quality and\nuniformity of the ground-truth. We then propose FLORAL, a factorized\nmulti-modal Transformer based decoder-only language model, which inherently\ncaptures the intra-modal and inter-modal dynamics within various input\nmodalities for the text summarization task. FLORAL utilizes an increasing\nnumber of self-attentions to capture multimodality and performs significantly\nbetter than traditional encoder-decoder based networks. Extensive experiments\nillustrate that FLORAL achieves significant improvement over the baselines in\nboth qualitative and quantitative evaluations on the existing How2 dataset for\nshort videos and newly introduced AVIATE dataset for videos with diverse\nduration, beating the best baseline on the two datasets by $1.39$ and $2.74$\nROUGE-L points respectively.\n