Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning

Recent approaches in literature have exploited the multi-modal information in\ndocuments (text, layout, image) to serve specific downstream document tasks.\nHowever, they are limited by their - (i) inability to learn cross-modal\nrepresentations across text, layout and image dimensions for documents and (ii)\ninability to process multi-page documents. Pre-training techniques have been\nshown in Natural Language Processing (NLP) domain to learn generic textual\nrepresentations from large unlabelled datasets, applicable to various\ndownstream NLP tasks. In this paper, we propose a multi-task learning-based\nframework that utilizes a combination of self-supervised and supervised\npre-training tasks to learn a generic document representation applicable to\nvarious downstream document tasks. Specifically, we introduce Document Topic\nModelling and Document Shuffle Prediction as novel pre-training tasks to learn\nrich image representations along with the text and layout representations for\ndocuments. We utilize the Longformer network architecture as the backbone to\nencode the multi-modal information from multi-page documents in an end-to-end\nfashion. We showcase the applicability of our pre-training framework on a\nvariety of different real-world document tasks such as document classification,\ndocument information extraction, and document retrieval. We evaluate our\nframework on different standard document datasets and conduct exhaustive\nexperiments to compare performance against various ablations of our framework\nand state-of-the-art baselines.\n