We address the challenging problem of Natural Language Comprehension beyond\nplain-text documents by introducing the TILT neural network architecture which\nsimultaneously learns layout information, visual features, and textual\nsemantics. Contrary to previous approaches, we rely on a decoder capable of\nunifying a variety of problems involving natural language. The layout is\nrepresented as an attention bias and complemented with contextualized visual\ninformation, while the core of our model is a pretrained encoder-decoder\nTransformer. Our novel approach achieves state-of-the-art results in extracting\ninformation from documents and answering questions which demand layout\nunderstanding (DocVQA, CORD, SROIE). At the same time, we simplify the process\nby employing an end-to-end model.\n