Information extraction (IE) for visually-rich documents (VRDs) has achieved\nSOTA performance recently thanks to the adaptation of Transformer-based\nlanguage models, which shows the great potential of pre-training methods. In\nthis paper, we present a new approach to improve the capability of language\nmodel pre-training on VRDs. Firstly, we introduce a new query-based IE model\nthat employs span extraction instead of using the common sequence labeling\napproach. Secondly, to further extend the span extraction formulation, we\npropose a new training task that focuses on modelling the relationships among\nsemantic entities within a document. This task enables target spans to be\nextracted recursively and can be used to pre-train the model or as an IE\ndownstream task. Evaluation on three datasets of popular business documents\n(invoices, receipts) shows that our proposed method achieves significant\nimprovements compared to existing models. The method also provides a mechanism\nfor knowledge accumulation from multiple downstream IE tasks.\n