We explore to what extent knowledge about the pre-trained language model that\nis used is beneficial for the task of abstractive summarization. To this end,\nwe experiment with conditioning the encoder and decoder of a Transformer-based\nneural model on the BERT language model. In addition, we propose a new method\nof BERT-windowing, which allows chunk-wise processing of texts longer than the\nBERT window size. We also explore how locality modelling, i.e., the explicit\nrestriction of calculations to the local context, can affect the summarization\nability of the Transformer. This is done by introducing 2-dimensional\nconvolutional self-attention into the first layers of the encoder. The results\nof our models are compared to a baseline and the state-of-the-art models on the\nCNN/Daily Mail dataset. We additionally train our model on the SwissText\ndataset to demonstrate usability on German. Both models outperform the baseline\nin ROUGE scores on two datasets and show its superiority in a manual\nqualitative analysis.\n