Pretraining Neural Language Models (NLMs) over a large corpus involves\nchunking the text into training examples, which are contiguous text segments of\nsizes processable by the neural architecture. We highlight a bias introduced by\nthis common practice: we prove that the pretrained NLM can model much stronger\ndependencies between text segments that appeared in the same training example,\nthan it can between text segments that appeared in different training examples.\nThis intuitive result has a twofold role. First, it formalizes the motivation\nbehind a broad line of recent successful NLM training heuristics, proposed for\nthe pretraining and fine-tuning stages, which do not necessarily appear related\nat first glance. Second, our result clearly indicates further improvements to\nbe made in NLM pretraining for the benefit of Natural Language Understanding\ntasks. As an example, we propose "kNN-Pretraining": we show that including\nsemantically related non-neighboring sentences in the same pretraining example\nyields improved sentence representations and open domain question answering\nabilities. This theoretically motivated degree of freedom for pretraining\nexample design indicates new training schemes for self-improving\nrepresentations.\n