Large-scale pretraining instills large amounts of knowledge in deep neural\nnetworks. This, in turn, improves the generalization behavior of these models\nin downstream tasks. What exactly are the limits to the generalization benefits\nof large-scale pretraining? Here, we report observations from some simple\nexperiments aimed at addressing this question in the context of two semantic\nparsing tasks involving natural language, SCAN and COGS. We show that language\nmodels pretrained exclusively with non-English corpora, or even with\nprogramming language corpora, significantly improve out-of-distribution\ngeneralization in these benchmarks, compared with models trained from scratch,\neven though both benchmarks are English-based. This demonstrates the\nsurprisingly broad transferability of pretrained representations and knowledge.\nPretraining with a large-scale protein sequence prediction task, on the other\nhand, mostly deteriorates the generalization performance in SCAN and COGS,\nsuggesting that pretrained representations do not transfer universally and that\nthere are constraints on the similarity between the pretraining and downstream\ndomains for successful transfer. Finally, we show that larger models are harder\nto train from scratch and their generalization accuracy is lower when trained\nup to convergence on the relatively small SCAN and COGS datasets, but the\nbenefits of large-scale pretraining become much clearer with larger models.\n