Self-supervised learning of speech representations has been a very active\nresearch area but most work is focused on a single domain such as read audio\nbooks for which there exist large quantities of labeled and unlabeled data. In\nthis paper, we explore more general setups where the domain of the unlabeled\ndata for pre-training data differs from the domain of the labeled data for\nfine-tuning, which in turn may differ from the test data domain. Our\nexperiments show that using target domain data during pre-training leads to\nlarge performance improvements across a variety of setups. On a large-scale\ncompetitive setup, we show that pre-training on unlabeled in-domain data\nreduces the gap between models trained on in-domain and out-of-domain labeled\ndata by 66%-73%. This has obvious practical implications since it is much\neasier to obtain unlabeled target domain data than labeled data. Moreover, we\nfind that pre-training on multiple domains improves generalization performance\non domains not seen during training. Code and models will be made available at\nhttps://github.com/pytorch/fairseq.\n