Recent work on training neural retrievers for open-domain question answering\n(OpenQA) has employed both supervised and unsupervised approaches. However, it\nremains unclear how unsupervised and supervised methods can be used most\neffectively for neural retrievers. In this work, we systematically study\nretriever pre-training. We first propose an approach of unsupervised\npre-training with the Inverse Cloze Task and masked salient spans, followed by\nsupervised finetuning using question-context pairs. This approach leads to\nabsolute gains of 2+ points over the previous best result in the top-20\nretrieval accuracy on Natural Questions and TriviaQA datasets.\n We also explore two approaches for end-to-end supervised training of the\nreader and retriever components in OpenQA models. In the first approach, the\nreader considers each retrieved document separately while in the second\napproach, the reader considers all the retrieved documents together. Our\nexperiments demonstrate the effectiveness of these approaches as we obtain new\nstate-of-the-art results. On the Natural Questions dataset, we obtain a top-20\nretrieval accuracy of 84, an improvement of 5 points over the recent DPR model.\nIn addition, we achieve good results on answer extraction, outperforming recent\nmodels like REALM and RAG by 3+ points. We further scale up end-to-end training\nto large models and show consistent gains in performance over smaller models.\n