Ensembles, where multiple neural networks are trained individually and their\npredictions are averaged, have been shown to be widely successful for improving\nboth the accuracy and predictive uncertainty of single neural networks.\nHowever, an ensemble's cost for both training and testing increases linearly\nwith the number of networks, which quickly becomes untenable.\n In this paper, we propose BatchEnsemble, an ensemble method whose\ncomputational and memory costs are significantly lower than typical ensembles.\nBatchEnsemble achieves this by defining each weight matrix to be the Hadamard\nproduct of a shared weight among all ensemble members and a rank-one matrix per\nmember. Unlike ensembles, BatchEnsemble is not only parallelizable across\ndevices, where one device trains one member, but also parallelizable within a\ndevice, where multiple ensemble members are updated simultaneously for a given\nmini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and\nout-of-distribution tasks, BatchEnsemble yields competitive accuracy and\nuncertainties as typical ensembles; the speedup at test time is 3X and memory\nreduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to\nlifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable\nperformance to progressive neural networks while having a much lower\ncomputational and memory costs. We further show that BatchEnsemble can easily\nscale up to lifelong learning on Split-ImageNet which involves 100 sequential\nlearning tasks.\n