BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Existing neural information retrieval (IR) models have often been studied in\nhomogeneous and narrow settings, which has considerably limited insights into\ntheir out-of-distribution (OOD) generalization capabilities. To address this,\nand to facilitate researchers to broadly evaluate the effectiveness of their\nmodels, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous\nevaluation benchmark for information retrieval. We leverage a careful selection\nof 18 publicly available datasets from diverse text retrieval tasks and domains\nand evaluate 10 state-of-the-art retrieval systems including lexical, sparse,\ndense, late-interaction and re-ranking architectures on the BEIR benchmark. Our\nresults show BM25 is a robust baseline and re-ranking and\nlate-interaction-based models on average achieve the best zero-shot\nperformances, however, at high computational costs. In contrast, dense and\nsparse-retrieval models are computationally more efficient but often\nunderperform other approaches, highlighting the considerable room for\nimprovement in their generalization capabilities. We hope this framework allows\nus to better evaluate and understand existing retrieval systems, and\ncontributes to accelerating progress towards better robust and generalizable\nsystems in the future. BEIR is publicly available at\nhttps://github.com/UKPLab/beir.\n