On the Reliability of Test Collections for Evaluating Systems of Different Types

As deep learning based models are increasingly being used for information\nretrieval (IR), a major challenge is to ensure the availability of test\ncollections for measuring their quality. Test collections are generated based\non pooling results of various retrieval systems, but until recently this did\nnot include deep learning systems. This raises a major challenge for reusable\nevaluation: Since deep learning based models use external resources (e.g. word\nembeddings) and advanced representations as opposed to traditional methods that\nare mainly based on lexical similarity, they may return different types of\nrelevant document that were not identified in the original pooling. If so, test\ncollections constructed using traditional methods are likely to lead to biased\nand unfair evaluation results for deep learning (neural) systems. This paper\nuses simulated pooling to test the fairness and reusability of test\ncollections, showing that pooling based on traditional systems only can lead to\nbiased evaluation of deep learning systems.\n