Search engines assess the quality of their results using nDCG@k for the queries in regular judgement sets. Since these sets are human annotated, the judgement sets are limited in size. Hence, these sets may not have proper representation from “Expert Topics”, e.g., , if an Expert Topic represents a tail segment. Therefore, query sets specific to expert topics are prepared to assess the search quality on such topics, which are judged by the domain experts. Such expert judgements are typically 5X-10X more expensive compared to regular judgements. Hence, “Expert Query Sets” must be prepared such that these queries are likely to be judged differently by an expert and a regular judge. In this paper, we define a novel problem, to identify a sub set of users queries, from a topic specific query set, such that the judgement difference between expert judges and regular judges, called yield rate, is high. Our results, across different expert topics, show that the expert query sets identified through our methodology have ∼15% more yield rate.
Paper
Full text
Identifying Expert Intent Queries
Semantic Scholar · Computer Science · 2020
Abstract
Search engines assess the quality of their results using nDCG@k for the queries in regular judgement sets. Since these sets are human annotated, the judgement sets are limited in size. Hence, these sets may not have proper representation from “Expert Topics”, e.g., , if an Expert Topic represents a tail segment. Therefore, query sets specific to expert topics are prepared to assess the search quality on such topics, which are judged by the domain experts. Such expert judgements are typically 5X-10X more expensive compared to regular judgements. Hence, “Expert Query Sets” must be prepared such that these queries are likely to be judged differently by an expert and a regular judge. In this paper, we define a novel problem, to identify a sub set of users queries, from a topic specific query set, such that the judgement difference between expert judges and regular judges, called yield rate, is high. Our results, across different expert topics, show that the expert query sets identified through our methodology have ∼15% more yield rate.