Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

The growth of domain-specific applications of semantic models, boosted by the\nrecent achievements of unsupervised embedding learning algorithms, demands\ndomain-specific evaluation datasets. In many cases, content-based recommenders\nbeing a prime example, these models are required to rank words or texts\naccording to their semantic relatedness to a given concept, with particular\nfocus on top ranks. In this work, we give a threefold contribution to address\nthese requirements: (i) we define a protocol for the construction, based on\nadaptive pairwise comparisons, of a relatedness-based evaluation dataset\ntailored on the available resources and optimized to be particularly accurate\nin top-rank evaluation; (ii) we define appropriate metrics, extensions of\nwell-known ranking correlation coefficients, to evaluate a semantic model via\nthe aforementioned dataset by taking into account the greater significance of\ntop ranks. Finally, (iii) we define a stochastic transitivity model to simulate\nsemantic-driven pairwise comparisons, which confirms the effectiveness of the\nproposed dataset construction protocol.\n