How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation

Manual evaluation is essential to judge progress on automatic text\nsummarization. However, we conduct a survey on recent summarization system\npapers that reveals little agreement on how to perform such evaluation studies.\nWe conduct two evaluation experiments on two aspects of summaries' linguistic\nquality (coherence and repetitiveness) to compare Likert-type and ranking\nannotations and show that best choice of evaluation method can vary from one\naspect to another. In our survey, we also find that study parameters such as\nthe overall number of annotators and distribution of annotators to annotation\nitems are often not fully reported and that subsequent statistical analysis\nignores grouping factors arising from one annotator judging multiple summaries.\nUsing our evaluation experiments, we show that the total number of annotators\ncan have a strong impact on study power and that current statistical analysis\nmethods can inflate type I error rates up to eight-fold. In addition, we\nhighlight that for the purpose of system comparison the current practice of\neliciting multiple judgements per summary leads to less powerful and reliable\nannotations given a fixed study budget.\n