ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them

This paper presents the winning system for the propaganda Technique\nClassification (TC) task and the second-placed system for the propaganda Span\nIdentification (SI) task. The purpose of TC task was to identify an applied\npropaganda technique given propaganda text fragment. The goal of SI task was to\nfind specific text fragments which contain at least one propaganda technique.\nBoth of the developed solutions used semi-supervised learning technique of\nself-training. Interestingly, although CRF is barely used with\ntransformer-based language models, the SI task was approached with RoBERTa-CRF\narchitecture. An ensemble of RoBERTa-based models was proposed for the TC task,\nwith one of them making use of Span CLS layers we introduce in the present\npaper. In addition to describing the submitted systems, an impact of\narchitectural decisions and training schemes is investigated along with remarks\nregarding training models of the same or better quality with lower\ncomputational budget. Finally, the results of error analysis are presented.\n