An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

Large-scale Multi-label Text Classification (LMTC) has a wide range of\nNatural Language Processing (NLP) applications and presents interesting\nchallenges. First, not all labels are well represented in the training set, due\nto the very large label set and the skewed label distributions of LMTC\ndatasets. Also, label hierarchies and differences in human labelling guidelines\nmay affect graph-aware annotation proximity. Finally, the label hierarchies are\nperiodically updated, requiring LMTC models capable of zero-shot\ngeneralization. Current state-of-the-art LMTC models employ Label-Wise\nAttention Networks (LWANs), which (1) typically treat LMTC as flat multi-label\nclassification; (2) may use the label hierarchy to improve zero-shot learning,\nalthough this practice is vastly understudied; and (3) have not been combined\nwith pre-trained Transformers (e.g. BERT), which have led to state-of-the-art\nresults in several NLP benchmarks. Here, for the first time, we empirically\nevaluate a battery of LMTC methods from vanilla LWANs to hierarchical\nclassification approaches and transfer learning, on frequent, few, and\nzero-shot learning on three datasets from different domains. We show that\nhierarchical methods based on Probabilistic Label Trees (PLTs) outperform\nLWANs. Furthermore, we show that Transformer-based approaches outperform the\nstate-of-the-art in two of the datasets, and we propose a new state-of-the-art\nmethod which combines BERT with LWANs. Finally, we propose new models that\nleverage the label hierarchy to improve few and zero-shot learning, considering\non each dataset a graph-aware annotation proximity measure that we introduce.\n