Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?

We investigate the self-attention mechanism of BERT in a fine-tuning scenario\nfor the classification of scientific articles over a taxonomy of research\ndisciplines. We observe how self-attention focuses on words that are highly\nrelated to the domain of the article. Particularly, a small subset of\nvocabulary words tends to receive most of the attention. We compare and\nevaluate the subset of the most attended words with feature selection methods\nnormally used for text classification in order to characterize self-attention\nas a possible feature selection approach. Using ConceptNet as ground truth, we\nalso find that attended words are more related to the research fields of the\narticles. However, conventional feature selection methods are still a better\noption to learn classifiers from scratch. This result suggests that, while\nself-attention identifies domain-relevant terms, the discriminatory information\nin BERT is encoded in the contextualized outputs and the classification layer.\nIt also raises the question whether injecting feature selection methods in the\nself-attention mechanism could further optimize single sequence classification\nusing transformers.\n