Exploring the Role of BERT Token Representations to Explain Sentence Probing Results

Several studies have been carried out on revealing linguistic features\ncaptured by BERT. This is usually achieved by training a diagnostic classifier\non the representations obtained from different layers of BERT. The subsequent\nclassification accuracy is then interpreted as the ability of the model in\nencoding the corresponding linguistic property. Despite providing insights,\nthese studies have left out the potential role of token representations. In\nthis paper, we provide a more in-depth analysis on the representation space of\nBERT in search for distinct and meaningful subspaces that can explain the\nreasons behind these probing results. Based on a set of probing tasks and with\nthe help of attribution methods we show that BERT tends to encode meaningful\nknowledge in specific token representations (which are often ignored in\nstandard classification setups), allowing the model to detect syntactic and\nsemantic abnormalities, and to distinctively separate grammatical number and\ntense subspaces.\n