Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings

Sense embedding learning methods learn different embeddings for the different\nsenses of an ambiguous word. One sense of an ambiguous word might be socially\nbiased while its other senses remain unbiased. In comparison to the numerous\nprior work evaluating the social biases in pretrained word embeddings, the\nbiases in sense embeddings have been relatively understudied. We create a\nbenchmark dataset for evaluating the social biases in sense embeddings and\npropose novel sense-specific bias evaluation measures. We conduct an extensive\nevaluation of multiple static and contextualised sense embeddings for various\ntypes of social biases using the proposed measures. Our experimental results\nshow that even in cases where no biases are found at word-level, there still\nexist worrying levels of social biases at sense-level, which are often ignored\nby the word-level bias evaluation measures.\n