OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings

Language representations are known to carry stereotypical biases and, as a\nresult, lead to biased predictions in downstream tasks. While existing methods\nare effective at mitigating biases by linear projection, such methods are too\naggressive: they not only remove bias, but also erase valuable information from\nword embeddings. We develop new measures for evaluating specific information\nretention that demonstrate the tradeoff between bias removal and information\nretention. To address this challenge, we propose OSCaR (Orthogonal Subspace\nCorrection and Rectification), a bias-mitigating method that focuses on\ndisentangling biased associations between concepts instead of removing concepts\nwholesale. Our experiments on gender biases show that OSCaR is a well-balanced\napproach that ensures that semantic information is retained in the embeddings\nand bias is also effectively mitigated.\n