VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning

Active Learning for discriminative models has largely been studied with the\nfocus on individual samples, with less emphasis on how classes are distributed\nor which classes are hard to deal with. In this work, we show that this is\nharmful. We propose a method based on the Bayes' rule, that can naturally\nincorporate class imbalance into the Active Learning framework. We derive that\nthree terms should be considered together when estimating the probability of a\nclassifier making a mistake for a given sample; i) probability of mislabelling\na class, ii) likelihood of the data given a predicted class, and iii) the prior\nprobability on the abundance of a predicted class. Implementing these terms\nrequires a generative model and an intractable likelihood estimation.\nTherefore, we train a Variational Auto Encoder (VAE) for this purpose. To\nfurther tie the VAE with the classifier and facilitate VAE training, we use the\nclassifiers' deep feature representations as input to the VAE. By considering\nall three probabilities, among them especially the data imbalance, we can\nsubstantially improve the potential of existing methods under limited data\nbudget. We show that our method can be applied to classification tasks on\nmultiple different datasets -- including one that is a real-world dataset with\nheavy data imbalance -- significantly outperforming the state of the art.\n