Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering

Active learning promises to alleviate the massive data needs of supervised\nmachine learning: it has successfully improved sample efficiency by an order of\nmagnitude on traditional tasks like topic classification and object\nrecognition. However, we uncover a striking contrast to this promise: across 5\nmodels and 4 datasets on the task of visual question answering, a wide variety\nof active learning approaches fail to outperform random selection. To\nunderstand this discrepancy, we profile 8 active learning methods on a\nper-example basis, and identify the problem as collective outliers -- groups of\nexamples that active learning methods prefer to acquire but models fail to\nlearn (e.g., questions that ask about text in images or require external\nknowledge). Through systematic ablation experiments and qualitative\nvisualizations, we verify that collective outliers are a general phenomenon\nresponsible for degrading pool-based active learning. Notably, we show that\nactive learning sample efficiency increases significantly as the number of\ncollective outliers in the active learning pool decreases. We conclude with a\ndiscussion and prescriptive recommendations for mitigating the effects of these\noutliers in future work.\n