Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets

The ImageNet dataset ushered in a flood of academic and industry interest in\ndeep learning for computer vision applications. Despite its significant impact,\nthere has not been a comprehensive investigation into the demographic\nattributes of images contained within the dataset. Such a study could lead to\nnew insights on inherent biases within ImageNet, particularly important given\nit is frequently used to pretrain models for a wide variety of computer vision\ntasks. In this work, we introduce a model-driven framework for the automatic\nannotation of apparent age and gender attributes in large-scale image datasets.\nUsing this framework, we conduct the first demographic audit of the 2012\nImageNet Large Scale Visual Recognition Challenge (ILSVRC) subset of ImageNet\nand the "person" hierarchical category of ImageNet. We find that 41.62% of\nfaces in ILSVRC appear as female, 1.71% appear as individuals above the age of\n60, and males aged 15 to 29 account for the largest subgroup with 27.11%. We\nnote that the presented model-driven framework is not fair for all\nintersectional groups, so annotation are subject to bias. We present this work\nas the starting point for future development of unbiased annotation models and\nfor the study of downstream effects of imbalances in the demographics of\nImageNet. Code and annotations are available at:\nhttp://bit.ly/ImageNetDemoAudit\n