This paper focuses on a novel and challenging detection scenario: A majority\nof true objects/instances is unlabeled in the datasets, so these\nmissing-labeled areas will be regarded as the background during training.\nPrevious art on this problem has proposed to use soft sampling to re-weight the\ngradients of RoIs based on the overlaps with positive instances, while their\nmethod is mainly based on the two-stage detector (i.e. Faster RCNN) which is\nmore robust and friendly for the missing label scenario. In this paper, we\nintroduce a superior solution called Background Recalibration Loss (BRL) that\ncan automatically re-calibrate the loss signals according to the pre-defined\nIoU threshold and input image. Our design is built on the one-stage detector\nwhich is faster and lighter. Inspired by the Focal Loss formulation, we make\nseveral significant modifications to fit on the missing-annotation\ncircumstance. We conduct extensive experiments on the curated PASCAL VOC and MS\nCOCO datasets. The results demonstrate that our proposed method outperforms the\nbaseline and other state-of-the-arts by a large margin. Code available:\nhttps://github.com/Dwrety/mmdetection-selective-iou.\n