One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks

Despite being widely used as a performance measure for visual detection\ntasks, Average Precision (AP) is limited in (i) reflecting localisation\nquality, (ii) interpretability and (iii) robustness to the design choices\nregarding its computation, and its applicability to outputs without confidence\nscores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic\nsegmentation (Kirillov et al., 2019), does not suffer from these limitations\nbut is limited to panoptic segmentation. In this paper, we propose Localisation\nRecall Precision (LRP) Error as the average matching error of a visual detector\ncomputed based on both its localisation and classification qualities for a\ngiven confidence score threshold. LRP Error, initially proposed only for object\ndetection by Oksuz et al. (2018), does not suffer from the aforementioned\nlimitations and is applicable to all visual detection tasks. We also introduce\nOptimal LRP (oLRP) Error as the minimum LRP Error obtained over confidence\nscores to evaluate visual detectors and obtain optimal thresholds for\ndeployment. We provide a detailed comparative analysis of LRP Error with AP and\nPQ, and use nearly 100 state-of-the-art visual detectors from seven visual\ndetection tasks (i.e. object detection, keypoint detection, instance\nsegmentation, panoptic segmentation, visual relationship detection, zero-shot\ndetection and generalised zero-shot detection) using ten datasets to\nempirically show that LRP Error provides richer and more discriminative\ninformation than its counterparts. Code available at:\nhttps://github.com/kemaloksuz/LRP-Error\n