UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

Despite the success of various text generation metrics such as BERTScore, it\nis still difficult to evaluate the image captions without enough reference\ncaptions due to the diversity of the descriptions. In this paper, we introduce\na new metric UMIC, an Unreferenced Metric for Image Captioning which does not\nrequire reference captions to evaluate image captions. Based on\nVision-and-Language BERT, we train UMIC to discriminate negative captions via\ncontrastive learning. Also, we observe critical problems of the previous\nbenchmark dataset (i.e., human annotations) on image captioning metric, and\nintroduce a new collection of human annotations on the generated captions. We\nvalidate UMIC on four datasets, including our new dataset, and show that UMIC\nhas a higher correlation than all previous metrics that require multiple\nreferences. We release the benchmark dataset and pre-trained models to compute\nthe UMIC.\n