Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning

Weakly supervised segmentation requires assigning a label to every pixel\nbased on training instances with partial annotations such as image-level tags,\nobject bounding boxes, labeled points and scribbles. This task is challenging,\nas coarse annotations (tags, boxes) lack precise pixel localization whereas\nsparse annotations (points, scribbles) lack broad region coverage. Existing\nmethods tackle these two types of weak supervision differently: Class\nactivation maps are used to localize coarse labels and iteratively refine the\nsegmentation model, whereas conditional random fields are used to propagate\nsparse labels to the entire image.\n We formulate weakly supervised segmentation as a semi-supervised metric\nlearning problem, where pixels of the same (different) semantics need to be\nmapped to the same (distinctive) features. We propose 4 types of contrastive\nrelationships between pixels and segments in the feature space, capturing\nlow-level image similarity, semantic annotation, co-occurrence, and feature\naffinity They act as priors; the pixel-wise feature can be learned from\ntraining images with any partial annotations in a data-driven fashion. In\nparticular, unlabeled pixels in training images participate not only in\ndata-driven grouping within each image, but also in discriminative feature\nlearning within and across images. We deliver a universal weakly supervised\nsegmenter with significant gains on Pascal VOC and DensePose. Our code is\npublicly available at https://github.com/twke18/SPML.\n