Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

This paper presents a simple yet effective approach to modeling space-time\ncorrespondences in the context of video object segmentation. Unlike most\nexisting approaches, we establish correspondences directly between frames\nwithout re-encoding the mask features for every object, leading to a highly\nefficient and robust framework. With the correspondences, every node in the\ncurrent query frame is inferred by aggregating features from the past in an\nassociative fashion. We cast the aggregation process as a voting problem and\nfind that the existing inner-product affinity leads to poor use of memory with\na small (fixed) subset of memory nodes dominating the votes, regardless of the\nquery. In light of this phenomenon, we propose using the negative squared\nEuclidean distance instead to compute the affinities. We validated that every\nmemory node now has a chance to contribute, and experimentally showed that such\ndiversified voting is beneficial to both memory efficiency and inference\naccuracy. The synergy of correspondence networks and diversified voting works\nexceedingly well, achieves new state-of-the-art results on both DAVIS and\nYouTubeVOS datasets while running significantly faster at 20+ FPS for multiple\nobjects without bells and whistles.\n