Deformable VisTR: Spatio temporal deformable attention for video instance segmentation

Video instance segmentation (VIS) task requires classifying, segmenting, and\ntracking object instances over all frames in a video clip. Recently, VisTR has\nbeen proposed as end-to-end transformer-based VIS framework, while\ndemonstrating state-of-the-art performance. However, VisTR is slow to converge\nduring training, requiring around 1000 GPU hours due to the high computational\ncost of its transformer attention module. To improve the training efficiency,\nwe propose Deformable VisTR, leveraging spatio-temporal deformable attention\nmodule that only attends to a small fixed set of key spatio-temporal sampling\npoints around a reference point. This enables Deformable VisTR to achieve\nlinear computation in the size of spatio-temporal feature maps. Moreover, it\ncan achieve on par performance as the original VisTR with 10$\\times$ less GPU\ntraining hours. We validate the effectiveness of our method on the Youtube-VIS\nbenchmark. Code is available at https://github.com/skrya/DefVIS.\n