DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion

We propose an online multi-view depth prediction approach on posed video\nstreams, where the scene geometry information computed in the previous time\nsteps is propagated to the current time step in an efficient and geometrically\nplausible way. The backbone of our approach is a real-time capable, lightweight\nencoder-decoder that relies on cost volumes computed from pairs of images. We\nextend it by placing a ConvLSTM cell at the bottleneck layer, which compresses\nan arbitrary amount of past information in its states. The novelty lies in\npropagating the hidden state of the cell by accounting for the viewpoint\nchanges between time steps. At a given time step, we warp the previous hidden\nstate into the current camera plane using the previous depth prediction. Our\nextension brings only a small overhead of computation time and memory\nconsumption, while improving the depth predictions significantly. As a result,\nwe outperform the existing state-of-the-art multi-view stereo methods on most\nof the evaluated metrics in hundreds of indoor scenes while maintaining a\nreal-time performance. Code available:\nhttps://github.com/ardaduz/deep-video-mvs\n