Summary
This paper approaches the task of predicting optical flow from a pair of images and depth from a single image. It proposes to do so using diffusion models, providing a training pipeline which includes contributions to deal with noisy training data. The proposed method is competitive with SOTA in depth prediction on NYU and KITTI, is similar or better to SOTA in optical flow prediction on Sintel and KITTI zero-shot, and is SOTA in optical flow after finetuning on KITTI. Additional experiments show the proposed pretraining pipeline meaningfully improves RAFT and the proposed model, that contributions dealing with noisy training data are helpful, and that the proposed method can predict multimodal outputs in cases of uncertainty.
Edit: Thanks to the authors for the rebuttal.
After reading the rebuttal and other reviews I will keep my rating at 7 - accept. I believe this paper should be accepted because it (1) introduces a simple but interesting method that performs SOTA on competitive benchmarks and (2) is very polished and experiments clearly defend all main contributions of the method.
Strengths
The paper provides a simple method that is similar-to or better than SOTA on competitive optical flow and depth tasks
- A new data pipeline is introduced which significantly boosts performance of prior SOTA e.g. RAFT, while improving further with the proposed method
- The method is built off a standard image-to-image diffusion method Palette, adds minimal detail needed to handle flow and depth data (e.g. L1 loss, infilling + unrolling, coarse-to-fine), and is competitive across a wide variety of experiments
- Real data is tricky to handle for diffusion models as it can contain incomplete flow and depth maps. The proposed method has an interesting idea of using its own predictions late in training. Combined with infilling, this is helpful.
The paper is very polished and experiments clearly defend all main contributions of the method.
- Writing is very clear, figures and tables are attractive and helpful
- Choice of training data is very important (Figure 3, Table 6, 7); it even improves RAFT meaningfully (Table 1)
- The method can also produce Multimodal samples faced with ambiguity e.g. transparent/translucent/reflective (Figure 1, 4, 5)
- Figure 6 and Table 5 shows the importance of coarse-to-fine refinement, which enables the method to produce better detail than RAFT (Figure 4, 5)
- Infilling + step-unrolling yields massive improvement on KITTI (Table 4)
- L1 loss (Supp Table 4), pretraining for depth (Supp Table 5, 6)
- Zero-shot depth completion is a cool application of the method
Weaknesses
Finetuning performance on real data yields weaker performance on Sintel vs. other methods (minor weakness, but addressing could further strengthen paper!)
- On Sintel (Table 2), most other methods use warm start, i.e. initializing flow prediction from previous frame. Is it possible to do this with diffusion, i.e. starting from a previous prediction, perhaps denoising fewer steps? If not, this is an important weakness of diffusion models in this setting
- Given most other method warm-start, it is harder to analyze the ability of the proposed method to finetune on real data. Perhaps it does not do as well relative to FlowFormer because (1) it does not handle real data as well as regression methods, even after contributions in this area, or (2) it does not overfit as well to specific datasets given others tend to have task-specific architecture. A comparison to more methods without warm-start, or by using the proposed method with warm-start, it would be very interesting to see this analyzed.
The main novel contribution of this paper beyond data pipeline is step-unrolled denoising diffusion training, which yields modest improvement over infilling alone (minor weakness)
- L1 loss is a design decision, infilling holes using bilinear interpolation is not a substantial contribution in my eyes. So the remaining technical contribution is unrolling.
- I understand the contributions are deliberately simple, which is a positive given performance gain. However, it is still important to analyze the reasons for the method’s success. In this case, the most novel component, “unroll”, improves optical depth REL from 0.077 to 0.075, RMS from 0.338 to 0.324, KITTI REL from 0.057 to 0.056, RMS from 2.744 to 2.700, AEPE from 1.53 to 1.47, and F1-all from 5.24% 4.74%. These contributions are nontrivial, but not substantial on their own.
Questions
- L179 is a bit confusing and could be rewritten “Training high resolution diffusion models is often slow and memory intensive but model performance has been shown to improve with resolution”
- Figure 4 (top) is dark and indistinct, making it a hard to see. Are there failure cases for RAFT on more easily readable examples?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.