Summary
This paper works on the optical flow estimation problem, especially in the unsupervised setting, for high-frequency spike camera inputs. Specifically, a dynamic timing representation module and a spike-based unsupervised loss are proposed to improve performance. A new synthetic dataset, namely SSES, is created to test extreme scenes in autonomous driving. Experiment results show the proposed method has achieved state-of-the-art performance compared with previous methods.
Strengths
1. The paper is well-motivated by introducing the importance of optical flow estimation for high-frequency inputs collected from latest developed sensors like spike cameras. This topic has not been extensively studied so far.
2. The paper is overall clear and well-structured.
3. The proposed method is reasonable. Explanations on each module design are provided.
4. The results are compelling.
Weaknesses
1. There are still some confusions in the text or figures that need to be clarified. See additional comments below.
2. It will be better to polish the language of the paper before publication. Correct grammatical errors and use more formal language. Avoid informal expressions like "a bunch of" (Line 138), "won't" (Line 140), etc. See additional comments below.
3. Authors are encouraged to share their code and generated datasets for better reproducibility. Such plan has not been declared in the paper.
Questions
1. If I understand it correctly, the method uses the predicted optical flow to determine whether each pixel belongs to a low-speed or high-speed region and then uses different ways to approximate light intensities $\tilde{I}$. At the starting iterations of training, the predictions could be totally off, so it is likely that the method will pick a wrong way to approximate intensities. Did you try to tackle this issue?
2. The data used in Fig 4 do not have ground-truth labels, right? Only qualitative evaluation is possible, so this evaluation may be weak.
3. There is also a large part of supervised results, so stating "unsupervised" in the title could be misleading.
4. I am curious how a normal RGB-based optical flow network would perform as a comparison. For example, a naïve approach is to first convert spike inputs to RGB and then apply a state-of-the-art RGB-based optical flow network of similar size (parameters, memory cost). What would the results be like? This could be a baseline to strengthen the motivation of developing optical flow networks specific to spike camera inputs.
Additional comments:
1. Grammatical errors: run-on sentences (Line 103-105, line 140-141, Line 163-165, Line 310-311).
2. Eq 1: The "mod" operator is traditionally only defined for integer division, but the value here is clearly continuous numbers, so it is better to define "mod" in the text in the continuous context.
3. Chapter 3.1: There are some confusions here. (1) I assume $A(\mathbf x, t)$ is continuous, but Line 119 says the output stream is binary, so there is missing gap in between that need explanations. Maybe adding a definition, something like $S_t^N(\mathbf x) = \mathbb I(\sup_{\tau\in[t-\Delta t, t+\Delta t]} A(\mathbf x, \tau) =\theta)$ will make it clear. (2) Line 113-114 say the camera "fires a spike" when the cumulative electrons hits the threshold, so I assume it is asynchronous for each pixel, but then Line 117-118 say the camera "fires a spike" at time T, which is synchronous and periodic. Maybe you could change the expression here to avoid confusion.
4. Line 130: "officially" -> "official".
5. Line 135: "dependence" -> "dependency".
6. Fig 2: Text is too small. It is better to make text size close to that of the main paper so that readers can read the figure clearly without needing to zoom in.
7. Line 147: It is better to add that your convolution is 1D on the time dimension to avoid confusion.
8. Line 182-183: Does your method guarantee $\omega_s + \omega_l = 1$? Eq 6: Does $\sum_k\omega_k=1$? If so, state so explicitly.
9. Eq 7 could be merged into the text. It does not have to an equation.
10. Line 175-184: What do you mean by "speed"? The frequency of camera readings, or fast motion? Please clarify.
11. Line 207: The notation $\omega_s, \omega_l, \omega_{k=1}, \omega_{k=2}$ is messy. Try not to use both textual abbreviations ("s", "l") and values ($k$) as subscripts for the same letter $\omega$. Use other letters like $\alpha_s, \alpha_l$, $\beta_k$ instead; or, you could do superscripts with parentheses $\omega^{(s)}, \omega^{(l)}, \omega_k$.
12. Line 209: "approximate" -> "approximated".
13. Line 212: define or cite "Charbonnier loss"; Line 213: define or cite "smoothness loss" specifically. There are many ways to define "smoothness".
14. Table 1 and 2: It is better to add citations into the table, so we know which data are generated from your experiments and which data are cited from other papers. The current table assumes that all data are credited to yourself.
15. Table 1 and 2: I recommend adding a vertical line between the "Top" and "Mean" columns, so we know "Top" is the name of the scene instead of the top error across all scenes.
16. Line 277: "through" -> "though".
17. Line 284: "Figure 4" -> "Table 4".
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.