We appreciate your dedicating effort to review and reply for our rebuttal.
However, we respectfully argue that the suggested framework to extend denoising steps does not mitigate the training-inference gap. To describe the reason, we will follow your suggestion step by step, sliding a denoising window every $n \geq 2$ denoising steps. We define the time step schedule as $0<\tau_0<\tau_1<\ldots<\tau_{nf}=T$.
1. Frame number $1 \sim f$ with diffusion time steps $[\tau_{nf-f+1}, \tau_{nf-f+2}, \ldots, \tau_{nf}]$ will be denoised $n$ times without sliding, reaching time steps of $[\tau_{nf-f+1-n}, \tau_{nf-f+2-n}, \ldots, \tau_{nf-n}]$.
2. Now we should slide the window, but we encounter two challenges. First, the foremost frame cannot be dequeued as it is not fully denoised. Second, adding a new random noise with the noise level of $\tau_{nf}$ results in an additional training-inference gap. Specifically, the queue now contains frames with time steps of $[\tau_{nf-f+2-n}, \tau_{nf-f+3-n}, \ldots, \tau_{nf-n},\tau_{nf}]$, where the last two frames have a time-step difference by $n$ while the rest has a gap of $1$. Moreover, the total noise level difference increases from $f-1$ to $f+n-2 > f-1$.
3. Assume that we repeat step 2 ignoring the aforementioned problems. Then, we will eventually have frames with time steps of $[\tau_{f}, \tau_{2f}, \ldots, \tau_{nf}]$ as a model input. The total time-step difference is now $(n-1)f$. In contrast, the time-step difference of latent partitioning is $f-1$ since frames with time steps of $[\tau_{kf+1}, \tau_{kf+2}, \ldots, \tau_{kf+f}]$ are model input.
Consequently, while your suggestion will increase denoising time steps, it cannot tackle the training-inference gap, which is latent partitioning’s motivation. Moreover, although your suggestion and latent partitioning require the same amount of computation, only our method allows parallelized inference on multiple GPUs.
Meanwhile, lookahead denoising is an independent component with latent partitioning, which can be applied solely to diagonal denoising. It clearly leverages the advantage of diagonal denoising that noisier frames are benefitted by clearer frames. We have demonstrated its effectiveness in ablation study (Section 5.4) both qualitatively and quantitatively. Moreover, we would like to clarify that all components in our paper have their own roles, and their necessities are well supported by theoretical and empirical analysis.
Finally, we want to clarify that we would rather consider our paper as concurrent work with [R1]. We have also submitted this paper in ICML 2024 with submission number 9782. Moreover, while there are some similarities, the two papers also have many differences as we described in the [global response](https://openreview.net/forum?id=uikhNa4wam¬eId=ovO0dl5OBe).