[R1] Tucker, R.; and Snavely, N. 2020. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 551–560.
[R2] Han, Y.; Wang, R.; and Yang, J. 2022. Single-view view synthesis in the wild with learned adaptive multiplane images. In ACM SIGGRAPH 2022 Conference Proceedings, 1–8.
[R3] Wang, X.; Wu, C.; Yin, S.; Ni, M.; Wang, J.; Li, L.; Yang, Z.; Yang, F.; Wang, L.; Liu, Z.; et al. 2023b. Learning 3D Photography Videos via Self-supervised Diffusion on Single Images. arXiv preprint arXiv:2302.10781
**Q5. High-level reason and more analysis of frame matrix.**
1) High-level motivations. In practice, 3D stereoscopic videos can be produced by recording with two cameras
(time-direction videos). Since both cameras are capturing the same scene, gradually moving the left camera toward the right camera also results in a coherent video (spatial-direction video). Likewise, it is essential to consider both the time and spatial directions when generating the 3D stereoscopic video to ensure that both perspectives represent the same scene. For this purpose, we propose the frame matrix, allowing the denoising process to be performed simultaneously along both spatial and temporal dimensions using a pre-trained video diffusion model, which has learned spatial and temporal priors from diverse static and dynamic scenes. Additionally, compared to direct inpainting a large region all at once, gradually expanding the inpainting area (spatial direction) tends to be easier and can lead to more stable and plausible results.
2) Theoretical analysis. Based on the above high-level motivation, each frame in the frame matrix should be as consistent as the denoising results from both time and spatial directions. Mathematically:
$$
L(i, j) = ||z_{t-1}(i, j) - z_{t-1}^{spatial}(i, j)||^2 + ||z_{t-1}(i, j) - z_{t-1}^{time}(i, j)||^2,
$$
$$
z_{t-1}^{spatial} = \Theta^{spatial}(z_t) ,
$$
$$
z_{t-1}^{time} = \Theta^{time}(z_t) ,
$$
Where $\Theta$ is a pre-trained video diffusion model, $\Theta^{time}$ and $\Theta^{spatial}$ indicate denoising frame matrix $z$ in the time and spatial directions respectively. Here, $L$ is a quadratic Least-Squares (LS) where the solution is as close as possible to all diffusion samples $z_{t-1}^{time}(i, j)$ and $z_{t-1}^{spatial}(i, j)$. In practice, we simultaneously denoise in spatial and time directions to obtain the optimal results.
**Q6. Lack citations**
We have included the missing articles in our new version (lines 689-691, 576-578, 700-701).
**Q7. The presentation contains certain misleading and deceptive elements.**
Could you please provide more details? We'd like to clarify any misunderstandings.
**Q8. Line 153-161, the viewpoint transfer part is widely used in novel view synthesis [R1, R2] and stereo image generation [R3].**
Lines 153-161 describe the overall workflow of our pipeline for creating stereoscopic videos, where we adopt the warp-then-inpaint approach, for which we do not claim contributions. Our primary focus and contribution lie in addressing the challenges of filling disoccluded regions with semantically reasonable and high-quality content by leveraging pre-trained video generation models in a zero-shot manner. To achieve this, we propose a novel frame matrix representation and a re-injection scheme.
**Q9. Insufficient evaluation metrics. In [R1], [R2], [R3], they will use SSIM, PSNR, LPIPS evaluation metrics in novel-view image, which can also be used in stereo video generation., the quantitative results such as Fréchet Video Distance should also be given.**
1) As shown in Fig. 22, the generated contents in disoccluded regions are not unique but visually reasonable. Therefore, metrics such as PSNR, SSIM, and LPIPS, which are calculated for measuring reconstruction error instead of generation quality, are not suitable for this task.
2) We added the suggested FVD in Table 2.
**Q10. Lack of comparison with highly related methods [R1, R2, R3]. The authors should give the results in the paper and clarify the strength of their methods.**
1) Please refer to Q4 for comparisons.
2) Methods [R1, R2, R3] are not designed for stereoscopic video and may encounter temporal consistency issues, as they process a single image and thus cannot leverage temporal information across frames (see Fig. 20). In contrast, we leverage a video diffusion model that enables effective utilization of temporal information.
3) Collecting stereoscopic videos for training these methods is costly, and insufficient training data can lead to performance issues due to the domain gap. In contrast, the pre-trained video diffusion model, trained on a vast amount of data, demonstrates great generalizability.