From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos

Summary

The paper introduces a large-scale, real-world, multiview, 360-degree outward dataset designed for static novel view synthesis and 3D reconstruction. To capture these data, the study develops a pipeline capable of identifying corresponding multiview frames from 360-degree outward videos with a fixed camera trajectory. To demonstrate the dataset's contribution and impact, the work proposes a diffusion-based model that achieves state-of-the-art performance using this dataset.

Strengths

Strength - The work introduces a real-world, multiview, 360-degree outward dataset that is significantly larger and more diverse than existing datasets for static novel view synthesis. This dataset is crucial for modern data-driven methods. - The work presents a novel and efficient pipeline that utilizes Dust3R to find corresponding frames and graphs to maintain long-range correspondence for data collection. This innovative pipeline makes the data collection process scalable. - The work proposes a diffusion-based model, ODIN, that leverages the video dataset for static novel view synthesis by incorporating motion masking and viewpoint conditions to achieve state-of-the-art performance. Additionally, the model can generate long-range novel views, further validating the dataset's positive impact.

Weaknesses

Dateset - The dataset is collected from 360 YouTube videos, captured beforehand by the camera operator. The camera trajectory (x, y, z) is controlled by the camera operator, thereby limiting the flexibility to select diverse viewpoints from different positions. - The quality of the data is unknown and can be unstable across different data points because the author cannot investigate each video one by one or frame by frame. This represents a tradeoff between capturing one's own data and crawling data from the Internet. For instance, the author acknowledges in lines 178-179 that it is infeasible to guarantee the uniqueness of video content. - Although the work claims that 1 fps is sufficient in lines 126-128, this claim lacks experimental support. The author should consider trying different fps rates and show the performance gap and computational time between them. - Larger dataset may need more training time and computational resource. Method - One of the most challenging aspects that prevent people from using video for static scene tasks is the impact of motion over time (multiview inconsistency). The work proposes a motion mask method to mitigate this impact. However, this aspect lacks analysis and discussion. For instance, the importance of the hyperparameter in Eq. 3 is unknown. - In Fig. 3, in the second row, the method is shown to synthesize humans as well. However, I wonder whether the motion masking has a negative impact on synthesizing dynamic objects when they are static. Writing - In lines 5-6, the claim may be too arbitrary. As mentioned in lines 87-90, there are some large-scale real-world datasets, such as CO3D, albeit with limited diversity. A related work, OmniObject3D, should also be mentioned. - In lines 8-12, the phrase "...diverse views" may be too broad and should be softened. The proposed dataset is collected from pre-captured 360 YouTube videos. The camera trajectory is decided by the camera operator beforehand, and only the camera orientation can be controlled after that. Hence, compared to general multiview datasets, which can have multiple viewpoints from different locations at the same timestamp, the views of the proposed dataset are limited. - Eq. 2 does not match the equation in Sec. 5.1. The equation in Sec. 5.1 contains an undefined term. - I hope the author can define their 360 multiview videos at the beginning since there are several kinds of 360 videos. For instance, DiVa360 captures 360-degree inward videos with multiple cameras at different locations at the same timestamp.

Questions

- I hope the author can verify that the viewpoints are diverse enough by discussing the difference between a multiview dataset that can capture images from different locations at the same timestamp and a dataset that can only change the camera orientation at the same timestamp. - The dataset contains a large amount of data, which may vary in quality. I hope the author can discuss the impact of the low-quality data in the dataset. Is it negligible? - Iines 178-179 mentions that it is infeasible to guarantee the uniqueness of video content. Hence, how do you make sure there is no overlap between training and testing set? How do you make sure the training dataset does not contain the testing cases in other dataset? Will it be unfair for the benchmark? - The dataset contains diverse data. How is the data distribution in terms of motion? Will high-motion data, such as scenes with rain, have a negative impact? - Lines 126-128 need experimental support. Please consider trying it on a subset of the dataset. - The work uses Depth Anything as the depth estimator. This is a monocular depth estimator with two types of checkpoints pretrained on two different datasets. I wonder if this causes a domain gap in the proposed dataset when using different checkpoints. Do you finetune the estimator on your dataset? Additionally, it is unknown how good the multiview consistency is on your data with this estimator. - What is the tradeoff between using this dataset and others? From the experiment section, we know that the dataset can increase performance. But how about the training time compared to existing datasets? Maybe the author can consider adding the training time from the appendix to the table. - How sensitive and stable is the motion masking hyperparameter in Eq. 3? How should it be tuned? - What is the performance when the input image contains many dynamic objects?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

I agree with the limitation section in the paper. - One potential limitation of the large-scale dataset is to maintain the data quality and high fps. - Another potential limitation is to make sure there is no overlap between training and testing set.

Summary

This paper considers novel view synthesis from a single image. The main contribution is a dataset sourced from 1 million 360 videos from youtube, which is used to create about 380 million pairs of images with corresponding relative poses. With this dataset the authors train a diffusion model similar to zero-1-to-3 [24], but for general scenes instead of objects, and real-world scenes instead of synthetic ones. State-of-the-art results are shown for novel view synthesis from single images.

Strengths

- The main contribution of the paper is that it introduces a dataset for novel view synthesis that is significantly larger (380 million image pairs extracted from 1 million 360-videos) than existing datasets for the task, and considers whole scenes rather than single objects as e.g. objaverse. - The authors train a diffusion model (ODIN) for novel view synthesis that is more general than existing work (zero-1-to-3 and follow up works which only consider single objects), and compared to scene based methods (zeroNVS), it is more accurate and general, permitting a larger set of possible relative poses (R,t). The improvements are mainly due to training on the larger dataset since the methodology, namely the diffusion model architecture and training, is very similar to zero-1-to-3. - The proposed diffusion model, ODIN, along with the 3d reconstruction method dust3r can generate realistic-looking 3d reconstruction from a single image.

Weaknesses

- There is a lack of qualitative examples. Specifically, a few things that would have been useful to show are 1) The generated images along the trajectories in fig. 4 in the form of both images and videos. Only the final 3d reconstruction by dust3r and not the intermediately generated images by ODIN are shown, 2) Images of the renderings of different methods in table 1 and 2, and 3) Generated images and the corresponding 3d reconstructions on Google Scanned Objects, so qualitative comparison corresponding to table 3 for all methods. - There are a few missing references and as a result too strong claims in the paper. 1) “Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models” (Yu et al. ICCV 2023), 2) “Geometry-Free View Synthesis: Transformers and no 3D Priors” (Rombach et al. ICCV 2021), and 3) “Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image” (Ren et al. CVPR 2022). These works, and also zeroNVS (which is cited) all address novel view synthesis from a single image, though trained on smaller datasets. ZeroNVS also constructs a 3d scene by training a NeRF with Score distillation sampling. Due to this I think e.g. the claim on line 48 that ODIN is the first to reasonably reconstruct a 3d scene from a single image is too strong.

Questions

- I wonder how important the use of dust3r is for the 3d reconstruction. Is the confidence measure by dust3r used, and if so, how? I would imagine that it would be useful to get a consistent 3d reconstruction even if the views generated by ODIN are not consistent. By consistent I mean both that the views might not adhere to a specific camera model well enough for traditional sfm method to work well, or that the different views contain contradicting contents. Is that why there are some holes in the reconstructions in fig. 4 (e.g. the middle part in the leftmost 3d model of the cathedral), because it looks from the positions of the cameras that they should not be empty? What would happen if colmap was used on the generated images, would it still result in a reasonable 3d model? - Did you notice any failure cases related to just conditioning on the previously generated image for the 3d generation? The camera motions in fig. 4 are fairly simple with little rotations and objects do not disappear and reappear, but if there would be more rotations I would imagine that it needs to be more carefully handled, e.g. as in the paper “WonderJourney: Going from Anywhere to Everywhere” (CVPR 2024). - For 3d reconstruction, many methods (e.g. zeroNVS) use a NeRF model and score distillation sampling instead of direct 3d reconstruction from multiple generated images like it is done in this paper. Were any experiments like that performed?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

This is adequately addressed

Summary

This paper introduces a new approach for efficiently finding corresponding frames from diverse viewpoints from YouTube videos at scale. The resulting 360-1M dataset contains over eighty million frames from approximately one million videos. Additionally, the paper also builds on the 360-1M dataset by proposing the ODIN model, which synthesizes novel views and reconstructs 3D scenes from a single input image. The ODIN model is trained by leveraging the diverse viewpoints in the dataset to handle significant camera view changes and complex real-world scenes. The authors demonstrate the utility of the 360-1M dataset as well as the benefits of their proposed ODIN model by comparing it to state-of-the-art 3D reconstruction approaches, where it outperforms the latter by a significant margin.

Strengths

1) The model figures are informative and especially helpful in helping the reader to understand the different stages of the data curation process as well as the intuition behind each stage. The paper is also well-organized and well-written. 2) The introduced algorithm to transform 360 degrees videos into the required multi-view format is especially significant. It opens up the possibility of scaling up available 3D object and scene datasets, which are often plagued by a lack of large-scale data. Furthermore, it is especially helpful that this algorithm appears to generalize well to videos from diverse subject categories. 3) The ODIN model also introduces a novel modeling objective which is conditioned on both rotation and translation. This differs from prior work which is unable to do so. This may be beneficial for generating more realistic samples of real world objects and scenes for further training.

Weaknesses

1) The paper relies heavily on the 360-degree videos collected from YouTube, which may contain varying video qualities and resolutions. The described preprocessing steps does not fully account for this diversity in video quality, which may affect the quality of the extracted multi-view data negatively. 2) It is mentioned in line 220 that optimizing Equation 2 directly results in a degenerate solution and Equation 3 is introduced to mitigate this problem. However, there is no ablation to demonstrate this. It may help make the paper more comprehensive. 3) In Tables 2 and 3, it is shown that ODIN outperforms other approaches on the LPIPS and PSNR metrics but not the SSIM metric consistently. However, there is no discussion of possible reasons for this discrepancy.

Questions

Please look at the above-mentioned limitations.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper addresses the challenge of training models with 3D object understanding using large-scale real data, proposing the use of 360-degree videos as a scalable and diverse data source. Its main contributions include a 360-1M video dataset, an efficient method to convert 360-degree videos into multi-view data, and a novel diffusion-based model for view synthesis. The proposed model is evaluated on NVS and 3D reconstruction tasks compared to prior approaches.

Strengths

1. The paper propose a novel large-scale real-world dataset collected from Internet along with an efficient pipeline to extract valid correspondence between frames. This dataset should be useful for future researches in the community. 2. The idea of utilizing 360 degree video to provide more correspondences is interesting. 3. Experiments demonstrate the proposed model is able to predict novel view images along a long trajectory, which is an important capability for large-scale scene/object understanding. 4. The paper is overall well written and easy to follow.

Weaknesses

1. Besides its major contribution as a large-scale and open-world training dataset compared to ZeroNVS, the model primarily aligns with Zero-1-to3, which somewhat limits its technical contributions. 2. Experiments: [-] Could you provide a reason why Zero-1-to-3 is not compared on the DTU benchmark, which is suitable for Zero-1-to-3 given its focus on single objects placed on tabletops? [-] I wonder how Zero-1-to-3 is applied in Fig. 3, as its camera pose involves both elevation and azimuth angles, which may differ from the video’s camera poses. Could the authors clarify how they calculate the camera pose? [-] This paper presents quantitative results only for experiments on DTU and MipNeRF360 (Tables 1 and 2). Including visual comparisons could help understanding. [-] Why does Table 4 in the supplementary materials show a higher Chamfer Distance compared to ZeroNVS? 2. Others [-] Considering the training requirements outlined in the appendix, it’s evident that training is highly resource-intensive, which is reasonable given the dataset’s scale. I wonder if there are potential methods to expedite this process, such as initializing from a pretrained Zero-1-to-3 model? [-] Could the authors elaborate on how long-range image pairs contribute to the model? It’s challenging to distinguish correspondence in Fig. 9 from the appendix, as the images appear not to overlap. While such pairs may assist the model in extrapolating unseen regions, could they potentially hinder learning the viewpoint changes of observable content? [-] The visualization in Fig. 3 could be enhanced by including relative changes in viewing angles, as it’s currently difficult for humans to find correspondence between the output and input view images.

Questions

See above. Overall, I believe the proposed dataset and the demonstrated performance of the model hold promise for future research. I look forward to the rebuttal addressing my concerns.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Continued Response to Reviewer whPn

Below we've included responses to the remaining questions and comments from reviewer whPn. We appreciate their detailed feedback and questions. **The dataset contains a large amount of data, which may vary in quality. I hope the author can discuss the impact of the low-quality data in the dataset. Is it negligible?** We found that low quality video frames such as blank frames, blurry frames, low resolution frames, or those with minimal camera movement were automatically filtered out in the Dust3r scoring phase. If the frames were low quality they naturally received a low score because the depth could not be well estimated. We verified this through manual inspection by randomly sampling frames which were selected by the Dust3r scoring mechanism. **Iines 178-179 mentions that it is infeasible to guarantee the uniqueness of video content. Hence, how do you make sure there is no overlap between training and testing set?** We ran deduplication over video titles, URLs, and thumbnails for all 1 million videos though this does not guarantee videos contain overlapping clips. The primary utility of a large multi-view dataset is for training models. We believe that datasets such as Mip-NeRF are better for evaluation. **Lines 126-128 [in theory we can align the views to look at the various regions of the scene to form multiple view correspondences] need experimental support. ** Figure 8 and 9 in the appendix show qualitative paired frames from the dataset. Quantitatively we found 80,567,325 paired frames from the videos by aligning the views. Upon manual inspection we found that the frames were aligned to contain overlapping content. We will add more qualitative paired frames from the dataset to the appendix for readers view. **The work uses Depth Anything as the depth estimator. I wonder if this causes a domain gap in the proposed dataset.** We used the outdoor version of Depth Anything off-the-shelf. We found that the depth estimation did not need to be extremely precise, as small errors in the scale of the scene did not impact training the model. **What is the tradeoff between using this dataset and others?** We believe our dataset is complementary to other datasets. For example, ZeroNVS combines multiple datasets to train their model. We see no reason why our dataset could not be similarly combined with other multi-view datasets to train even better models. In terms of training, the training time scales proportionally with dataset size so more data means longer training. We will add a FLOPS comparison to the appendix.

Post-rebuttal

I thank the authors' efforts in addressing my concerns, especially over the experiment details. After reading the opinions from other reviewers, I raise my rating to weak accept.