DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even $360^{\circ}$ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic video-text dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images. Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses.
Paper
Similar papers
Peer review
Summary
This paper introduces the DiffPano framework, designed to generate consistent panoramic images from multiple viewpoints based on a text description of a scene. The authors first create a panoramic video-text dataset from 3D scenes using Habitat Simulator, BLIP2, and an LLM. Using this dataset, they fine-tune the original Stable Diffusion model with LoRA to develop a “single-view panorama-based Stable Diffusion.” This single-view model is then extended to a “multi-view consistent panorama Stable Diffusion” by incorporating a spherical epipolar attention module into the U-Net. Experiments show that DiffPano produces more consistent and higher-quality multi-view panoramic images compared to baseline methods.
Strengths
1. The paper defines an interesting task of generating consistent panoramic images from multiple viewpoints, which could be useful for practical applications such as VR environments. 2. Providing a new dataset specifically designed for multi-view panoramic generation would be quite meaningful, facilitating further studies on panorama generation. 3. While epipolar-aware attention has been employed in prior works for perspective images [1], this paper effectively adapts a similar concept to panoramic images through a mathematical derivation for the spherical epipolar line. [1] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion, Huang et al., CVPR 2024
Weaknesses
1. In Sec. 4.1, the authors describe panorama generation as a “style transformation” from the original perspective generation. However, generating a panoramic image requires more than a style change, as a 360-degree panorama must adhere to certain geometric constraints (i.e., when a portion of the panorama is projected into a perspective image, it should appear realistic). To address this, other than the user study, there need to be clear quantitative or qualitative evaluations of the realism of the generated multi-view panoramas. For instance, FAED [1] could be used to compare the realism of the panoramas themselves. Without such evidence, it is concerning whether PanoDiff is indeed generating more geometrically “realistic” panoramic images compared to the MVDream-based baselines for multi-view generation. 2. This work does not provide any qualitative examples of the perspective projections of the generated panoramas. In previous works on 360 panorama generation, the realism of the perspective view has been a crucial source for assessing the realism of the generated panoramas [2, 3]. Including perspective views would enhance the proper evaluation of the generation quality of the proposed method. 3. While the paper claims that DiffPano shows high generalizability, there are concerns regarding its performance on diverse text prompts. The single-view model is initially fine-tuned on panoramic images using LoRA, and the multi-view model is subsequently trained based on this fine-tuned model. This two-stage fine-tuning process on panoramic data might hinder the original generation diversity of Stable Diffusion, which is primarily trained on perspective images. [1] Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning, Oh et al., ECCV 2022 [2] MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion, Tang et al., NeurIPS 2023 [3] Taming Stable Diffusion for Text to 360 Panorama Image Generation, Zhang et al., CVPR 2024
Questions
Missing citation: The data collection process for “Panorama Video Construction and Caption” (Fig. 2) appears to be quite related to the data captioning method proposed in Cap3D [1]. Can you elaborate on how it differs from or is related to their approach? If it is based on their approach, proper citation for the dataset generation pipeline is necessary. [1] Scalable 3D Captioning with Pretrained Models, Luo et al., NeurIPS 2023
Rating
5
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
The paper discusses the limitation of the proposed method (hallucination of content).
Please let us know if your concerns have been addressed
Dear Reviewer Yygs, Thank you again for your review. **We hope that our rebuttal could address your questions and concerns** such as the quantitative or qualitative evaluations of the realism of the generated multi-view panoramas, the qualitative examples of the perspective projections of the generated panoramas, the generalizability of DiffPano, and the citation of Cap3D. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address**. We have **provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage.** This anonymous link contains some generated outdoor panoramic samples. **It would be appreciated if you could raise your score on our paper**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Summary
The paper focusing on the simulation of panoramic images and the annotation of text descriptions. The primary contribution is the introduction of a LoRA-based fine-tuning technique, aimed at enhancing the performance of benchmark datasets. The authors propose a pipeline that integrates simulated panoramic data with annotated textual descriptions to improve the accuracy and robustness of 3D models.
Strengths
The idea of simulating panoramic images combined with text annotation is interesting, potentially offering new ways to enhance 3D vision using panorama as representation. The paper is well-structured, with a clear explanation of the proposed method and the experimental setup.
Weaknesses
Lack of Technical Contribution: The paper primarily focuses on the application of existing techniques (panoramic simulation, text annotation and LoRA-based fine-tuning) rather than introducing new technical innovations. This limits its contribution to the field. Typos and Clarity Issues: There are several typos and unclear sentences throughout the paper (e.g., "??" in line 34), which detract from its overall readability and professionalism. The scope of this paper lies in the indoor scene panorama generation, and it is suggested to highlight this keyword in both title and abstract.
Questions
Can the authors clarify the experimental setup and parameters used for LoRA-based fine-tuning? More details on the hyperparameters and training process would be helpful. Are there any specific challenges or limitations encountered when simulating panoramic images (especially the difficulty for outdoor/synthetic simulation)? How were these addressed in the proposed pipeline?
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
Although this paper introduce an interesting dataset and problem, it is unclear how was the dataset quality, as the supplementary materials are empty. Could the author clarify the date for releasing the datasets? The scope of the paper is more close to the benchmark and dataset track where the datasets can be used to insight the 3D computer vision researches. Besides, the authors should discuss the scalability of their approach to larger and more complex scenarios (indoor, outdoor, synthetic, cartoon). Additionally, they should address any potential biases introduced by the simulated data and how these might affect the generalizability of the results.
To SAC for pointing out the factual error of Reviewer K8Nd
We thank all reviewers for their constructive comments and recognition of our work ("first work, pioneer work, interesting task", "novel and technically sound", "plausible and visually pleasant panoramic results"), and ACs and SACs for their hard work. However, Reviewer K8Nd seems to have misunderstood the core content of this paper, the problem it solves, and the core innovation. We believe this review may be pure LLM generated. We hope that ACs, SACs, and all reviewers can read our paper and the rebuttal and then have in-depth discussions on K8Nd's misunderstandings to ensure that reviewer K8Nd can eliminate the factual errors and re-evaluate our work to obtain a fair review. We sincerely thank you. Here, we point out the factual error. Reviewer K8Nd argues that our approach lacks innovation and is a technique that has been studied in previous approaches. The goal of this paper is to perform scalable multi-view panorama generation based on text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-image panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single-view panorama generation work cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. To this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and embed it into the network as an attention layer to realize the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results. Our method is different from previous methods in terms of both technique and dataset (as recognized by Reviewer 2LuU and Reviewer 7HQE), but Reviewer K8Nd did not realize the innovativeness of our technique. We hope that our rebuttal and explanation can correct the reviewer K8Nd's misunderstanding so that our work can be properly reviewed and obtain a satisfactory review result. We sincerely hope that ACs and SACs can have in-depth discussions with all reviewers to review our work more fairly.
Please let us know if your concerns have been addressed
Dear Reviewer K8Nd, **We hope that our rebuttal addresses your questions and concerns.** As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** **It would be appreciated if you could raise your score on our paper if we address your concerns**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Summary
This work proposes a text-driven panorama generation framework to achieve scalable, consistent, and diverse panoramic scene generation. In particular, a spherical epipolar attention module with relative poses is designed to ensure multi-view consistency. Moreover, a comprehensive panoramic video-text dataset is constructed, which contains millions of consecutive panoramic frames with corresponding depths, camera poses, and text descriptions. Extensive experiments demonstrate the superiority of the proposed method beyond previous works.
Strengths
+ This work is a pioneer in exploring the scalable multi-view panorama generation task from text descriptions. + A diverse and rich panoramic video-text dataset, which shows promising potential to promote the community of panoramic generation. + The proposed framework looks compact and practical to implement. + Overall, I am in favor of the presentation of this work, it is well-structured and easy to follow. The generated panoramic results also look plausible and visually pleasant.
Weaknesses
- For the motivation of constructing the panoramic video-text dataset, the authors argue that the corresponding text descriptions in previous panoramic datasets are not precise enough. However, such a conclusion needs to be supported by some quantitative metrics or data statistics. - The seamless content is an important property in the generated panoramas. Namely, the panorama can be stitched from left to right boundaries without a noticeable edge effect. This work performs data augmentation to improve such a left-right continuity. Have the authors considered the cylinder convolutions like previous panoramic vision works? For example, "Cylin-Painting: Seamless 360° Panoramic Image Outpainting and Beyond" and "Spherical Image Generation From a Few Normal-Field-of-View Images by Considering Scene Symmetry" use a cylinder-like convolution or circular padding to ensure the seamless content generation of panoramas. The brief reviews and discussions of these works are also expected to be presented in this work. - Some previous works propose specific metrics to evaluate the panorama model. For example, "PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation" proposes the Left-Right Consistency Error (LRCE) to quantitatively measure the consistency of the left-right boundaries by calculating the horizontal gradient. The authors are suggested to add this metric for generated panorama evaluations in experiments. - Minor points: There are some typos in the manuscript. For example, in line 34: "(see Sec. ??)".
Questions
Can the proposed framework be extended to panoramic video generation? More insightful discussions are expected to be provided.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
This work lacks some customized metrics for evaluating the property of panoramic images, which should be different from those designed for the perspective images.
Please let us know if your concerns have been addressed
Dear Reviewer 7HQE, Thank you again for your review. **We hope that our rebuttal addresses your questions and concerns**, such as the motivation for constructing the panoramic video-text dataset, circular padding vs data augmentation, and specific metrics to evaluate the panorama model and panoramic video generation. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** We have **provided an anonymous link to the AC to forward the link to the reviewers**, due to the requirements during the rebuttal stage. This anonymous link contains **the difference between PanFusion's dataset and our proposed dataset**, **panoramic video generation**, etc. **It would be appreciated if you could raise your score on our paper if we address your concerns**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Thanks for the detailed rebuttal. I would like to increase the rating. The updated experimental results and discussions of the related works are expected to be presented in the final version.
Reply to Reviewer 7HQE
Dear Reviewer 7HQE, Thanks for increasing the rating and for your review. **We will add the updated experiment results and discussions of the related works in the revised version.** **It seems that our rebuttal addresses your questions and concerns.** As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address**. **It would be appreciated if you could recommend accepting our paper in the reviewers-pc discussion**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Summary
The paper presents a novel framework called DiffPano for scalable and consistent text-to-panorama generation. The authors first build a panoramic video-text dataset, then propose a spherical epipolar-aware diffusion model to generate multi-view consistent panoramic images, addressing the limitations of existing single-view panorama generation methods. Extensive experiments demonstrate that the proposed DiffPano framework can generate scalable, consistent, and diverse panoramic images from text descriptions.
Strengths
- This paper is well-written and easy to follow. - This is the first work that tackles the research direction of text to multi-view panorama generation. - The authors have created a large-scale panoramic video-text dataset, which is a valuable resource for the community. - The proposed spherical epipolar-aware diffusion model is a novel and technically sound approach to address the challenge of generating multi-view consistent panoramic images from text. - The experimental results show the effectiveness of the DiffPano framework in generating scalable, consistent, and diverse panoramic images.
Weaknesses
- It seems like this work can only do in-distribution generation (indoor scene generation only) and cannot generalize to arbitrary unbounded outdoor scenes. - This paper does not address why generating multi-view ERP panoramas is meaningful. Is the multi-view output format more helpful for later 3D reconstruction? And why? - Typos: L34: see Sec. ??
Questions
What is the use case for this work? What kinds of downstream applications need multi-view panorama images?
Rating
6
Confidence
3
Soundness
4
Presentation
4
Contribution
3
Limitations
N/A
Please let us know if your concerns have been addressed
Dear Reviewer 2LuU, Thank you again for your review. **We hope that our rebuttal could address your questions and concerns**, such as the generalizability of DiffPano, why generating multi-view ERP panoramas is meaningful, and the downstream applications of multi-view panorama images. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** We have **provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage**. This anonymous link which contains **outdoor panorama generation and single-view vs multi-view panoramas**. **It would be appreciated if you could raise your score on our paper if we address your concerns**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Please let us know if your concerns have been addressed
Dear Reviewer 2LuU, Thank you again for your review. **We hope that our rebuttal could address your questions and concerns**, such as the generalizability of DiffPano, why generating multi-view ERP panoramas is meaningful, and the downstream applications of multi-view panorama images. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** We have provided an anonymous link to the AC to forward the link to the reviewers, due to the requirements during the rebuttal stage. This anonymous link contains outdoor panorama generation and single-view vs multi-view panoramas. **It would be appreciated if you could raise your score on our paper if we address your concerns.** We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Thank you for your rebuttal. Most of my concerns have been addressed. However, I would appreciate more discussion in the revised version on the use case of multi-view panorama. For instance, while multi-view ERP cannot be directly used for VR roaming, could it enhance immersive 3D scene generation (as opposed to using just a single view)? Additionally, it would be beneficial to include more related work in panoramic 3D scene generation to further strengthen this paper, as generating panorama image is not the terminal of the real-world application.
Reply to Reviewer 2LuU
Dear Reviewer 2LuU, Thank you again for your review. We are glad that **our rebuttal has addressed your concerns**. We will **discuss more in the revised version of the use case of the multi-view panorama and include more related work(such as recent single-view panorama generation work[1, 2, 3, 4, 5, 6]) in panoramic 3D scene generation**. [1] Zhou, Haiyang, et al. "HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions." arXiv preprint arXiv:2407.15187 (2024). [2] Li, Renjie, et al. "4K4DGen: Panoramic 4D Generation at 4K Resolution." arXiv preprint arXiv:2406.13527 (2024). [3] Wu, Tianhao, Chuanxia Zheng, and Tat-Jen Cham. "Ipo-ldm: Depth-aided 360-degree indoor rgb panorama outpainting via latent diffusion model." arXiv preprint arXiv:2307.03177 (2023). [4] Wang, Jionghao, et al. "360-degree panorama generation from few unregistered nfov images." arXiv preprint arXiv:2308.14686 (2023). [5] Lu, Zhuqiang, et al. "Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 13. 2024. [6] Wang, Hai, et al. "Customizing 360-degree panoramas through text-to-image diffusion models." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024. Compared with single-view panorama generation, **our method can generate consistent multi-view ERP panoramas, which can be transferred to the perspective view for immersive VR roaming with unlimited scapes**. There is **related work [7] on using single-view panoramas for VR roaming**. We can use this work for VR roaming. [7] DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation, VR2024. Additionally, we would like to **discuss the use of multi-view panoramas for interior home design**. Given a floor plan of a house, users can customize different rooms throughout the house according to their needs. Then given a textual description of the tour from different rooms (similar to the specified trajectory and textual description in our panoramic video-text pipeline), our method can be used to generate multi-view panoramic videos. In this way, users can preview their envisioned home design. More importantly, the single-view panoramic image generated by previous methods mainly supports 3DoF roaming. Our method generates multi-view panoramic images that support 6DoF roaming, serving as inputs for 360-degree Gaussian Splatting[8] or 360-degree NeRF[9, 10]. Our method also has a great potential value in relightable 360-degree NVS with the combination of 360-degree multi-view inverse rendering method [11]. [8] Xiong, Haolin. SparseGS: Real-time 360° sparse view synthesis using Gaussian splatting. MS thesis. University of California, Los Angeles, 2024. [9] Huang, Huajian, et al. "360Roam: Real-Time Indoor Roaming Using Geometry-Aware ${360^\circ} $ Radiance Fields." arXiv preprint arXiv:2208.02705 (2022). [10] Chen, Zheng, et al. "PanoGRF: generalizable spherical radiance fields for wide-baseline panoramas." Advances in Neural Information Processing Systems 36 (2023): 6961-6985. [11] Li, Zhen, et al. "Multi-view inverse rendering for large-scale real-world indoor scenes." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. **We hope our reply could address your questions**. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** **It would be appreciated if you could recommend accepting our paper in the reviewers-pc discussion**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
To ACs, SACs and Reviewers for pointing out the factual error of Reviewer K8Nd
We thank all reviewers for their constructive comments and recognition of our work ("first work, pioneer work, interesting task", "novel and technically sound", "plausible and visually pleasant panoramic results"), and ACs and SACs for their hard work. However, Reviewer K8Nd seems to have misunderstood the core content of this paper, the problem it solves, and the core innovation. We believe this review may be pure LLM generated. We hope that ACs, SACs, and all reviewers can read our paper and the rebuttal and then have in-depth discussions on K8Nd's misunderstandings to ensure that reviewer K8Nd can eliminate the factual errors and re-evaluate our work to obtain a fair review. We sincerely thank you. Here, we point out the factual error. Reviewer K8Nd argues that our approach lacks innovation and is a technique that has been studied in previous approaches. The goal of this paper is to perform scalable multi-view panorama generation based on text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-image panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single-view panorama generation work cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. To this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and embed it into the network as an attention layer to realize the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results. Our method is different from previous methods in terms of both technique and dataset (as recognized by Reviewer 2LuU and Reviewer 7HQE), but Reviewer K8Nd did not realize the innovativeness of our technique. We hope that our rebuttal and explanation can correct the reviewer K8Nd's misunderstanding so that our work can be properly reviewed and obtain a satisfactory review result. We sincerely hope that ACs and SACs can have in-depth discussions with all reviewers to review our work more fairly.
Please comment on our rebuttal and discuss with reviewer K8Nd to correct his factual errors
Dear Program Chairs, Senior Area Chairs, Area Chairs and Reviewers, As the deadline for author-reviewer discussion is approaching, we **have not received any comments from the reviewer**, and **reviewer K8Nd seems to have misunderstood the core content, problems solved, and core innovations of this article**. We sincerely hope that you can **remind the reviewer to comment on our rebuttal** and **discuss with reviewer K8Nd to correct his factual errors** so that our article can receive reasonable evaluation. Best regards, DiffPano Authors
Please comment on our rebuttal and discuss with reviewer K8Nd to correct his factual errors
Dear Program Chairs, Senior Area Chairs, Area Chairs and Reviewers, As the deadline for author-reviewer discussion is approaching, we **have not received any comments from the reviewers**, and **reviewer K8Nd seems to have misunderstood the core content, problems solved, and core innovations of this paper**. We sincerely hope that you can **remind the reviewer to comment on our rebuttal** and **discuss with reviewer K8Nd to correct his factual errors** so that our paper can receive reasonable evaluations. Best regards, DiffPano Authors
Replies
Thank you to the authors for providing comprehensive responses to clarify the work. The proposed work clearly remains within the scope of **text-to-panoramic images generation**, creating a large-scale dataset using multi-view panoramic images for each sample. The application of LoRA-based fine-tuning combined with spherical epipolar attention to address new generation problem. After reviewing the clarifications and the visualized PDF, some of the concerns regarding the contribution formulation—specifically whether the methodology/task or the panorama generation pipeline is being overemphasized, and the generalization for outdoor cases—have been partially addressed. However, there are still issues that need attention, such as the generated realism, the lack of qualitative examples, and the discussion on generalization to avoid overclaiming in the paper. These should be thoroughly discussed in the revised submission: - High-resolution (original) data samples should be provided in the supplementary materials to allow for detailed inspection. - Why does Figure 6 in the attached PDF contain significant artifacts at the boundaries? Are these artifacts caused by the low resolution of the generated images, or is it due to the method treating the generation process as a "style transformation," potentially lacking geometric constraints? - There is a small typo in the attached Figure 5 where the closing bracket is missing. If the issues mentioned above can be further addressed, I would be willing to raise my recommendation to borderline accept based on the current quality of the submission, the visualizations provided in the response, and feedbacks from other reviewers. Addressing these new concerns in depth would greatly benefit the discussion within the reviewer-AC panel. Thanks.
reply to the reviewer K8Nd
Dear Reviewer K8Nd， Thank you for your reply and for expressing the recommendation to borderline accept our paper. We are glad to see that our comprehensive responses can **correct your misunderstanding to some extent**. However, we still want to clarify that this paper does belong to the field of text-to-panoramic image generation. However, unlike **existing methods focusing on single-view panorama generation**, **this paper focuses on scalable multi-view panorama generation**. **This paper is the first work on text-to-multi-view panorama generation.** To this end, we **propose a novel panoramic video-text pipeline to build a large-scale panoramic video-text dataset.** To generate multi-view panoramas, we **propose a spherical epipolar multi-view diffusion model** for the multi-view panorama generation, which **improves the multi-view consistency of the generated panoramas** by exploiting the geometric properties of the spherical epipolar of the panoramas. We **treat a single-view panorama generation as style transfer and use simple data augmentation to improve the geometric properties of the left-right consistency of the panorama, making scalable panorama generation possible**. It should be noted that **our network is compact, faster (~3x speed) than single-view panorama generation method like PanFusion, and easier to extend to multi-view panoramas**. As for the generated realism, as the rebuttal mentioned, our synthetic panoramic video-text dataset is based on Matterport3D, which is reconstructed from the captured real images. There will be **no domain gap problem at least when generating indoor panoramas**. As for generalization, we perform outdoor panorama generation, which shows **our method has a certain degree of generalization in outdoor panorama generation, we will add this in the revised version.** Of course, we will **explore high-fidelity outdoor reconstruction for creating more diverse panoramic video-text datasets, which may make our model more generalizable. We leave it as future work.** As for qualitative examples, we **showcase text-to-multi-view panorama generation in the supplementary material**, and **provide an anonymous link to the AC to forward the link to the reviewers**, due to the requirements during the rebuttal stage. We will **showcase more high-resolution panoramic video datasets with detailed text descriptions in the supplementary materials of the revised version**. We also plan to **release the train/test dataset and open source a complete panoramic video construction and caption pipelines and release the complete framework** upon acceptance, so the community can create large-scale panoramic video-text datasets according to their needs and use and follow our work. Since our method is to directly generate panoramas with text description, **directly converting the panorama into perspective views may face a certain precision loss due to numerical interpolation**. In addition, **the resolution of the currently generated panorama is relatively low, and directly converting it to a perspective view will also cause artefacts**. **Training high-resolution panorama generation requires more resources and more time**. We can later **explore the upsampling method in video generation to fine-tune higher-resolution panorama generation to reduce artefacts**. Although our method regards the single-view generation as a style transfer from perspective view to panorama, since we added data augmentation during the training process to improve the geometric constraints of the left and right consistency of the panorama, this impact is relatively small. We will polish the writing and proofread the revised version and the supplementary material, including the typos. We hope that our response addresses your concerns. **It would be appreciated if you could further raise your score on our paper.** We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Thank you to the authors for providing a detailed rebuttal on the raised concerns. I have carefully read the rebuttal and would like to add a few comments. Firstly, considering the new qualitative results on perspective views and diverse generation outputs, I acknowledge that DiffPano indeed demonstrates the capability in generating panoramas that are realistic to a certain extent. Additionally, the additional experiment for FAED suggests that DiffPano outperforms its MVDream baseline. **In light of this, I raise my evaluation to "Borderline Accept."** **However, there are still a few aspects that remain unclear. If the authors could provide some comments on the below issues, it would greatly assist in the following discussion between the reviewers and ACs:** * I am not entirely convinced that DiffPano's "highest CLIP score" can support the claim of generating "realistic" panoramas. While CLIP score measures text-image alignment, it is not clear how it directly correlates with the realism of the generated panoramas. * The statement that *"making scalable panoramas possible"* remains ambiguous. Given its importance in justifying the lower scores of DiffPano compared to PanFusion, it would be helpful if you could elaborate on this point. I still question whether achieving fast inference is worth the significant trade-off in generation quality/realism (as seen in the notable difference between PanFusion and DiffPano on the FAED metric). Thank you again for the additional information and for your continued efforts in addressing these concerns.
Reply to Reviewer Yygs and please let us know if your concerns have been addressed
Dear Reviewer Yygs, Thanks for your reply. **Fr´echet Inception Distance (FID) and Fr´echet Auto-Encoder Distance (FAED) are used to compare realism.** FID is widely used for image generation, it relies on an Inception network trained on perspective images, thus less applicable for panoramic images. Therefore, a variant of FID customized for panorama, **FAED is used to better compare the realism.** **Inception Score (IS) is used to measure realism the diversity.** **CLIP Score (CS) is used to evaluate the text-image consistency.** From the rebuttal, **our CLIP Score achieves the highest results, which means that our method can generate panoramas that are more consistent with the text description.** Although the panorama generated by PanFusion can obtain slightly higher FID and FAED than our method, we found that **when the panorama is converted to perspective, we can obtain higher FID, which shows that our perspective has better realism**. It should be noted that **PanFusion is only a single-view panorama generation. It improves the realism of the panorama by adding many complex modules but reduces the realism of the perspective view converted from panorama**. In addition, **the top and bottom of the panorama generated by PanFusion is blurred**, and **the speed of its panorama generation is too low, only about 1/3 of the speed of our single panorama generation method**. **We can achieve a relatively realistic panorama that is more consistent with the text description through style transfer and simple data augmentation, and the speed is greatly improved**. **A faster pre-training module makes fine-tuning faster and better during multi-view generative network training.** **The core of this paper is to perform scalable multi-view panorama generation**. With the faster generation speed, we can **generate longer panoramic videos in a shorter time, which is beneficial to subsequent downstream applications.** Why does fast inference make scalable panoramas possible? As mentioned above, although PanFusion can obtain a slightly higher FAED score for panoramas, it should be noted that **the top and bottom of the panorama generated by PanFusion is blurred**, and **the speed of its panorama generation is too low**. While our single-view panorama generation can generate a relatively realistic panorama that is more consistent with the text description. The speed is about 3x of PanFusion's. **Our generated panoramas have a clear top and bottom**. Since **this paper focuses on multi-view panorama generation, PanFusion can not keep the consistency of multi-view panorama generation, we are the first method that can generate consistent and scalable multi-view panoramas**. To achieve this, we **propose spherical epipolar multi-view diffusion fine-tuned on the single-view panorama generation method. A faster pre-training module makes the fine-tuning of multi-view diffusion faster and better during the training of the multi-view generative network**. **The core of this paper is to perform scalable multi-view panorama generation**. With the faster generation speed, we can **generate longer panoramic videos in a shorter time, which is beneficial to subsequent downstream applications, such as VR roaming.** **We hope that our rebuttal addresses your questions and concerns**. As the discussion phase is nearing its end, **we would be grateful to hear your feedback and wondered if you might still have any concerns we could address.** **It would be appreciated if you could raise your score on our paper**. We thank you again for your effort in reviewing our paper. Best regards, DiffPano Authors
Decision
Accept (poster)