PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation

Summary

This paper proposes a nove approach for creating an infinite number of diverse panoramic environments conditioned on text for visual-language-navigation (VLN). Specifically, the authors use stable diffusion with captions of images from a existing dataset to generate in-door panoramic views. Recursive inpainting is used to ensure the consistency across views. The authors also present pretraining and finetuning strategies that are designed to maximize the benefits of the synthesized data, and they demonstrate its effectiveness across various benchmarks.

Strengths

1. This paper demonstates that strong generative models (like Stable Diffusion) are potential to augment traning data for robotics learning (VLN in this paper). Similar ideas could be adopted in other areas like autonomous driving. 2. The experiments over different benchmarks show that incorporating virtual panoramic scenes generated by Stable Diffusion is useful.

Weaknesses

1. There are no examples for mPLUG captioning. It would be great to contain several samples of {previous_img, previous_caption, generated_img, generated_caption}. With such kind of samples can reader better understand how mPLUG creates suitable captions for generated images with different layouts and appearances. 2. This method creates novel data with different appearance, scene layouts, and instructions compared to original data. And experiments show that incorporating these three factors together is helpful. However, It is not very clear which specific part or all parts have played a helpful role. If novel layouts and instructions do not help, I believe incorporating ControlNet/GLIGEN can provide novel appearances with the same arrangements, no need for finetuning mPLUG and captioning, making the overall method simpler and easier to follow.

Questions

1. In this paper, panoramic generation is conditioned on captions from existing data. However, it is also possible to create panoramic views in an unconditioned manner. Then mPLUG can also create instructions over this data. Will such data help VLN training? 2. Is it necessary to have novel layouts with instructions, as I mentioned in Weakness (2)? Can we just use ControlNet/GLIGEN?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed about the limitations and potential impact in the supp.

Summary

This paper proposed a data augmentation method named PANOGEN, which generates panoramic environments. The proposed method employs a recursive image inpainting technique to generate coherent panoramic environments and incorporates these augmented environments in both the pre-training and fine-tuning stages. Experimental evaluations demonstrate the effectiveness of the proposed method in enhancing performance in VLN tasks.

Strengths

1. PANOGEN generates new environments without human annotation, which is novel and solves the problem of environment scarcity. 2. The paper is well-written and easy to follow. 3. The experiment is sufficient and ablation studies are strict.

Weaknesses

1. It seems that it lacks a discussion about the selection of the image caption model since the caption is a critical section. And I am wondering how it would be to directly generate the panorama view from the instruction. 2. More discussion on the effectiveness of panorama replacement in fine-tuning is needed. I'm still confused about why using the generated environment is better than the original one because of the larger data. 3. It would be better if the visualization results were made into a panorama-instruction format so that readers can compare easily, and there are no failure cases. 4. I could not find any examples of generated instructions or a measure of the quality of the generation, thus it is hard to evaluate the performance of the Speaker.

Questions

It would be better to indicate in Table that DUET-CLIP is the baseline to make it easier to make comparisons.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Summary

This paper presents a new data augmentation method for VLN tasks. The proposed method first generates captions for each view and then recursively generates new images to ensure the consistency among multi-views. The authors demonstrate two ways to utilize the newly generated panorama on three benchmarks: R2R, CVDN, R4R. Better results are achieved compared to the previous SoTA.

Strengths

1. The paper is well-written and easy to follow, with clearly stated objectives and technical descriptions. 2. Strong performances are achieved on three different VLN tasks. 3. Comprehensive and insightful ablation study is provided. 4. Quantitative and qualitative results show the importance of multi-view consistency to the VLN tasks.

Weaknesses

1. Will the generated panorama in two consecutive steps be largely different? If yes, will it be better if reduce the difference across steps? 2. What are the impacts on high-level VLN tasks, such as REVERIE? 3. It could be better to make comparisons to previous data augmentation methods, such as EnvDrop and PREVALENT. ====== after rebuttal ============ Thanks for the authors' responses. All my concerns have been well addressed.

Questions

See details in weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See details in weaknesses.

Summary

In this paper, the authors propose to leverage the generative model to create panoramic images for agent training. A recursive inpainting method is adopted to generate 360-degree panorama views, which aims to ensure the co-occurrence of objects and enough diversity. Experiments are conducted on R2R, R4R, and CVDN datasets, confirming the effectiveness of the proposed method.

Strengths

- The proposed pipeline is reasonable. - The performance improvement is promising when using PANOGEN. - The paper is well-written and easy to follow.

Weaknesses

- My main concern is the lack of novelty. In my view, the authors simply leverage and modify the existing text2img model to conduct data augmentation for VLN agents, and there is no specific design for the VLN method. Thus I believe this work does not provide enough technical contribution to the VLN community. - As shown in Table 2 and 3, compared to DUET-CLIP on R2R, the proposed method does not show performance improvement on the SPL. - More qualitative results of generated panoramic environments need to be provided for better examining the spatiotemporal consistency of panoramas. - Lacking experiments on datasets with high-level instruction such as REVERIE. - Missing some references such as [1-4]. [1] HOP+: History-enhanced and Order-aware Pre-training for Vision-and-Language Navigation. In TPAMI. [2] Adaptive Zone-Aware Hierarchical Planner for Vision-Language Navigation. In CVPR 2023. [3] LANA: A Language-Capable Navigator for Instruction Following and Generation. In CVPR 2023. [4] Reinforced Structured State-Evolution for Vision-Language Navigation. In CVPR 2022.

Questions

- Why were the performance results of DUET-CLIP on the test set not provided?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

This paper proposes a creative system-level solution for VL-navigation agent training, by incorporating the recent T2I model to help increase the data diversity while follow the context and human intuition, which facilitate effective pre-training for performing domain-specific tasks. The proposed pipeline with image capturing and recursive in-painting can potentially create an infinite number of diverse panoramic environments conditioned on text. The author has further proposed two ways to demonstrate the usage of this generated panoramic data, by replacing existing images or generating new trajectory-instruction pairs. They have well demonstrated its effectiveness on existing VL-Nav benchmarks, with better generalization to unseen environments.

Strengths

The proposed method of leveraging existing T2I models for VL-Nav tasks is creative and provides a cost-effective way to generate rich data for training embodied agents. In addition to the effective method of generating panoramas for indoor environments, the author also provides a complete pipeline for using the data and evaluates their idea on well-known public benchmarks. This required considerable effort, but will inspire follow-up work in the community to leverage pre-trained generative models to create diverse training data for downstream tasks. The proposed framework has achieved exciting results in improving existing methods in instructed navigation tasks, such as improving goal progress by 1.59 meters on the CVDN test set. The ablation experiments with different ways and ratios of using the generated panorama data are also commendable, as they help to achieve a better system design.

Weaknesses

- It seems that the comparison to previous practices is missing. There have been multiple different practices used to avoid overfitting and increase data diversity, such as manually doing domain randomization, doing re-lighting to existing environments, or doing joint training over multiple different datasets for the same task. If possible, it would be helpful to know how this proposed framework compares to existing practices. - Another concern is the quality of the synthetic data. While nearby view in-painting might make sense, the final generated panorama can sometimes be counterintuitive and lack loop closure characteristics. The generated trajectory-instruction pair can also sometimes make no sense, and in the replaced view case, the image may differ from the original one a lot regarding the layout. It would be helpful to know if there is a way to measure the quality of the generated data besides the final evaluation in the target tasks, since some generated data might be harmful to the tasks.

Questions

When generating realistic trajectory from the panorama images, will any depth information (like from pre-trained depth) be used as well?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

NA

Reply to Rebuttal by Authors

I appreciate the clarification provided. I still have a concern. It appears that when training the model using both original instructions and mPLUG-synthesized instructions, the resulting difference in success rate is only 0.8%. While I am not an expert in VLN, based on my previous experience in robotics, I perceive this difference to be relatively insignificant. In order to determine the true contribution of including mPLUG, can you conduct additional experiments using different seeds, and report the mean and standard deviation of the success rate, as this will provide a more comprehensive understanding of how the inclusion of mPLUG impacts the overall performance of the model.