Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image

Summary

Given that previous works for standalone SI2T (Scene-to-Image-to-Text) or ST2I (Scene-to-Text-to-Image) perform imperfectly in spatial understanding due to the difficulty of 3D spatial feature modeling, this paper proposes tp model SI2T and ST2I together under a dual learning framework. Within this dual framework, a novel 3D scene graph (3DSG) representation is introduced, to capture 3D spatial scene features. Moreover, a Spatial Dual Discrete Diffusion framework is proposed to utilize the intermediate features of the 3D→X processes to guide the hard X→3D processes. Extensive experiments show the proposed method outperforms the mainstream T2I and I2T methods significantly.

Strengths

+ The proposed dual learning framework with 3D scene graph (3DSG) representation to enhance the 3D spatial feature modeling is novel. + The proposed Spatial Dual Discrete Diffusion framework is simple but effective, which can utilize the intermediate features of the 3D→X processes to guide the hard X→3D processes + Extensive experiments on VSD dataset have valid the effectiveness of this method. Compared with previous works (e.g., DALLE, CogView), the proposed works show more competitive performances.

Weaknesses

- Although the proposed method in this paper has achieved good results, the methods compared in Table 2 are somewhat outdated and not the latest SOTA methods (e.g., DALLE-3, CogView-2). Could the authors compare with some of the latest text-to-image methods, such as the Stable Diffusion series, SD-1.5[1], SDXL[2], etc.? - The authors should present some failure cases to analyze the shortcomings of the proposed method. For example, can the 3D scene graph always generate perfect outputs? If there are issues in the generated outputs, what impact will it have on the final results? What are the subsequent strategies to address these issues? [1] https://huggingface.co/runwayml/stable-diffusion-v1-5 [2] Dustin Podell, Zion English and etc. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, in arxiv, 2023.

Questions

Please refer tot the "weakness" section.

Rating

4

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

Please refer tot the "weakness" section.

Summary

This paper presented a novel model for the SI2T and ST2I tasks. The proposed model combines the two dual tasks and let them mutually learn via intermediate feature sharing. Through this framework, both SI2T and ST2I are enhanced. The author also provide analysis that how this method works.

Strengths

1. The paper proposed a novel dual learning idea and the implemented model is reasonable. 2. Generally comprehensive analysis on the designed model.

Weaknesses

There is no major issue in my view. Some questions and advices are listed following: About the motivation: The author claims that the 3D feature is important for spatial understanding, and I agree with that. I wonder if 3D modeling is necessary for these two tasks and what special information could the 3D modeling provide. In this framework, it seems the model acquires the capability of 3D understanding via the pretraining of the 3DSG generator. Is the scene graph a reasonable way to modeling 3D features, and how does it match your task? The author should discuss the above points. The DGAE is pretrained by the gold 3DSG dataset. Intuitively, the quality of this 3DSG dataset significantly affects the DGAE results and further the final results. Figure 7 analyzes this problem by an ablation study on the manual noise data. But I still have a concern that the performance of DGAE may be the bottleneck of the whole model. About the discrete modeling: To my knowledge, for the respective I2T or T2I task, the continuous diffusion models could achieve better performance, while the Table 6 presents the better final performance of the proposed dual model. Ignoring the efficiency problems, what superiority does discrete modeling has? This work provides a novel way to solve dual tasks. From your perspective, what kind of tasks can be solved with this dual learning method? Typos: Line 767 “alignede” should be “aligned” Line 810 “eh” should be “the”

Questions

Please refer the weakness.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

No Limitations

Summary

This paper presented a novel dual learning framework for spatial image-to-text and spatial text-to-image generation. The main model is a combination of three discrete diffusion models, where an intermediate 3D presentation is first generated and then the image and text outputs are generated based on the 3D feature. The proposed method creatively divides the SI2T and ST2I to two pairs of dual stages, i.e., the Text-3D with 3D-Text and Image-3D and 3D-Image, and then takes a dual training strategy to enhance the hard X-3D processes with the easy 3D-X processes. The experimental results show that the proposed methods outperform current I2T and T2I models significantly.

Strengths

1. The novel and effective dual learning solution for the paired ST2I and SI2T tasks. 2. The proposed methods provide an interesting perspective for the similar dual tasks, which is of value to the community. 3. The paper is well-organized and easy to read. Overall, the idea is novel and the presentation is good. I tend to believe the quality of the paper meets the standard of the conference if the author could address the concerns I raised in the weaknesses.

Weaknesses

Some confusion about the technical details: 1. How many node categories does the 3DSG has and how to define the high-level spatial concepts? Does it follow the definition of previous works? Lack of related discussion and references. 2. Do the VSG/TSG and 3DSG have the compatible node sets? Do you map the nodes among the three types of SG with a rule? 3. About the codebooks of the discrete modeling. How do the codebooks (graph, image, and text) be initialized and updated during the whole training process? The training of diffusion model is time consuming. I am a little worried about the efficiency problem. Can the author provide the efficiency analysis for each training stage? The author takes GPT2 as the text decoder. Could it be replaced by other PLMs and how does it influence the performance? Minor Issues: 1. Two small font size in Figure 2 2. Line 248, the subscript “T23D” and “I23D” are ambiguous. 3. Bars overlap in Figure 5.

Questions

Please see my weakness comments.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

Limitations have been discussed and I do not foresee any other negative impact from this work.

Summary

The paper introduces a dual learning framework and a 3D scene graph representation for enhancing spatial image-to-text and text-to-image tasks in visual-spatial understanding. The proposed Spatial Dual Discrete Diffusion (SD3) system outperforms existing methods on the VSD dataset, demonstrating the effectiveness of the dual learning strategy in improving spatial feature modeling and task performance.

Strengths

1. The authors are commended for providing the code, which facilitates further investigation and reproducibility of the research findings. 2. This manuscript is the first to achieve synergy between two spatial-aware cross-modal dual generations. 3. The manuscript provides a comprehensive summary of the differences between the 2DSG and the proposed 3DSG.

Weaknesses

1. Could the authors consider prioritizing the most relevant or impactful references and possibly discuss the contribution of each cited work in more detail? 2. The introduction would benefit from a concise summary of the manuscript's contributions at its conclusion. 3. Could the authors briefly discuss how the treatment of spatial features in existing methods compares to the approach taken in this paper, and highlight the distinct contributions of this work? 4. The complexity of the SD3 framework may impact training and inference efficiency. The authors should provide a computational complexity analysis to assess its practicality. 5. Please provide complete results for Table 2 and consider additional datasets to validate the findings. 6. The current reliance on human evaluation for spatial assessment could be complemented with automated evaluation techniques to provide a more comprehensive assessment approach.

Questions

See the weaknesses.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately discussed the limitations and potential negative societal impact.

Re-response to the author:

Thank the author’s careful response. Your answer has basically addressed my concerns. I have reconsidered this model. Overall, its key lies in the performance of the graph generation module. The purpose of its design is to utilize various methods to enable it to learn sufficient 3D information, and of course, the quality of the 3DSG data involved is also very important. As the author mentioned, the model supports open object tags, but the currently used data does not achieve this. I think in future explorations, we can try to use better methods, including high-quality data, to further enhance the performance of the graph model.

Please let us know whether we address all the issues

Dear reviewer, Thank you for the comments on our paper. We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. Thank you

Re-response to the author:

Thanks for responding. I have carefully read the rebuttal and the comments of other reviewers, and my concerns have been well resolved. However, in consideration of enhancing the solidity of the work, as other reviewers have mentioned, I suggest that the authors consider scaling up the current diffusion model in future attempts, to further verify the effectiveness of this method in larger-scale models. Moreover, if possible, it is also recommended that the authors apply this method to more general tasks. Of course, considering the length and time constraints, these tasks can be fully addressed in future work. I thank the authors again for their responses.