Summary
This paper proposes a method for generating multi-view images conditioned on both image and text prompts. Building on the reference attention mechanism from prior work, it incorporates additional text conditioning to enable controllable generation through text prompts. To enhance text captions, the authors use GPT-4V to annotate 3D assets and render objects with two different material properties, allowing for varied material appearances. Quantitative experiments demonstrate improved performance over several baseline methods in view synthesis, 3D reconstruction, and text-to-multi-view tasks.
Weaknesses
- The primary contributions rely on integrating existing techniques (GPT-4V for captioning and a previously established reference-guided mechanism), rather than proposing fundamentally new methodologies.
- While the detailed captioning using GPT-4V is beneficial, the approach does not introduce a novel annotation strategy beyond leveraging GPT’s generative capacity.
- The core of the proposed approach relies heavily on previously established methods, specifically the reference view guidance. The main component, known as the "key-value (k, v) appending mechanism," which enables reference view guidance, was first introduced in prior work by Zhang et al. [1]. This paper primarily extends the mechanism by adding additional text prompts for control, but even this extension is not entirely novel; the concept of using text prompts alongside multi-view guidance has been previously explored in works such as Direct2.5 [2] and MVControl [3].
- The paper lacks quantitative evaluation to assess the effectiveness of material properties (e.g., metallic, roughness). Additionally, the approach does not appear scalable, as supporting a new material requires generating an entirely new set of images in Blender. It is also challenging to predict how well material conditioning via text prompts would perform for a broader range of material combinations.
[1] Lyumin Zhang. Reference-only control. In Reference-only control, pp. https://github.com/Mikubill/sd–webui–controlnet/discussions/1236. github, 2023.
[2] Lu, Y., Zhang, J., Li, S., Fang, T., McKinnon, D., Tsin, Y., Quan, L., Cao, X. and Yao, Y., 2024. Direct2. 5: Diverse text-to-3d generation via multi-view 2.5 d diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8744-8753).
[3] Li, Z., Chen, Y., Zhao, L. and Liu, P., 2023. Mvcontrol: Adding conditional control to multi-view diffusion for controllable text-to-3d generation. arXiv preprint arXiv:2311.14494.
Questions
- Given that much of the proposed approach relies on established techniques, could the authors clarify what specific aspects of the methodology are novel? In what ways does the integration of text prompts and reference guidance go beyond previous work.
- While GPT-4V is used for enhanced captioning, did you consider alternative or custom annotation strategies to achieve richer or more context-specific annotations for 3D assets? If so, why were they not pursued, and if not, how might they enhance your model’s performance?
- Currently, the approach requires generating new images in Blender for each material property. Have you considered any strategies to make the model more scalable in terms of material variations, possibly by automating or simulating material properties in the model itself?
- How does your model handle situations where the text and image prompts may conflict or suggest different visual details? Have you tested such cases, and if so, what did you observe about the model’s ability to reconcile or prioritize these inputs?