Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials

Summary

This paper aims to assign material information to 3D objects that already have a base color. They do not generate material maps through training generative models but instead use a pre-established material library and build an automated recognition and indexing system to index from the material library. To use this framework, the authors pre-built a material library and proposed some prompt templates to prompt GPT-4V. In this process, the segmentation of parts is mainly performed using semantic-SAM. The authors iteratively segment different views and eventually fuse them into a single UV map.

Strengths

1. The core of the entire system is retrieval rather than generation from a generative model. This approach generally offers better controllability, adjustability, and interpretability. 2. Some techniques to accelerate querying are proposed, such as hierarchical prompting.

Weaknesses

1. This task is very limited because it requires a 3D mesh with an already existing base color. The requirement for a base color is not very practical. In line 113, the authors mention, "Given the advancements in generating high-quality 3D shapes with albedo maps, the restoration of realistic material properties remains a challenge." I disagree with this point. One of the main difficulties in current 3D generation or texture generation methods is generating a base color (lightning, shadow-free). Most methods, whether through 2D SDS or 3D rendering training, fail to generate the base color. The more pressing issue is actually estimating albedo and material properties from the final RGB image. 2. In line 136, the authors mention, "These patches are then merged based on similar colors to obtain the final material grouping." This is very limiting, as it implies that parts with the same base color are assumed to have the same material. 3. The technical contributions of the paper are minimal, mainly leveraging the capabilities of GPT-4V for VQA. I do not believe designing prompt templates is a sufficient contribution. Additionally, it is hard to reproduce the authors' implementation since GPT-4V is close-sourced. I believe the authors should at least demonstrate the feasibility with another open-source VLM model. 4. Most of the key implementation details are placed in the supplementary file. However, the step of retrieving the corresponding material based on the given albedo map and the material library established by the authors is crucial (i.e., section B.2). At the very least, this part needs to be included in the main text. Additionally, the procedure is hard to understand (please see my questions in the "Questions" section). 5. I have some concerns about the evaluation of this task because it is difficult to assess the ability to assign materials. From the results provided by the authors, it is hard to see a significant difference between different parts. It seems that compared to the albedo-only baseline, the main difference is the more pronounced lighting effect caused by the higher metallicity in the authors' results. Besides, I feel the quality is poor from the video demo at 00:56. 6. This work is more related to texture and material generation, rather than performing inverse rendering from existing photo-realistic rendered images. In the related work section, studies on texture generation need to be mentioned.

Questions

1. In line 39, the authors mention "Additionally, shadows and lighting can affect judgment." I cannot understand how shadows and lighting come into play here. If the authors assume that only the base color is used, the view can be directly obtained through rasterization without the influence of shadows and lighting. 2. For the section B.2, what is the key diffuse? Is it an existing diffuse map from your library? If this is the case, how to select the key diffuse based on the current query diffuse? Besides, what is "3) The third step involves using the obtained indices to obtain the corresponding 533 values from the rest of the material maps."? Please elaborate more about this part.

Rating

3

Confidence

5

Soundness

2

Presentation

1

Contribution

1

Limitations

The authors point out the limitations of their work in the supplementary material and leave it for future work.

Summary

This paper presents a novel framework leveraging MLLM priors (GPT-4V) to build a material library and proposes an automatic pipeline to refine and synthesize new PBR maps for initial 3D models with diffuse albedo only. The pipeline integrates existing tools, such as GPT-4V and Semantic-SAM, while introducing novel techniques to refine and complete segmented masks, resulting in a set of full SVBRDF maps. Experimental results demonstrate that this approach can automatically refine both generated and CAD models to achieve photorealism under dynamic lighting conditions.

Strengths

1. The provided results exhibit a decent quality, particularly for objects with metallic or specular materials 2. The paper is well-written, covering all necessary details for reproduction. 3. The potential application is clear: it can be used in a plug-and-play manner for 3D generative models to enhance their photorealism.

Weaknesses

1. The novelty is kind of limited. The major components rely on powerful backbones such as GPT-4V and Semantic-SAM, with the novel contributions primarily found in the segmentation and material retrieval parts. 2. The (quantitative) evaluation is sort of limited, with only GPT-4V-based and user-based study results reported in Table 1. Given that this is a novel application without standard metrics, I will not critique it too harshly. 3. The quality of the results depends heavily on the initial albedo map. If the given albedo map is not clean (e.g., it contains baked-in lighting effects), the proposed method cannot correct it, resulting in weird SVBRDF maps.

Questions

Generally speaking, I have no further questions regarding this submission; my concerns are already listed above. While the novelty might be somewhat limited, the clear potential applications and its widespread applicability lead me to still stand on the positive side of this submission.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations and potential negative social impact are well discussed.

Summary

The proposed work leverages GPT-4V to extract and infer materials in albedo-only scenarios, utilizing existing material libraries to generate SVBRDF maps with a region-to-pixel algorithm. This approach enhances 3D mesh realism, ensures precise part-specific material matching, and is compatible with rendering engines, generating six comprehensive material maps. Developers need only paint albedo textures, with other material properties automatically generated, saving significant time. The contributions include using multimodal large language models for material recognition, creating a detailed material library, and developing a pipeline for high-quality material application to 3D assets.

Strengths

- The method greatly enhances the realism of 3D objects by utilizing GPT-4V's extensive visual perception, supported by experimental evidence. - It generates detailed PBR material maps (roughness, metallic, specular, normal, displacement, height) that are compatible with various rendering engines in a low-cost way.

Weaknesses

- The method's effectiveness is largely dependent on the capabilities of MLLMs such as GPT-4V, which may not always produce accurate results. While the paper claims that the generated materials, represented as PBR maps, are compatible with downstream engines, it should be noted that this generation method does not rely on any physical principles of light transport and therefore cannot be genuinely considered as PBR maps. - The method's performance might be constrained by the quality and detail of the initial albedo maps.

Questions

- How are the PBR texture maps in the material library generated and how to ensure the accuracy of the material property? - How does the proposed method perform with low-quality input albedo maps, such as those with many small UV pieces?

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

- The proposed method relies on GPT-4V for material generation from a generated database. However, many offline tools like ShaderMap can accomplish the same task more cost-effectively. These offline tools require no training and can produce comprehensive, high-quality material decompositions. Given that the generated materials are not physically-based, what advantages does the proposed method offer over efficient offline software that requires no training or fine-tuning?

Summary

This paper introduces the large-scale multimodal language models to realistic material rendering of 3D objects. Specifically, this paper employs MLLM to retrieve materials from a material library for different parts of . By combining 2D-3D alignment and diffuse reflection reference techniques, it generates and applies material texture maps to objects, achieving realistic rendering of 3D objects.

Strengths

1. This paper introduces the MLLM into the texture inpainting pipeline, which is a interesting direction in the future. 2. The experiment results demonstrates the effectiveness of this method. 3. The paper is well-written and clearly articulated.

Weaknesses

1. This paper shares some similarities with Mapa[1]. Could the authors further demonstrate the differences between Make-it-real and Mapa and the superiority of this method? 2. The necessity of MLLM in the pipeline requires further validation. How does the performance of the MLLM-based retrieval method compare to other retrieval methods? More experiments on the retrieval results are needed to show the superiority of the MLLM -based methods. [1]Zhang, Shangzhan, et al. "MaPa: Text-driven Photorealistic Material Painting for 3D Shapes." arXiv preprint arXiv:2404.17569 (2024).

Questions

See the weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have not addressed the limitation in the paper.

Summary

This paper tackles the problem of recovering materials for 3D meshes with known geometry and base color. The proposed solution consists of three steps: - Utilizing Semantic-SAM to segment the 3D objects, identifying and isolating various material regions. - Using hierarchical prompting to retrieve and match materials from a material library. - Generating the actual SVBRDF map based on the albedo and the retrieved materials.

Strengths

- The qualitative results look promising for both artist-created assets and objects generated by 3D generative models. - The paper explores and demonstrates the potential of leveraging large multimodal language models to describe material properties of the objects—an exciting finding that could inspire future research. - The design of hierarchical prompting for material library retrieval presents an intuitive and effective way to improve the efficiency of the query.

Weaknesses

- In the user study, participants are asked to compare the base mesh and the refined mesh. However, users may naturally prefer objects with enhanced lighting effects, which does not effectively evaluate the quality of the generated materials. A more informative approach might be to compare an artist-created mesh with material to the same shape with a re-predicted material, which could provide more insights into material quality. - For the application presented in this paper, a common issue is that 3D models with a single albedo map often have baked-in shadows and other lighting effects. This is common in both artist-created models and especially in outputs from 3D generative models (e.g. is the shadow in Fig 7 bottom baked in albedo?). It's unclear how the proposed method addresses or might even be misled by these artifacts since it assumes that similar colors represent similar BRDF values in estimation. This could be a limitation when compared to alternative methods that directly optimize or predict the texture map. A discussion on this could be helpful to clarify. - The discussion on performance metrics is missing, such as runtime and the average number of parts and queries per object.

Questions

- How are viewpoints selected? - Does the method depend on good UV mapping? - How are the categories within the material library constructed?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors discuss the limitations of their method and broader impact in the paper.

Thanks for the comment

Thank you for your thoughtful feedback. We acknowledge that the current version of our paper lacks detailed information regarding the human data and transparency, we address you concerns below and in the revised paper. **Human Derived Data and Human Subjects:** The datasets utilized in our work, specifically Objaverse, is composed entirely of inanimate objects. Importantly, the Objaverse dataset we used have undergone rigorous ethical filtering as part of the Cap3D [1]. Cap3D’s ethical filtering process removed all potentially problematic data, including any identifiable human elements and NSFW content (detailed in Sec 3.2 of Cap3D). This process included the removal of objects with licenses that do not permit commercial usage, the exclusion of objects lacking sufficient camera information for rendering, and the application of face detection and NSFW classifiers with high thresholds to ensure thorough filtering. The final dataset used in our work, therefore, does not contain any human-derived data or data related to human subjects. **Deception and Fraud:** We acknowledge the reviewers' concern regarding the potential misuse of the technology to create realistic fake human representations. However, given that our dataset does not include any human data, this risk is substantially mitigated. **Lack of Discussion on Ethical Concerns:** While our current work strictly involves inanimate objects that have been ethically filtered to exclude any human-derived data, we recognize the importance of addressing the broader implications of this technology. We have discussed some ethical issues in Appendix F. Moreover, we will further enhance the discussion in this section by addressing the risks of deception and fraud associated with the creation of realistic human representations. This section will explore potential misuse scenarios and outline strategies to mitigate these risks, underscoring our commitment to responsible AI development. **Transparency and Reproducibility:** We will expand this discussion in the revised version of our data handling processes, including detailed information on the ethical filtering procedures and any potential biases. This will also include demographic details of the human experts involved, where relevant, to ensure a thorough and responsible approach to AI development. [1] Luo T, Rockwell C, Lee H, et al. Scalable 3d captioning with pretrained models[J]. Advances in Neural Information Processing Systems, 2024, 36.

Thanks for the comment

Thank you for your thoughtful feedback. We acknowledge that the current version of our paper lacks detailed information regarding the human experts who participate in the evaluation of our results. To address this, we provide additional details below and in the revised paper: 1. **Number and Selection of Experts**: We engage 16 volunteers, each with a solid background in computer science, specifically related to 3D modeling and generation. These experts have at least one year of research or work experience in the 3D field and hold advanced educational backgrounds. We ensure gender balance through random selection to avoid any bias in the evaluations. To protect their privacy, no personally identifiable information is disclosed. Each expert evaluates 400 data points(detailed in Sec 4.1), and we obtain their explicit consent to use the study data and publish the evaluation results. 2. **Procedures and Instructions**: We base our evaluation criteria primarily on whether the material appearance of the objects is photorealistic, with a focus on accurately replicating the attributes of real physical materials, similar to the instructions and examples shown in Figure 14. Each expert undergoes thorough training, starting with an analysis of 40 ground truth cases from Objaverse, with and without materials for comparison. During this training phase, we provide the correct answers, indicating which objects are closer to reality. This training helps experts determine which object's appearance is closer to real materials, enabling more accurate assessment of realism, which further prepares them to evaluate the subsequent test data. 3. **Ethics of Test Data**: As detailed in Sec 4.1, our data sources adhere strictly to NeurIPS guidelines and ethical standards for data usage. Notably, our study involves no human subjects; all test data consists solely of inanimate 3D mesh representations. We hope these additions clarify the procedure of the our evaluations and suggest that the results align with the standards of rigorous scientific research. We also ensure that these updates fully comply with the NeurIPS ethical guidelines regarding research involving human evaluators. We will include this information in the revised version of the paper. Thank you again for your valuable suggestions. We remain committed to improving the transparency and reproducibility of our work.