Summary
The paper mainly works on text-to-3D with PBR materials. It follows the diffusion-based multiview generation and LRM-based reconstruction paradigm. It is a fast feed-forward solution for PBR generation: The diffusion model predicts both shaded and albedo, and the LRM predicts PBR via differentiable rendering with an efficient pixel-wise deferred shading loss. Moreover, it designs a UNet-based texture refiner to project input images to mesh, bringing sharp textures. Also, it adapts LightplaneLRM with the VolSDF renderer and SDF loss to enhance geometry. The generation only takes 30 seconds. It outperforms state-of-the-art feed-forward baselines and performs comparably to top optimization-based industry solutions.
Strengths
1. Good performance.
a. It compares with and outperforms some very recent SOTA approaches like InstantMesh and MeshLRM as well as some commercial products like Meshy and Luma Genie.
b. It showcases that it can predict well on objects with mixed materials, which is very impressive. Its PBR prediction is SOTA performance. Also, it is not optimization-based and thus fast.
c. Its texture refinement module is robust and preserves many details (Figure 4, 8), which helps win out in terms of visual appearance.
2. It conducts many quantitative and qualitative ablation studies to validate each component’s effectiveness.
3. The writing is very clear and easy to follow. Notations are used properly and are well explained. Each design choice is well-motivated and elaborated. It also includes a nice preliminary reading of the used BRDF model.
Weaknesses
1. Image-to-3D may have a degraded material decomposition. Since the proposed approach does not predict albedo for input images, PBR prediction correctness will drop significantly as pointed out in the ablation study (Table 1, Fig. 3).
2. The diffusion model is inherently stochastic, which leads to variance in albedo prediction. How is this variance? How will this variance impact the material decomposition afterward? This point is not quantitatively evaluated.
3. The paper evaluates PSNR for metallic. It is great. But metallic is usually binary. A simple 2x2 confusion matrix on metallic prediction on single-material objects can show the performance more intuitively.
4. If input images are taken under non-uniform lighting conditions, I doubt if albedo can still be correctly decomposed.
5. Lack of failure case analysis. Following the previous point, I wonder if there are some typical failure cases for PBR prediction.
Questions
- Fewer than 10% meshes are selected as high-quality samples to finetune Emu (L226). What is the filtering standard? In terms of appearance quality, material correctness, and composition complexity? Please elaborate more.
- L221: What dataset is used? Is it publicly available, like Objaverse? Is the dataset used under proper license?
- How is the text-to-image model (3-channel output) adapted to output 6 channels? Please provide more details on that.
- For results in Figure 7., if I understand correctly, they are deterministic? What is their PBR PSNR? (How cherry-picked are they?)
- May consider adding some qualitative comparisons to optimization-based PBR generation baselines like Richdreamer. Although they are time-costly and may not be as competitive as commercial solutions, such a comparison can strengthen the claim.
- In the Image-to-3D part (excluding texture refinement), Are albedo predictions only used as inputs to MLP?
- Some qualitative examples of Luma Genie 1.0, stage 2 vs the proposed approach (A.5). It would help to show where optimization helps to get an edge.
- 30-sec Runtime breakdown to each step.
Limitations
Some suggestions are included in the Questions above.