Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention

Summary

This paper develops a new image-to-multiview diffusion model with two key highlights. First, it proposes a novel method for estimating the focal length and elevation. Second, it introduces a new cross-view attention mechanism. Experiments demonstrate that the method outperforms previous SOTAs.

Strengths

Previous image-to-3D methods struggle when the object in the image is distorted due to a small focal length. Therefore, this paper focuses on a practical problem. Given the development of previous image-to-multiview methods, it is common to apply a cross-view attention mechanism to address the multiview consistency problem. This paper reconsiders the shortcomings of previous cross-view attention methods and proposes a better solution.

Weaknesses

My main concern is with the presentation of experiments. As mentioned in L136, the motivation is to improve performance on real-world data, where small focal-length cases exist. Fig. 2 shows a good example. Before reading the experiments, I expected to see more results like Fig. 2. However, I think only the toaster example in Fig. 5 demonstrates the superiority of Era3D in this regard. The main paper should present more results. By comparison, the results in Fig. 6 seem more focused on demonstrating the generalization ability of Era3D and how other methods suffer from out-of-distribution issues. Although I understand that the robust generalization ability of Era3D might be due to EFReg, given that other methods are also trained on Objaverse, I think this presentation is not explicit. In particular, the focal length of the images output by SDXL cannot be controlled, so it is unclear whether Era3D's better performance is due to EFReg or better preprocessing of the training data. Moreover, I disagree with the claim in L97 that Era3D is the first method to solve distortion artifacts. For example, "Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion" (CVPR23) also explores this issue by training directly on Pscal3D+, a real-world dataset with varying focal lengths. VQA-Diff (ECCV24) explores a similar problem and proposes to utilize prior knowledge of LLMs. Despite this, I find the proposed EFReg to be an interesting and novel solution.

Questions

(1) I think Wonder3D is a highly related paper, as both deal with the image-to-multiview problem. Given that the authors claim one of Era3D's superiorities is its high resolution, I'm wondering why they did not use PSNR and SSIM to evaluate the synthetic novel views as Wonder3D did. (2) As explained by the authors in L207, the RMA is based on a simplification. Table 3 shows the benefits of RMA. However, my intuition is that RMA sacrifices quality to some degree compared to Dense and Epipolar. I did not find an ablation study regarding this in the main paper or supplementary material. I think the authors should demonstrate that the quality of RMA, Dense/Epipolar are comparable. For example, if the quality deterioration of RMA exists, it should be shown to be acceptable. Otherwise, the authors should demonstrate that RMA outperforms Dense and Epipolar in terms of quality and explain the reason.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

This paper addresses the issue of varying focal lengths in real-world data, which is a good motivation in my view. However, I do not think Era3D fully closed this problem, given that only a limited number of focal lengths are considered when preparing the training data. The authors should include this as a limitation or discuss it as a possible future work.

Summary

The authors proposed a method that can estimate the camera intrinsic matrix of the render of a given object, which attempted to solve the problem of other image-to-3d methods that only trained on fixed camera intrinsic matrix.

Strengths

The proposed method is the first work that considers the change of the camera intrinsic matrixes for the image-to-3D generation, and proposed an attention strategy to reduce the computational cost.

Weaknesses

1. I do not think row-wise attention should be a contribution as it is a special case of the epipolar attention from another paper. 2. Lacks the comparison with SV3D which is released before the deadline of the submission and can generate high resolution videos. Although Unique3D is released after the deadline, but it would be better if authors can also optionally include the comparison with it. 3. There is only a demo video in the supplementary material. If authors want refer reviewers to the arxiv version, it actually violates the double-blind policy.

Questions

1. Authors mentioned that they used the feature maps of middle-level transformer block for intrinsic estimation. Can you specify it? Which level did you use? How many levels did you use? Any experiments about the impact of different levels? 2. Is it possible to directly generate images under perspective cameras?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1. I am not sure the novelty of the paper is enough for publication. The only contribution that I can tell is the camera intrinsics estimation. Row-wise attention is borrowed from other works. The authors mentioned that the proposed work can generate high resolution image, but sv3d can also do this.

Summary

This work introduces a novel take on multiview diffusion models, highlighting the potential to realize high-resolution images from one image. The method comes with a new design for the diffusion-based camera prediction module, focal length, and elevation of the input image elevation, together with row-wise attention in enforcing epipolar priors in the MV diffusion. The results show that the approach is very high in quality, detailed in 3D meshing ability, and multiview image generation with larger resolution ability while consuming much less computation compared to other presented approaches.

Strengths

**Clear Motivation and Innovative Module Design** In this paper, the authors address several significant challenges associated with MV Diffusion in 3D content generation. These challenges include issues such as low resolution, inefficient generation processes, and inconsistent camera settings. For each of these problems, the authors propose novel designs aimed at providing effective solutions. **State-of-the-Art Results** The authors claim to have achieved state-of-the-art performance in single-view image generation tasks, as evidenced by the results in Table 1 and Table 2. However, due to the rapid advancements in 3D generation, their work does not compare their generation quality with recent methods such as LRM. **Novel Contribution to 3D MV Diffusion Generation** To the best of my knowledge, this paper is the first to address and propose solutions for the distortion problem specifically in 3D MV Diffusion generation.

Weaknesses

### Missing Critical Details The paper lacks several essential points, including optimization time for inference, details on hyperparameters, and robustness testing. ### Insufficient Ablation Study The ablation study does not adequately justify the architectural design choices. More comprehensive experiments are needed to support these decisions.

Questions

1. The abstract (line 16) states, "Era3D generates high-quality... up to a 512×512 resolution while reducing computation complexity by **12x times**." I was really confused about this context. In the paper, the authors claim the design of row-wise multiview attention results in a 12-fold reduction in running time compared with Dense MV Attention. However, the author didn't show the complexity of other parts' time in their pipeline. Further Claims are needed. 2. Why did you choose viewpoints on an elevation of 0? Will it affect the generation for a specific side (e.g., bottom side)? 3. Have the authors tried to perform any sparse view reconstruction? And why you choose 6 views: {β, β + 45◦, β + 90◦, β − 45◦, β − 90◦, β + 180◦}? More ablation studies or claims may be needed. 4. I checked your demo video, but it only shows your result. Could you also compare it with other results in the video for better display?

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

response to rebuttal

The authors addressed most of my concerns. However, I believe there is still a limitation/weakness remaining. [Strength and Suggestions] Fig. 2 looks good, and I think it provides a more meaningful comparison than the current Figs. 5 and 6 in the main paper. I highly recommend moving Fig. 6 to the supplementary material, as it does not seem directly related to the main topic of this paper. If this paper were about a robust and powerful zero-shot generative model, then showing out-of-distribution cases (Fig. 6) of other methods would be reasonable. However, please note that the focus of this paper is on addressing the challenging short-focal length cases. Q2 and Q4 refer to a similar issue. I noticed that you did not include another important baseline in Table I of the rebuttal: w/o EFReg and w/o various focal length data in training. The reason I mentioned NFI and VQA-Diff is that I believe these studies have shown that adding various focal-length data during training can improve performance when dealing with short-focal-length scenarios. As they are prior works, I think it is necessary to demonstrate that Era3D processed one more step upon this. By doing so, we can see the necessity of EFReg more explicitly. The author should add this row to Table I and include the discussion in the main paper. [Weakness] I still have an issue with the response to Q5. Consider the classic MobileNet. When it was first proposed, do you think readers would have accepted it if the authors had only demonstrated that MobileNet requires fewer FLOPs and has a shorter inference time? Demonstrating performance is always important because there is usually a trade-off between performance and efficiency (though it is definitely desirable to improve both simultaneously). I understand that the resolution issue prevents the authors from comparing the methods using the current setup. However, I believe that for this type of comparison, it is acceptable to adjust the setup as long as the comparison remains fair. Specifically, the authors could train the models with a smaller resolution (perhaps on a smaller dataset as well, if it doesn’t compromise the results). In summary, fairness is the only consideration in this test, and for a top-tier conference, a thorough experiment is necessary. I will increase my rating if the authors can address the weakness mentioned in this comment. For now, I will keep it at 6.

Ethics Statement

Dear Reviewer, We sincerely appreciate your thorough review and the important ethical considerations regarding our work. We acknowledge that we did not adequately address potential societal impacts in our initial submission, and we are grateful for the opportunity to address these concerns. 1. **Acknowledgment and Discussion of Potential Misuse** As suggested, we will include the **"Ethics Statement"** in our paper to discuss and claim the potential misuse. ``` The objective of Era3D is to equip users with a powerful tool for creating detailed 3D models. Our method allows users to generate 3D objects based on a single image. However, there is a potential risk that these generated models could be misused to deceive viewers. It is important to note that this issue is not unique to our methodology but prevalent in other generative model methodologies. Therefore, it is absolutely essential for current and future research in the field of 3D generative modeling to address and reassess these considerations consistently. ``` 2. **Implementation of Safeguards** To mitigate the risks associated with our technology, we are implementing the following safeguards: + Implementing an email checking system requiring users to acknowledge and agree to user guidelines before accessing our pre-trained weight. + Integrating pre-trained facial recognition and human body detection models (e.g., MTCNN, OpenPose) to identify and flag potentially sensitive content. + Incorporating [safeguard strategies in Stable Diffusion](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) to filter sensitive and harmful input images, including Content Filtering, NSFW (Not Safe For Work) Detection, Watermarking and Tracing. 3. **Code Release and Licensing** Our code and pre-trained weight will be released under strict licenses (like, Ethical Source License) that explicitly prohibit illegal or unethical use. We believe these measures will significantly mitigate the potential for misuse while allowing for beneficial applications in gaming, virtual reality, and robotics. Finally, we appreciate your time and insightful reviews. If there are more issues, we are glad to discuss them here and continue to refine our submission to ensure full compliance with ethical considerations.

Response to the rebuttal

I appreciate the feedback from the authors. I still have some concerns as follows: 1. The author mentioned that "We leave the proof in the supplementary material.", but no such a thing. 2. How come the final layer of the UNet has the lowest resolution, shouldn't it be the highest resolution? 3. Can you please list the related methods that focus on orthogonal setups?