Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering

Summary

This paper proposes a test-time approach that adaptively modifies the attention map during inference to enhance the consistency and plausibility of novel view synthesis. Experimental results on GSO demonstrate improved performance with the proposed method, and the authors conducted ablation studies to assess the impact of different modules on performance.

Strengths

1. The authors carefully analyze the challenges and issues of the current zero-1-to-3 method on novel view synthesis. 2. The idea proposed in the paper to filter attention maps akin to SGD is interesting as it can enhance the robustness of attention maps to a certain extent.

Weaknesses

1. Since (Re-)Sampling increases the computational load, with the hyperparameter R determining the number of resampling iterations, it would be beneficial for the authors to conduct an ablation study on how this parameter affects both results and computational overhead. 2. The method proposed by the authors has shown limited improvement in results. They need to validate the effectiveness of their proposed approach on a larger and more diverse dataset. 3. When conducting the ablation study, the authors' selection of only 10 objects from GSO could introduce bias into the results. It would be preferable for them to include all objects to ensure a more comprehensive evaluation. 4. The authors' proposal of Mutual Self-Attention (MSA), which has been used in other papers such as Consistent123, Tune-A-Video, and MasaCtrl, cannot be considered a unique contribution in their paper. 5. In the ablation study (Table 2), Mutual Self-Attention (MSA) shows the largest improvement, with a greater increase in PSNR (MSA 17.82 vs AMF 17.58, last two rows). This suggests that the method's improvement primarily stems from MSA rather than the core contribution point, AMF.

Questions

The paper includes several hyperparameters such as the number of resampling iterations (R) and the alpha for cross-step aggregation. How did the authors set these hyperparameters, and how significantly do they impact the final results?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The method proposed by the authors seems intriguing, but the actual improvement in results is limited. Moreover, the largest gain comes from a commonly used trick for enhancing viewpoint consistency, Mutual Self-Attention (MSA).

Summary

This paper proposes a novel approach to generate realistic images from arbitrary views based on a single source image. The authors introduce a test-time method that enhances view synthesis by manipulating attention maps during the denoising process of diffusion models. This process improves geometric consistency and realism without requiring retraining or significant computational resources. The key contributions include an attention map filtering mechanism, a modification of the self-attention mechanism to integrate source view information, and a specialized sampling schedule. Experimental results show substantial improvements in fidelity and consistency, demonstrating the effectiveness of the proposed method.

Strengths

1. The paper introduces a unique test-time method for enhancing view synthesis, specifically manipulating attention maps during the denoising process, which is innovative and effective. 2. The paper is generally well-written and easy to follow.

Weaknesses

1. Lack of qualitive results in the experimental part and even in Appendix I cannot find more comparison. 2. Only compared and equipped with zero123 and zero123-XL is not enough also.

Questions

1. Can Zero-to-Hero be equipped with other multi-view diffusion models, e.g,,MVDream, Wonder3D? 2. The quantitive metrics is calculated in which views? 3. Will Zero-to-Hero hurt the diversity of Zero123 model with relatively consistent outputs among different seeds?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

Please refers to W and Q. I will consider to raise my score if the author could address my concerns and provide more convinced results.

Summary

This paper proposes attention map filtering to enhance the novel view synthesis performance of Zero-1-to-3. The method is composed of several changes to the original sampling method, but the main contribution is an aggregation of attention map strategy by resampling the same denoising step multiple times. The paper draws an analogy to SGD to argue that resampling resembles the batch training in SGD, enhances the stability of the sampling process, thus improving the performance. Controlled experiments show that the proposed technique is effective in improving the quality of the generation

Strengths

- The paper is well-written. Related works include sufficient references. The method is clear and well-structured. The figures are also well-made and clear. I had no trouble understanding the method and the results presented in the paper. - The method is simple yet seems to be effective. The technique can be plug-and-play to apply to diffusion-based NVS methods such as Zero123. - The analysis is great. The authors perform interesting analysis on attention maps of the denoising layers, the decreased diversity in generation, and limitations. I think these analysis helps a lot in building intuition behind the method and can be useful for future works to understand the nature of the problem. - The experiments are controlled. Though the authors didn't compare against the various latest 3D generative models, they focus on evaluating the effectiveness of the proposed technique by running controlled studies. At a time when there are so many papers coming out every day, controlled and scientific experiments are valuable.

Weaknesses

- Following the analogy introduced by the authors between diffusion and SGD, it seems unavoidable that aggregating attention maps across multiple sampling steps will compromise the model on generation diversity. As we can imagine, optimization with a larger batch size is more stable. So, even though the authors have proposed techniques to mitigate the problem of lack of diversity, I believe it is a more fundamental limitation. - Since the techniques proposed seem general to all diffusion models, what prevents one from applying the technique to all diffusion generation tasks? Given the vast amount of literature and problems being solved by diffusion models, the potential impact could be significant. Are there any specific constraints such that the method is only applicable to the problem of novel view synthesis? - Does the method lead to lack of visual details in the generated views? I noticed that the eyes of the generated chickens are missing in the last row of figure 1. Is this a general problem of the proposed techniques? - It's a little unfair to claim that the proposed method can maintain the same computational efficiency. The proposed techniques to accelerate the diffusion process is not exclusive to the proposed techniques and can be applied to the base model as well.

Questions

Please see the weakness section for questions.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper has discussed the limitations.

Summary

This paper experimentally analyzes which parts are important and responsible for generation artifacts. To solve it, this paper propose an attention map filtering process. This process share the similar idea with SGD, which reduces the error of the generation process by repeated sampling. This paper also propose some other things to enhance the results, including identity view information injection and a specialized sampling schedule. The whole pipeline is training-free, so it’s very easy to apply to the pre-trained diffusion model.

Strengths

This paper propose a method to improve the novel-view diffusion model without external model and training, it is very easy to use and can be applied to different novel-view diffusion models easily. The motivation of this article is clear, and the process and details of thinking and discovery is demonstrated.

Weaknesses

* The model has no training cost, but there is a lack of thorough analysis of the additional inference cost. * Qualitative results are limited. Given that the quantitative results do not show much improvement, more qualitative comparisons would be better.

Questions

* Ablation studies of zero123 and zero123-XL showed different trends. MSA even impaired the results, but MSA with AMD improved the results. What is the reason do you think? * Form the quantitative results, the improvement brought by MSA is significantly greater than that of AMF. Are there any qualitative results on MSA and AMF? * How does this method affect the speed of the original diffusion model? Does the choice of R affect the model results? * Fig 3 and 7 seem to share some of the same samples, maybe we can show more cases and perspectives?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

This method is limited by the generative capabilities of the pre-trained model, if the results wrong too much, it does not have the ability to correct it.