Response to Reviewer nQgK
We thank the reviewer for the time and helpful comments. Here are our responses to the reviewer's concerns:
1. **Multi-view consistency** \
It is true that MVDream does not include any architecture design to inject induct bias to ensure multi-view consistency. The key insight here is that 3D-specific architecture design prohibits the model from share the architecture and weights with 2D image generation models, and consequently limited in terms of generalizability. \
Therefore, in MVDream, we seek a pure data-driven approach, where we simply re-use image diffusion models for multi-view generation. We believe that this is achievable because dense multi-view generation can be regarded as a special case of video generation, which has already been shown successful. Besides, Zero123 does not have any inductive bias to ensure view consistency either but it is able to generate promising novel views.\
Once our data-drive approach is proved feasible (which we believe is proved by our generated images), it will be more flexible to be combine with different 2D diffusion models to extend 2D vision prior to 3D.
2. **Re-using 2D attention** \
This means (1) we inherit the weights of 2D self-attention layers from Image Diffusion UNet, and (2) during reshaping BxFxHxWxC to Bx(FxHxW)xC for self-attention. (for original 2D models, it was Bx(HxW)xC for self-attention).
3. **Training Set** \
We will add the training set description into the main paper.
4. **3D dataset only** \
We have compared with such a model trained on 3D data only, which is discussed in Table 1 in Sec 4.1 and Figure 12 in Appendix A.3.
5. **Quantitative Evaluation** \
The evaluation was conducted on a subset from the training set because it was a common practice to calculate FID on training set[1][2]. This being said, we have updated the revised paper by recalculating the metrics on a held-out validation set. We then also evaluated Zero123-XL on the same validation set. But it is worth noting that Zero-123 is an image conditioned model, so the comparison will not be fair. The results are as follows:
| Method | Condition | FID | IS | CLIP |
| --- | --- | --- | --- | --- |
| Zero123-XL | Image | 13.91 | 11.77±0.66 | 26.89±3.25 |
| Ours | Text | 39.04 | 12.77±0.60 | 30.38±3.50 |
To evaluate the Zero123 model, we feed one of the ground-truth view (image) of each sample to generate 4 novel views. Because of this, Zero123's results will have a smaller FID, which measures semantic difference between the generated and ground-truth distributions. In contrast, our models shows higher Inception Score and CLIP, which are metrics to evaluate image quality and text-image consistency.
[1] Karras, Tero, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Analyzing and improving the image quality of stylegan." In CVPR 2020.
[2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." In NeurIPS 2020.